JP6722084B2 - Object detection device - Google Patents

Description

The present invention relates to an object detection device mounted on a vehicle, for example.

Conventionally, a stereo camera is mounted on a vehicle, an object existing around the vehicle is detected based on a pair of captured images (a reference image and a comparison image), and the traveling safety of the vehicle is improved with respect to the detected object. It has been proposed to perform vehicle control for.

Here, in a stereo camera, a shield (dust, dust, water drop, etc.) may adhere to the windshield in front of the camera or the lens surface of the camera. In such a case, the shielding may be reflected in the captured image, which may affect the detection of the object.

For example, in the device described in Patent Document 1, it is determined that a shield has adhered to the windshield in front of the vehicle-mounted camera based on the difference in brightness value between the reference image and the comparison image. Specifically, the prediction region in which the object detected based on the pair of captured images is captured in the current frame is set. Then, when the difference between the average values of the brightness values of the prediction area on the reference image and the corresponding prediction area on the comparative image is equal to or more than a threshold value, when a predetermined number of frames or more continues, the shield adheres. To determine.

JP, 2009-110173, A

By the way, the captured image in the camera changes greatly due to the movement of the vehicle, the object, or the like. In this case, if the position of the object on the image changes, the result of the comparison between the pair of images may change. Then, depending on the position of the object or the shield on the image, the shield state may not be properly detected only by comparing the pair of images.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an object detection device capable of accurately determining the presence or absence of a shield and appropriately performing object detection.

The object detection device of the present invention is applied to a vehicle (50) equipped with a first camera (11) and a second camera (12) that capture a first image and a second image including a parallax in the left-right direction, and An object detection device (20) for detecting an object existing around the vehicle based on the first image and the second image, wherein the first image and the second image at a current time point and a past time point at the current time point An acquisition unit that acquires the first image and the second image, and each acquired image is divided into predetermined sections, and for each section, the degree of similarity between the first image and the second image at the present time Calculate a certain first similarity, a second similarity that is the similarity between the first image and the second image in the past, and a third similarity that is the similarity of the first image at the current time and the past. It is determined whether or not the first image and the second image match at the present time and the past, respectively, based on the calculation unit and the first similarity and the second similarity, and the third similarity is calculated. According to, a match determination unit that determines whether or not the first image at the current time and the first image at the past match, and that the first image and the second image do not match at the current time and the past, respectively. Is determined and it is determined that the first image at the present time and the first image in the past match, the field of view of the camera is provided in front of either the first camera or the second camera. And a shielding determination unit that determines that there is a shielding object that interferes with.

When the first camera or the second camera is in the shielded state due to adhesion of dust or the like, the first image and the second image do not match at the present time and the past. In addition, since the adhered state of dust and the like is maintained for a while, the position of dust and the like on the image does not change in time series, that is, the first image at the present time and the first image in the past match. ..

In consideration of such a situation, by adopting the above configuration, it is possible to properly determine whether or not the shield is present in front of either the first camera or the second camera. Further, in the case where a shield is attached to any of the cameras, considering that the region shielded by the shield is a part of the captured image, the above-described configuration divides the captured image into Since the similarities are compared, it is possible to improve the accuracy of comparison of the similarities as compared with the case of not dividing.

The figure which shows schematic structure of the object detection system of a vehicle. The figure for explaining the outline of this embodiment. The figure for demonstrating the aspect of an image division. The figure which showed the parameter of SSIM between each image. The figure which showed the parameter of the entropy between each image. 9 is a flowchart showing the procedure of image occlusion determination processing. The figure which showed the shielding state for every division in each parameter. FIG. 8 is a diagram for explaining a modified example of a mode of dividing an image. FIG. 6 is a diagram for explaining a mode of variable setting of a threshold value.

Hereinafter, embodiments embodying the present invention will be described with reference to the drawings. The present embodiment embodies an object detection system mounted on a vehicle. The system is an example of a vehicle system mounted on a vehicle, for example, and detects an object (for example, another vehicle or a road structure) existing around the vehicle.

First, a schematic configuration of an object detection system for a vehicle according to this embodiment will be described with reference to FIG. The vehicle 50 includes a stereo camera 10, an ECU 20, and a controlled object 30. Note that in the embodiment shown in FIG. 1, the ECU 20 functions as an object detection device.

The stereo camera 10 is installed in the vehicle compartment with its imaging axis facing the front of the vehicle 50 so that the front of the vehicle 50 can be imaged. Further, the stereo camera 10 includes a right camera 11 and a left camera 12, which have different positions in the left-right (horizontal) direction. The right camera 11 and the left camera 12 simultaneously capture images in a predetermined cycle, and the right image captured by the right camera 11 and the left image captured by the left camera 12 are output to the ECU 20, respectively. Each of the right camera 11 and the left camera 12 is composed of, for example, a CCD image sensor or a CMOS image sensor. In the present embodiment, the right camera 11 and the left camera 12 correspond to the first camera and the second camera, respectively, and the right image and the left image correspond to the first image and the second image, respectively.

The ECU 20 is mainly composed of a computer including a CPU and various memories. The ECU 20 detects an object located in front of the vehicle based on the right image and the left image output by the right camera 11 and the left camera 12 of the stereo camera 10, and calculates the distance to the object.

A method using a well-known stereo matching process is applied to the calculation of the distance to the object. Briefly described, the ECU 20 acquires the right image and the left image captured at the same timing, and based on the displacement amount (parallax information) calculated from these images, the distance to the object using the principle of triangulation. To calculate.

Further, the ECU 20 carries out a contact avoidance process or the like for the object based on the calculated distance to the object. Specifically, the alarm device 31 and the brake device 32, which are the controlled objects 30, are operated at predetermined operation timings. The warning device 31 warns the driver of the existence of an object in front of the vehicle according to a control command from the ECU 20. The alarm device 31 is composed of, for example, a speaker provided in the vehicle compartment and a display unit that displays an image. In addition, the brake device 32 includes a brake mechanism that changes the braking force of the vehicle 50 and a brake ECU that controls the operation of the brake mechanism. The brake ECU is communicatively connected to the ECU 20, and controls the brake mechanism according to a control command from the ECU 20.

By the way, in the object detection by the stereo camera 10, a stereo matching process is performed on a pair of images composed of a right image and a left image, whereby the object can be properly detected. In other words, if the stereo matching process cannot be appropriately performed on the pair of images, the object detection may be adversely affected.

For example, while the vehicle 50 is traveling, dust, dust, water droplets, or other shields may adhere to the windshield in front of either the right camera 11 or the left camera 12. Similarly, dust or the like may directly adhere to the lens surface of either one of the cameras. In these cases, the shielding may be reflected in the image, which may adversely affect the object detection. That is, it is considered that the stereo matching process cannot be properly performed because the shielding object is reflected in one image while the shielding object is not reflected in the other image. .. As a result, vehicle control based on object detection may not be properly performed.

Therefore, in the present embodiment, it is determined whether or not there is an obstacle in front of one of the cameras based on the similarity between the right image and the left image for each section at the present time and in the past past the present time. .. That is, the right image and the left image at the current time point, and the right image and the left image that are past the current time point are each divided into predetermined sections, and the similarity, which is the degree of similarity between the right image and the left image at the present time point, is divided for each section. The degree SA, the degree of similarity SB which is the degree of similarity between the right image and the left image in the past, and the degree of similarity SC which is the degree of similarity between the right image at the present time and the past are calculated. Then, it is determined based on the similarity SA and the similarity SB that the right image and the left image do not match at each of the present time and the past, and based on the similarity SC, the current right image (or left image). When it is determined that the previous right image (or the left image) and the previous right image match each other, it is determined that the shield exists in front of either the right camera 11 or the left camera 12.

The outline of this embodiment will be described with reference to FIG. FIG. 2 assumes that dust is attached to the windshield in front of the right camera 11. Here, four images are shown: a right image and a left image at the current time point, and a right image and a left image at a past time point from the current time point. In this case, since dust is reflected in the right image, a large difference occurs in the image as compared with the left image. For example, regarding the corresponding sections A, B, C, and D between the images, the section A and the section B do not match, and the section C and the section D do not match. On the other hand, since the state in which dust adheres continues for a while, the section B and the section D are in the same state. In the present embodiment, the presence of the shield can be properly determined because the shield is determined under such a condition.

A specific configuration of the ECU 20 in this embodiment will be described with reference to FIG. Returning to FIG. 1, the ECU 20 functions as the acquisition unit 21, the calculation unit 22, the coincidence determination unit 23, the shielding determination unit 24, the control stop unit 25, the shielding notification unit 26, and the shielding cancellation unit 27.

The acquisition unit 21 acquires the right image and the left image captured by the right camera 11 and the left camera 12, respectively. The right image and the left image are images captured at the same timing, and the acquisition unit 21 receives them at a predetermined cycle. In the present embodiment, the acquisition unit 21 acquires the right image and the left image at the current time point and the right image and the left image at a past time point than the current time point. That is, four images are acquired as images. Here, the image past the present time is, for example, an image captured in the previous time or two times before in the imaging cycle and stored in the memory. That is, the past here means a past having a correlation with the current image.

The calculation unit 22 divides each image acquired by the acquisition unit 21, that is, the right image and the left image at the present time point, and the right image and the left image that are past the present time point into predetermined sections. 3A and 3B show an example of division of the acquired image. FIG. 3A shows a case in which the vertical length of the image is divided into ¼ and the horizontal length is divided into ⅛, and the image is equally divided into 32 in a grid pattern. Further, FIG. 3B shows a case where the horizontal length of the image is divided into ⅛ and is evenly divided into eight strips. In this way, the calculation unit 22 divides each image according to the preset division mode.

It should be noted that regarding division between images, it is preferable that division be performed so that association between divisions described below can be appropriately performed. For example, when the right image is divided as a reference, the left image may be divided after considering the amount of left/right deviation.

For each of the divided sections, the calculation unit 22 (1) between the right image and the left image at the present time, (2) between the right image and the left image past the present time, (3) between the present time and the past right image (4) Identify corresponding sections between the images such as the current and past left images. A known method can be applied to identify the corresponding section. For example, the Viterbi algorithm can be applied.

Further, the calculation unit 22 calculates the degree of similarity between the images for each of the associated sections. Specifically, the similarity SA of the right image and the left image at the current time, the similarity SB of the past right image and the left image, and the similarity SC of the right image at the current time and the past are calculated for each section. The similarity SC may be the similarity of the left image at the present time and the past.

The similarity is an index showing how similar the two sections to be compared are, and if the similarity is high, it means that the two sections are similar, and if the similarity is low, the two sections are similar. Means no. A well-known index can be used as the similarity. In this embodiment, SSIM (structural similarity) and entropy (information entropy, mutual information amount) are calculated as the similarity.

The SSIM is calculated as a similarity including the elements of the brightness similarity, the contrast similarity, and the structure similarity of the corresponding two sections. In other words, it also includes elements that are not affected by brightness. FIG. 4 shows parameters between SIM images. In FIG. 4, a section corresponding to the right image and the left image at the present time is M(L,R), and a section corresponding to the right image and the left image earlier than the present time is M(Lp,Rp). And the corresponding section between the past right images is M (R, Rp), and the corresponding section between the current and past left images is M (L, Lp). Here, for example, M(L,R) is calculated based on the following equation (1).

Note that SSIM(L,R) in the above formula (1) can be obtained using a well-known mathematical formula for SSIM. Specifically, it is calculated based on the following equations (2) and (3).

The definition of each symbol in the above equations (2) and (3) will be briefly described. Here, the corresponding section between the right image and the left image at the present time is shown. l(L,R) represents the similarity of luminance, c(L,R) represents the similarity of contrast, and s(L,R) represents the similarity of image structure. Further, μ _L and μ _R are luminance average values, δ _L and δ _R are luminance standard deviations, and δ _LR is a luminance covariance of a partition between the left image and the right image. Further, α, β, and γ in the above formula (2) are arbitrary constants.

Note that, similarly to M(L,R), M(Lp,Rp), M(L,Rp), and M(Lp,R) are also calculated based on equations (1) to (3).

Here, the SSIM value is a value greater than 0 and equal to or less than 1, and for example, when the SSIM value is 1, it means that the two sections are the same. Also, the greater the SSIM value, the higher the similarity between the two sections. Here, for example, replacing it with M(L,R), if the corresponding sections between the right image and the left image at the present time are the same, M(L,R) becomes zero. Also, the higher the similarity of the corresponding section, the closer M(L,R) becomes to zero.

Further, the calculation unit 22 calculates entropy (information entropy, mutual information amount) as the similarity.

Information entropy is the expected value of the amount of information for all events, and the larger the number, the more ambiguous the information. In terms of an image, which pixel value appears with what probability? That is, it means ambiguity as to what value the pixel value actually takes, and an image with a large entropy has a large variation in the pixel value.

On the other hand, the mutual information indicates the correlation between two images, and the smaller the numerical value is, the more similar the two images are.

Regarding entropy, the parameters between each image are shown in FIG. In FIG. 5, the information entropy of the current right image is E(R), the information entropy of the current left image is E(L), and the information entropy of the right image older than the current time is E(Rp). Let E(Lp) be the information entropy of the left image in the past. Further, the mutual information amount of the corresponding section between the right image and the left image at the current time is J(L,R), and the mutual information amount of the corresponding section between the right image and the left image earlier than the current time is J(Lp). , Rp), and the mutual information amount of the corresponding section between the current and past right images is J(R, Rp), and the mutual information amount of the corresponding section between the current and past left images is J(L, Lp). ). Here, for example, E(R) and J(L,R) are calculated based on the following equations (4) and (5).

Similar to E(R) and J(L,R), other parameters are calculated based on equations (4) and (5).

The match determination unit 23 determines whether or not each section between images matches based on the similarity calculated by the calculation unit 22. Specifically, based on the similarity SA and the similarity SB, it is determined whether the right image and the left image match at the present time and the past, respectively. Further, it is determined based on the similarity SC whether or not the right image (or left image) at the current time and the right image (or left image) in the past match.

Here, a case where SSIM is used as the similarity will be described. For example, if M(L,R) as the similarity SA is equal to or less than the threshold value Mth1, it is determined that the corresponding sections of the right image and the left image at the current time match. The threshold value Mth1 is set as a determination value for determining whether or not the two sections match. Similarly, M(Lp, Rp) as the similarity SB and M(L, Rp) or M(Lp, R) as the similarity SC are also compared with the threshold value Mth1. On the other hand, when the entropy is used, it is possible to determine whether the two partitions match by comparing the calculated entropy with a predetermined threshold. It should be noted that it may be possible to derive whether or not there is a match using a map that represents the correspondence between the calculated degree of similarity and the matching of sections.

It should be noted that either the SSIM or the entropy may be used or both may be used to determine whether the corresponding partitions match. Here, in the case where both are used, for example, when the SSIM determines that they match, but the entropy determines that they do not match, when the two determinations are different, the occlusion determination is likely to be performed. It is advisable to select the judgment that is

The occlusion determination unit 24 determines that the right image and the left image do not match at the present time and the past by the coincidence determination unit 23, and the right image at the present time (or the left image) and the past right image (or the left image). If it is determined that there is a match with the image), it is determined that a shield that obstructs the field of view of the camera is present in front of either the right camera 11 or the left camera 12. That is, in such a case, the state in which the right image and the left image do not match continues in time series, and it is considered that the shield is attached. In this determination, the shielding determination unit 24 calculates the probability P of whether each section is in the shielding state.

The calculation of the probability P will be described below. First, SCOREssim and SCOREentropy are calculated using the calculated similarity for all sections, and the maximum score value is selected for each section. These scores are calculated based on the following equations (6) and (7), respectively.

Then, the shielding determination unit 24 uses the calculated score to calculate the probability P for determining whether or not each section is in the shielding state by the following formula (8).

Note that P(A) in the above formula represents SCOREssim calculated by SSIM, and P(B) represents SCORE entropy calculated by entropy. Then, the shielding determination unit 24 compares the calculated probability P with the threshold Pth1, and when the probability P is larger than the threshold Pth1, determines that the partition is in the shielding state.

The control stopping unit 25 stops a part or all of the stereo matching processing when it is determined that the state is the shielded state. That is, the stereo matching process cannot be properly performed in the region where the shielded state occurs, so the process is stopped at least in the region where the shielded state occurs. Here, when only a part of the stereo matching process is stopped, the stereo matching process is performed in the area where the shielding state is not generated. That is, the object detection is continued.

When the shielding determination unit 24 determines that the shielding state is in the shielding state, the shielding notification unit 26 transmits a control command to the alarm device 31 to notify the driver that the shielding state is in the shielding state. With this, by operating the alarm device 31, the driver can recognize that the vehicle is in the shielded state. As a result, it is possible to prompt the driver to perform an operation for removing the shielded state, such as wiping off dust and dirt.

When the shielding determination unit 24 determines that the shielding state is in the shielding state, the shielding removal unit 27 transmits a control command for canceling the shielding state. For example, in a configuration including a wiper and a heater, when it is determined that the wiper and the heater are in the shielded state, the shielded state is canceled by operating the wiper and the heater.

Next, the procedure of the occlusion determination processing of the object detection device in this embodiment will be described with reference to the flowchart in FIG. This processing is repeatedly executed by the ECU 20 at predetermined intervals.

First, in step S11, the right image and the left image at the current time point and the right image and the left image at a past time point before the current time point are acquired. The past image is stored in the memory in the ECU 20, for example. In step S12, each acquired image is divided into predetermined sections. Here, for example, the vertical length of the image is divided into ¼, and the horizontal length thereof is divided into ⅛, and is equally divided into 32.

In step S13, a corresponding section is identified between the images. For example, corresponding sections between the right image and the left image can be specified from the same horizontal axis. In step S14, the similarity is calculated for each corresponding section between the images. Here, SSIM and entropy are calculated as the similarity by the method described above. Specifically, the similarity SA, the similarity SB, and the similarity SC are calculated.

In step S15, the similarity SA is used to determine whether the right image and the left image at the present time do not match, and the similarity SB is used to determine whether the past right image and the left image do not match. Then, the similarity SC is used to determine whether or not the right image (or left image) at the present time and the right image (or left image) in the past match. If all of these determinations are affirmative, step S15 is YES, and if at least one of them is negative, step S15 is NO.

If YES in step S15, the process proceeds to step S16, and if NO in step S15, it is determined that the right image and the left image are not in the shielded state, and the present process ends. In step S16, the probability P for determining whether each section is in the shielded state is calculated. Here, the probability P is calculated for each section using SCOREssim and SCOREentropy by the method described above.

In step S17, it is determined whether the probability P is larger than the threshold Pth1. If YES in step S17, it is determined that the partition is in the shielded state, and the process proceeds to step S18. On the other hand, if step S17 is NO, it is determined that the partition is not in the shielded state, and the present process is terminated. In step S18, the stereo matching process is stopped for the section determined to be in the shielded state. In other words, the stereo matching process is performed on the section determined not to be in the occluded state, that is, the object detection is performed. In step S19, a control command for notifying the driver of the shielded state is transmitted to the alarm device 31.

According to the shielding determination process in the present embodiment, it is possible to determine the presence or absence of the shielding state for each section. For example, FIG. 7A shows a case where the shield S is attached to one of the pair of images at the position shown in the figure. Here, FIGS. 7B and 7C respectively show an index of the probability P of each section when SCOREssim calculated by SSIM is used and when SCORE entropy calculated by entropy is used. There is. Here, the probability P is divided into four levels of numerical values 1 to 4 to be indexed, and the larger the numerical value, the higher the probability of being in the shielding state. It should be noted that in these figures, each image is equally divided into 18 for convenience.

Here, if the numerical values 3 and 4 of the indexed probabilities P are in the shielded state, for example, it is determined that the partition of the region X is in the shielded state in FIG. 7B, and FIG. Then, it is determined that the section of the region Y is in the shielded state. In this way, by comparing the similarities for each section, the occluded region can be accurately specified.

Furthermore, in the present embodiment, when the determination result indicating that the state is the shielded state is different at different degrees of similarity, in other words, when it is determined that the state is the shielded state at at least one similarity, it is determined that the state is the shielded state. .. Therefore, in the case of FIG. 7, it is consequently determined that the section of the region Y is in the shielded state. As described above, by using different similarities, it is possible to consider each determination result, and it is possible to appropriately determine the presence/absence of the shielding state.

The above configuration may be changed in the case where the determination result indicating the shielded state is different for different degrees of similarity. For example, the determination result indicating that it is not in the shielded state may be prioritized, or the determination result prioritized may be set depending on the similarity used.

According to this embodiment described in detail above, the following excellent effects can be obtained.

When the right camera 11 or the left camera 12 is in the shielded state due to adhesion of dust or the like, the first image and the second image do not match at the present time and the past. Further, by maintaining the state in which dust or the like is attached, the first image at the present time and the first image in the past match. In consideration of such a situation, by adopting the above configuration, it is possible to properly determine that the shield is present in front of either the right camera 11 or the left camera 12. Further, in the case where a shield is attached to any of the cameras, considering that the region shielded by the shield is a part of the captured image, the above-described configuration divides the captured image into Since the similarities are compared, it is possible to improve the accuracy of comparison of the similarities as compared with the case of not dividing.

When the shield is attached, the area shielded by the shield is a part of the captured image. In such a case, it is considered that the object can be detected in the unshielded portion. In this respect, in the above configuration, since the object is detected based on a section other than the section in which the shield is determined to exist, the object detection is uniformly stopped regardless of the shielded area. Unlike, it can increase the chance of object detection.

There are various methods for calculating the degree of similarity in a plurality of images, and it is considered that the determination as to whether or not a shield is present may be different depending on the degree of similarity used. In this respect, in the above configuration, when the determination result of whether or not the shield is present is different, that is, when it is determined that the shield is present by at least one method, it is determined that the shield is present. I decided to do it. In this case, by prioritizing the determination that the shield is present, object detection is limited in a situation where the certainty that the shield is not present is poor. It is considered that this can suppress the occurrence of unnecessary operation based on erroneous object detection.

The above embodiment may be modified as follows, for example.

In the above-described embodiment, in step S12 of FIG. 6, each acquired image is equally divided into each section, but this may be changed. For example, the ECU 20 may variably set the size of the section according to the position on the image, or the ECU 20 may be configured to change the division mode according to the traveling scene.

The road region on the image is considered to be highly important in object detection. Considering this point, for example, as shown in FIG. 8A, the image is divided into a road region and other regions (for example, a roadside band region and a sky region), and the road region is further divided into smaller sections. The configuration may be divided into. The road area is specified by using a known image recognition technique such as lane marking recognition. For example, the area sandwiched by the lane markings at both ends recognized by the lane marking recognition is defined as a road area. The roadside zone is specified as a region surrounded by a straight line obtained by reversing the dividing line with the horizontal line including the vanishing point of the dividing line at the end of the road region as an axis, and the dividing line. The sky region is a region other than the road region and the roadside band region. According to this configuration, it is possible to improve the accuracy of comparison of the similarity between sections in a road area that is highly important in object detection, as compared to areas other than the road area.

Further, in the image, the lower region represents the front side of the vehicle 50, and the upper region represents the rear side of the vehicle 50. Therefore, in the object detection, the lower region of the image is considered to be more important than the upper region of the image. In consideration of this point, for example, as shown in FIG. 8B, the lower region of the image may be divided into smaller sections than the upper region above the image. According to this configuration, it is possible to improve the accuracy of comparison of the degree of similarity between the sections in the lower area of the image, which is highly important in object detection, as compared with the upper area above it.

In this way, by setting the size of the section variably according to the position on the image in consideration of the importance in object detection, it is possible to improve the accuracy of similarity comparison, and thus the accuracy of occlusion determination. .. On the other hand, since the degree of similarity is calculated for each of the divided sections, the calculation load of the ECU 20 increases as the number of sections increases. Considering this point, by changing the size of the section according to the position of the image, the calculation load can be reduced as compared with the case where the section is uniformly reduced. This makes it possible to achieve both the accuracy of the occlusion determination and the calculation load.

On the other hand, as an example of changing the mode of division according to the traveling scene, there is a configuration in which the size of the divided section is changed according to whether or not another vehicle or a pedestrian exists in front of the vehicle. In such a configuration, for example, when another vehicle or a pedestrian exists in front of the vehicle, the division may be divided into smaller sections as compared with a case where no other vehicle or a pedestrian exists. As an example of this, when another vehicle or the like does not exist in front of the vehicle 50, it is divided as shown in FIG. 3B, and when another vehicle or the like exists in front of the vehicle 50, as shown in FIG. 3A. It can be divided into When another vehicle or a pedestrian exists in front of the vehicle, it is desirable that the object detection be appropriately performed. Therefore, with the above configuration, it is conceivable that the obstacle can be accurately determined, and that the object can be appropriately detected. Other vehicles and pedestrians are recognized by well-known pattern matching or the like.

In addition, as another example of the traveling scene, in a traveling environment in which a shield is likely to adhere (for example, during snowfall), the size of the partition to be divided is changed compared to the case where it is not (for example, a small partition). (Divided into two) is conceivable.

In the above-described embodiment, in the shielding determination in step S17 of FIG. 6, the threshold value Pth1 set in advance is used to determine whether or not the vehicle is in the shielding state, but this may be changed. For example, the ECU 20 may variably set the threshold value Pth1 according to the position on the image of the section, or the ECU 20 may be configured to change the threshold value Pth1 according to the traveling scene.

Regarding the position of the section on the image, it is conceivable to variably set the threshold Pth1 in consideration of the importance of the position on the image in object detection. For example, the relationship between the threshold Pth1 and the position on the image is shown in FIGS. 9(a) and 9(b). FIG. 9A shows the relationship between the threshold Pth1 and the horizontal axis (x axis) of the image. In this case, the threshold value Pth1 is set to be smaller as it approaches the horizontal center of the image. That is, it is easier to determine that the central portion of the horizontal axis of the image is in the shielded state than the peripheral portion. Further, FIG. 9B shows the relationship between the threshold Pth1 and the vertical axis (y-axis) of the image. In this case, the threshold value Pth1 is set to increase as it approaches the upper end of the image. In other words, the upper region of the image is considered to be less important in object detection than the lower region below it, so that it is difficult to determine that the upper region of the image is in the occluded state. In this way, by setting the threshold Pth1 variably according to the position of the section on the image, it is possible to appropriately determine the occlusion state while considering the importance of the position on the image in object detection.

Further, regarding the traveling scene, the size of the threshold Pth1 may be changed depending on whether or not another vehicle or a pedestrian exists in front of the vehicle. In such a configuration, for example, when another vehicle or a pedestrian exists in front of the vehicle, the threshold value Pth1 may be changed to be smaller than that when no other vehicle or a pedestrian exists. When another vehicle or a pedestrian exists in front of the vehicle, it is desirable that the object detection be appropriately performed. With such a configuration, it is possible to appropriately determine the shielding state while considering the traveling scene. To be The threshold value Pth1 may be changed so that the threshold value Pth1 is increased according to the traveling scene.

In the above configuration, the right image and the left image at the present time, and the right image and the left image before the present time are used to calculate the similarity between the images, but this may be changed. Good. For example, the images at two different times in the past may be acquired as the images past the current time, and six images may be used together with the current time to calculate the similarity between the images. Even in such a configuration, it is determined that the right image and the left image do not match at the present time and the past based on the calculated similarity, and the right image at the present time (or the left image) and the past right image are determined. If it is determined that the image (or the left image) matches, the occlusion state is determined.

In the configuration described above, in the shielding determination in step S17 of FIG. 6, the configuration is determined to be the shielding state when the probability P is larger than the threshold Pth1, but this may be changed. For example, the number of times that the probability P becomes larger than the threshold value Pth1 may be counted, and when the number of times counted is equal to or more than a predetermined number, it may be determined that the state is the shielding state. With this configuration, it is possible to further improve the accuracy of determination of the shielding state.

-In the shielding determination process of FIG. 6, the number of sections may be taken into consideration. For example, between steps S17 and S18 of FIG. 6, a step of determining whether or not the number of sections for which the probability P is determined to be larger than the threshold value Pth1 is a predetermined number N or more may be provided. In this case, if the number of sections is smaller than the predetermined number N, this processing is ended as it is. On the other hand, if the number of partitions is equal to or more than the predetermined number N, it is determined that the partition is in the shielded state, and the process proceeds to step S18 to execute the subsequent processing.

Further, instead of or in addition to the above-mentioned number of sections, a section area may be taken into consideration. For example, a step of determining whether or not the section area of the section whose probability P is determined to be larger than the threshold value Pth1 is greater than or equal to a predetermined area may be provided between steps S17 and S18 of FIG. In this case, if the area of the section is smaller than the predetermined area, this process is terminated as it is, and if it is equal to or larger than the predetermined area, it is determined that the area is in the shielded state, and the process proceeds to step S18 to execute the subsequent process.

Furthermore, the position of the section on the image may be taken into consideration. For example, between steps S17 and S18 of FIG. 6, a step of determining whether or not the position on the image of the section for which the probability P is determined to be larger than the threshold value Pth1 is the sky region may be provided. In this case, if it is determined that the area is the sky area, the present process is terminated. On the other hand, if it is determined that the area is not the sky area, the process proceeds to step S18 and the subsequent processing is executed. Note that the determination of the sky region may be appropriately changed to the determination of other regions.

-In the above-mentioned embodiment, although SSIM and entropy were used as a similarity, this may be changed and the similarity of either SSIM and entropy may be used. For example, in the case of using only SSIM, the probability P is calculated based on only SCOREssim.

As the similarity, a known similarity other than SSIM and entropy can be used. For example, SSD (sum of squared difference), SAD (sum of absolute difference), NCC (Normalized Cross-Correlation), image uniformity ratio, etc. can be used as the similarity.

11... right camera, 12... left camera, 20... ECU, 50... vehicle.

Claims (6)

The first image and the second image are applied to a vehicle (50) equipped with a first camera (11) and a second camera (12) that capture a first image and a second image including a parallax in the left-right direction. An object detection device (20) for detecting an object existing around the vehicle based on
An acquisition unit that acquires the first image and the second image at the present time point and the first image and the second image that are earlier than the present time point;
Each acquired image is divided into predetermined sections, and for each section, the first similarity, which is the similarity between the first image and the second image at the present time, and the first image and the second image in the past. A second similarity that is the similarity of the first image and a third similarity that is the similarity of the first image at the present time and the past, and
Based on the first similarity and the second similarity, it is determined for each section whether or not the first image and the second image match at the present time and the past, and the third similarity is determined. A match determination unit that determines whether or not the first image at the present time and the first image in the past match for each section based on the similarity.
For each of the sections, it is determined that the first image and the second image do not match at the current time and the past, and the first image at the current time and the first image at the past match. When it is determined that there is, a shielding determination unit that determines that a shielding object that blocks the field of view of the camera is present in front of either the first camera or the second camera,
An object detection device comprising. The shielding determination unit is for determining whether or not the shielding object exists for each of the sections of the acquired images,
The object detection device according to claim 1, wherein, when it is determined that the shielding object is present, the object is detected based on a section other than the section where the shielding object is determined to be present. The calculating unit calculates the first similarity, the second similarity, and the third similarity using at least two different methods,
The object detection according to claim 1 or 2, wherein the shielding determination unit determines that the shielding object exists when the determination result of whether the shielding object exists in each of the methods is different. apparatus. The object detection apparatus according to claim 1, wherein the calculation unit divides a lower region of the acquired image into sections smaller than upper regions of the acquired images. .. A road area recognition unit that recognizes a road area in front of the vehicle based on the acquired images,
The object detection device according to claim 1, wherein the calculation unit divides the road area into sections smaller than areas other than the road area in each of the acquired images. The object detection device according to any one of claims 1 to 5, wherein the calculation unit changes a mode of division of the acquired images according to a traveling state of the vehicle.