JP7456716B2 - Obstacle detection device and moving body - Google Patents

Description

The present invention relates to an obstacle detection device, and particularly to obstacle detection using a stereo camera.

When a small mobility vehicle such as a senior car travels, it is necessary to avoid colliding with utility poles, guardrails, pedestrians, steps, cars, etc. in various situations. These obstacles can be detected by, for example, a three-dimensional positioning sensor such as a stereo camera, ToF (Time of Flight), or 3DLiDAR. A stereo camera can detect the distance to an object using the parallax between images captured by two cameras. For example, the vehicle obstacle detection device disclosed in Patent Document 1 includes an infrared camera that images the front of the vehicle and a stereo camera that images the front of the vehicle, and identifies the running route from the infrared image taken by the infrared camera and creates a stereo image. Discloses a technology for detecting obstacles on a road that have been identified.

Japanese Patent Application Publication No. 2013-20543

Some conventional obstacle detection systems utilize infrared (IR) cameras as stereo cameras. In an infrared (IR) image taken with an infrared camera, areas that reflect infrared rays appear white, and areas that absorb infrared rays appear black. White areas have high brightness and black areas have low brightness.

The obstacle detection system combines left and right IR images captured by stereo cameras and generates point cloud data that can represent the three-dimensional position (or coordinates) of the imaging target. The depth of the imaging target is calculated using the parallax between the left and right IR images. For example, when a stereo camera images a pedestrian as shown in FIG. 1, point cloud data as shown in FIG. 2 is generated. In the point cloud data, a point group representing the distance (depth) to the pedestrian is generated in the area corresponding to the pedestrian.

The obstacle detection system determines the presence or absence of obstacles (including obstacles on the ground, holes and grooves) based on the generated point cloud data, and if there is an obstacle, it determines the direction of the obstacle and detects the obstacle. Detects the distance to objects and the size of obstacles, and outputs the detection results. For example, when an obstacle detection system is installed in a transportation device such as a car, it is used to warn passengers of obstacles or to automatically slow down or stop the transportation device.

However, in positioning using stereo cameras, when direct sunlight falls on the camera, gaps and noise occur in the point cloud data due to the influence of the direct sunlight, and the noise and gaps are incorrectly detected as obstacles. Sometimes. FIG. 3 is an example of an IR image when direct sunlight enters the image, and the portion exposed to direct sunlight appears pure white, and changes in the brightness of the road surface become large due to the influence of direct sunlight. FIG. 4 shows point cloud data generated from the IR image in FIG. 3, where the point cloud in areas corresponding to direct sunlight is missing, and noise is generated in parts of the road surface where the brightness difference is large due to direct sunlight. A point cloud with noise represents an abnormal depth (distance).

In this situation, the obstacle detection function is not working properly, so there is a possibility that the obstacle will be detected incorrectly, and the user or driving support system will be notified that the detection function is not working properly. It is desirable to do so. However, just because direct sunlight enters the camera image, it does not necessarily mean that point clouds are missing or noise occurs, so simply determining whether direct sunlight enters the camera image is insufficient to determine if the obstacle detection function is working properly. It is insufficient to determine whether or not the In other words, it is necessary to determine whether or not point clouds are missing or noise is occurring due to direct sunlight. On the other hand, determining whether a point group is missing or the occurrence of noise places a large load on an arithmetic processing unit such as a CPU, so it is desirable to perform the determination using processing with as low a load as possible.

SUMMARY OF THE INVENTION An object of the present invention is to provide an obstacle detection device that can solve these conventional problems and prevent erroneous detection of obstacles caused by direct sunlight.

An obstacle detection device according to the present invention includes a stereo camera, a generation means for generating point cloud data based on a stereo image photographed by the stereo camera, and an obstacle detection device for detecting an obstacle based on the generated point cloud data. a detection means; a direct sunlight determination means for determining the presence or absence of direct sunlight based on the brightness of an image taken by the stereo camera; and influence determination means for determining influence, and the obstacle detection means outputs an obstacle detection result according to the determination result of the influence determination means.

In one embodiment, the obstacle detection means outputs that the obstacle detection result is not normal or that the reliability of the detection result is low when the influence means determines that there is an influence from the road surface. In one embodiment, the direct sunlight determination means determines the presence or absence of direct sunlight based on whether a road in a predetermined range of the captured image has high brightness. In one embodiment, the direct sunlight determination means detects the average brightness of the entire image, the average brightness of the sky area, and the average brightness of the road in a certain range on the near side of the image, and determines the brightness of the entire image based on the average brightness of the sky area. When the average brightness of the road is high and the average brightness of the road is higher than the average brightness of the entire image, it is determined that the road is high brightness. In one embodiment, the direct sunlight determination means further determines the presence or absence of direct sunlight based on the brightness of an empty area when the road is not highly bright. In one embodiment, the direct sunlight determining means changes the range for detecting the brightness of the sky area according to the brightness of the road area, and determines that there is direct sunlight when the brightness of the sky area is high. judge. In one embodiment, the influence determining means divides a certain range of road surface area in front of the captured image into a plurality of areas, and determines that there is an influence on the road surface if the average brightness of the divided areas is high. judge. In one embodiment, the impact determining means determines that there is an impact on the road surface when the average brightness of the divided area is not high and the difference between the maximum value and the minimum value of the average brightness of the divided area is equal to or greater than a threshold value. judge. In one embodiment, the influence determination means divides the road surface area into a plurality of parts in the vertical direction, and divides the road surface area into a left area, a right area, and a center area in the horizontal direction, Determine whether or not the affected area is affected. In one embodiment, the influence determining means calculates a minimum threshold and a maximum threshold from the average brightness of the divided area, and further calculates the minimum number of pixels of the divided area with a brightness lower than the minimum threshold and the maximum pixel of the divided area with a brightness higher than the maximum threshold. When the minimum number of pixels is equal to or greater than the threshold or the maximum number of pixels is equal to or greater than the threshold, it is determined that there is an influence on the road surface. In one embodiment, the influence determining means determines the influence of direct sunlight on the road surface based on the number of points on the road surface in the point cloud data. In one embodiment, the influence determining means determines the influence of direct sunlight on the road surface based on the brightness characteristics of the road surface in the captured image and the number of points on the road surface in the point cloud data. In one embodiment, the point cloud data is three-dimensional information including distance information obtained by combining images captured by stereo cameras. In one embodiment, the stereo camera is an infrared camera. In one embodiment, the stereo camera images the front of the moving object.

According to the present invention, the presence or absence of direct sunlight is determined, and if it is determined that there is direct sunlight, the impact on the road surface is determined, and if it is determined that there is an impact on the road surface, the detection of obstacles is normal. Since the output indicates that the object is not present, it is possible to prevent erroneous detection of obstacles caused by direct sunlight.

It is a figure which shows the example of the IR image which imaged the pedestrian. 2 is a diagram showing point cloud data obtained by combining the IR stereo images shown in FIG. 1. FIG. FIG. 3 is a diagram showing an example of an IR image in which direct sunlight has entered. 4 is a diagram showing point cloud data obtained by combining the IR stereo images shown in FIG. 3. FIG. FIG. 1 is a block diagram showing the configuration of an obstacle detection device according to an embodiment of the present invention. 3 is a flowchart showing the overall operation of the obstacle detection device according to the embodiment of the present invention. It is a flowchart for determining the presence or absence of direct sunlight according to an embodiment of the present invention. FIG. 8(A) is a diagram illustrating high brightness detection of a road, and FIG. 8(B) is a diagram illustrating brightness detection of a sky area. This is an example of an image when the road has high brightness and an image when the road is not bright. These are examples of images with direct sunlight and images without direct sunlight. It is a flowchart for determining the influence of direct sunlight according to an embodiment of the present invention. This is an example of an image for determining the influence of a small area. It is a figure explaining the determination method of the influence of a road surface. It is a flowchart for determining the influence of the road surface in the left and right areas according to the embodiment of the present invention. 13 is a flow chart showing a process for determining the influence of the road surface in the central area according to an embodiment of the present invention. FIG. 6 is a diagram illustrating determination of influence in a small area according to an embodiment of the present invention. It is a flowchart for determining the influence of a small range according to an embodiment of the present invention. This is an example of an image when there is an influence on the road surface from 2 m to 3 m in the central area. 1 is a block diagram showing a configuration of a driving assistance system to which an obstacle detection device according to an embodiment of the present invention is applied;

The obstacle detection device according to the present invention detects obstacles using stereo images captured by a stereo camera. In one embodiment, the obstacle detection device is mounted on a moving body or transportation device such as a senior car (small mobility such as an electric wheelchair) or an automobile, and the detection results of the obstacle detection device are used in the movement control system of the moving body or the user. (passenger) etc.

Next, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 5 is a block diagram showing the configuration of an obstacle detection device according to an embodiment of the present invention. The obstacle detection device 100 of this embodiment combines IR cameras 110L and 110R (generally referred to as stereo cameras 110) that capture left and right IR images, and stereo images captured by the stereo camera 110, and calculates the depth by combining the stereo images captured by the stereo camera 110. a point cloud data generation unit 120 that generates point cloud data; an obstacle detection unit 130 that detects obstacles based on the point cloud data; a direct sunlight determination unit 140 that determines the presence or absence of direct sunlight based on an IR image; When the direct sunlight determining section 140 determines that there is direct sunlight, the influence determining section 150 determines whether direct sunlight has an influence on the road surface.

The point cloud data generation section 120, the obstacle detection section 130, the direct sunlight determination section 140, and the influence determination section 150 are implemented by hardware and/or software in a computer device or an electronic device. Hardware specifically includes storage media such as a GPU (graphics processor unit), CPU, and memory, and software includes programs and applications stored in storage media, etc., or downloaded from a server, etc. via a network. This includes programs and applications that have been created.

Although the mounting position of the stereo camera 110 is not particularly limited, for example, the stereo camera 110 may be mounted in front of an electric wheelchair or a vehicle to image the front of the vehicle. Since the IR cameras 110L and 110R use infrared rays, they can photograph objects even at night. Further, the IR cameras 110L and 110R may emit infrared patterns or patterns, but since such infrared light is invisible to the human eye, it has the advantage of not causing discomfort. The IR cameras 110L and 110R generate a plurality of frames per second, and each pixel constituting one frame represents the object to be photographed with 256 gradations of brightness using 8-bit data, for example. In other words, an IR image taken by an IR camera is gradation image data representing black and white shading, in which areas that reflect infrared rays appear white and areas that absorb infrared rays appear black. The number of pixels and optical characteristics of the two IR cameras 110L and 110R are the same, and the coordinate positions of the IR cameras 110L and 110R are known. For example, the two IR cameras 110L and 110R are placed at the same height (horizontally) and separated by a certain distance.

The point cloud data generation unit 120 receives IR images captured by the IR cameras 110L and 110R. The point cloud data generation unit 120 matches the left and right IR images, calculates the depth (distance) to the imaging target from the parallax of both images, and finally determines the three-dimensional position of the imaging target by combining the left and right IR images. Generate point cloud data to represent.

The obstacle detection unit 130 determines the presence or absence of an obstacle based on the point cloud data generated by the point cloud data generation unit 120. If there is an obstacle, the direction of the obstacle, the distance to the obstacle, and the size of the obstacle are detected. Further, even if an obstacle is detected based on point cloud data, the obstacle detection unit 130 detects at least an obstacle if the influence determination unit 150 determines that direct sunlight has an influence on the road surface. Outputs that object detection is not performed correctly. When direct sunlight is captured in an IR image, the brightness of the direct sunlight may cause point cloud data to be missing or noise to occur in the point cloud data, making it impossible to correctly detect obstacles from the point cloud data. There is. The output result of the obstacle detection unit 130 is used for warning to the vehicle occupants, the vehicle driving support system, or the automatic driving system.

In positioning using a stereo camera, noise or omissions occur in point cloud data indicating a three-dimensional position due to direct sunlight at locations where the difference in brightness from the periphery of the left and right camera images is large. This is because in such a location, it is impossible to capture the image shift (parallax), and the depth (distance) cannot be calculated or the error becomes large. This is especially likely to occur when direct sunlight enters the camera. Therefore, in this embodiment, the presence or absence of direct sunlight is determined, and when it is determined that there is direct sunlight, it is determined whether there is an influence of the direct sunlight.

FIG. 6 is a flowchart showing the overall operation of the obstacle detection device of this embodiment. The direct sunlight determination unit 140 determines the presence or absence of direct sunlight based on the IR image captured by the IR camera 110L or 110R (S100). When the direct sunlight determining section 140 determines that there is direct sunlight (S110), the influence determining section 150 determines whether there is an influence due to direct sunlight (S120), and when it is determined that there is no direct sunlight, the influence determining section 150 It is determined that there is no influence of direct sunlight (S130). The determination result of the influence determining section 150 is output to the obstacle detecting section 130. When the obstacle detection unit 130 receives the determination result that there is an impact, it outputs information including that the obstacle detection result based on the point cloud data is not normal (or unreliable), and outputs the determination result that there is no impact. If received, the obstacle detection results based on the point cloud data are output as usual.

Next, details of the direct sunlight determining section 140 will be explained. FIG. 7 is an operation flow of the direct sunlight determining section 140. The direct sunlight determination unit 140 performs high-intensity detection of the road based on an IR image that is synchronized with the IR image used by the point cloud data generation unit 120 (S200).

Detection of whether a road is high brightness uses (1) the average brightness of the entire IR image, (2) the average brightness of the sky, and (3) the average brightness of the road in a certain range on the near side of the IR image. The direct sunlight determination unit 140 divides the IR image into areas in front of and behind the horizon, as shown in FIG. 8(A). The relationship between the two-dimensional coordinates of the IR image and the imaging range captured by the IR camera is uniquely determined by the mounting positions and mounting angles of the stereo cameras 110L and 110R. The horizon line is the boundary between the sky and roads, buildings, etc. The area in front of the horizon line shows roads and buildings, and the area behind the horizon line shows the sky. The coordinates of this horizon line are determined in advance. be done. Moreover, "1 m", "2 m", "3 m", "4 m", and "6 m" in the depth direction represent the distance from the stereo camera. In this example, the range up to 2 m from the stereo camera is used as the average brightness of the road. The road is reliably imaged in this range of 1 m to 2 m.

The direct sunlight determination unit 140 determines that when there is a relationship of average brightness of the sky<average brightness of the entire image<average brightness of the road up to 2 m, that is, the average brightness of the sky is the lowest and the average brightness of the road up to 2 m is the lowest. When it is the largest, it is determined that the road has high brightness (S210). FIG. 9 shows an example of an IR image when the road is bright and an example of an IR image when the road is not bright.

When the direct sunlight determination unit 140 determines that the road has high brightness (S210), it determines that there is no direct sunlight (S250). In other words, since the average brightness of the road is higher than the average brightness of the sky, it is determined that direct sunlight is not incident.

On the other hand, if the direct sunlight determination unit 140 determines that the road is not high brightness (S210), it determines the brightness of the divided area of the sky (S230), and if the divided area is high brightness, it determines the brightness of the divided area of the sky. If the divided area is not high brightness, it is determined that there is no direct sunlight (S250).

FIG. 8B shows a method for determining the brightness of divided areas in the sky. As shown in the figure, the sky area is divided vertically into 3 and horizontally into 12 to generate 36 divided areas, and the average brightness of each divided area is used. In a preferred embodiment, the brightness is checked by changing the range of the divided areas depending on whether the average brightness of the road up to 2 m is dark or not. That is, when the road is dark, the average brightness of a wide range of divided areas is checked, and when the road is not dark, the average brightness of a narrow range of divided areas is checked.

If the average brightness of the road up to 2 m is dark (lower than the set threshold), it is detected whether there is a high brightness divided area using the range A of divided areas (18 divided areas). For example, if one pixel represents 256 gradations of brightness, it is detected whether or not there is a divided area with an average brightness of 220 or more within range A. If at least one exists, it is determined that there is direct sunlight.

In addition, if the average brightness of the road up to 2 m is not dark (if it is greater than the set threshold), it is detected whether there is a high brightness divided area using range B of divided areas (6 divided areas). . For example, it is detected whether there is a divided area with an average brightness of 220 or more within range B, and if there is at least one such divided area with high brightness within range B, it is determined that there is direct sunlight. It is determined that If there is no high-intensity divided area within range A or range B, it is determined that there is no direct sunlight.

FIG. 10 shows an example of an IR image when there is direct sunlight and an example of an IR image when there is no direct sunlight. When there is direct sunlight, it can be seen that there are areas of high brightness in the sky. Furthermore, it can be seen that the brightness of roads up to 2 m is low. On the other hand, it can be seen that when there is no direct sunlight, the brightness of the road up to 2 meters is higher than the brightness of the sky.

In the above example, the direct sunlight determination unit 140 determines the presence or absence of direct sunlight only from the IR image, but it is also possible to consider the exposure time to supplement the reliability of this determination. If the IR camera is equipped with an automatic exposure adjustment function, the automatic exposure adjustment function shortens the exposure time to reduce the exposure when direct sunlight enters. For example, the exposure time is 1600 usec or less. Therefore, the direct sunlight determination unit 140 may determine that there is direct sunlight using as a weighted condition that the exposure time has been shortened to a certain value or less.

Next, details of the influence determination section 150 will be explained. FIG. 11 is an operation flow of the influence determining section 150. When the direct sunlight determining unit 140 determines that there is direct sunlight, the influence determining unit 150 determines the influence of direct sunlight on the road surface (S300). If it is determined that there is an influence from the road surface, this is notified to the obstacle detection unit 130, and the process ends. On the other hand, when it is determined that there is no influence from the road surface (S310), it is further determined whether there is an influence in a small range (S320), and based on the determination result, it is determined whether there is an influence from the road surface. The small area impact determination is to determine the impact that occurs in a relatively small area, such as direct sunlight reflected on the lens or shadow reflected on the lens, and is determined by road surface impact determination (S300). Detect what you can't. FIG. 12 shows an example of an IR image that is subject to small-range influence determination. The upper figure shows an example in which flare occurs in an IR image due to direct sunlight, and the lower figure shows an example in which a dark shadow appears in the IR image due to a shadow reflected on the lens.

FIG. 13 is a diagram illustrating a method for determining the influence of the road surface. As shown in Figure 13 (A), near the center of the horizon in the IR image, the part where the road intersects with the horizon is assumed to be the vanishing point where the road disappears, and from 1 m to 6 m above the road surface based on this vanishing point. Divide the left area, center area, and right area into multiple areas within the range. For example, (1) the vertical direction is divided into four areas: 1 m to 2 m, 2 m to 3 m, 3 m to 4 m, and 4 m to 6 m; (2) the horizontal central area is divided into 4 areas, and the left and right areas are divided into two. In other words, the left area is divided into 8 by dividing lines L1, L2, and L3 from the vanishing point to the left area, the central area is divided into 16 by dividing lines L4, L5, and L6 to the center area, and the dividing line to the rear right The right area is divided into eight by L7, L8, and L9.

The influence determination unit 150 determines whether there is an influence on the road surface using the average brightness of each divided area. In one aspect, the influence determination unit 150 checks the average brightness of the left area, right area, and center area in order from 1 m to 2 m, 2 m to 3 m, 3 m to 4 m, and 4 m to 6 m from the near side, and determines whether there is an influence. When the divided area is found, the determination of whether or not there is an impact ends. In Fig. 13(B), when eight divided areas ranging from 1 m to 2 m are represented by numbers 1, 2, ..., 7, 8, the average brightness of divided areas 1 and 2 in the left area is checked first. Then, the average brightness of divided areas 7 and 8 in the right area is checked, and then the average brightness of divided areas 3, 4, 5, and 6 in the center area is checked. After that, the average brightness of the ranges from 2 m to 3 m, from 3 m to 4 m, and from 4 m to 6 m is checked in the same order.

Note that the order in which the average brightness of the divided areas is checked is not limited to the above; for example, it may be checked in the order of the left area, the center area, and the right area, or vice versa. There may be. Furthermore, instead of checking the average brightness of the divided areas horizontally, it may be checked from the front side to the back side. For example, the average brightness of the left area from 1 m to 2 m, 2 m to 3 m, 3 m to 4 m, and 4 m to 6 m may be checked, and then the average brightness of the right area may be checked from the front side to the back side.

FIG. 14 shows a flow for determining the influence of the road surface in the left area or the right area. The influence determination unit 150 first sets the higher average luminance of the two divided areas to nMax (S400). For example, when determining the influence of the road surface in the left area, the higher average luminance of divided areas 1 and 2 in the range of 1 m to 2 m shown in FIG. 13(B) is set as nMax. Next, the influence determination unit 150 determines whether the divided area of nMax has high brightness (S410). For example, when nMax≧180, it is determined that the brightness is high. If nMax is high brightness, it is determined that there is an influence from the road surface in the left area or the right area (S440). For example, in determining the influence of the road surface in the left area, when the average luminance of divided area 1 or 2 is 180 or more, it is determined that there is an influence of the road surface in the left area.

On the other hand, if the divided area of nMax is not high brightness, next, the minimum average brightness among the four divided areas of the two divided areas and the central area is set to nMin (S420). In the example of FIG. 13(B), the minimum average brightness among divided areas 1, 2, 3, 4, 5, and 6 in the range of 1 m to 2 m is set to nMin. Next, the influence determination unit 150 determines whether the difference between nMax and nMin is greater than or equal to a threshold value (S430). If this brightness difference is equal to or greater than the threshold value, it is determined that there is an influence on the road surface in the left area or the right area (S440), and if not, it is determined that there is no influence on the road surface (S450).

As described above, the influence determination unit 150 determines the influence of the road surface in the left area, then the influence of the road surface in the right area, and then the influence of the road surface in the center area. Figure 15 shows the flow of determining the influence of the road surface in the center area. The influence determination unit 150 sets the maximum average brightness of the four divided areas to nMax (S500). In the example of Figure 13 (B), when determining the influence of the road surface in the center area, the maximum average brightness of divided areas 3, 4, 5, and 6 in the range of 1 m to 2 m is set to nMax. Next, the influence determination unit 150 determines whether the divided area of nMax is high brightness or not (S510). For example, if nMax ≧ 180, it is determined to be high brightness. If nMax is high brightness, it is determined to be influenced (S540).

On the other hand, if the divided area of nMax is not high brightness, the minimum average brightness among each of the four divided areas and the adjacent left and right areas is set to nMin (S520). In the example of FIG. 13(B), the minimum average brightness is set to nMin in divisions 2, 3, 4, 5, 6, and 7 in the range of 1 m to 2 m. Next, the influence determination unit 150 determines whether the difference between nMax and nMin is greater than or equal to a threshold value (S530). If this brightness difference is equal to or greater than the threshold value, it is determined that there is an influence of the road surface on the central area (S540), and if not, it is determined that there is no influence (S550).

Next, a method for determining influence in a small area shown in FIG. 11 will be described. Small-area impact determination is carried out in the range of 1 m to 4 m on the road surface, (1) in the four-part area of the central area in order from left to right, (2) in the order of 1 m to 2 m, 2 m to 3 m, 3 m to 4 m, Judgment ends when there is an impact. FIG. 16 shows an area for determining the influence of a small range. First, the presence or absence of influence is determined in the order of divided areas 1, 2, 3, and 4 in the range of 1 m to 2 m, then the presence or absence of influence is determined in the range of 2 m to 3 m, and then 3 m to 4 m.

FIG. 17 is a flowchart for determining the influence of a small area. The influence determination unit 150 calculates the average brightness of the divided areas, and calculates a minimum threshold value and a maximum threshold value from this average brightness (S600). In the example of FIG. 16, first, the average brightness of divided area 1 is calculated. The minimum threshold value is, for example, 1/2 of the average brightness, and the maximum threshold value is, for example, twice the average brightness.

Next, the influence determination unit 150 examines all pixels in the divided area, calculates the number of pixels with a luminance smaller than the minimum threshold value, and sets this number of pixels to nNumMin (S610). Similarly, all the numbers of pixels in the divided area are checked, the number of pixels with luminance greater than the maximum threshold is calculated, and this number of pixels is set to nNumMax (S620). In the example of divided area 1, all pixels in divided area 1 are checked, and the number of pixels with a luminance smaller than nNumMin and the number of pixels with luminance larger than nNumMax are calculated. Next, the influence determination unit 150 determines that there is an influence if nNumMin is greater than or equal to the threshold value or NnumMax is greater than or equal to the threshold value (S640), and otherwise determines that there is no influence (S650). If it is determined that there is no influence, a similar determination is made for the next divided area. FIG. 18 is an example of an image when it is determined that there is a small area affected from 2 m to 3 m in the central area, and here it can be seen that a manhole (small area) on the road is affected.

In this way, the influence determination section 150 determines the influence of direct sunlight on the road surface, and provides the determination result to the obstacle detection section 130. The obstacle detection unit 130 detects obstacles based on point cloud data, but if the impact determination unit 150 determines that there is an impact on the road surface, for example, the obstacle detection result may not be output. , or output that there is an effect of direct sunlight along with the obstacle detection result, or output that the reliability of the obstacle detection result is not good.

In the above example, the influence determination unit 150 determines the influence on the road surface from the brightness characteristics of the IR image, but in order to supplement the reliability of this determination, the influence determination unit 150 considers the number of generated point clouds. Good too. Missing images occur when direct sunlight is reflected on the road surface, causing overexposure. Therefore, the impact determination unit 150 receives the point cloud data generated by the point cloud data generation unit 120, extracts the road surface part from the point cloud data, and calculates the point cloud caused by direct sunlight based on the number of point clouds on the road surface. Estimate the presence or absence of omissions and noise. For example, when there is a road surface up to 7 meters ahead, the number of points in the cloud is approximately 10,000. In such a case, if the number of positioned point groups is less than 3 m (for example, the number of point groups is 3500 or less), it is assumed that there is a missing point group or noise due to direct sunlight. Therefore, even if the influence determination unit 150 determines that there is no influence on the road surface based on the characteristics of the brightness of the road surface, if the number of point clouds on the road surface is less than or equal to the threshold, it is determined that there is an influence on the road surface due to direct sunlight. It may be determined. Alternatively, the influence determination unit 150 may determine that there is an influence on the road surface using as a weighted condition that the number of points on the road surface is less than or equal to a threshold value.

In the above embodiment, an example is shown in which an IR camera is used as a stereo camera, but the stereo camera is not limited to an IR camera, but may be an RGB camera that captures color images. In this case, the direct sunlight determining unit 140 determines the presence or absence of direct sunlight based on the brightness of the RGB image captured by the RGB camera (RGB data is converted to brightness data using a predetermined formula), and the influence determining unit 150 The presence or absence of the influence of direct sunlight is determined based on the brightness of the RGB image.

Next, FIG. 19 shows a driving support system to which the obstacle detection device of this embodiment is applied. The driving support system 200 includes the obstacle detection device 100 described in the previous embodiment, a warning section 210 that issues a warning to the passenger, and a driving support module 220 that supports driving of the vehicle M. The warning unit 210 includes, for example, a display unit and an audio output unit, and when an obstacle is detected in front of the vehicle M, it notifies the user of the presence of the obstacle using image information and audio information. Further, the driving support module 220 cooperates with the driving control of the vehicle M to decelerate the vehicle M in order to prevent a collision with an obstacle, or change the traveling direction of the vehicle M to avoid the obstacle. let

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to specific embodiments, and various modifications and variations can be made within the scope of the gist of the invention described in the claims. Changes are possible.

100: Obstacle detection device 110L, 110R: Stereo camera 120: Point cloud data generation section 130: Obstacle detection section 140: Invalid area identification section 150: Object recognition section 160: RGB camera

Claims (15)

stereo camera and
generation means for generating point cloud data based on the stereo image taken by the stereo camera;
Obstacle detection means for detecting obstacles based on the generated point cloud data;
direct sunlight determining means for determining the presence or absence of direct sunlight based on the brightness of the image taken by the stereo camera;
influence determining means for determining the influence of direct sunlight on the road surface when the direct sunlight determining means determines that there is direct sunlight;
The obstacle detection means outputs an obstacle detection result according to the determination result of the influence determination means,
The influence determining means divides the area of the road surface in a certain range on the near side of the captured image into a plurality of areas, and when the average brightness of the divided areas is high brightness, the obstacle is determined to have an influence on the road surface. Detection device. stereo camera and
generation means for generating point cloud data based on the stereo image taken by the stereo camera;
Obstacle detection means for detecting obstacles based on the generated point cloud data;
direct sunlight determining means for determining the presence or absence of direct sunlight based on the brightness of the image taken by the stereo camera;
an influence determination means for determining an influence of the direct sunlight on a road surface when the direct sunlight determination means determines that there is direct sunlight;
The obstacle detection means outputs an obstacle detection result according to the determination result of the influence determination means,
The influence determining means is an obstacle detection device that determines the influence of direct sunlight on the road surface based on the number of points on the road surface of the point cloud data. stereo camera and
generation means for generating point cloud data based on the stereo image taken by the stereo camera;
Obstacle detection means for detecting obstacles based on the generated point cloud data;
direct sunlight determining means for determining the presence or absence of direct sunlight based on the brightness of the image taken by the stereo camera;
influence determining means for determining the influence of direct sunlight on the road surface when the direct sunlight determining means determines that there is direct sunlight;
The obstacle detection means outputs an obstacle detection result according to the determination result of the influence determination means,
The influence determining means is an obstacle detection device that determines the influence of direct sunlight on the road surface based on the brightness characteristics of the road surface in the captured image and the number of points on the road surface in the point cloud data. Any one of claims 1 to 3, wherein the obstacle detection means outputs a message indicating that the obstacle detection result is not normal or that the reliability of the detection result is low when the influence means determines that there is an influence from the road surface. The obstacle detection device according to item 1 . 4. The obstacle detection device according to claim 1, wherein the direct sunlight determination means determines the presence or absence of direct sunlight based on whether a road in a predetermined range of the captured image has high brightness. The direct sunlight determination means detects the average brightness of the entire image, the average brightness of the sky area, and the average brightness of the road in a certain range on the near side of the image, and determines whether the average brightness of the entire image is higher than the average brightness of the sky area. 6. The obstacle detection device according to claim 5 , wherein the road is determined to have high brightness when the average brightness of the road is higher than the average brightness of the entire image. 7. The obstacle detection device according to claim 5 , wherein the direct sunlight determining means further determines whether there is direct sunlight based on the brightness of an empty area when the road is not highly bright. The direct sunlight determination means changes the range for detecting the brightness of the sky area according to the brightness of the road area, and determines that there is direct sunlight when the brightness of the sky area is high brightness. 7. The obstacle detection device according to 7 . The influence determining means determines that there is an influence on the road surface when the average luminance of the divided area is not high and the difference between the maximum value and the minimum value of the average luminance of the divided area is equal to or more than a threshold value. 1. The obstacle detection device according to 1 . The influence determination means divides the road surface area into a plurality of areas in the vertical direction and divides the area in the horizontal direction into a left area, a right area, and a center area, and determines the influence of the divided areas of the left area, right area, and center area. The obstacle detection device according to claim 1 , which determines the presence or absence of an obstacle. The influence determination means calculates a minimum threshold value and a maximum threshold value from the average brightness of the divided area, and further calculates a minimum number of pixels with a brightness lower than the minimum threshold value and a maximum number of pixels with a brightness higher than the maximum threshold value of the divided area, The obstacle detection device according to claim 1 , which determines that there is an influence on the road surface when the minimum number of pixels is greater than or equal to a threshold value or the maximum number of pixels is greater than or equal to a threshold value. 4. The obstacle detection device according to claim 1, wherein the point cloud data is three-dimensional information including distance information obtained by combining captured images of stereo cameras. The obstacle detection device according to any one of claims 1 to 3, wherein the stereo camera is an infrared camera. 4. The obstacle detection device according to claim 1, wherein the stereo camera captures an image of an area ahead of a moving object. An obstacle detection device according to any one of claims 1 to 14 ,
A control means for controlling the movement of the mobile object using the detection result by the obstacle detection device;
moving objects including

Images