JP6949090B2 - Obstacle detection device and obstacle detection method - Google Patents

Description

The present application relates to an obstacle detection device and an obstacle detection method.

In recent years, as one of the in-vehicle monitoring devices, the development of a technology using an in-vehicle camera device which is a sensor for recognizing the shape of an object has been progressing. The in-vehicle camera device eliminates blind spots by mounting a plurality of in-vehicle cameras around the vehicle. The in-vehicle camera device recognizes an object to be photographed such as an obstacle by a dedicated recognition processing device, so that the vehicle can be operated while avoiding contact with the obstacle.

A camera for monitoring the surroundings of a vehicle is attached to a vehicle mounting location as a small camera module. The installation location is centered on the grill guards and door mirrors provided on the front and rear of the vehicle. In many cases, the lens angle of view of the camera module, the vehicle mounting angle, and the like are optimized for each vehicle in order to compensate for the blind spot.

As a method of recognizing obstacles using this camera, an optical flow that captures the feature points of an object photographed as an obstacle and examines how the feature points move between individual images that are video frames. The method is disclosed, and for example, the relative positional relationship with an obstacle is calculated based on the optical flow of the captured image (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2011-203766

In Patent Document 1, the relative positional relationship with an obstacle can be calculated based on the optical flow. However, in the case of obstacle detection by optical flow, there is a problem that it is necessary to improve the detection accuracy because optical flow is generated and erroneously detected even for cracks, dirt, or road markings and patterns at equal intervals on the road surface. ..

The present application has been made to solve the above-mentioned problems, and an object of the present application is to obtain an obstacle detection device having improved detection accuracy for obstacles without additionally providing a sensor other than a camera. It is supposed to be.

The obstacle detection device disclosed in the present application is installed in a vehicle, takes an image of the surroundings of the vehicle, acquires a plurality of images in a time series, and calculates an optical flow of feature points extracted from the plurality of images. From the optical flow calculation unit that extracts the feature points approaching the vehicle as approach feature points and outputs the information of the approach feature points, and the road surface area that can determine the surroundings of the vehicle as the road surface in each of the plurality of images. A road surface detection unit that outputs the result of dividing into obstacle areas where objects other than the road surface are shown, and an obstacle detection unit that detects obstacles around the vehicle based on the information of approach feature points and the information of the road surface area. The obstacle detection unit corrects the coordinates of the four corners of the rectangular bounding box surrounding the detected obstacle based on the road surface area and the obstacle area output from the road surface detection unit .

According to the obstacle detection device disclosed in the present application, the detection accuracy for obstacles is improved without additionally providing a sensor other than the camera.

It is a figure which shows the outline of the structure of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the outline of the process of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the photographing range of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the image taken by the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the outline of the optical flow processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the outline of the road surface detection processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the pattern of the histogram of the road surface detection processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of another pattern of the histogram of the road surface detection processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of another pattern of the histogram of the road surface detection processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the outline of the obstacle detection processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows another example of the image taken by the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the outline of the processing added to the obstacle detection apparatus which concerns on Embodiment 2. FIG. It is a figure which shows the example of the image taken by the obstacle detection apparatus which concerns on Embodiment 2. FIG. It is a figure which shows the outline of the structure of the obstacle detection apparatus which concerns on Embodiment 3. It is a figure which shows the outline of the structure of the obstacle detection apparatus which concerns on Embodiment 4. FIG. It is a figure which shows the example of the image taken by the obstacle detection apparatus which concerns on Embodiment 4. FIG. It is a figure which shows the example of the image taken by the obstacle detection apparatus which concerns on Embodiment 5. It is a figure which shows another example of the image taken by the obstacle detection apparatus which concerns on embodiment 5. It is a block diagram which shows an example of the hardware of an obstacle detection ECU.

Hereinafter, the obstacle detection device and the obstacle detection method according to the embodiment of the present application will be described with reference to the drawings. In each figure, the same or corresponding members and parts will be described with the same reference numerals.

Embodiment 1.
FIG. 1 is a diagram showing an outline of the configuration of the obstacle detection device 100 according to the first embodiment, FIG. 2 is a diagram showing an outline of processing of the obstacle detection device 100, and FIG. 3 is an image pickup unit 1 of the obstacle detection device 100. FIG. 4 is a diagram showing an example of a shooting range 12 of the camera 11, and FIG. 4 is a diagram showing an example of an image captured by the obstacle detection device 100. The obstacle detection device 100 is a device mounted on a vehicle driven by a driver (hereinafter referred to as the own vehicle 10) and detects an obstacle approaching the own vehicle 10. The obstacle is, for example, as shown in FIG. 4, a moving object such as another vehicle including a two-wheeled vehicle (hereinafter, referred to as an approaching vehicle 22). As shown in FIG. 1, the obstacle detection device 100 includes an imaging unit 1 that captures the surroundings of the own vehicle 10, a frame buffer 2 that stores image data captured in time series, and feature points that approach the own vehicle 10. Optical flow calculation unit 3 that extracts information on approach feature points by extracting them as approach feature points, road surface detection unit 4 that extracts the road surface area around the own vehicle 10, and obstacles that detect obstacles around the own vehicle 10. It includes a detection unit 5, an image display unit 6, an alarm unit 7, and a control unit 8 that utilize the detection result of an obstacle. Among these components, the processing performed by the frame buffer 2, the optical flow calculation unit 3, the road surface detection unit 4, and the obstacle detection unit 5 is an obstacle detection ECU (Electronic Control Unit) having an image processing function. At 9, it is executed by the program. Since the control unit 8 is not included in the obstacle detection ECU 9, it is connected to the obstacle detection ECU 9 via an in-vehicle network.

As shown in FIG. 19, an example of the hardware of the obstacle detection ECU 9 is composed of a processor 110 and a storage device 111. The storage device 111 including the program executed by the obstacle detection ECU 9 and the image data stored in the frame buffer 2 or the like is, for example, a volatile storage device such as a random access memory and a non-volatile auxiliary such as a flash memory. It is equipped with a storage device. Further, an auxiliary storage device of a hard disk may be provided instead of the flash memory. The processor 110 executes a program input from the storage device 111, and the obstacle detection unit 5 provided in the obstacle detection ECU 9 detects obstacles around the own vehicle 10. In this case, a program is input from the auxiliary storage device to the processor 110 via the volatile storage device. Further, the processor 110 may output data such as a calculation result to the volatile storage device of the storage device 111, or may store the data in the auxiliary storage device via the volatile storage device.

The outline of the process for detecting an obstacle performed by each part of the obstacle detection device 100 shown in FIG. 1 will be described with reference to FIG. The detection process includes four steps from step S1 to step S4. The step S1 performed by the imaging unit 1 is a step of photographing the surroundings of the own vehicle 10 and acquiring and outputting a plurality of images in chronological order. In step S2 performed by the optical flow calculation unit 3, the feature points approaching the own vehicle 10 are extracted as approach feature points from the optical flow calculation of the feature points extracted from a plurality of images, and the information of the approach feature points is output. It is a step. In step S3 performed by the road surface detection unit 4, for example, in the acquired image shown in FIG. 4, the alarm detection area 14 set as the detection target road surface is divided around the own vehicle 10 and a plurality of road surface detection areas 15 to 17 are divided. Then, a histogram of the brightness value of each road surface detection area is created, and the road surface area is extracted from a plurality of road surface detection areas 15 to 17 by comparing with the histogram as reference data that can be judged as the road surface, and the warning detection area 14 is set. This is a step of outputting the result of dividing into the road surface area and the obstacle area in which an object other than the road surface is shown. In step S4 performed by the obstacle detection unit 5, a bounding box 19 is created as an obstacle information area surrounded by obstacle coordinates based on the information of the approach feature point and the information of the road surface area, and the surroundings of the own vehicle 10 are generated. This is a step to detect an obstacle. When an obstacle is detected around the own vehicle 10, a step of warning the driver by sound or vibration (S5), a step of displaying a video to notify the driver (S6), and the own vehicle 10 The step (S7) of controlling the own vehicle 10 so as to avoid contact with the obstacle indicated by the bounding box 19 is performed by each of the alarm unit 7, the image display unit 6, and the control unit 8. Each step of S5, S6, and S7 is not performed at all, and any of the steps may be performed. Further, the time when an obstacle is detected around the own vehicle 10 may be defined as the time when at least a part of the alarm detection area 14 is included in the created bounding box 19. Hereinafter, the details of the configuration of the obstacle detection device 100 and the details of the processing will be described.

The imaging unit 1 is a camera 11 that is installed in the own vehicle 10 and photographs the surroundings of the own vehicle 10 to acquire a plurality of images in chronological order. As shown in FIG. 3, the shooting range 12 of the camera 11 installed at the position of the door mirror is the rear left side of the own vehicle 10. The shooting range 12 is set for the purpose of compensating for the blind spot of the driver when the own vehicle 10 is a vehicle with a right-hand drive. The installation location of the camera 11 is not limited to the position of the door mirror, and the installation location may be provided at another location on the left side surface of the own vehicle 10 to photograph the rear left side. Further, the photographing range 12 is not limited to the rear left side of the own vehicle 10, and the area for detecting an obstacle may be changed by photographing the rear right side of the own vehicle 10 or the rear side of the own vehicle 10.

The image acquired by the imaging unit 1 will be described with reference to FIG. In the example of the image shown in FIG. 4, the area required for the process of detecting an obstacle is shown by a broken line. The angle of view of the image is set so that the side surface 10a of the own vehicle 10 is included. This is to determine the alarm detection area 14 and the optical flow calculation area 20 used for the calculation of the optical flow with reference to the side surface 10a of the own vehicle 10. The warning detection area 14 is an area that is predetermined as an area necessary for detecting obstacles around the own vehicle 10. The alarm detection area 14 in the present embodiment is an area corresponding to 1 m laterally from the side surface 10a of the own vehicle 10 shown in the image and 30 m behind the own vehicle 10 from the camera 11. The end of the extension of the alarm detection area 14 in the image is the vanishing point 13. The alarm detection area 14 includes a crack 21a which is a non-obstacle 21 and a white line 21b. The alarm detection area 14 is not limited to this, and may be changed or not specified. For example, when the target area for obstacle detection is the entire area shown in the image without defining the warning detection area 14, the warning detection area 14 is the area on the right side excluding the own vehicle 10 in FIG. 4 and disappears. The region below the point 13 may be used. This is because the road surface appears in the region below the vanishing point 13 in the image. The warning detection area 14 is divided into a plurality of blocks according to the distance behind the own vehicle 10. The road surface detection area 15, the road surface detection area 16, and the road surface detection area 17 are set from the side closer to the own vehicle 10.

The image data captured by the camera 11 is output to the frame buffer 2. The frame buffer 2 stores image data for the past several frames. The image data is output to the optical flow calculation unit 3 and the road surface detection unit 4 in the subsequent stage via the frame buffer 2.

The optical flow calculation unit 3 extracts the feature points approaching the own vehicle 10 as the approach feature points from the optical flow calculation of the feature points extracted from the plurality of images, and outputs the information of the approach feature points. FIG. 5 is a diagram showing an outline of optical flow processing of the obstacle detection device 100 according to the first embodiment. The details of the processing performed by the optical flow calculation unit 3 will be described with reference to FIG. First, the optical flow calculation area 20 used for the optical flow calculation is acquired as a partial image from the image (step S101). In the present embodiment, the optical flow calculation area 20 is the area on the right side of the image excluding the own vehicle 10, but any area in the image may be used as long as it is an area for detecting an obstacle. Next, the corner points, which are the points where the edges on the image intersect, are detected in the optical flow calculation area 20, and the corner points are extracted as feature points (step S102). There are many methods for detecting corner points, such as Harris's corner detection algorithm, but any method may be used.

Next, the feature points extracted from the latest image are matched with the feature points extracted from the past image, and the optical flow is calculated (step S103). Finally, the optical flow approaching the own vehicle 10 is extracted from the calculated result (step S104). In FIG. 3, the vertical direction is the Y direction, the downward direction is positive, the horizontal direction is the X direction, and the right direction is positive. In this definition, the optical flow of an obstacle approaching the own vehicle 10 from behind has an optical flow in which the Y component is positive. On the contrary, the optical flow of a structure away from the own vehicle 10, such as a building and a vehicle parked on the road, has an optical flow in which the Y component is negative. Therefore, in the extraction of the optical flow, the optical flow in which the Y component is positive is extracted. The optical flow calculation unit 3 outputs the information of the approach feature points included in the extracted optical flow to the obstacle detection unit 5 (step S105). The information of the approaching feature point is the coordinates of the approaching feature point, the Y component length of the optical flow, the X component length, the direction, the length of the movement amount, and the like.

In each of the plurality of images, the road surface detection unit 4 divides the warning detection area 14 set as the detection target road surface around the own vehicle 10 into a plurality of road surface detection areas 15 to 17, and each road surface detection area A histogram of the brightness value of is created, compared with the histogram as reference data that can be judged as the road surface, the road surface area is extracted from a plurality of road surface detection areas 15 to 17, and the warning detection area 14 is defined as a road surface area and an object other than the road surface. The result of dividing into the obstacle area where is shown is output. FIG. 6 is a diagram showing an outline of the road surface detection process of the obstacle detection device 100 according to the first embodiment. The details of the processing performed by the road surface detection unit 4 will be described with reference to FIG. First, the alarm detection area 14 is acquired as a partial image from the image (step S201). Next, the road surface histogram extraction area is set (step S202). The road surface detection area set closest to the own vehicle 10 is used as the road surface histogram extraction area, and the histogram obtained from the road surface histogram extraction area is used as reference data. In the road surface detection method of the present embodiment, it is premised that the area closest to the own vehicle 10 is the road surface. Therefore, a histogram created from an image of the nearest region of the own vehicle 10 is used as reference data. In the warning detection area 14 shown in FIG. 3, the road surface detection area 15 which is the area closest to the own vehicle 10 is set as the histogram extraction area. Since the histogram extraction area may be the area closest to the own vehicle 10 in the acquired image, the setting may be changed according to the angle of view of the camera, the installation position of the camera, or the orientation of the camera.

Next, a histogram is created in each of the road surface detection areas 15 to 17 of the warning detection area 14 (step S203). Luminance values are assigned to each pixel in the road surface detection areas 15 to 17, and a histogram of the luminance values is created with the horizontal axis as the luminance value and the vertical axis as the frequency. The luminance value is represented, for example, in the range of 0 (black) to 255 (white). FIG. 7 is a diagram showing a histogram for the road surface detection region 15 as an example of the histogram pattern created by the road surface detection process of the obstacle detection device 100 according to the first embodiment. Since the road surface detection region 15 occupies most of the road surface region, the brightness of gray is large and the frequency peak of G1 appears. The histogram is a diagram that captures the characteristics of the image by showing the brightness value on the horizontal axis as the class and the frequency on the vertical axis, but the road surface is painted with a yellow line in addition to the white line, and further with red or blue paint. Since there is also a road surface, R, G, and B may be combined to form a class, and the frequency of each class may be taken to capture the characteristics of the road surface including the color.

FIG. 8 is a diagram showing a histogram for the road surface detection area 16 as an example of a histogram pattern created by the road surface detection process of the obstacle detection device 100 according to the first embodiment, and FIG. 9 is a histogram for the road surface detection area 17. It is a figure which shows. In the histogram of the road surface detection area 16, the frequency peak of G2 indicating the characteristic brightness as the road surface appears as in G1 of the road surface detection area 15. Further, since the road surface detection region 16 includes the white line 21b, the frequency peak of G3 corresponding to the brightness of the white line 21b appears. As shown in FIG. 9, in the histogram of the road surface detection area 17, the frequency peak of G5 showing the characteristic brightness as the road surface appears like G1 of the road surface detection area 15 and G2 of the road surface detection area 16, and the brightness of the white line 21b. The frequency peak of G6 corresponding to is appearing. Further, since the road surface detection region 17 includes the approaching vehicle 22 approaching the own vehicle 10, two frequency peaks G4 and G7 corresponding to the approaching vehicle 22 appear. G4 is the lowest frequency peak of the brightness that is the tire and shadow of the approaching vehicle 22 at the boundary between the approaching vehicle 22 and the road surface. G7 is a side surface of the own vehicle 10 that is generally in a color that easily reflects light, and is a frequency peak with high brightness.

Finally, using the histogram of the road surface detection area 15 as reference data, an area that is the road surface is extracted from the warning detection area 14 (step S204). There are many methods to extract the road surface area using the histogram, but here, the histogram of the reference data is compared with the histogram of other areas, and the area of the histogram similar to the reference data is determined as the road surface, and as the road surface area. Extract. The similarity of the histogram can be determined from the shape of the histogram and the range of the distributed luminance values, but the method for determining the similarity of the histogram is not limited to this method. A binarized image is created by setting the brightness value of the area determined to be the road surface to 255 and the brightness value of the area determined to be not the road surface (hereinafter referred to as an obstacle area) to 0. The road surface detection unit 4 outputs a binarized image, which is the result of dividing the alarm detection area 14 into the road surface area and the obstacle area, to the obstacle detection unit 5 (step S205). Further, the road surface detection unit 4 transmits information on the brightness value of each pixel of the alarm detection area 14 used for creating the histogram to the obstacle detection unit 5 in order to use it for correcting the obstacle coordinates at the four corners of the bounding box, which will be described later. Output.

Although the warning detection area 14 is divided into three to define the road surface detection areas 15 to 17, the road surface area may be extracted for a small area obtained by further dividing the road surface detection area. By creating a histogram from the brightness value of a small area and determining whether it is a road surface from the similarity of the histogram, the distance resolution between the road surface and the own vehicle is improved. Further, instead of creating a histogram for each small area, the road surface can be determined for each pixel from the brightness value of each pixel in the alarm detection area 14 by using a generally known back projection method of the histogram. I do not care. It is preferable to select the optimum road surface extraction method according to the distance resolution required for road surface detection.

In the present embodiment, the road surface detection method for dividing the warning detection area 14 into the road surface area and the obstacle area is described by an example using a histogram, but the road surface detection method is not limited to this method. As long as the requirements of this method are satisfied, the road surface detection method does not matter. Other road surface detection methods include a road surface detection method using motion stereo, a road surface detection method in which color features of the road surface are learned, and a road surface detection method using image segmentation using deep learning.

The obstacle detection unit 5 is an obstacle information area surrounded by obstacle coordinates based on the approach feature point information output from the optical flow calculation unit 3 and the road surface area information output from the road surface detection unit 4. A certain bounding box is created to detect an approaching vehicle 22 which is an obstacle around the own vehicle 10. FIG. 10 is a diagram showing an outline of an obstacle detection process of the obstacle detection device 100 according to the first embodiment. The details of the processing performed by the obstacle detection unit 5 will be described with reference to FIG. First, the information on the approaching feature points in the road surface region is deleted from the input approaching feature point information (step S301). When the coordinates of the approaching feature point are in the road surface area, the information of the feature point is determined to be an erroneous flow generated from a pattern on the road surface or a non-obstacle such as a crack and deleted. Next, from the coordinates of the approaching feature points left without being deleted and the flow direction of the approaching feature points, the approaching feature points that are close to each other and have the same flow direction are grouped together (step). S302). A rectangular bounding box surrounding the approaching feature points is created by classifying a group of a plurality of approaching feature points generated by grouping (step S303). In this way, the bounding box is created by using the information of the approach feature points included in the area other than the road surface area. The created information of each bounding box is output to the video display unit 6, the alarm unit 7, and the control unit 8 as the detection result of obstacles around the own vehicle 10 (step S304). In FIG. 4, the information of the created bounding box 19 is output. The information of the bounding box is the coordinates, the size, the number, the average length of the flow, and the average direction of the flow.

When an obstacle is detected around the own vehicle 10, the alarm unit 7 notifies the driver of the own vehicle 10 by sound or vibration. When notifying by vibration, for example, a signal is sent to EPS (electric power steering), and the steering wheel is vibrated by generating high-frequency vibration in EPS to notify the driver. In order to vibrate the handle, a handle vibration motor may be attached to the handle and the handle may be vibrated by vibrating the motor. In addition, notification by sound or vibration may be used properly. For example, when the bounding box contains at least a part of the alarm detection area 14, a sound is first notified, and when the approaching vehicle 22 approaches a place where the own vehicle 10 may come into contact, a vibration is notified. You may.

When an obstacle is detected around the own vehicle 10, the image display unit 6 notifies the driver of the own vehicle 10 by displaying the position of the bounding box as an image. Further, the control unit 8 controls the own vehicle 10 so that the contact between the own vehicle 10 and the approaching vehicle 22, which is an obstacle indicated by the bounding box, is avoided. The own vehicle 10 may not include all of the image display unit 6, the alarm unit 7, and the control unit 8, and may include any one or two of them.

When the coordinates of the approaching feature point are in the road surface region, the approaching feature point related to the crack 21a and the white line 21b shown in FIG. 3 is deleted by deleting the information of the approaching feature point. The approach feature points to be deleted are not limited to these examples, and the puddle in the road surface area and the approach feature points related to the zebra pattern drawn on the road surface are also deleted. FIG. 11 is a diagram showing another example of an image taken by the obstacle detection device 100 according to the first embodiment. A curb 21c on the shoulder of the road is regularly and repeatedly formed at the end of the alarm detection area 14. The feature points for the curb 21c may also be included in the approach feature point information, but even for non-obstacles such as the regularly formed curb 21c, if the coordinates of the approach feature points are in the road surface detection area, The information of the approach feature point is deleted.

In the above, the optical flow calculation and the road surface extraction are performed in parallel for the input image. First, the road surface area is extracted, and based on the output result, the obstacle area is obtained. By calculating the optical flow, it is possible to reduce the erroneous flow due to the pattern of the road surface area or the like. In addition, the bounding box was created after deleting the approach feature points included in the road surface area, but is not limited to this. The bounding box is created without deleting the approach feature points, and then it is included only in the road surface area. You can remove the bounding box to eliminate erroneous flows. In addition, although obstacles were detected by creating a bounding box, obstacle detection is not limited to the creation of a bounding box, even if it is an obstacle detection using a point cloud or a binarized image. I do not care.

As described above, the obstacle detection device 100 is based on the information of the approach feature point output from the optical flow calculation unit 3 and the information of the road surface area output from the road surface detection unit 4, and is based on the information around the own vehicle 10. Since the approaching vehicle 22 which is an obstacle is detected, the detection accuracy for the approaching vehicle 22 which is an obstacle can be improved without additionally providing a sensor other than the camera. Further, since the bounding box is created based on the information of the approaching feature points included in the area other than the road surface area, the information of the approaching feature points included in the road surface area is deleted, and the obstacle detection accuracy can be improved. .. Further, when an obstacle is detected around the own vehicle 10, an alarm unit 7 for notifying the driver of the own vehicle 10 by sound or vibration is provided, so that the driver is notified of the existence of the approaching vehicle 22 at an appropriate timing. be able to. Further, when an obstacle is detected around the own vehicle 10, the driver of the own vehicle 10 is provided with an image display unit 6 that displays the position of the bounding box as an image and notifies the driver of the approaching vehicle 22. The position can be notified at an appropriate timing. Further, since the detection accuracy for the approaching vehicle 22 is improved, false alarms are reduced. Further, when an obstacle is detected around the own vehicle 10, the control unit 8 for controlling the own vehicle 10 is provided so that the contact between the own vehicle 10 and the obstacle indicated by the bounding box is avoided. It is possible to avoid contact with obstacles at an appropriate timing without the intervention of a driver.

Embodiment 2.
The obstacle detection device 100 according to the second embodiment will be described. FIG. 12 is a diagram showing an outline of processing added to the obstacle detection device 100. The process of the obstacle detection device 100 according to the second embodiment has a configuration in which, in addition to the process of the first embodiment, a process of correcting the coordinates of the detected obstacle is added.

The obstacle detection unit 5 corrects the coordinates of the detected obstacle based on the road surface area and the obstacle area output from the road surface detection unit 4. Specifically, for example, the obstacle coordinates at the four corners of the bounding box created in step S303 are corrected by using the information of the luminance value of a part of the road surface detection area output from the road surface detection unit 4 (step S401). In the example of the image shown in FIG. 4, the bounding box 19 is created in step S303 shown in FIG. In step S203 shown in FIG. 6, a luminance value is assigned to each pixel of each road surface detection area, and a histogram of each road surface detection area is created. From the comparison between the histogram of the road surface detection area 17 including the frequency peak having the characteristics of the approaching vehicle 22 and the histogram of the road surface detection area 16 having no frequency peak of the approaching vehicle 22, the tires and shadows of the approaching vehicle 22 are obtained. It can be seen that the frequency peak G4 having the lowest brightness is only in the road surface detection region 17. From the brightness value assigned to each pixel of the road surface detection area when creating the histogram, some obstacle other than the road surface, that is, G4 is obtained near the boundary between the road surface detection area 17 and the road surface detection area 16 of the warning detection area 14. It can be recognized that there is a designated area. Based on this information, it can be determined that the estimation that the obstacle coordinates of the bounding box 19 are at the obstacle coordinates of the bounding box 18 matches the histogram obtained by the road surface detection process. The obstacle coordinates at the four corners of the bounding box 19 of the approaching vehicle 22 are corrected to the obstacle coordinates at the four corners of the bounding box 18, and the corrected bounding box 18 information is output as the processing result of the obstacle detection unit 5 (step S402). ). As a result, the accuracy of the distance between the own vehicle 10 and the approaching vehicle 22 can be improved. The correction is not limited to the method using the information of the luminance value, and when the road surface detection method not using the histogram is used, the correction may be performed according to the method used.

FIG. 13 is a diagram showing an example of an image taken by the obstacle detection device 100 according to the second embodiment. The approaching vehicle in FIG. 13 is a two-wheeled vehicle 22a. In the processing of the optical flow calculation unit 3, in the case of the motorcycle 22a, it is particularly difficult to extract the corner points of the tires. Therefore, in the example of the image shown in FIG. 13, the bounding box 24 is created in step S303 shown in FIG. In step S203 shown in FIG. 6, histograms of the road surface detection regions 15 to 17 are created. From the comparison between the histogram of the road surface detection area 16 including the power peak having the characteristics of the motorcycle 22a and the histogram of the road surface detection area 15 having no power peak of the motorcycle 22a, the tire of the motorcycle 22a and the brightness that is the shadow are the most. It can be seen that the low frequency peak is only in the road surface detection region 16. From the brightness values assigned to each pixel of the road surface detection area when creating the histogram, it can be recognized that there is some obstacle other than the road surface near the boundary between the road surface detection area 16 and the road surface detection area 15 of the warning detection area 14. .. Based on this information, it can be determined that it is better to estimate that the obstacle coordinates at the four corners of the bounding box 24 are at the obstacle coordinates at the four corners of the bounding box 23, which is consistent with the histogram obtained by the road surface detection process. The obstacle coordinates at the four corners of the bounding box 24 of the motorcycle 22a are corrected to the obstacle coordinates at the four corners of the bounding box 23, and the corrected bounding box 23 information is output as the processing result of the obstacle detection unit 5. Since the coordinates of the lower end of the bounding box 23 are used to estimate the distance from the own vehicle 10, by correcting the position of the motorcycle 22a in the warning detection area 14 to be closer to 1 m or more, an accurate obstacle to the driver can be caused. Can tell the position of an object.

As described above, in the obstacle detection device 100, the obstacle detection unit 5 outputs the obstacle coordinates of the four corners of the bounding box created by the optical flow calculation unit 3 from the road surface detection unit 4 to a part of the road surface detection area. Since the correction is made by using the information of the brightness value of, the accuracy of the distance between the own vehicle 10 and the approaching vehicle 22 can be improved.

Embodiment 3.
The obstacle detection device 100 according to the third embodiment will be described. FIG. 14 is a diagram showing an outline of the configuration of the obstacle detection device 100. The obstacle detection device 100 according to the third embodiment has a configuration including a vehicle information acquisition unit 25 in addition to the configuration of the first embodiment.

The vehicle information acquisition unit 25 acquires the traveling information of the own vehicle 10. The traveling information is information such as the vehicle speed, yaw rate, and GPS of the own vehicle 10. The vehicle information acquisition unit 25 acquires driving information via, for example, an in-vehicle network of the own vehicle 10. The acquired travel information is output to each of the optical flow calculation unit 3, the road surface detection unit 4, and the obstacle detection unit 5. The optical flow calculation unit 3 extracts approach feature points from the optical flow calculation of the feature points extracted from a plurality of images by using the traveling information. By using the driving information for the obstacle detection process, it is possible to prevent erroneous detection or non-detection of the obstacle and improve the accuracy of the obstacle detection. Hereinafter, a specific example using the driving information will be described.

When the vehicle is turning, the optical flow is different from when the vehicle is traveling straight. When the vehicle turns, an optical flow downward on the screen, such as an approaching obstacle, may be detected from a structure such as a building. Therefore, the erroneous detection is reduced by determining the turning of the own vehicle 10 using the yaw rate as the traveling information and removing the optical flow in the direction corresponding to the turning of the vehicle at the time of the optical flow calculation according to the yaw rate value. Obstacle detection accuracy is improved.

As described above, the obstacle detection device 100 includes a vehicle information acquisition unit 25 that acquires the travel information of the own vehicle 10, and the optical flow calculation unit 3 uses the travel information to extract features extracted from a plurality of images. Since the information on the approaching feature points is extracted from the optical flow calculation of the points, the obstacle detection accuracy can be improved.

Embodiment 4.
The obstacle detection device 100 according to the fourth embodiment will be described. FIG. 15 is a diagram showing an outline of the configuration of the obstacle detection device 100. The obstacle detection device 100 according to the fourth embodiment has a configuration including a road surface boundary line tracking unit 26 in addition to the configuration of the third embodiment.

The road surface boundary line tracking unit 26 is provided between the road surface detection unit 4 and the obstacle detection unit 5, and based on the binarized image output by the road surface detection unit 4, the road surface area and obstacles in the warning detection area 14 The road surface boundary line located between the object area and the object area is recorded in a plurality of continuous images. The road surface boundary line tracking unit 26 outputs the road surface boundary lines of a plurality of images to the obstacle detection unit 5 together with the binarized images output by the road surface detection unit 4.

The obstacle detection unit 5 performs a process of associating obstacles by associating the bounding boxes created in each of the two images between two consecutive images among the plurality of images. In the process, when the bounding box associated with the two images cannot be associated with the latest image, the positions of the road surface boundary lines of the two consecutive images recorded by the road surface boundary line tracking unit 26 are compared. If there is no change in the position of the road surface boundary between the two images, create a bounding box at the same position as the bounding box of the image immediately before the latest image in the latest image, and if there is an obstacle judge. Hereinafter, a specific example using the road surface boundary line will be described.

FIG. 16 is a diagram showing an example of an image taken by the obstacle detection device 100 according to the fourth embodiment. The approaching vehicle in FIG. 16 is a two-wheeled vehicle 22b running in parallel with the own vehicle 10. When the two-wheeled vehicle 22b is running in parallel, the optical flow cannot be calculated because the relative speed is not generated between the own vehicle 10 and the two-wheeled vehicle 22b, and the obstacle detection unit 5 detects an obstacle using the optical flow. Can not. When the obstacles are captured in chronological order, the relative speed is generated when the vehicle approaches the own vehicle 10, so that the obstacles can be detected from the optical flow in the past images. By recording the road surface boundary line detected in the past image, the flow until the appearance or disappearance of the bounding box can be tracked.

Since the two-wheeled vehicle 22b is not affected by the road surface detection areas 15 and 17 of the warning detection area 14, the histogram of FIG. 7 having the characteristics of the road surface is created. From the road surface detection region 16, a histogram similar to FIG. 9 having the characteristics of the motorcycle 22b is generated. When the own vehicle 10 and the two-wheeled vehicle 22b are running in parallel, it is considered that the characteristics of the pattern of the histogram do not change significantly unless the positions of the own vehicle 10 and the two-wheeled vehicle 22b change. Even when an obstacle cannot be detected by the optical flow, the existence and position of the motorcycle 22b can be estimated by comparing the road surface boundary line of a plurality of consecutive images with the road surface boundary line of the latest image.

It can be estimated that there is an obstacle when the bounding box has already been created and the obstacle is recognized. Even when the bounding box suddenly disappears, if there is no change between the past road surface boundary line recorded by the road surface boundary line tracking unit 26 and the current road surface boundary line, the obstacle exists at a relative velocity of 0 from the previous image. Judgment is made, a bounding box is created at the same position as the bounding box created in the previous image, and the bounding box information is output. As a result, obstacles that are in a parallel running state can be continuously detected, so that undetected obstacles are reduced.

As described above, the obstacle detection device 100 includes the road surface boundary line tracking unit 26, and when there is no change in the position of the road surface boundary line of a plurality of images and the position of the road surface boundary line of the latest image, the bounding box is displayed. Since it is determined that the image exists at the same position as the previous image of the latest image, and a bounding box is created at the same position as the previous image in the latest image to determine that an obstacle exists, the approaching vehicle is running in parallel. Even if the obstacle cannot be detected by the optical flow, the obstacle can be continuously detected and the undetected obstacle can be reduced.

Although the case where there is no change in the positions of the road surface boundary lines of the two images has been described, the following processing is performed when there is a change. If the position of the road surface boundary line recedes compared to the previous image, it is judged that the obstacle has receded, and a bounding is created at the position of the road surface boundary line of the latest image. When the road surface boundary line disappears, it is determined that the obstacle has escaped from the warning detection area, and the information in the bounding box is discarded. When the position of the road surface boundary line advances compared to the previous image, the position of the obstacle in the latest image is estimated from the moving speed of the obstacle in the previous image. If the road surface boundary line is on the upper side of the image or on the rear side of the own vehicle in real coordinates from the estimated position, it is judged that the optical flow is not detected but the vehicle is moving slightly forward, and the bounding box is placed on the road surface boundary line. create. If the road surface boundary line is in front of the estimated position, it is determined that another vehicle has retreated from the front of the vehicle, and a bounding box is created on the road surface boundary line.

Embodiment 5.
The obstacle detection device 100 according to the fifth embodiment will be described. The process of the obstacle detection device 100 according to the fifth embodiment has a configuration in which, in addition to the process of the fourth embodiment, a process is added when an obstacle cannot be detected because the bounding box is not created.

When the optical flow calculation unit 3 cannot extract the approach feature point and the obstacle detection unit 5 does not create a bounding box and the obstacle cannot be detected, the obstacle detection unit 5 uses the road surface boundary line of a plurality of consecutive images. The distance between the position and the own vehicle 10 is compared, and when the distance is shortened, the coordinates of the road surface boundary line are detected as the coordinates of the obstacle. A specific example of this process will be described below.

FIG. 17 is a diagram showing an example of an image taken by the obstacle detection device 100 according to the fifth embodiment, and FIG. 18 is a diagram showing another example of the image. The approaching vehicle in FIGS. 17 and 18 is a two-wheeled vehicle 22c approaching the own vehicle 10 from the vicinity of the vanishing point of the image. For obstacles that approach the camera straight from the vicinity of the vanishing point, the optical flow calculation unit 3 cannot extract the approach feature points, and the obstacle detection unit 5 creates a bounding box to detect the obstacle. Can't. This is because the optical flow cannot be calculated because it approaches the camera straight. When the bounding box is not created and the obstacle cannot be detected, the obstacle is detected by using the road surface boundary line.

In the vicinity of the vanishing point, the resolution of the luminance gradation of the alarm detection area 14 is increased, and the alarm detection area 14 in the image is further divided narrowly. When the motorcycle 22c is included in the road surface detection region 17, the brightness near the tire contact is lower than the brightness of the road surface, and a histogram similar to FIG. 9 is generated. This histogram moves from the road surface detection area 17 to the road surface detection area 16 in FIGS. 17 to 18. Along with this, the position of the road surface boundary line also moves. When the distance between the position of the road surface boundary line and the own vehicle 10 is shortened in this way, it is estimated that there is an obstacle at the position of the coordinates of the road surface boundary line, and the coordinates of the road surface boundary line are used as the coordinates of the obstacle. .. By recognizing the approaching state of the road surface boundary line, even if it is difficult for the optical flow calculation unit 3 to create a bounding box, it is possible to estimate and detect an approaching vehicle with respect to the own vehicle 10.

As described above, even when the obstacle detection device 100 cannot extract the approaching feature points, the bounding box is not created, and the obstacle cannot be detected, the position of the road surface boundary line of a plurality of images and the own vehicle 10 When the distance from the vehicle is reduced, the coordinates of the road surface boundary line are detected as the coordinates of the obstacle, so that the obstacle can be detected.

The present application also describes various exemplary embodiments and examples, although the various features, embodiments, and functions described in one or more embodiments are those of a particular embodiment. It is not limited to application, but can be applied to embodiments alone or in various combinations.
Therefore, innumerable variations not illustrated are envisioned within the scope of the techniques disclosed herein. For example, it is assumed that at least one component is modified, added or omitted, and further, at least one component is extracted and combined with the components of other embodiments.

1 Imaging unit, 2 Frame buffer, 3 Optical flow calculation unit, 4 Road surface detection unit, 5 Obstacle detection unit, 6 Video display unit, 7 Alarm unit, 8 Control unit, 9 Obstacle detection ECU, 10 Own vehicle, 10a side , 11 camera, 12 shooting range, 13 vanishing point, 14 alarm detection area, 15 road surface detection area, 16 road surface detection area, 17 road surface detection area, 18 bounding box, 19 bounding box, 20 optical flow calculation area, 21 non-obstacle , 21a crack, 21b white line, 21c curb, 22 approaching vehicle, 22a two-wheeled vehicle, 22b two-wheeled vehicle, 22c two-wheeled vehicle, 23 bounding box, 24 bounding box, 25 vehicle information acquisition unit, 26 road surface boundary line tracking unit, 100 obstacle detection device, 110 processor, 111 storage

Claims (10)

An imaging unit installed in a vehicle that photographs the surroundings of the vehicle and acquires a plurality of images in chronological order.
From the calculation of the optical flow of the feature points extracted from the plurality of images, the optical flow calculation unit that extracts the feature points approaching the vehicle as the approach feature points and outputs the information of the approach feature points.
In each of the plurality of images, a road surface detection unit that outputs the result of dividing the surroundings of the vehicle into a road surface area that can be determined as a road surface and an obstacle area in which an object other than the road surface is shown.
An obstacle detection unit that detects an obstacle around the vehicle based on the information of the approach feature point and the information of the road surface area is provided .
The obstacle detection unit is characterized in that the coordinates of the four corners of the rectangular bounding box surrounding the detected obstacle are corrected based on the road surface region and the obstacle region output from the road surface detection unit. Obstacle detection device. The obstacle detection device according to claim 1, wherein the obstacle detection unit detects the obstacle by using the information of the approach feature point included in a region other than the road surface region. It is equipped with a vehicle information acquisition unit that acquires the traveling information of the vehicle.
The optical flow computing unit uses the running information, wherein the optical flow calculation of feature points extracted from the plurality of images, to claim 1 or claim 2, characterized in that extracting the proximity feature point Obstacle detection device. The road surface boundary line tracking unit for recording the road surface boundary line between the road surface area and the obstacle area in a plurality of continuous images is provided.
The obstacle detection unit performs a process of associating the obstacle detected in each of the two images between two consecutive images of the plurality of images, and in the process, the above 2 When the obstacle associated with one image cannot be associated with the latest image
The positions of the road surface boundary lines of the two consecutive images recorded by the road surface boundary line tracking unit are compared, and the positions of the road surface boundary lines are compared.
When there is no change in the position of the road surface boundary line of the two images, it is determined that the obstacle exists at the same position as the position of the obstacle in the image immediately before the latest image in the latest image. The obstacle detection device according to any one of claims 1 to 3, wherein the obstacle detection device. When the optical flow calculation unit cannot extract the approach feature point and the obstacle detection unit cannot detect the obstacle.
The obstacle detection unit compares the position of the road surface boundary line of the plurality of continuous images with the distance to the vehicle, and when the distance is shortened, the coordinates of the road surface boundary line are used as the coordinates of the obstacle. The obstacle detection device according to claim 4 , wherein the obstacle detection device is detected as coordinates. The invention according to any one of claims 1 to 5 , wherein an alarm unit is provided to notify the driver of the vehicle by sound when the obstacle is detected around the vehicle. Obstacle detection device. Any one of claims 1 to 5 , wherein an alarm unit is provided to notify the driver of the vehicle by vibrating the steering wheel when the obstacle is detected around the vehicle. Obstacle detection device described in. According to claim 1, the image display unit is provided to notify the driver of the vehicle of the position of the obstacle area by displaying the position of the obstacle area as an image when the obstacle is detected around the vehicle. The obstacle detection device according to any one of claims 7. When an obstacle is detected around the vehicle, the vehicle is provided with a control unit that controls the vehicle so that contact between the vehicle and the obstacle indicated by the obstacle area is avoided. The obstacle detection device according to any one of claims 1 to 8. It is an obstacle detection method that detects obstacles approaching the vehicle.
A step of photographing the surroundings of the vehicle, acquiring and outputting a plurality of images in chronological order, and
From the calculation of the optical flow of the feature points extracted from the plurality of images, the step of extracting the feature points approaching the vehicle as the approach feature points and outputting the information of the approach feature points.
In each of the plurality of images, a step of outputting the result of dividing the surroundings of the vehicle into a road surface area that can be determined as a road surface and an obstacle area in which an object other than the road surface is shown.
A step of detecting the obstacle around the vehicle based on the information of the approach feature point and the information of the road surface area, and
A step of correcting the coordinates of the four corners of the detected rectangular bounding box surrounding the obstacle based on the output results for the road surface area and the obstacle area, and
An obstacle detection method characterized by being equipped with.

Images