JP6987173B2 - Obstacle detection device, obstacle detection system equipped with it, obstacle detection method - Google Patents

Description

The present application relates to an obstacle detection device, an obstacle detection system equipped with the obstacle detection device, and an obstacle detection method.

In recent years, as one of the in-vehicle monitoring devices, the development of a technology using an in-vehicle camera device which is a sensor for recognizing the shape of an object has been progressing. The in-vehicle camera device eliminates blind spots by mounting multiple in-vehicle cameras on the vehicle, and avoids contact with obstacles by recognizing obstacles taken by the in-vehicle camera as images with a dedicated recognition processing device. It is possible to operate the vehicle.

The camera for monitoring the surroundings of the vehicle is attached to the grill guards and door mirrors provided in the front and rear of the vehicle as a small camera module. The lens angle of view of the camera module, the vehicle mounting angle, and the like are optimized for each vehicle in order to compensate for the blind spot.

As a method of recognizing obstacles using this camera, there is an optical flow method that captures the feature points of an object taken as an obstacle and investigates how the feature points move between individual images that are video frames. Are known. For example, in Patent Document 1, the relative positional relationship between the vehicle and an obstacle is calculated based on the optical flow of the captured image.

Japanese Unexamined Patent Publication No. 2011-203766

However, in obstacle detection by optical flow, erroneous feature points are associated with complicated patterns such as cracks and dirt on the road surface, which may result in erroneous detection. In addition, optical flow is generated by the fact that road markings drawn at equal intervals, walls with patterns at equal intervals, and structures such as poles or pylon installed at equal intervals appear close to each other due to the stroboscopic effect. False detection may occur, and improving the detection accuracy has been an issue.

In order to improve the detection accuracy of obstacles, it is effective to provide sensors other than cameras, such as LiDAR (Light Detection and Ringing), millimeter-wave radar, and ultrasonic sensors, but there is a problem that the cost is high. ..

The present application discloses a technique for solving the above-mentioned problems, and is an obstacle detection device and an obstacle detection device having improved detection accuracy for obstacles without additionally providing a sensor other than a camera. The purpose is to obtain an object detection method.

In addition, an obstacle detection system equipped with an obstacle detection device with improved detection accuracy for obstacles, which can suppress false detections and false alarms and notify the driver of the existence of obstacles at an appropriate timing. The purpose is to obtain.

The obstacle detection device disclosed in the present application is an obstacle detection device that detects an obstacle approaching a vehicle based on a plurality of time-series images taken by an image pickup unit installed in the vehicle. The optical flow calculation unit that calculates the optical flow of the feature points extracted from the image of, and extracts the feature points approaching the vehicle as the approach feature points, and the road surface area for each of the plurality of images, and the road surface area. Obstacles may exist based on the road surface detection unit that separates the area other than the road surface area, the information of the approach feature points extracted by the optical flow calculation unit, and the information of the road surface area extracted by the road surface detection unit. The target object is an obstacle based on the characteristics of the object learned in advance for the obstacle area limiting part that sets the obstacle area surrounding a certain range and the obstacle area set by the obstacle area limiting part. It is equipped with an object detection unit that determines whether or not it is, and outputs object information including the position and type of the target object, and an alarm determination unit that determines whether or not to notify an alarm based on the object information. be.

Further, the obstacle detection system disclosed in the present application is installed in a vehicle and captures the surroundings of the vehicle to acquire a plurality of images in chronological order, and the vehicle is based on a plurality of images captured by the imaging unit. As an obstacle detection device, the obstacle detection device disclosed in the present application includes an obstacle detection device for detecting an approaching obstacle and an alarm unit for notifying an alarm regarding an obstacle detected by the obstacle detection device. It is prepared.

Further, the obstacle detection method disclosed in the present application is an obstacle detection method for detecting an obstacle approaching a vehicle, and includes an image acquisition step of photographing the surroundings of the vehicle and acquiring a plurality of images in chronological order. , The approach feature point extraction step that calculates the optical flow of the feature points extracted from a plurality of images and extracts the feature points approaching the vehicle as the approach feature points, and the road surface area is extracted for each of the plurality of images. An obstacle is created based on the road surface detection step that separates the road surface area and the area other than the road surface area, the approach feature point information extracted in the approach feature point extraction step, and the road surface area information extracted in the road surface detection step. The target object based on the characteristics of the object learned in advance for the obstacle area limited step that sets the obstacle area that surrounds the range that may exist and the obstacle area that is set in the obstacle area limited step. Includes an object detection step that determines whether or not is an obstacle and outputs object information including the position and type of the target object, and the object detection step includes the approach feature points extracted in the approach feature point extraction step. Moreover, the object existing in the area determined to be other than the road surface area in the road surface detection step is set as the target object.

According to the obstacle detection device and the obstacle detection method disclosed in the present application, even if the approach feature point extracted by the optical flow is not due to an obstacle, the approach feature is detected by the road surface detection and the object detection based on learning. Since the points can be removed, the detection accuracy for obstacles can be improved without adding a sensor other than the camera.

According to the obstacle detection system disclosed in the present application, false detection and false alarm can be suppressed, and the driver can be notified of the existence of an obstacle at an appropriate timing.

It is a block diagram which shows the structure of the obstacle detection system which concerns on Embodiment 1. FIG. It is a figure which shows the hardware configuration example of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the photographing range of the camera of the obstacle detection system which concerns on Embodiment 1. FIG. It is a figure which shows the example of the image taken by the camera of the obstacle detection system which concerns on Embodiment 1. FIG. It is a figure explaining the flow of the whole processing in the obstacle detection system which concerns on Embodiment 1. FIG. It is a figure explaining the flow of processing by the optical flow calculation unit of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure explaining the flow of processing by the road surface detection part of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the pattern example of the histogram created by the road surface detection processing of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure explaining the flow of processing by the obstacle area limitation part of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the obstacle area extracted by the obstacle area limitation part of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows another example of the obstacle area extracted by the obstacle area limitation part of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows still another example of the obstacle area extracted by the obstacle area limitation part of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure explaining the flow of processing by the object detection part of the obstacle detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the result of the object detection processing by the object detection part of the obstacle detection apparatus which concerns on Embodiment 1. FIG.

Embodiment 1.
Hereinafter, the obstacle detection device according to the first embodiment and the obstacle detection system provided with the obstacle detection device will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration of an obstacle detection system according to the first embodiment, and FIG. 2 is a diagram showing a hardware configuration example of the obstacle detection device according to the first embodiment. In the figure, the same and corresponding parts are designated by the same reference numerals.

As shown in FIG. 1, the obstacle detection system 200 includes an obstacle detection device 100, an image pickup unit 1, an alarm unit 8, a control unit 9, and a display unit 10. The obstacle detection device 100 includes a frame buffer 2, an optical flow calculation unit 3, a road surface detection unit 4, an obstacle area limitation unit 5, an object detection unit 6, and an alarm determination unit 7.

The obstacle detection device 100 is mounted on a vehicle driven by a driver (hereinafter referred to as own vehicle 11) and detects an obstacle approaching the own vehicle 11. The obstacle is, for example, an object including a peripheral vehicle, a two-wheeled vehicle, a pedestrian, or the like approaching the own vehicle 11.

The hardware configuration of the obstacle detection device 100 includes a processor 101 and a storage device 102, as shown in FIG. 2 as an example. The storage device 102 includes a volatile storage device such as a random access memory and a non-volatile auxiliary storage device such as a flash memory. Further, the auxiliary storage device of the hard disk may be provided instead of the flash memory.

The processor 101 executes the program input from the storage device 102. In this case, the program is input from the auxiliary storage device to the processor 101 via the volatile storage device. Further, the processor 101 may output data such as a calculation result to the volatile storage device of the storage device 102, or may store the data in the auxiliary storage device via the volatile storage device.

More specifically, among the components of the obstacle detection device 100, the frame buffer 2, the optical flow calculation unit 3, the road surface detection unit 4, the obstacle area limitation unit 5, the object detection unit 6, and the alarm determination unit 7 The processing performed in the above is executed by a program in an obstacle detection ECU (Electronic Control Unit) having an image processing function. The image pickup unit 1, the alarm unit 8, the control unit 9, and the display unit 10 are connected to the obstacle detection ECU via an in-vehicle network.

The image pickup unit 1 of the obstacle detection system 200 is a camera 12 installed in the own vehicle 11, and photographs the surroundings of the own vehicle 11 to acquire a plurality of images in chronological order. In the example shown in FIG. 3, the shooting range 13 of the camera 12 installed at the position of the door mirror is the left rear side of the own vehicle 11. The shooting range 13 is set for the purpose of supplementing the blind spot of the driver when the own vehicle 11 is a vehicle with a right-hand drive.

The location where the camera 12 is installed is not limited to the position of the door mirror, and the camera 12 may be installed at another location on the left side surface of the own vehicle 11 to photograph the rear left side. Further, the shooting range 13 by the camera 12 is not limited to the rear left side of the own vehicle 11, but the rear right side of the own vehicle 11 or the rear side of the own vehicle 11 is photographed, and the area for detecting an obstacle is changed. You may.

FIG. 4 shows an example of an image taken by the camera 12 of the obstacle detection system 200. The angle of view of the image 14 is set so that the side surface of the own vehicle 11 is inserted. The warning area 15 is set according to the distance behind the own vehicle 11. In the image 14 taken by the camera 12, in addition to the approaching vehicle 20, there are non-obstacle road cracks 17, white lines 18, poles 19, and the like.

The image data taken by the camera 12 is output to the frame buffer 2. The frame buffer 2 stores image data for, for example, several frames taken by the camera 12. The image data is output to each of the optical flow calculation unit 3, the road surface detection unit 4, and the object detection unit 6 in the subsequent stage via the frame buffer 2.

The optical flow calculation unit 3 calculates the optical flow of the feature points extracted from a plurality of images, extracts the feature points approaching the own vehicle 11 as the approach feature points, and outputs the information of the approach feature points.

The road surface detection unit 4 extracts a road surface region for each of the plurality of images and divides the road surface region into a road surface region and a region other than the road surface region. There are a plurality of methods for extracting the road surface area, but here, the road surface area is extracted based on the color of the road surface acquired from the peripheral area of the own vehicle 11. Specifically, assuming that the area closest to the own vehicle 11 is the road surface, a histogram is created from an image of the nearest road surface detection area 16 (see FIG. 4) of the own vehicle 11 and used as reference data. do.

The obstacle area limiting unit 5 may have an obstacle based on the information of the approach feature point extracted by the optical flow calculation unit 3 and the information of the road surface area extracted by the road surface detection unit 4. Set the obstacle area. Specifically, a bounding box, which is a rectangular area, is created to limit the obstacle area.

As one of the methods for setting the obstacle area, the obstacle area limiting unit 5 removes the approach feature points existing in the road surface region from the approach feature points extracted by the optical flow calculation unit 3, and removes the remaining approach feature points. Set the bounding box including all as the obstacle area.

Further, as another method, the approach feature points existing in the road surface region are removed from the approach feature points extracted by the optical flow calculation unit 3, the remaining approach feature points are grouped by the density, and the approach feature points for each group are obtained. Set the bounding box including all as the obstacle area. Further, the bounding box may include the boundary of the road surface area.

The object detection unit 6 determines whether or not the target object is an obstacle based on the characteristics of the object learned in advance for each of the plurality of images, and determines the position information (obstacle coordinates) and type information of the target object. Outputs object information including. When the object detection unit 6 determines that the target object is an obstacle, the object information becomes obstacle information. The obstacle information output by the object detection unit 6 includes the position information of the obstacle, the type information, and the information of the obstacle area (bounding box) output by the object detection unit 6.

Here, the target object of the object detection unit 6 is an object that includes the approach feature points extracted by the optical flow calculation unit 3 and exists in a region determined by the road surface detection unit 4 to be other than the road surface region. The object detection unit 6 discriminates the target object from the bounding box output by the obstacle area limiting unit 5 based on the characteristics of the object learned in advance.

The object detection process based on learning uses parameters learned based on images prepared in advance and annotation information of obstacles. For example, a method based on deep learning may be used, or HOG (Histograms of Oriented Grounds) and SVN. (Support Vector Machine) may be used. Parameters include vehicles, motorcycles, pedestrians, etc., as well as various objects that may be present on the road, such as poles, pylon, guardrails, etc.

The alarm determination unit 7 determines whether to notify an alarm or whether to perform control based on the obstacle information output by the object detection unit 6. The alarm determination unit 7 needs an alarm when the bounding box in which the object is detected is in the alarm area 15 (see FIG. 4), or when the optical flow included in the object information is in the alarm area 15. Judge that.

The alarm unit 8 notifies an alarm regarding an obstacle detected by the obstacle detection device 100. Specifically, when the alarm determination unit 7 determines that an alarm is necessary, the driver of the own vehicle 11 is notified of the alarm by sound or vibration. When notifying the alarm by vibration, for example, a signal is sent to EPS (electric power steering), and the steering wheel is vibrated by generating high-frequency vibration in EPS to notify the driver. As a method of vibrating the handle, there is also a method of attaching a handle vibration motor to the handle.

Further, the notification by sound and the notification by vibration may be used properly. For example, when the bounding box contains at least a part of the warning area 15, a sound is first notified, and when an obstacle such as an approaching vehicle approaches a place where the own vehicle 11 may come into contact, a vibration is notified. You may.

When the alarm determination unit 7 determines that control is necessary, the control unit 9 controls the own vehicle 11 so that contact between the obstacle detected by the obstacle detection device 100 and the own vehicle 11 is avoided. do. The display unit 10 notifies the driver of the position information of the obstacle detected by the obstacle detection device 100 by video. Further, when the alarm determination unit 7 determines that an alarm is necessary, the alarm may be superimposed and displayed on the image of the display unit 10 to notify the driver of the own vehicle 11.

Each process performed in the obstacle detection system 200 will be described with reference to FIGS. 5 to 14. FIG. 5 is a flowchart illustrating an overall processing flow in the obstacle detection system 200. In FIG. 5, step S1 is a process by the imaging unit 1, steps S2 to S7 are processes by the obstacle detection device 100, step S8 is a process by the alarm unit 8 and the display unit 10, and step S9 is a process by the control unit 9. be.

Further, FIG. 6 is a flowchart explaining the flow of processing by the optical flow calculation unit 3, FIG. 7 is a flowchart explaining the flow of processing by the road surface detection unit 4, and FIG. 9 is a flowchart of processing by the obstacle area limiting unit 5. A flowchart for explaining the flow and FIG. 13 are a flowchart for explaining the flow of processing by the object detection unit.

In step S1 (image acquisition step) of FIG. 5, the imaging unit 1 photographs the surroundings of the own vehicle 11 and acquires and outputs a plurality of images in chronological order. Subsequently, in step S2 (approach feature point extraction step), the optical flow calculation unit 3 calculates the optical flow of the feature points extracted from a plurality of images, and sets the feature points approaching the own vehicle 11 as the approach feature points. Extracts and outputs information on approach feature points.

The approach feature point extraction process by the optical flow calculation unit 3 will be described in detail with reference to FIG. In step S21 of FIG. 6, the optical flow calculation area used for the optical flow calculation is acquired as a partial image from the image. Here, the optical flow calculation area is an area excluding the own vehicle 11 from the image 14 shown in FIG. 4, but any area in the image may be used as long as it is necessary to detect an obstacle.

Next, in step S22, the corner points on the image are extracted as feature points with respect to the partial image acquired in step S21. As a method for detecting a corner point, for example, there is a corner detection algorithm of Harris, but any method may be used. Further, a method using feature points other than corner points, a method of evenly distributing the feature points in the calculation area without using the feature points, and the like can also be used.

Next, in step S23, the feature points extracted from the latest image are matched with the feature points extracted from the past image, and the optical flow is calculated. Subsequently, in step S24, the optical flow approaching the own vehicle 11 is extracted from the calculation result in step S23.

In the image 14 shown in FIG. 4, the vertical direction is the Y direction and the downward direction is positive, and the left-right direction is the X direction and the right direction is positive. In this definition, an obstacle approaching the own vehicle 11 from behind has a positive optical flow in the Y component. On the contrary, the structure away from the own vehicle 11, such as a building and a vehicle parked on the road, has an optical flow in which the Y component is negative. Therefore, in step S24, an optical flow in which the Y component is positive is extracted.

Subsequently, in step S25, the information of the approach feature points extracted in step S24 is output to the obstacle area limiting unit 5. The approach feature point information includes the coordinates of the approach feature point, the Y component length of the optical flow, the X component length, the direction, the length of the movement amount, and the like. This completes the approach feature point extraction process by the optical flow calculation unit 3.

Next, in step S3 (road surface detection step) of FIG. 5, the road surface detection unit 4 extracts a road surface region for each of the plurality of images acquired in step S1 and separates the road surface region and the region other than the road surface region. The result is output. The road surface detection process by the road surface detection unit 4 will be described in detail with reference to FIGS. 4, 7, and 8.

In step S31 of FIG. 7, a partial image including the alarm area 15 is acquired from the image. Here, the partial image is a region obtained by excluding the own vehicle 11 from the image 14 shown in FIG. 4, similarly to the optical flow calculation region. Next, in step S32, the road surface detection region 16 set in the immediate vicinity of the own vehicle 11 is set as the road surface histogram extraction region. The road surface histogram extraction area may be a peripheral area of the own vehicle 11 in the acquired image, and is set according to the angle of view, the installation position, or the orientation of the camera 12.

Subsequently, in step S33, a histogram of the road surface detection region 16 which is the road surface histogram extraction region is created. Specifically, a luminance value is assigned to each pixel of the road surface detection region 16, and a histogram of the luminance value is created by assigning a luminance value on the horizontal axis and a frequency on the vertical axis. The luminance value is represented, for example, in the range of 0 (black) to 255 (white). The histogram obtained from the road surface detection area 16 is used as reference data.

FIG. 8 shows a histogram of the luminance value of the road surface detection region 16 as an example of the histogram pattern created by the road surface detection process. The histogram is a diagram in which the luminance value on the horizontal axis is shown by class and the frequency is shown on the vertical axis to capture the characteristics of the image. Since the road surface detection region 16 occupies most of the road surface region, the brightness of gray is large, and the frequency peak as shown by G1 appears in FIG.

In addition to the white line, there is also a yellow line on the road surface, and there is also a road surface painted with red or blue paint, so make a class by combining R, G, B, take the frequency for each class, and include the color. You may capture the characteristics of the road surface, or you may use a color space other than RGB. Further, only one frequency peak may be acquired, or a plurality of peaks may be acquired. Further, since the color information changes for each frame, a plurality of color information may be recorded as reference data.

Subsequently, in step S34, the road surface region is extracted from the warning region 15 using the histogram of the road surface detection region 16 as reference data. The method of extracting the road surface region using the histogram is not particularly limited, but here, the histogram of the reference data is compared with the histogram of another region, and the region of the histogram similar to the reference data is determined as the road surface. And extract it as a road surface area.

The similarity of the histogram can be determined from the shape of the histogram and the range of the distributed luminance values, but the method for determining the similarity of the histogram is not limited to this method. A binarized image is created by setting the luminance value of the region determined to be the road surface to 255 and the luminance value of the region determined not to be the road surface to 0.

Finally, in step S35, the information of the road surface area is output to the obstacle area limiting unit 5. The information of the road surface region is, for example, a binarized image obtained by dividing a partial image into a road surface region and a region other than the road surface region. This completes the road surface detection process by the road surface detection unit 4.

Although the method of road surface detection processing using the histogram has been described here, the method of road surface detection is not limited to this method. Other methods include a method using motion stereo, a method learning the amount of color features on the road surface, and a method using image segmentation using deep learning.

Next, in step S4 (obstacle area limiting step) of FIG. 5, the obstacle area limiting unit 5 uses the information of the approach feature points extracted in step S2 and the information of the road surface area extracted in step S3. Based on this, set the obstacle area where obstacles may exist. The obstacle area limiting process by the obstacle area limiting unit 5 will be described in detail with reference to FIGS. 9 to 12.

In step S41 of FIG. 9, information on the approach feature points of the partial image output from the optical flow calculation unit 3 is acquired. Further, in step S42, the information of the road surface area of the partial image output from the road surface detection unit 4 is acquired.

Subsequently, in step S43, the information of the approach feature point in the road surface region is removed. When the coordinates of the approaching feature point are in the road surface region, the information of the approaching feature point is determined to be an erroneous flow generated from a non-obstacle such as a pattern or a crack on the road surface and is removed.

Next, in step S44, the approach feature points that remain unremoved are grouped by density. More specifically, from the coordinates of the approaching feature points and the flow direction of the approaching feature points, the approaching feature points that are close to each other and have the same flow direction are grouped. Subsequently, in step S45, a group of a plurality of approach feature points generated in step S44 is classified, and a bounding box surrounding the approach feature points is created.

Examples of the bounding box are shown in FIGS. 10 to 12. The bounding box 23 shown in FIG. 10 is an circumscribing rectangle including all the remaining approach feature points from which the approach feature points existing in the road surface region 22 are removed. This technique is simple and can include all approach obstacles present in the partial image. When setting the circumscribed rectangle, the circumscribed rectangle may be set with a certain margin from the approach feature point.

Further, in the bounding boxes 23a and 23b shown in FIG. 11, the obstacle region is divided into a plurality of groups according to the density of the approach feature points. As compared with the example shown in FIG. 10, the obstacle area can be effectively limited. Further, the bounding boxes 23c and 23d shown in FIG. 12 are those in which the lower ends of the bounding boxes 23a and 23b shown in FIG. 11 are moved so as to include the boundary of the road surface region 22. As a result, the lower end of the obstacle can easily enter the bounding boxes 23c and 23d, and the obstacle can be easily detected in the object detection process.

Finally, in step S46, the information of the created bounding box is output to the object detection unit 6 as the information of the obstacle area. The information of the obstacle area includes the coordinates of the bounding box on the image, the size and number of optical flows contained in the bounding box, the average length of the flow, the average direction of the flow, and the like. This completes the obstacle area limiting process by the obstacle area limiting unit 5.

Next, in step S5 (object detection step) of FIG. 5, the object detection unit 6 performs object detection processing based on learning on the obstacle area set by the obstacle area limitation unit 5, and the position of the target object and Output object information including type. The object detection process by the object detection unit 6 will be described in detail with reference to FIGS. 13 and 14.

In step S51 of FIG. 13, a partial image of the obstacle area is acquired based on the information of the obstacle area output by the obstacle area limiting unit 5. Then, in step S52. Extract the characteristics of the target object existing in the bounding box. The feature extraction may be HOG or the RGB image as it is.

Subsequently, in step S53, the features extracted in step S52 are compared with the learned parameters, and the position and type of the target object are determined. As a result, it is determined whether or not the target object is an obstacle. Finally, in step S54, the obstacle information including the position information, the type information, and the obstacle area information of the target object determined to be an obstacle is output to the warning determination unit 7. This completes the object detection process by the object detection unit 6.

FIG. 14 shows the result of the object detection process by the object detection unit 6. The bounding box 24 shown in FIG. 14 includes an approaching vehicle 20 determined to be an obstacle and an optical flow 21 thereof. Since the target object of the bounding box 23d shown in FIG. 12 is the pole 19, it is determined that the object is not an obstacle as a result of the object detection process by the object detection unit 6.

Further, in step S6 of FIG. 5, the alarm determination unit 7 determines the necessity of an alarm for the obstacle detected in step S5. Specifically, in step S7, it is determined whether or not the obstacle exists in the warning area 15 based on the object information of the obstacle. If it is determined in step S7 that the obstacle does not exist in the alarm area 15 (NO), all the processing for the image acquired this time is terminated. If it is determined that the obstacle exists in the alarm area 15 (YES), the process proceeds to step S8 or step S9.

In step S8, the alarm unit 8 notifies the alarm by sound or vibration, and the display unit 10 superimposes the alarm on the image to notify the alarm. At this time, the image displayed on the display unit 10 may be the original image stored in the frame buffer 2, or may be an image obtained by cutting out the original image and enlarging, reducing, or deforming the original image. In step S9, the control unit 9 controls the own vehicle 11 so that the detected obstacle and the own vehicle 11 are avoided from coming into contact with each other. In step S8, any one or more of sound, vibration, and video notifications may be performed. Further, either step S8 or step S9 may be performed.

As described above, when the approach feature point extracted by the optical flow calculation unit 3 exists in the road surface region, the obstacle detection device 100 according to the first embodiment removes the information of the approach feature point. The approach feature points related to the crack 17 and the white line 18 of the road shown in FIG. 4 are removed. The approach feature points to be removed include, for example, a puddle in the road surface region, a zebra pattern drawn on the road surface, and approach feature points related to road markings drawn at equal intervals.

Furthermore, even for approach feature points that exist in areas other than the road surface area, approach feature points appear due to the effect of the stroboscopic effect on regularly installed poles and pylon, and evenly spaced patterns drawn on the wall. However, such an object is determined not to be an obstacle by the object detection process based on learning. Therefore, false positives due to optical flow processing can be removed by road surface detection processing and object detection processing based on learning.

Further, the object detection process based on learning has a larger amount of processing than the optical flow process or the road surface detection process, but is processed by performing object detection only on the obstacle area set by the obstacle area limiting unit 5. The efficiency is improved and the processing amount can be reduced without deteriorating the detection performance.

Further, when an obstacle is detected around the own vehicle 11, the warning unit 8 and the display unit 10 for notifying the driver of an alarm by sound, vibration, or video are provided, so that the driver can approach the vehicle 20 or the like. It is possible to notify the existence of obstacles at an appropriate timing. Further, since the control unit 9 for controlling the own vehicle 11 is provided so that the contact between the own vehicle 11 and the obstacle is avoided when an obstacle is detected around the own vehicle 11, the driver operates the vehicle. It is possible to avoid contact with obstacles at an appropriate timing.

From the above, according to the first embodiment, the obstacle detection device 100 having improved detection accuracy for obstacles can be obtained without additionally providing a sensor other than the camera. Further, by providing the obstacle detection device 100, false detection and false alarm can be suppressed, and an obstacle detection system 200 capable of notifying the driver of the existence of an obstacle at an appropriate timing can be obtained.

Although the present disclosure describes exemplary embodiments, the various features, embodiments, and functions described in the embodiments are not limited to the application of a particular embodiment, but alone. Alternatively, various combinations can be applied to the embodiments.
Therefore, innumerable variations not exemplified are envisioned within the scope of the techniques disclosed herein. For example, it is assumed that at least one component is modified, added or omitted.

The present application can be used as an obstacle detection device mounted on a vehicle and an obstacle detection system including the obstacle detection device.

1 Imaging unit, 2 Frame buffer, 3 Optical flow calculation unit, 4 Road surface detection unit, 5 Obstacle area limitation unit, 6 Object detection unit, 7 Alarm judgment unit, 8 Alarm unit, 9 Control unit, 10 Display unit, 11 Self Vehicle, 12 cameras, 13 shooting range, 14 images, 15 warning area, 16 road surface detection area, 17 cracks, 18 white lines, 19 poles, 20 approaching vehicles, 21 optical flow, 22 road surface areas, 23, 23a, 23b, 23c, 23d, 24 bounding box, 100 obstacle detector, 101 processor, 102 storage, 200 obstacle detector system

Claims (10)

It is an obstacle detection device that detects obstacles approaching the vehicle based on a plurality of time-series images taken by an image pickup unit installed in the vehicle.
An optical flow calculation unit that calculates the optical flow of feature points extracted from the plurality of images and extracts the feature points approaching the vehicle as approach feature points.
A road surface detection unit that extracts a road surface region for each of the plurality of images and separates the road surface region and a region other than the road surface region.
An obstacle that surrounds a range in which the obstacle may exist, based on the information of the approach feature point extracted by the optical flow calculation unit and the information of the road surface region extracted by the road surface detection unit. Obstacle area limitation part to set the area and
With respect to the obstacle area set by the obstacle region limiting unit performs determination target object whether the obstacle based on the characteristics of pre-learned object, the position and type of the target object An object detector that outputs the included object information, and
An obstacle detection device including an alarm determination unit for determining whether or not to notify an alarm based on the object information. The object detection unit is characterized in that an object existing in a region other than the road surface region determined by the road surface detection unit to include the approach feature points extracted by the optical flow calculation unit is the target object. The obstacle detection device according to claim 1. The obstacle region limiting portion is a rectangular obstacle region including all the remaining approach feature points after removing the approach feature points existing in the road surface region from the approach feature points extracted by the optical flow calculation unit. The obstacle detection device according to claim 1, wherein the device is set. The obstacle region limiting portion removes the approach feature points existing in the road surface region from the approach feature points extracted by the optical flow calculation unit, and groups the remaining approach feature points by density for each group. wherein the proximity feature points and sets a rectangular obstacle region including all claim 1 obstacle detection apparatus as claimed in. The obstacle detection device according to claim 4, wherein the obstacle area limiting portion sets the obstacle area so as to include a boundary of the road surface area. The obstacle detection device according to any one of claims 1 to 5 , wherein the road surface detection unit extracts the road surface region based on the color of the road surface acquired from the peripheral region of the vehicle. .. An image pickup unit installed in a vehicle that captures the surroundings of the vehicle and acquires multiple images in chronological order.
An obstacle detection device that detects an obstacle approaching the vehicle based on the plurality of images taken by the image pickup unit, and an obstacle detection device.
Including an alarm unit for notifying an alarm regarding the obstacle detected by the obstacle detection device.
An obstacle detection system comprising the obstacle detection device according to any one of claims 1 to 6 as the obstacle detection device. The obstacle detection system according to claim 7 , further comprising a control unit that controls the vehicle so that contact between the obstacle detected by the obstacle detection device and the vehicle is avoided. Obstacle detection system according to claim 7 or claim 8, characterized in that the position information of the obstacle detected by the obstacle detecting device comprising a display unit that notifies the video. It is an obstacle detection method that detects obstacles approaching the vehicle.
An image acquisition step of photographing the surroundings of the vehicle and acquiring a plurality of images in chronological order,
An approach feature point extraction step that calculates an optical flow of feature points extracted from the plurality of images and extracts feature points approaching the vehicle as approach feature points.
A road surface detection step of extracting a road surface region for each of the plurality of images and separating the road surface region and a region other than the road surface region.
An obstacle that surrounds a range in which the obstacle may exist, based on the information of the approach feature point extracted in the approach feature point extraction step and the information of the road surface region extracted in the road surface detection step. Obstacle area limitation step to set area and
The obstacle region limiting the obstacle region set in step to perform the judgment target object whether the obstacle based on the characteristics of pre-learned object, the position and type of the target object Including an object detection step that outputs object information, including
In the object detection step, an object including the approach feature points extracted in the approach feature point extraction step and existing in a region determined to be other than the road surface region in the road surface detection step is defined as the target object. An obstacle detection method characterized by.

Images