JP7395913B2 - object detection device - Google Patents

Description

Embodiments of the present invention relate to object detection devices.

Conventionally, a technology that detects objects (e.g., obstacles such as pedestrians and other vehicles) by setting, for example, a rectangular bounding box on a captured image captured by an imaging unit (camera) installed in a vehicle. has been put into practical use. For example, some systems perform object recognition by extracting a video corresponding to a predetermined object from a captured image and setting a bounding box that surrounds the entire video.

Japanese Patent Application Publication No. 2018-142309

By the way, in the case of conventional object recognition using a bounding box as described above, for example, when trying to estimate the distance to an object without using a distance sensor, etc., the coordinates of the center part of the lower end of the bounding box are As the position, for example, the distance to the object may be estimated by converting from two-dimensional coordinates to three-dimensional coordinates. However, as described above, the bounding box is set to extract an image corresponding to a predetermined object and surround the entire image. Then, for example, the center position of the bottom of the bounding box may be estimated as the position of the object. Therefore, distance estimation may be performed based on a position different from the actual position of the object. As a result, the distance error to the object may become large, and for example, when the vehicle is driven using driving support or the like, there is a problem in that the risk of contact with the object may become high. Therefore, in object detection using a bounding box, if an object detection device that can estimate the object's position with higher accuracy can be provided, it would be meaningful to realize more reliable and safer object avoidance control. be.

The object detection device according to the embodiment of the present invention includes, for example, an acquisition unit that acquires a captured image captured by an imaging unit capable of capturing a surrounding situation including a road surface on which a vehicle is present, and the road surface included in the captured image. A three-dimensional object boundary line indicating the boundary between the object area recognized as the area in which the three-dimensional object exists and the road surface, and a three-dimensional object boundary line for selecting a predetermined target object from among the three-dimensional objects included in the object area. a setting unit for setting a rectangular bounding box; and extracting a partial boundary line included in the bounding box from among the three-dimensional object boundary lines, and avoiding contact between the vehicle and the target object on the partial boundary line. a control unit that estimates and outputs a point of interest to be focused on in order to calculate a distance from the vehicle to the target object in a three-dimensional space based on the point of interest . According to this configuration, for example, after extracting the target object using a bounding box, it is possible to further specify the boundary between the object and the road surface, that is, the position of the foot of the object, so that the accuracy of detecting the position of the target object can be improved. In other words, the position is estimated to be a more accurate point of interest to avoid contact with the target object , and the distance from the vehicle to the target object in three-dimensional space is calculated . As a result, more reliable and safer object avoidance control can be realized.

For example, the control unit of the object detection device according to the embodiment of the present invention may set the lowermost position of the bounding box in the vertical axis direction of the partial boundary line as the point of interest. According to this configuration, for example, among the partial boundary lines indicating the position of the object, the position closest to the vehicle (self-vehicle), that is, the position with a high possibility of contact can be easily determined as the point of interest. As a result, more reliable and safer object avoidance control can be realized.

For example, when there is a switching position in the partial boundary line where the slope of the partial boundary line is switched between positive and negative, the control unit of the object detection device according to the embodiment of the present invention changes the switching position to the point of interest. You can also use it as According to this configuration, for example, a corner portion (for example, a corner portion) of a target object can be extracted as a point of interest, and more reliable and safe object avoidance control can be realized.

The control unit of the object detection device according to the embodiment of the present invention may, for example, determine the partial boundary line based on the bounding box and an expected travel area determined by an expected travel line indicating the traveling direction based on the steering angle of the vehicle. Alternatively, the position of the point of interest may be estimated. According to this configuration, for example, it is possible to extract a partial boundary line for a region in which a vehicle is likely to proceed (drive), and it is possible to realize object avoidance control with higher reliability and higher safety.

FIG. 1 is a schematic plan view showing an example of a vehicle in which an object detection device according to an embodiment can be mounted. FIG. 2 is an exemplary block diagram of the configuration of a control system including the object detection device according to the embodiment. FIG. 3 is a block diagram exemplarily showing a configuration when the object detection device (object detection section) according to the embodiment is implemented by a CPU. FIG. 4 is an exemplary and schematic explanatory diagram showing the setting of solid object boundaries and bounding boxes in the object detection device according to the embodiment. FIG. 5 is an exemplary and schematic explanatory diagram showing the setting of partial boundaries in the object detection device according to the embodiment. FIG. 6 is an exemplary and schematic diagram illustrating that the lowest position of the bounding box in the vertical axis direction is the point of interest in the object detection device according to the embodiment. FIG. 7 is an exemplary and schematic diagram illustrating another example in which the lowest position of the bounding box in the vertical axis direction is the point of interest in the object detection device according to the embodiment. FIG. 8 is a schematic explanatory diagram illustrating setting a partial boundary line based on the predicted travel area and the bounding box in the object detection device according to the embodiment. FIG. 9 is a flowchart illustrating an example of the flow of the attention point estimation process of the object detection device according to the embodiment.

Exemplary embodiments of the invention are disclosed below. The configuration of the embodiment shown below, and the actions, results, and effects brought about by the configuration are examples. The present invention can be realized with configurations other than those disclosed in the embodiments below, and it is possible to obtain at least one of various effects based on the basic configuration and derivative effects. .

The object detection device of this embodiment detects, for example, a three-dimensional object boundary line indicating a boundary between an object area and the road surface, which is recognized as an area where a three-dimensional object exists on the road surface included in the captured image, and an object included in the object area. The position of the partial boundary line where the target object is in contact with the road surface is extracted using a bounding box for selecting (extracting) the object. Then, the points of interest that should be noted in order to avoid contact between the vehicle and the target object on the partial boundary line are estimated and output.

FIG. 1 is a schematic plan view of a vehicle 10 in which an object detection device of this embodiment is mounted. The vehicle 10 may be, for example, an automobile (internal combustion engine automobile) whose driving source is an internal combustion engine (engine, not shown), or an automobile (electric automobile) whose driving source is an electric motor (motor, not shown). , fuel cell vehicle, etc.), or a vehicle (hybrid vehicle) that uses both of these as drive sources. Further, the vehicle 10 can be equipped with various transmission devices, and can be equipped with various devices (systems, parts, etc.) necessary to drive an internal combustion engine or an electric motor. Further, the system, number, layout, etc. of devices related to driving the wheels 12 (front wheels 12F, rear wheels 12R) in the vehicle 10 can be set in various ways.

As illustrated in FIG. 1, the vehicle 10 is provided with, for example, four imaging units 14a to 14d as the plurality of imaging units 14. The imaging unit 14 is, for example, a digital camera incorporating an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS image sensor). The imaging unit 14 can output moving image data (captured image data) at a predetermined frame rate. The imaging units 14 each have a wide-angle lens or a fisheye lens, and can take images in a range of, for example, 140° to 220° in the horizontal direction. Furthermore, for example, the optical axis of the imaging section 14 (14a to 14d) disposed on the outer periphery of the vehicle 10 may be set diagonally downward. Therefore, the imaging unit 14 (14a to 14d) detects road surfaces on which the vehicle 10 can move, marks attached to the road surfaces (arrows, division lines, parking frames indicating parking spaces, lane separation lines, etc.) and objects (as obstacles). , pedestrians, other vehicles, etc.) are sequentially photographed and output as captured image data.

The imaging unit 14a is, for example, provided on the front side of the vehicle 10, that is, on the front side in the longitudinal direction of the vehicle, at an end portion approximately at the center in the vehicle width direction, such as the front bumper 10a, the front grille, etc. It is possible to capture a front image including the front bumper 10a). Further, the imaging unit 14b is provided, for example, at the rear side of the vehicle 10, that is, at the rear side in the longitudinal direction of the vehicle, at a substantially central end in the vehicle width direction, for example, at a position above the rear bumper 10b, and at the rear end of the vehicle 10. (For example, the rear bumper 10b) can be imaged. Further, the imaging unit 14c is provided, for example, at the right end of the vehicle 10, for example, the right side door mirror 10c, and is located on the right side including an area centered on the right side of the vehicle 10 (for example, an area from the right front to the right rear). It is possible to capture images from both sides. The imaging unit 14d is provided, for example, at the left end of the vehicle 10, for example, the left side door mirror 10d, and captures a left side image including an area centered on the left side of the vehicle 10 (for example, an area from the front left to the rear left). can be imaged.

For example, each captured image data obtained by the imaging units 14a to 14d is subjected to arithmetic processing and image processing to display images in each direction around the vehicle 10 or to perform surrounding monitoring. I can do it. In addition, by performing arithmetic processing and image processing based on each captured image data, images with a wider viewing angle can be generated, and virtual images ( It is possible to generate and display a bird's-eye view image (a planar image), a side view image, a front view image, etc., and perform peripheral monitoring.

As described above, the captured image data captured by each imaging unit 14 is displayed on a display device in the vehicle interior in order to provide users, such as the driver, with information about the surroundings of the vehicle 10. In addition, the captured image data is provided to a processing device (processing unit) that performs various detections and detections such as detecting objects (for example, obstacles such as other vehicles and pedestrians), specifying the position, and measuring distance. It can be used for control.

FIG. 2 is an exemplary block diagram of the configuration of control system 100 including an object detection device mounted on vehicle 10. A display device 16 and an audio output device 18 are provided inside the vehicle 10 . The display device 16 is, for example, an LCD (liquid crystal display) or an OELD (organic electroluminescent display). The audio output device 18 is, for example, a speaker. Further, the display device 16 is covered with a transparent operation input section 20 such as a touch panel. A user (for example, a driver) can visually recognize an image displayed on the display screen of the display device 16 via the operation input unit 20. Further, the user can perform operation input by touching, pressing, or moving the operation input section 20 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 16. I can do it. The display device 16, the audio output device 18, the operation input unit 20, and the like are provided, for example, in a monitor device 22 located at the center of the dashboard of the vehicle 10 in the vehicle width direction, that is, in the left-right direction. The monitor device 22 can have an operation input section (not shown) such as a switch, dial, joystick, push button, or the like. The monitor device 22 can be used, for example, as a navigation system or an audio system.

Further, as illustrated in FIG. 2, the control system 100 (including the object detection device) includes, in addition to the imaging section 14 (14a to 14d) and the monitor device 22, an ECU 24 (electronic control unit), a wheel speed sensor 26, It includes a steering angle sensor 28, a shift sensor 30, a driving support section 32, and the like. In the control system 100, the ECU 24, monitor device 22, wheel speed sensor 26, steering angle sensor 28, shift sensor 30, driving support section 32, etc. are electrically connected via an in-vehicle network 34 as a telecommunications line. . The in-vehicle network 34 is configured as, for example, a CAN (controller area network). The ECU 24 can control various systems by sending control signals through the in-vehicle network 34. Further, the ECU 24 can receive operation signals from the operation input unit 20 and various switches, detection signals from various sensors such as the wheel speed sensor 26, the steering angle sensor 28, the shift sensor 30, etc. via the in-vehicle network 34. . Note that various systems (steering system, brake system, drive system, etc.) and various sensors for driving the vehicle 10 are connected to the in-vehicle network 34, and in FIG. Configurations that are less relevant to the device are not shown in the drawings, and their descriptions are also omitted.

The ECU 24 is composed of a computer or the like, and controls overall control of the vehicle 10 through cooperation of hardware and software. Specifically, the ECU 24 includes a CPU (Central Processing Unit) 24a, a ROM (Read Only Memory) 24b, a RAM (Random Access Memory) 24c, a display control section 24d, an audio control section 24e, and an SSD (Solid State Drive) 24f. Equipped with

The CPU 24a reads a program stored (installed) in a nonvolatile storage device such as a ROM 24b, and executes arithmetic processing according to the program. For example, the CPU 24a can perform image processing on the captured image captured by the imaging unit 14, perform object recognition, and estimate the location of the object, the distance to the object, and the like. Additionally, it is possible to estimate the position of a point of interest to avoid contact with the recognized object. Furthermore, based on the point of interest, it is possible to execute a driving support process or the like for driving the vehicle 10 without contacting an object. For example, information necessary for controlling and operating a steering system, brake system, drive system, etc. can be provided.

The ROM 24b stores each program and parameters necessary for executing the program. The RAM 24c is used as a work area when the CPU 24a executes object detection processing, and also stores various data used in calculations by the CPU 24a (captured image data sequentially (chronologically) captured by the imaging unit 14, etc.). ) is used as a temporary storage area. Among the calculation processing performed by the ECU 24, the display control unit 24d mainly processes image data acquired from the imaging unit 14 and output to the CPU 24a, and performs image processing for displaying image data acquired from the CPU 24a on the display device 16. Executes conversion to data, etc. Among the calculation processing performed by the ECU 24, the audio control unit 24e mainly executes audio processing that is acquired from the CPU 24a and output to the audio output device 18. The SSD 24f is a rewritable non-volatile storage unit, and continues to store data acquired from the CPU 24a even when the power of the ECU 24 is turned off. Note that the CPU 24a, ROM 24b, RAM 24c, etc. can be integrated in the same package. Further, the ECU 24 may have a configuration in which other logic processors such as a DSP (digital signal processor), logic circuits, etc. are used instead of the CPU 24a. Further, an HDD (hard disk drive) may be provided in place of the SSD 24f, or the SSD 24f and the HDD may be provided separately from the ECU 24.

The wheel speed sensor 26 is a sensor that detects the amount of rotation of the wheel 12 and the number of rotations per unit time. The wheel speed sensor 26 is arranged at each wheel 12 and outputs the number of wheel speed pulses indicating the rotation speed detected by each wheel 12 as a sensor value. The wheel speed sensor 26 may be configured using, for example, a Hall element. The CPU 24a calculates the vehicle speed, acceleration, etc. of the vehicle 10 based on the detected values obtained from the wheel speed sensors 26, and executes various controls.

The steering angle sensor 28 is, for example, a sensor that detects the amount of steering of a steering unit such as a steering wheel. The steering angle sensor 28 is configured using, for example, a Hall element. The CPU 24a acquires from the steering angle sensor 28 the amount of steering of the steering unit by the driver, the amount of steering of the front wheels 12F during automatic steering when executing parking assistance, for example, and executes various controls.

The shift sensor 30 is a sensor that detects the position of the movable parts (bars, arms, buttons, etc.) of the gear shift operation unit, and detects the operating state of the transmission, the gear position state, and the direction in which the vehicle 10 can travel (D range: forward Detects information indicating the direction, R range (reverse direction), etc.

The driving support unit 32 provides driving support for the steering system, brake system, drive system, etc. in order to realize driving support for moving the vehicle 10 based on the movement route calculated by the control system 100 or the movement route provided from the outside. Provide control information. For example, the driving support unit 32 executes fully automatic control that automatically controls all of the steering system, brake system, drive system, etc., or semi-automatic control that automatically controls part of the steering system, brake system, drive system, etc. or execute it. In addition, the driving support unit 32 provides the driver with operation guidance for the steering system, brake system, drive system, etc. so that the vehicle 10 can move along the travel route, and manually prompts the driver to perform driving operations. Control may also be executed. In this case, the driving support section 32 may provide operation information to the display device 16 and the audio output device 18. Further, even when performing semi-automatic control, the driving support unit 32 can provide the driver with information regarding, for example, accelerator operation, via the display device 16 and the audio output device 18.

FIG. 3 is a block diagram illustratively and schematically showing a configuration when the object detection device (object detection unit 40) according to the embodiment is implemented by the CPU 24a. By executing the object detection program read from the ROM 24b, the CPU 24a realizes the object detection section 40 including modules such as an image acquisition section 42, a setting section 44, a control section 46, etc., as shown in FIG. Further, the setting section 44 includes detailed modules such as a three-dimensional object boundary line setting section 44a, a bounding box setting section 44b, and an expected travel area setting section 44c. Further, the control unit 46 includes detailed modules such as a partial boundary line extraction unit 46a, a point of interest estimation unit 46b, and an output unit 46c. Note that part or all of the image acquisition section 42, setting section 44, and control section 46 may be configured by hardware such as a circuit. Although not shown in FIG. 3, the CPU 24a can also implement various modules necessary for the vehicle 10 to travel. Further, although FIG. 2 mainly shows the CPU 24a that executes object detection processing, it may also include a CPU for realizing various modules necessary for running the vehicle 10, or may include an ECU separate from the ECU 24. It's okay.

The image acquisition unit 42 acquires a captured image (surrounding image) showing the surrounding situation of the vehicle 10 including the road surface on which the vehicle 10 exists, captured by the imaging unit 14, and provides it to the setting unit 44 and the control unit 46. Note that the image acquisition section 42 may sequentially acquire the captured images captured by each of the imaging sections 14 (14a to 14d) and provide them to the setting section 44 and the control section 46. In another example, the image acquisition unit 42 detects objects in the possible travel direction of the vehicle 10 based on information regarding the possible travel direction of the vehicle 10 (forward direction or backward direction) that can be obtained from the shift sensor 30. Captured images in possible directions may be selectively acquired and provided to the setting unit 44 and the control unit 46.

The setting unit 44 performs image processing and the like on the captured image provided from the image acquisition unit 42 to set various boundaries and areas. FIGS. 4 and 5 are exemplary and schematic diagrams illustrating settings of various boundary lines and areas. Note that FIGS. 4 and 5 are examples in which various boundary lines and areas are set for the rear image when the vehicle 10 is in a state where it can move backward (when the detection result of the shift sensor 30 is in the R range).

As shown in FIG. 4, the three-dimensional object boundary line setting unit 44a recognizes an area where a three-dimensional object (another vehicle 54, a fence 56, etc.) exists on a road surface 52 included in a captured image 50 (for example, a rear image). A three-dimensional object boundary line 58 indicating the boundary between the object region BR and the road surface 52 is set. The three-dimensional object boundary line 58 can be set using a well-known image processing technique such as edge detection, linear approximation, machine learning, or the like. The three-dimensional object boundary line 58 can be regarded as a line connecting the foot position of the three-dimensional object existing on the road surface 52 (the position where it contacts the road surface 52 or the position where the three-dimensional object floating from the road surface 52 is projected onto the road surface 52). In the case of Fig. 4, the three-dimensional object boundary line 58 is expressed as a smooth line for convenience of drawing, but in reality, it may vary depending on the shape and position of the three-dimensional object, the resolution and recognition accuracy of the captured image, noise, etc. The line may be non-smooth. In such a case, a well-known smoothing process may be performed.

The bounding box setting unit 44b is configured to set a rectangular shape for selecting a predetermined target object (for example, an object corresponding to the other vehicle 54) from among the three-dimensional objects (other vehicle 54, fence 56, etc.) included in the object region BR. A bounding box 60 is set. Target objects are obstacles that may exist on the road surface 52 and should be avoided when the vehicle 10 is traveling, and include objects such as other vehicles 54 and fences 56, as well as bicycles, pedestrians, telephone poles, These objects correspond to roadside trees, flower beds, etc., and are objects that can be extracted in advance using a model trained by, for example, machine learning. The bounding box setting unit 44b applies a model trained according to a well-known technique to the captured image 50, calculates the degree of coincidence between the feature points of the target object and the feature points of the image included in the captured image 50, and It is detected where on the image 50 a known target object exists. Then, a bounding box 60 is set to surround the detected object, that is, an object that can be considered as a target object (for example, another vehicle 54), as shown in FIG. 4, FIG. 5, and FIG. 8, which will be described later, show an example in which a bounding box 60 is set for an object extracted at a position closest to the vehicle 10, as an example of detecting an object that avoids contact with the vehicle 10. ing.

The expected travel area setting unit 44c sets an area through which the vehicle 10 is expected to travel. For example, based on the current steering angle of the vehicle 10, the direction in which the vehicle 10 will travel and the area through which the vehicle 10 will pass can be estimated. Therefore, it is possible to limit the area where it is necessary to avoid contact with a target object (for example, another vehicle 54) by the passage area of the vehicle 10, and it is possible to improve the estimation accuracy of the contact avoidance position and to avoid processing load can be reduced. Note that the details of the control using the predicted travel area setting section 44c will be described later.

The control unit 46 extracts a partial boundary line 62 that is a limit line for avoiding contact between the vehicle 10 and the target object, and also identifies points on the partial boundary line 62 that should be noted in order to avoid contact with the vehicle 10. The point of interest T is estimated and output. Note that the point of interest T is a typical point (position) when avoiding contact with a target object, and if contact with the point of interest T can be avoided, contact with the vehicle 10 can be avoided.

As shown in FIGS. 4 and 5, the partial boundary line extraction unit 46a selects the captured image in which the bounding box setting unit 44b is the same among the three-dimensional object boundaries 58 set by the three-dimensional object boundary line setting unit 44a in the captured image 50. 50 is extracted and set as a partial boundary line 62. That is, a part of the three-dimensional object boundary line 58 is selected as a partial boundary line 62 by the bounding box 60, and as shown in FIG. This line segment indicates the contact position between the wheels and the road surface 52 or the projected position of the body of the other vehicle 54. That is, when the vehicle 10 approaches (for example, another vehicle 54), the partial boundary line 62 extracted by the partial boundary line extraction unit 46a avoids contact with the target object (for example, other vehicle 54). It can be seen as a limit line for Note that when the bounding box 60 is set, a target object exists. If a target object (three-dimensional object) exists, it is set as a three-dimensional object boundary line 58. Therefore, if the bounding box 60 can be set (if a target object exists), the partial boundary line 62 can be extracted.

As shown in FIG. 5, the attention point estimation unit 46b estimates the attention point T on the partial boundary line 62 to avoid contact between the vehicle 1 and the target object. As described above, when the vehicle 10 approaches a target object, the possibility of contact is small unless the vehicle 10 crosses the partial boundary line 62. Therefore, the point of interest estimating section 46b sets (estimates) the point of interest T on the partial boundary line 62 indicating the foot of the target object set by the three-dimensional object boundary line setting section 44a. The attention point estimation unit 46b sets the attention point T according to predetermined rules. For example, as shown in FIG. 5, the point of interest estimation unit 46b calculates that in the two-dimensional coordinates (XY coordinates) of the captured image 50, when the X coordinates of both ends of the partial boundary line 62 are X _i and X _j , , let the position of the midpoint be the X coordinate _X0 of the point of interest T. Therefore, the two-dimensional coordinates of the point of interest T can be estimated as (X ₀ , Y ₀ ).

The output unit 46c may output information on two-dimensional coordinates to the driving support unit 32 as information on the point of interest T, or change the two-dimensional coordinates into three-dimensional coordinates by well-known calculation processing and output the two-dimensional coordinate information as the information on the point of interest T. The three-dimensional coordinates of the vehicle may be output to the driving support section 32. The output unit 46c may also calculate the distance from the vehicle 10 to the target object in the three-dimensional space (the distance from the end reference position P to the point of interest T) and output the calculated distance to the driving support unit 32.

In this way, by combining the three-dimensional object boundary line 58 and the bounding box 60 and extracting the partial boundary line 62, it is possible to recognize the location under the feet of the target object (for example, another vehicle 54). Then, by executing driving support using the point of interest T estimated on the partial boundary line 62, the positional relationship between the vehicle 10 and the target object can be more accurately recognized, and contact with the target object can be more easily avoided. It can be achieved with high precision. In particular, since the positional relationship (relative distance) when the vehicle 10 approaches the target object can be accurately detected, more accurate control can be executed.

In addition, as another embodiment, for example, as shown in FIG. 5, in the partial boundary line 62, when there is a switching position where the slope of the partial boundary line 62 changes between positive and negative, The switching position may be estimated as the point of interest _T1 . As shown in FIG. 5, the position (coordinates (X ₁ , Y ₁ ) where the slope of the partial boundary line 62 switches between positive and negative is the position of the partial boundary line 62 corresponding to the front of the vehicle of the target object (for example, the other vehicle 54). This corresponds to the contact position between the dividing line 62a and the dividing line 62b corresponding to the side surface of the vehicle.In other words, the position where the slope of the partial boundary line 62 changes corresponds to the corner position of the target object (for example, the position of the front right corner of the other vehicle 54). ).In other words, when the vehicle 10 travels close to the target object (other vehicle 54), the corner position of the target object corresponding to the tilt switching position is the closest position from the rear bumper 10b of the vehicle 10. By setting this position as the point of interest _T1 and performing travel control to avoid contact, it is possible to more reliably avoid contact with the target object.

The position where the slope of the partial boundary line 62 switches between positive and negative may be estimated, for example, by calculating an inflection point on the partial boundary line 62, or by calculating the slope between any two points on the partial boundary line 62. It may be estimated by sequentially calculating and detecting positive/negative changes in the slope.

Further, as another embodiment for estimating the point of interest T, for example, as shown in FIG. It may also be _T2 . As shown in FIG. 5, in the bounding box 60, the lower part in the y-axis direction is closer to the vehicle 10. That is, the lowest position of the partial boundary lines 62 included in the bounding box 60 is the position closest to the vehicle 10 and indicates the foot of the target object (other vehicle 54). Therefore, by controlling the travel to avoid the point of interest _T2 , it is possible to more reliably avoid contact with the target object.

FIG. 6 is another exemplary and schematic diagram illustrating that the lowest position in the vertical axis direction of the bounding box 60 is set as the point of interest T. FIG. 6 shows a case where a fence 64 is extracted as a target object by the bounding box 60. In the case of an object such as the fence 64, which has no or little depth, the position where the slope of the partial boundary line 62 changes from positive to negative may not exist or may not be detected. In such a case, as in the other embodiment described above, it is possible to set the lowest position of the partial boundary line 62 in the vertical axis direction (y-axis direction) in the bounding box 60 as the point of interest _T3. . Therefore, by performing travel control to avoid contact with the point of interest _T3 , contact with the target object (fence 64) can be avoided more reliably.

Further, FIG. 7 shows a case where a pedestrian 66 is extracted as the target object. In the case of a pedestrian 66, the partial boundary line 62 (three-dimensional object boundary line 58) based on the walking motion may be short. Even in this case, it is possible to set the lowest position in the vertical axis direction (y-axis direction) of the bounding box 60 among the partial boundary lines 62 set for the pedestrian 66 as the point of interest _T4 . In other words, the point of interest _T4 can be regarded as the position closest to the vehicle 10 indicating the feet of the pedestrian 66. Therefore, by performing travel control to avoid contact with the point of interest _T4 , contact with the target object (pedestrian 66) can be more reliably avoided. Note that depending on the behavior of a moving object such as a pedestrian 66, it may be possible to sufficiently extract the partial boundary line 62 (three-dimensional object boundary line 58) based on the movement, and there is a position where the slope of the partial boundary line 62 switches between positive and negative. There are cases where In such a case, it is possible to estimate the point of interest _T4 using the position of the inflection point. Therefore, by performing travel control to avoid contact with the point of interest _T4 , contact with the target object (moving object) can be similarly avoided.

FIG. 8 is an exemplary and schematic explanatory diagram showing a modification that allows the point of interest T to be estimated more efficiently.

As described above, the expected travel area setting unit 44c of the setting unit 44 sets the area through which the vehicle 10 is expected to travel. For example, based on the current steering angle of the vehicle 10 provided from the steering angle sensor 28, it is possible to estimate the direction in which the vehicle 10 travels (the position and direction in which the wheels 12 pass). For example, as shown in FIG. 8, the predicted driving area setting section 44c defines a region surrounded by predicted driving lines 66a and 66b based on a predicted driving line 68a along which the right rear wheel travels and a predicted driving line 66b along which the left rear wheel travels. A predicted driving area 68 can be set. That is, when the vehicle 10 travels (for example, when reversing), the vehicle 10 does not pass through any area other than the expected travel area 68. Therefore, the partial boundary line extraction unit 46a can determine the partial boundary line 62c based on the three-dimensional object boundary line 58 included in the area where the expected travel area 68 and the bounding box 60 overlap. As a result, the point of interest estimation unit 46b can extract the partial boundary line 62c and estimate the point of interest T in a more limited area where the vehicle 10 is expected to pass. As a result, the accuracy of estimating the point of interest T can be improved, and the processing load for estimating the point of interest T can be reduced. Even when the partial boundary line 62c is extracted using the predicted driving area 68, the midpoint on the X coordinate of the partial boundary line 62c is calculated, and the inflection point of the partial boundary line 62c is calculated, as in the above example. The position of the point of interest T can be estimated by calculating the lowest position.

An example of the flow of the process of estimating the point of interest T by the object detection device (object detection unit 40) configured as described above will be described using the flowchart of FIG. 9.

First, the object detection unit 40 confirms the current direction in which the vehicle 10 can travel. For example, based on the shift position of the speed change operation unit that can be obtained from the shift sensor 30, it is confirmed whether the vehicle 10 is in a state where it can travel backwards (S100). If the vehicle is in a state where it is possible to travel backwards (Yes in S100), that is, if the R range is selected in the gear shift operation section, the image acquisition section 42 acquires the rear image captured by the imaging section 14b as an image of the object detection target. (S102). On the other hand, in S100, if it is not possible to drive backwards (No in S100), for example, in the shift operation section, the D range that allows forward running is selected, or a range other than R ranges such as P range or N range is selected. If so, the image acquisition unit 42 acquires the front image captured by the imaging unit 14a as an image of the object detection target (S104).

When the image of the object detection target is acquired, the three-dimensional object boundary line setting section 44a sets the three-dimensional object boundary line 58 on the captured image acquired by the image acquisition section 42 (S106), and also sets the three-dimensional object boundary line 58 in the captured image acquired by the image acquisition section 42 (S106). 44b sets the bounding box 60 (S108). Further, the predicted travel area setting unit 44c sets a predicted travel line 68a and a predicted travel line 68b based on the current steering angle of the vehicle 10 acquired from the steering angle sensor 28, and based on the predicted travel lines 68a and 68b. A predicted travel area 68 is set (S110).

The control unit 46 checks whether there is a bounding box 60 that overlaps the predicted travel region 68 (S112). If an overlapping region exists (Yes in S112), a target object to be detected exists in a region where the vehicle 10 is likely to travel. In this case, the partial boundary line extracting unit 46a further limits the partial boundary line 62 determined by the three-dimensional object boundary line 58 included in the bounding box 60 by the predicted travel area 68, so that The location (part) of the object is specified, and a partial boundary line 62c for the location is set (S114).

Then, the attention point estimation unit 46b estimates the attention point T based on the partial boundary line 62c extracted by the partial boundary line extraction unit 46a (S116). As mentioned above, the point of interest T can _be estimated by using the coordinates of _T (X ₀ , Y ₀ ) may be estimated. Alternatively, the position where the slope of the partial boundary line 62 (62c) switches between positive and negative may be estimated as the coordinate of T. Alternatively, the lowest position of the partial boundary line 62 in the bounding box 60 in the vertical axis direction (y-axis direction) may be set as the point of interest T.

When the point of interest T is estimated, the output unit 46c converts the coordinate information of the point of interest T estimated on two-dimensional coordinates into coordinate information on three-dimensional coordinates, and converts the coordinate information of the point of interest T estimated on two-dimensional coordinates to The distance to the object (distance from the end reference position P to the point of interest T) is calculated and output to the driving support unit 32 (S118), and the series of object detection processes is temporarily ended. In another example, the output unit 46c outputs information on the point of interest T in two-dimensional coordinates and information on the point of interest T in three-dimensional coordinates to the driving support unit 32, and performs coordinate conversion processing on the driving support unit 32 side. Alternatively, distance calculation may be performed. Based on the information based on the acquired point of interest T, the driving support unit 32 controls the steering system, brake system, drive system, etc. or provides operation guidance to the driver to avoid contact with the target object. .

In S112, if there is no overlap between the bounding box 60 and the expected travel area 68 (No in S112), it is assumed that there is no target object that may come into contact with the vehicle 10 during travel based on the current steering angle of the vehicle 10. , the process moves to S116, and information indicating that the point of interest T does not exist is outputted to the driving support unit 32 via the output unit 46c, and the series of object detection processes is temporarily ended.

In this way, according to the object detection device of the embodiment, it is possible to detect the position of the foot of the target object recognized by the bounding box 60, that is, the position closer to the target object. Therefore, the accuracy of detecting the point of interest T indicating the position of the target object can be improved compared to the case where the position of the target object is detected using only the bounding box 60. As a result, the position and distance of the target object relative to the vehicle 10 can be more accurately determined, and contact with the target object can be avoided more reliably. In particular, it becomes possible to more accurately recognize the positional relationship and distance when the vehicle 10 approaches a target object, and control with improved reliability and safety can be realized.

In the embodiment described above, an example was given in which the information on the estimated point of interest T is used to control the steering system, brake system, drive system, etc., and to provide operation guidance to the driver. It may also be used for control, for example, processing such as image display and alarm output. In this case as well, the improved detection accuracy of the point of interest T can contribute to improved image display accuracy and warning accuracy.

The captured images 50 shown in FIGS. 4, 5, and 8 are images for explaining the flow of object recognition, and although they do not need to be displayed on the display device 16, they can be displayed on the display device 16 to help detect objects. This makes it possible to present the situation to the driver, contributing to improving confidence in object detection processing and a sense of security. Furthermore, the highly accurate object detection described in this embodiment is particularly effective when the vehicle 10 travels at a low speed and approaches a target object in parking assistance, exit assistance, or the like. Therefore, for example, according to the vehicle speed provided by the wheel speed sensor 26, the object detection process of this embodiment may be executed when the vehicle speed is below a predetermined speed (for example, 12 km/h or below). Conversely, when the vehicle is running at a speed exceeding a predetermined speed, for example, a simple object detection process using only the bounding box 60 may be performed.

Further, in the above-described embodiment, an example was shown in which object detection is performed using the front image or the rear image, but the same object detection processing is possible for the right side image and the left side image, and the same effect can be achieved. Obtainable.

The object detection program for object detection processing executed by the object detection unit 40 (CPU 24a) of this embodiment is a file in an installable format or an executable format, and is stored on a CD-ROM, a flexible disk (FD), or a CD-ROM. The information may be provided by being recorded on a computer-readable recording medium such as R, DVD (Digital Versatile Disk), or the like.

Furthermore, the object detection program may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network. Further, the object detection program executed in this embodiment may be provided or distributed via a network such as the Internet.

Although the embodiments and modifications of the present invention have been described, these embodiments and modifications are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

DESCRIPTION OF SYMBOLS 10... Vehicle, 14, 14a, 14b, 14c, 14d... Imaging unit, 24... ECU, 24a... CPU, 26... Wheel speed sensor, 28... Rudder angle sensor, 30... Shift sensor, 32... Driving support unit, 40... Object detection unit, 42... Image acquisition unit, 44... Setting unit, 44a... Three-dimensional object boundary line setting unit, 44b... Bounding box setting unit, 44c... Driving expected area setting unit, 46... Control unit, 46a... Partial boundary line extraction Section, 46b... Point of interest estimation section, 46c... Output section, 50... Captured image, 52... Road surface, 58... Three-dimensional object boundary line, 60... Bounding box, 62... Partial boundary line, 100... Control system, T... Point of interest .

Claims (4)

an acquisition unit that acquires an image captured by an imaging unit capable of capturing an image of the surrounding situation including the road surface where the vehicle is present;
A three-dimensional object boundary line indicating a boundary between the road surface and an object area recognized as a region where a three-dimensional object exists on the road surface included in the captured image, and a three-dimensional object boundary line that indicates a boundary between the three-dimensional object included in the object area. a rectangular bounding box for selecting a target object;
extracting a partial boundary line included in the bounding box from the three-dimensional object boundary line, and estimating and outputting a point of interest to be focused on to avoid contact between the vehicle and the target object on the partial boundary line; a control unit;
Equipped with
The control unit is an object detection device that calculates a distance from the vehicle to the target object in three-dimensional space based on the point of interest . The object detection device according to claim 1, wherein the control unit sets the lowest position in the vertical axis direction of the bounding box among the partial boundaries as the point of interest. 2. The object detection device according to claim 1, wherein when there is a switching position in the partial boundary line where the slope of the partial boundary line is switched between positive and negative, the control unit sets the switching position as the point of interest. The control unit estimates the position of the point of interest by determining the partial boundary line based on the bounding box and a predicted travel area determined by a predicted travel line indicating a traveling direction based on a steering angle of the vehicle. The object detection device according to any one of claims 1 to 3.

Images