JP2022154605A - Detection apparatus, detection method, and program - Google Patents

Abstract

To provide a detection apparatus capable of more surely suppressing reduction in recognition accuracy of an object around an own vehicle, and provide a detection method and a program.SOLUTION: The present invention relates to a detection apparatus comprising: a first feature point extraction section for extracting as a first feature point a feature point of an object detected by a first object detection device which detects an object ahead of an own vehicle, based on a detection result of the first object detection device; a second feature point extraction section for extracting the feature point of the object as a second feature point based on a detection result of a second object detection device including a part of or an entire detection range of the first object detection device as a detection range; and a determination section for determining that a block factor with respect to the detection of the object is present, regarding the object detection device acquiring a detection result relating to an extraction of a feature point group missing a corresponding feature point between a first feature point group which is a point group of the first feature points and a second feature point group which is a point group of the second feature points, between the first object detection device and the second object detection device.SELECTED DRAWING: Figure 1

Description

The present invention relates to a detection device, a detection method, and a program.

Conventionally, for the purpose of preventing deterioration in the detection performance of surrounding information of a vehicle, a technique has been developed that activates a washing device when dirt is detected based on the defect rate of surrounding information (Patent Document 1).

JP 2016-179767 A

However, in the conventional technology, the defect rate of the surrounding information output by the sensor to be cleaned is detected based on the output status of the signal from the sensor. In some cases, it was not possible to avoid a decrease in the recognition rate of peripheral information.

The present invention has been made in consideration of such circumstances, and aims to provide a detection device, a detection method, and a program that can more reliably suppress deterioration in the accuracy of recognizing objects around the vehicle. one of the purposes.

A detection device, a detection method, and a program according to the present invention employ the following configurations.
(1): A detection device according to an aspect of the present invention detects feature points of an object detected by the first object detection device based on the detection result of the first object detection device that detects an object in front of the vehicle. a first feature point extraction unit for extracting as first feature points; and a second object detection device for detecting the object, the detection range including part or all of the detection range of the first object detection device. A second feature point extraction unit for extracting the feature points of the object as second feature points based on the detection result of the object detection device of 2, and the first object detection device and the second object detection device, Extraction of a feature point group in which corresponding feature points are missing from a first feature point group that is a point group of the first feature points and a second feature point group that is a point group of the second feature points a determination unit that determines that an obstacle to detection of the object exists in the object detection device that has acquired the detection result.

(2): In the aspect of (1) above, the determination unit determines whether the inhibition It determines that a factor exists.

(3): In the aspect (1) or (2) above, the determination unit determines that the first feature point or the second feature point that does not have a corresponding feature point is a missing feature point that should be detected. Feature points are detected in chronological order, the number of detected missing feature points is totaled for each partial area obtained by dividing the detection range, and when the total number of missing feature points exceeds a threshold, the It is determined that the obstruction factor exists in the partial area for which the total number has been obtained.

(4): In any one of the above aspects (1) to (3), the first object detection device is a camera that detects an object in front of the vehicle through a front window from inside the vehicle, and A factor that hinders object detection by the first object detection device is water droplets adhering to the front window.

(5): In the aspect of (4) above, the second object detection device is a radar device behind a radio wave transmitting portion provided in a portion of the bumper, and the object detected by the second object detection device A detection impediment factor is an adhering substance to the radio wave transmitting portion.

(6): In the aspect of (4) above, the second object detection device is a lidar device provided on a bumper, and the obstacle to object detection by the second object detection device is the lidar device. Adherence to the light-emitting part or light-receiving part.

(7): In any one of the above (4) to (6), the heater controller further comprises a heater control unit that operates a heater device that blows hot air onto the front window according to the determination result of the determination unit. The control unit operates the heater device when the determination unit determines that the obstruction factor exists with respect to object detection by the first object detection device.

(8): A detection method according to an aspect of the present invention, in which a computer detects an object detected by a first object detection device based on a detection result of a first object detection device that detects an object in front of the vehicle. A second object detection device for extracting a feature point as a first feature point and detecting the object, the second object including part or all of a detection range of the first object detection device in the detection range. Based on the detection result of the detection device, the feature points of the object are extracted as second feature points, and the point group of the first feature points of the first object detection device and the second object detection device is obtained. Object detection of the first feature point group and the second feature point group, which is the point group of the second feature points, for which the detection result related to the extraction of the feature point group in which the corresponding feature point is missing is acquired. For the device, it is determined that there is an obstacle to the detection of the object.

(9): A program according to an aspect of the present invention provides a computer with characteristics of an object detected by the first object detection device based on the detection result of the first object detection device for detecting an object in front of the vehicle. A second object detection device for extracting a point as a first feature point and detecting the object, wherein the detection range includes part or all of a detection range of the first object detection device. Based on the detection result of the device, the feature points of the object are extracted as the second feature points, and the point group of the first feature points, which is the point group of the first object detection device and the second object detection device, is extracted. Regarding the object detection device that acquires the detection result related to the extraction of the feature point group in which the corresponding feature point is missing from the second feature point group, which is the point group of the first feature point group and the second feature point group, It is determined that there is an obstacle to the detection of the object.

According to aspects (1) to (9) above, based on the detection result of the first object detection device that detects an object in front of the vehicle, the characteristic points of the object detected by the first object detection device are Detection by a second object detection device that extracts one feature point and detects the object, the detection range including part or all of the detection range of the first object detection device Based on the result, the feature points of the object are extracted as second feature points, and the first feature points, which are the point group of the first feature points, out of the first object detection device and the second object detection device With respect to an object detection device that acquires a detection result related to extraction of a feature point group in which a corresponding feature point is missing from among a second feature point group that is a point group of the second feature point group and the second feature point group, By determining that there is an obstacle to detection, it is possible to more reliably suppress a decrease in accuracy in recognizing objects around the vehicle.

1 is a configuration diagram of a vehicle system using a vehicle control device according to an embodiment; FIG. 3 is a functional configuration diagram of a first control unit and a second control unit; FIG. It is a figure which shows an example of the correspondence of a driving mode, the control state of the own vehicle, and a task. It is a block diagram which shows an example of the structure with which the object recognition apparatus of embodiment detects an obstruction factor. FIG. 10 is a diagram showing an example of a case where the obstruction factor determination unit determines that there is no obstruction factor in the object recognition device of the embodiment; FIG. 10 is a diagram showing an example of a case in which an obstruction factor determination unit determines that there is an obstruction factor in the object recognition device of the embodiment; 4 is a flowchart showing an example of the flow of non-corresponding feature point detection processing executed by the object recognition device of the embodiment; 6 is a flow chart showing an example of the flow of hindrance factor determination processing executed by the object recognition device of the embodiment. FIG. 5 is an image diagram showing a specific example of hindrance factor determination processing in the embodiment;

Hereinafter, embodiments of the detection device, detection method, and program of the present invention will be described with reference to the drawings.

[overall structure]
FIG. 1 is a configuration diagram of a vehicle system 1 using a vehicle control device according to an embodiment. A vehicle on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or electric power discharged from a secondary battery or a fuel cell.

The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, and a vehicle sensor 40. , a navigation device 50, an MPU (Map Positioning Unit) 60, a driver monitor camera 70, a driving operator 80, an automatic driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. and a heater device 300 . These apparatuses and devices are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, and the like. Note that the configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted, or another configuration may be added.

The camera 10 is, for example, a digital camera using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary location of a vehicle (hereinafter referred to as own vehicle) on which the vehicle system 1 is mounted. When imaging the front, the camera 10 is attached to the upper part of the front windshield, the rear surface of the rearview mirror, or the like. The camera 10, for example, periodically and repeatedly captures the surroundings of the own vehicle. Camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves around the vehicle and detects radio waves (reflected waves) reflected by an object to detect at least the position (distance and direction) of the object. The radar device 12 may be attached to any location on the vehicle. The radar device 12 may detect the position and velocity of an object by the FM-CW (Frequency Modulated Continuous Wave) method. For example, the radar device 12 is installed behind a radio wave transmitting portion provided in a portion of the bumper.

The LIDAR 14 irradiates light (or electromagnetic waves with wavelengths close to light) around the vehicle and measures scattered light. The LIDAR 14 detects the distance to the object based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. LIDAR14 may be attached to the arbitrary parts of self-vehicles.

The object recognition device 16 performs sensor fusion processing on the detection results of some or all of the camera 10, the radar device 12, and the LIDAR 14, and recognizes the position, type, speed, and the like of the object. The object recognition device 16 outputs recognition results to the automatic driving control device 100 . Object recognition device 16 may output the detection result of camera 10, radar installation 12, and LIDAR14 to automatic operation control device 100 as it is. The object recognition device 16 may be omitted from the vehicle system 1 .

In addition to the object recognition function described above, the object recognition device 16 in the present embodiment has a function of detecting the presence or absence of an object that may hinder object detection by the camera 10, the radar device 12, and the LIDAR 14. For example, object detection by the camera 10 can be hindered by an object that is present in the detection range (imaging range) of the camera 10 and that impairs visibility to the object that is originally desired to be detected.

For example, water droplets adhering to the front window and the lens are examples of factors that hinder object detection by the camera 10 . Further, when the radar device 12 is installed behind a radio wave transmitting portion provided in a part of a bumper, an example of a factor that hinders object detection by the radar device 12 is dirt adhering to the radio wave transmitting portion. Further, an example of a factor that hinders object detection by the LIDAR 14 is dirt adhering to the light emitting unit or the light receiving unit of the LIDAR 14 .

The object recognition device 16 determines the presence or absence of these hindrance factors based on the detection results of object detection devices such as the camera 10, the radar device 12, and the LIDAR 14. The details of the method for determining the presence or absence of a hindrance factor will be described later. The object recognition device 16 is an example of a "detection device." The object recognition device 16 may be configured separately into an object recognition device having an object recognition function and a detection device for detecting the presence or absence of an obstruction factor.

The communication device 20 uses, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like, to communicate with other vehicles existing in the vicinity of the own vehicle, or to communicate with a wireless base. It communicates with various server devices through the station.

The HMI 30 presents various types of information to the occupants of the own vehicle and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys, and the like.

The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects acceleration, a yaw rate sensor that detects angular velocity about a vertical axis, a direction sensor that detects the orientation of the vehicle, and the like.

The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51 , a navigation HMI 52 and a route determining section 53 . The navigation device 50 holds first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 identifies the position of the own vehicle based on the signals received from the GNSS satellites. The position of the own vehicle may be specified or complemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40 . The navigation HMI 52 includes a display device, speaker, touch panel, keys, and the like. The navigation HMI 52 may be partially or entirely shared with the HMI 30 described above. For example, the route determination unit 53 determines a route (hereinafter referred to as a map upper route) is determined with reference to the first map information 54 . The first map information 54 is, for example, information in which road shapes are represented by links indicating roads and nodes connected by the links. The first map information 54 may include road curvature, POI (Point Of Interest) information, and the like. A route on the map is output to the MPU 60 . The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the function of a terminal device such as a smart phone or a tablet terminal owned by the passenger. The navigation device 50 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire a route equivalent to the route on the map from the navigation server.

The MPU 60 includes, for example, a recommended lane determination unit 61, and holds second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determining unit 61 divides the route on the map provided from the navigation device 50 into a plurality of blocks (for example, by dividing each block by 100 [m] in the vehicle traveling direction), and refers to the second map information 62. Determine recommended lanes for each block. The recommended lane decision unit 61 decides which lane to drive from the left. The recommended lane determination unit 61 determines a recommended lane so that the vehicle can travel on a rational route to the branch when there is a branch on the route on the map.

The second map information 62 is map information with higher precision than the first map information 54 . The second map information 62 includes, for example, lane center information or lane boundary information. Further, the second map information 62 includes road information, traffic control information, address information (address/zip code), facility information, telephone number information, information on prohibited sections where mode A or mode B is prohibited, and the like. may be included. The second map information 62 may be updated at any time by the communication device 20 communicating with other devices.

The driver monitor camera 70 is, for example, a digital camera using a solid-state imaging device such as CCD or CMOS. The driver monitor camera 70 is positioned and oriented such that the head of the passenger (hereinafter referred to as the driver) seated in the driver's seat of the vehicle can be imaged from the front (in an orientation that captures an image of the face), and is positioned at an arbitrary location in the vehicle. It is attached. For example, the driver monitor camera 70 is attached to the upper part of the display device provided in the central part of the instrument panel of the own vehicle.

The driving operator 80 includes, for example, a steering wheel 82, an accelerator pedal, a brake pedal, a shift lever, and other operators. A sensor that detects the amount of operation or the presence or absence of operation is attached to the driving operation element 80, and the detection result is applied to the automatic driving control device 100, or the driving force output device 200, the brake device 210, and the steering device. 220 to some or all of them. The steering wheel 82 is an example of "operator for receiving steering operation by the driver". The manipulator does not necessarily have to be ring-shaped, and may be in the form of a deformed steering wheel, joystick, button, or the like. A steering grip sensor 84 is attached to the steering wheel 82 . The steering grip sensor 84 is realized by a capacitance sensor or the like, and automatically generates a signal capable of detecting whether or not the driver is gripping the steering wheel 82 (meaning that the driver is in contact with the steering wheel 82 in a state where force is applied). Output to the operation control device 100 .

Automatic operation control device 100 is provided with the 1st control part 120 and the 2nd control part 160, for example. The first control unit 120 and the second control unit 160 are each implemented by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). (including circuitry), or by cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as the HDD or flash memory of the automatic operation control device 100, or may be detachable such as a DVD or CD-ROM. It is stored in a storage medium, and may be installed in the HDD or flash memory of the automatic operation control device 100 by attaching the storage medium (non-transitory storage medium) to the drive device.

FIG. 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. As shown in FIG. The 1st control part 120 is provided with the recognition part 130, the action plan production|generation part 140, and the mode determination part 150, for example. The first control unit 120, for example, realizes in parallel a function based on AI (Artificial Intelligence) and a function based on a model given in advance. For example, the "recognition of intersections" function performs recognition of intersections by deep learning, etc., and recognition based on predetermined conditions (signals that can be pattern-matched, road markings, etc.) in parallel. It may be realized by scoring and evaluating comprehensively. This ensures the reliability of automated driving.

The recognition unit 130 recognizes the position, speed, acceleration, and other states of objects around the vehicle based on information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16. . The position of the object is recognized, for example, as a position on absolute coordinates with a representative point (the center of gravity, the center of the drive shaft, etc.) of the own vehicle as the origin, and used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. The "state" of the object may include acceleration or jerk of the object, or "behavioral state" (eg, whether it is changing lanes or about to change lanes).

The recognition unit 130 also recognizes, for example, the lane in which the host vehicle is traveling (driving lane). For example, the recognition unit 130 recognizes the pattern of road markings obtained from the second map information 62 (for example, an array of solid and broken lines) and the road markings around the vehicle recognized from the image captured by the camera 10. By comparing with the pattern, the driving lane is recognized. Note that the recognition unit 130 may recognize the driving lane by recognizing road boundaries (road boundaries) including road division lines, road shoulders, curbs, medians, guardrails, etc., not limited to road division lines. . In this recognition, the position of the own vehicle acquired from the navigation device 50 and the processing result by the INS may be taken into consideration. The recognition unit 130 also recognizes stop lines, obstacles, red lights, toll gates, and other road events.

The recognition unit 130 recognizes the position and posture of the own vehicle with respect to the driving lane when recognizing the driving lane. The recognizing unit 130 uses, for example, the deviation of the vehicle's reference point from the lane center and the angle formed with the line connecting the lane center in the traveling direction of the vehicle as the relative position and attitude of the vehicle with respect to the driving lane. may recognize. Instead, the recognition unit 130 recognizes the position of the vehicle's reference point with respect to one of the side edges of the driving lane (road division line or road boundary) as the relative position of the vehicle with respect to the driving lane. good too.

In principle, the action plan generation unit 140 drives the recommended lane determined by the recommended lane determination unit 61, and furthermore, the own vehicle automatically (operated by the driver) so as to respond to the surrounding conditions of the own vehicle. ) to generate a target trajectory to travel in the future. The target trajectory includes, for example, velocity elements. For example, the target trajectory is represented by arranging points (trajectory points) that the vehicle should reach in order. A trajectory point is a point to be reached by the own vehicle for each predetermined travel distance (for example, about several [m]) along the road, and separately from that, for a predetermined sampling time (for example, about 0 comma [sec]). A target velocity and acceleration for each is generated as part of the target trajectory. Also, the trajectory point may be a position that the vehicle should reach at each predetermined sampling time. In this case, the information on the target velocity and target acceleration is represented by the intervals between the trajectory points.

The action plan generator 140 may set an automatic driving event when generating the target trajectory. Autonomous driving events include constant-speed driving events, low-speed following driving events, lane change events, branching events, merging events, and takeover events. The action plan generator 140 generates a target trajectory according to the activated event.

The mode determination unit 150 determines the driving mode of the own vehicle to one of a plurality of driving modes with different tasks imposed on the driver. The mode determination unit 150 includes, for example, a driver state determination unit 152 and a mode change processing unit 154 . These individual functions are described below.

FIG. 3 is a diagram showing an example of correspondence relationships between driving modes, control states of the own vehicle, and tasks. There are, for example, five modes from mode A to mode E in the driving mode of the own vehicle. The control state, that is, the degree of automation of the driving control of the host vehicle, is highest in mode A, followed by mode B, mode C, and mode D in descending order, and mode E is lowest. Conversely, the tasks imposed on the driver are lightest in Mode A, followed by Mode B, Mode C, Mode D in order of severity, and Mode E being the most severe. In Modes D and E, since the control state is not automatic operation, the automatic operation control device 100 is responsible for terminating control related to automatic operation and shifting to driving assistance or manual operation. The contents of each operation mode are exemplified below.

In mode A, the vehicle is automatically driven, and the driver is not required to look ahead or grip the steering wheel 82 (gripping the steering wheel in the figure). However, even in mode A, the driver is required to be in a posture that can quickly shift to manual driving in response to a request from the system centered on the automatic driving control device 100 . The term "automatic driving" as used herein means that both steering and acceleration/deceleration are controlled independently of the driver's operation. The front means the space in the traveling direction of the own vehicle that is visually recognized through the front windshield. In mode A, for example, on a motorway such as a highway, the own vehicle is traveling at a predetermined speed (for example, about 50 [km/h]) or less, and there is a preceding vehicle to be followed. It is an operation mode that can be executed when it is satisfied, and is sometimes referred to as TJP (Traffic Jam Pilot). When this condition is no longer satisfied, the mode determination unit 150 changes the driving mode of the host vehicle to mode B.

In mode B, the driver is tasked with monitoring the front of the host vehicle (hereinafter referred to as forward monitoring), but is not tasked with gripping the steering wheel 82 . In mode C, the vehicle is in a state of driving assistance, and the driver is tasked with a task of looking ahead and a task of gripping the steering wheel 82 . Mode D is a driving mode that requires a certain amount of driving operation by the driver regarding at least one of steering and acceleration/deceleration of the own vehicle. For example, mode D provides driving assistance such as ACC (Adaptive Cruise Control) and LKAS (Lane Keeping Assist System). In mode E, the vehicle is in a manual operation state in which the driver needs to operate the vehicle for both steering and acceleration/deceleration. In both modes D and E, the driver is naturally tasked with monitoring the front of the vehicle.

The automatic driving control device 100 (and the driving support device (not shown)) executes an automatic lane change according to the driving mode. The automatic lane change includes an automatic lane change (1) requested by the system and an automatic lane change (2) requested by the driver. Automatic lane change (1) includes an automatic lane change for overtaking and an automatic lane change for advancing toward the destination, which is performed when the speed of the preceding vehicle is lower than the speed of the own vehicle by a standard or higher. Auto lane change (auto lane change due to change of recommended lane). Auto Lane Change (2) changes the lane of the vehicle in the direction in which the driver operates the direction indicator when the conditions related to speed, positional relationship with surrounding vehicles, etc. are satisfied. It is a thing.

In mode A, the automatic driving control device 100 performs neither automatic lane change (1) nor (2). In modes B and C, the automatic driving control device 100 executes both automatic lane changes (1) and (2). In mode D, the driving support device (not shown) executes automatic lane change (2) without executing automatic lane change (1). In mode E, neither auto lane change (1) nor (2) is performed.

The mode determining unit 150 changes the driving mode of the host vehicle to a driving mode with more severe tasks when the driver does not execute the task related to the determined driving mode (hereinafter referred to as the current driving mode).

For example, in mode A, if the driver is in a position that cannot shift to manual driving in response to a request from the system (for example, if he continues to look aside outside the allowable area, or if a sign of difficulty driving is detected) ), the mode determination unit 150 uses the HMI 30 to urge the driver to shift to manual driving, and if the driver does not respond, the host vehicle is pulled over to the road shoulder and gradually stopped, thereby stopping automatic driving. conduct. After the automatic operation is stopped, the own vehicle enters the state of mode D or E, and the own vehicle can be started by manual operation of the driver. Hereinafter, the same applies to "stop automatic operation". If the driver is not looking ahead in mode B, the mode determining unit 150 uses the HMI 30 to prompt the driver to look ahead, and if the driver does not respond, the vehicle is pulled over to the road shoulder and gradually stopped. It performs control such as stopping automatic operation. If the driver is not looking ahead or is not gripping the steering wheel 82 in mode C, the mode determiner 150 uses the HMI 30 to instruct the driver to look ahead and/or grip the steering wheel 82. If the driver does not respond, the vehicle is pulled over to the side of the road and gradually stopped, and the automatic operation is stopped.

The driver state determination unit 152 monitors the driver state for the above mode change and determines whether or not the driver state corresponds to the task. For example, the driver state determination unit 152 analyzes the image captured by the driver monitor camera 70, performs posture estimation processing, and determines whether the driver is in a posture that cannot shift to manual driving in response to a request from the system. judge. Further, the driver state determination unit 152 analyzes the image captured by the driver monitor camera 70, performs line-of-sight estimation processing, and determines whether or not the driver is monitoring the front.

The mode change processing unit 154 performs various processes for mode change. For example, the mode change processing unit 154 instructs the action plan generation unit 140 to generate a target trajectory for stopping on the shoulder, instructs a driving support device (not shown) to operate, or instructs the driver to take action. The HMI 30 is controlled for prompting.

The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the host vehicle passes the target trajectory generated by the action plan generating unit 140 at the scheduled time. do.

Returning to FIG. 2, the second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information on the target trajectory (trajectory point) generated by the action plan generation unit 140 and stores it in a memory (not shown). Speed control unit 164 controls running driving force output device 200 or brake device 210 based on the speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curve of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized by, for example, a combination of feedforward control and feedback control. As an example, the steering control unit 166 performs a combination of feedforward control according to the curvature of the road ahead of the host vehicle and feedback control based on deviation from the target trajectory.

The running driving force output device 200 outputs running driving force (torque) for running the vehicle to the drive wheels. Traveling driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls these. The ECU controls the above configuration in accordance with information input from the second control unit 160 or information input from the operation operator 80 .

The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motors according to information input from the second control unit 160 or information input from the driving operator 80 so that brake torque corresponding to the braking operation is output to each wheel. The brake device 210 may include, as a backup, a mechanism that transmits hydraulic pressure generated by operating a brake pedal included in the operation operator 80 to the cylinders via a master cylinder. The brake device 210 is not limited to the configuration described above, and is an electronically controlled hydraulic brake device that controls the actuator according to information input from the second control unit 160 to transmit the hydraulic pressure of the master cylinder to the cylinder. good too.

The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the orientation of the steered wheels. The steering ECU drives the electric motor according to information input from the second control unit 160 or information input from the driving operator 80 to change the direction of the steered wheels.

The heater device 300 is a device that blows hot air onto the front windshield in order to remove water droplets adhering to the windshield. In the present embodiment, the heater device 300 is configured to start or stop air blowing in accordance with the operation of the driver and in accordance with the instruction of the object recognition device 16 .

FIG. 4 is a block diagram showing an example of a configuration that the object recognition device 16 according to this embodiment has for detecting an obstruction factor. The object recognition device 16 includes, for example, a storage section 610 and a control section 620 . Control unit 620 is implemented by, for example, a hardware processor such as a CPU executing a program (software). Also, some or all of these components may be realized by hardware (circuitry) such as LSI, ASIC, FPGA, GPU, etc., or by cooperation of software and hardware may be The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as the HDD or flash memory of the automatic operation control device 100, or may be detachable such as a DVD or CD-ROM. It is stored in a storage medium, and may be installed in the HDD or flash memory of the automatic operation control device 100 by attaching the storage medium (non-transitory storage medium) to the drive device.

The storage unit 610 is implemented by, for example, an HDD, a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The storage unit 610 includes, for example, an object Information including criteria and threshold values used by the recognition device 16 in the process of detecting an obstruction factor is stored.

The control unit 620 includes, for example, a first feature point extraction unit 621, a second feature point extraction unit 622, a feature point superimposition unit 623, an obstruction factor determination unit 624, and a heater control unit 625. The first feature point extraction unit 621 extracts the feature points of the object detected by the camera 10 as first feature points based on the detection result of the object by the camera 10 . Similarly, the second feature point extraction unit 622 extracts the feature points of the object detected by the radar device 12 as second feature points based on the detection result of the object by the radar device 12 . The second feature point extraction unit 622 may be configured to extract the feature points of the object detected by the LIDAR 14 as the second feature points based on the detection result of the object by the LIDAR 14 .

Here, the camera 10 is an example of a "first object detection device" that detects an object in front of the vehicle based on the reception result of electromagnetic waves, and the radar device 12 and LIDAR 14 are examples of a "second object detection device". An example. Moreover, the radar device 12 and the LIDAR 14 are installed at positions and attitudes that include part or all of the detection range of the camera 10 in the detection range. First feature point extraction section 621 and second feature point extraction section 622 output information indicating the extracted first feature point group and second feature point group, respectively, to feature point superimposition section 623 .

The feature point superimposing unit 623 superimposes the first feature point group and the second feature point group in the same coordinate space, thereby obtaining feature points indicating the same object in the first feature point group and the second feature point group (hereinafter referred to as " ) are extracted. Further, the feature point superimposing unit 623 detects feature points that are not extracted as a set of corresponding feature points among the feature points included in the first feature point group or the second feature point group, as “non-corresponding feature points”. That is, a non-corresponding feature point is a feature point belonging to one of the first feature point group and the second feature point group, and having no corresponding feature point in the other feature point group. is.

The obstruction factor determination unit 624 determines whether there is an obstruction factor for object detection by the camera 10 and the radar device 12 based on the detection result of the non-corresponding feature points by the feature point superimposition unit 623 . Specifically, when the feature point superimposing unit 623 detects a non-corresponding feature point, the hindrance factor determination unit 624 detects the non-corresponding feature point from the first feature point group and the second feature point group. It is determined that there is an impeding factor in the object detection means related to the feature point group that did not exist.

FIG. 5 is a diagram showing an example of a case where the obstruction factor determination unit 624 determines that there is no obstruction factor for any of the detection means as a result of the determination. FIG. 5 shows detection results of the camera 10 and the radar device 12 at times t ₁₀ and t ₁₁ (>t ₁₀ ). FIG. 5 shows detection results in a situation in which the own vehicle is traveling in the direction from the front side of the paper to the back of the paper on the lane R1 demarcated by the road division lines L1 and L2. A vehicle M1 in the drawing is a vehicle that runs in front of the own vehicle.

In this example, the obstruction factor determination unit 624 detects the same object (existing on the left side of the lane R1) in both the detection result of the camera ₁₀ and the detection result of the radar device ₁₂ at both times t10 and t11. In this case, the feature point superimposing unit 623 does not detect non-corresponding feature points because the camera 10 and the radar device 12 extract the same feature points from the street tree T1. In this case, the obstruction factor determination unit 624 determines that there is no obstruction factor for both the detection means of the camera 10 and the radar device 12 .

On the other hand, FIG. 6 is a diagram showing an example of a case where the obstruction factor determination unit 624 determines that there is an obstruction factor in object detection by the camera 10 as a result of the determination. FIG. 6 shows detection results of the camera 10 and the radar device 12 at times t ₂₀ and t ₂₁ (>t ₂₀ ). FIG. 6 shows the results of object detection performed by the own vehicle in the same running situation as in FIG. An obstacle W1 in the drawing is, for example, water droplets adhering to the front window.

_In this example, at time t20, the water droplet W1 does not block the line of sight to the roadside tree T1. There is no hindrance factor”. _On the other hand, at time t11, the water droplet W1 blocks the line of sight to the roadside tree T1. It becomes a feature point of the part that is not On the other hand, since the radar device 12 detects the entire street tree T1, the feature point group F3 that does not correspond to the feature point group F1 out of the feature point group F2 of the street tree T1 extracted with respect to the detection result by the radar device 12 is a non-corresponding feature point. Therefore, in this case, the obstruction factor determination unit 624 determines that there is an obstruction factor with respect to the object detection by the camera 10 .

The heater control unit 625 controls the heater device 300 based on the determination result of the presence or absence of the obstruction factor by the obstruction factor determination part 624 . Specifically, the heater control unit 625 operates the heater device 300 when the obstruction factor determination unit 624 determines that there is an obstruction factor. As a result, when the detected impeding factor is water droplets adhering to the front window, the impeding factor can be removed.

[Method for Determining Inhibitory Factors]
FIG. 7 is a flow chart showing an example of the flow of processing for detecting non-corresponding feature points (hereinafter referred to as "non-corresponding feature point detection processing") by the object recognition device 16 of this embodiment. First, the feature point superimposing unit 623 acquires information on the first feature point group from the first feature point extraction unit 621 and acquires information on the second feature point group from the second feature point extraction unit 622 (step S101). ).

Subsequently, the feature point superimposing unit 623 selects one feature point from the first feature point group (step S102), and for the selected first feature point, a corresponding feature point exists in the second feature point group. It is determined whether or not (step S103). Here, if it is determined that the corresponding feature point does not exist in the second feature point group for the selected first feature point, the feature point superimposing unit 623 records the selected first feature point as a non-corresponding feature point. Then (step S104), the process proceeds to step S105. On the other hand, if it is determined in step S103 that the corresponding feature point exists in the second feature point group for the selected first feature point, the feature point superimposing unit 623 skips step S104 and proceeds to step S105. proceed.

Subsequently, the feature point superimposing unit 623 determines whether or not the corresponding feature points have been checked for all the feature points included in the first feature point group (step S105). Here, when it is determined that there is an unchecked feature point in the first feature point group, the feature point superimposing unit 623 returns the processing to step S102, and repeats steps S102 to S104 until there is no unchecked feature point. do. On the other hand, if it is determined in step S105 that the corresponding feature points have been inspected for all feature points included in the first feature point group, the feature point superimposing unit 623 advances the process to step S106.

Subsequently, the feature point superimposing unit 623 selects one feature point from the second feature point group (step S106), and for the selected second feature point, the corresponding feature point exists in the first feature point group. It is determined whether or not (step S107). Here, if it is determined that the corresponding feature point does not exist in the first feature point group for the selected second feature point, the feature point superimposing unit 623 records the selected second feature point as a non-corresponding feature point. Then (step S108), the process proceeds to step S109. On the other hand, if it is determined in step S107 that the corresponding feature point exists in the first feature point group for the selected second feature point, the feature point superimposing unit 623 skips step S108 and proceeds to step S109. proceed.

Subsequently, the feature point superimposing unit 623 determines whether or not the corresponding feature points have been inspected for all the feature points included in the second feature point group (step S109). Here, if it is determined that there is an unchecked feature point in the second feature point group, the feature point superimposing unit 623 returns the process to step S106, and repeats steps S106 to S108 until there is no unchecked feature point. do. On the other hand, if it is determined in step S109 that the corresponding feature points have been inspected for all the feature points included in the second feature point group, the feature point superimposing unit 623 ends the non-corresponding feature point detection processing.

Note that the non-corresponding feature point detection processing described with reference to FIG. 7 is processing performed for each detection result acquired in time series for the camera 10 and the radar device 12 . The feature point superimposing unit 623 performs the non-corresponding feature point detection process of FIG. (In this embodiment, that is, the camera 10 or the radar device 12) are associated with each other and recorded. In this way, by cumulatively recording the detection information of the non-corresponding feature points, the hindrance factor determination unit 624 can count the number of non-corresponding feature points that have occurred during the specified period. It is possible to determine the presence or absence of a hindrance factor based on the number of occurrences of non-corresponding feature points in a period. Specifically, the obstruction factor determination unit 624 can detect the presence or absence of an obstruction factor by executing the obstruction factor determination process shown in FIG. 8 below.

In the example of FIG. 7, the feature point superimposing unit 623 inspects the second feature point after inspecting the first feature point, but either feature point inspection may be performed first. However, both may be performed in parallel.

FIG. 8 is a flowchart showing an example of the flow of obstruction factor determination processing executed by the object recognition device 16 of this embodiment. First, the obstruction factor determination unit 624 acquires coordinate information for all non-corresponding feature points detected with respect to the camera 10 and the radar device 12 from the feature point superimposition unit 623 (step S201). Subsequently, the hindrance factor determination unit 624 selects a non-corresponding feature point to be processed from the group of non-corresponding feature points for which coordinate information has been acquired (step S202), and for the selected non-corresponding feature point, a correspondence that should be detected The coordinates of the feature point (hereinafter referred to as "missing feature point") are recorded on the recording surface corresponding to the detection means (step S203). Here, the recording surface is a memory area for recording the presence of missing feature points in correspondence with the positions to be originally detected, and is secured for each detection means of the camera 10 and the radar device 12. be.

Subsequently, the hindrance factor determination unit 624 sums up the number of missing feature points recorded on the recording surface of each detection means for each partial area set within the detection range, and the total number is equal to or greater than the threshold. It is determined whether or not there is a partial area in which the image is present (step S204). Here, when it is determined that there is a partial area in which the total number of missing feature points is equal to or greater than the threshold value, the hindrance factor determination unit 624 determines that the detection means that should originally detect the missing feature points in the partial area is It is determined that "there is an impediment factor" regarding detection of the area (step S205), and the impediment factor determination process is terminated. On the other hand, if it is determined in step S204 that there is no partial region in which the total number of missing feature points is greater than or equal to the threshold value, the impediment factor determination unit 624 determines that "there is no impediment factor" (step S206), The hindrance factor determination process is terminated.

FIG. 9 is an image diagram showing a specific example of the hindrance factor determination process. Here, for example, for each of the camera 10 and the radar device 12, frames indicating detection results are input to the object recognition device 16 at predetermined time intervals in order of N (N represents an integer equal to or greater than 1), N+1, and N+2. It is assumed that Hereinafter, the frames input in the order of N, N+1, and N+2 are referred to as N frame, N+1 frame, and N+2 frame, respectively, and each frame represents the detection result obtained from the common detection area by the camera 10 and the radar device 12. shall be Also, the example of FIG. 9 represents a case where the camera 10 captures images of the hindrance factor B2 in the detection of the N frame, the N+1 frame, and the N+2 frame.

In this case, the hindrance factor determination unit 624 secures the recording planes M1 and M2 as recording planes for recording the missing feature points of the camera 10 and the radar device 12, respectively, corresponding to the detection range common to the detection means. . Subsequently, the hindrance factor determination unit 624 determines the corresponding missing feature for each of the second feature points determined as non-corresponding feature points in the second feature point group extracted based on the object detection result by the radar device 12. A point is recorded on the recording surface M1 of the camera 10. FIG.

In the example of FIG. 9, since the obstruction factor B1 exists with respect to the object detection of the camera 10, in the second feature point group, the non-corresponding feature points F11-1, F11-2, F11-1, F11-2, F12-1 to F12-3 and F13 are detected, and missing feature points D11-1, D11-2, D12-1 to D12-3 and D13 corresponding to them are recorded on the recording surface M1. A recording surface M1 shown in FIG. 8 represents the result of performing processing for recording missing feature points for the N frame, the N+1 frame, and the N+2 frame. Missing feature points D11-1 and D11-2 on the recording surface M1 are missing feature points corresponding to the non-corresponding feature points F11-1 and F11-2 detected in the N frames of the radar device 12. FIG. Missing feature points D12-1 to D12-3 are missing feature points corresponding to non-corresponding feature points F12-1 to F12-3 detected in the N+1 frame of the radar device 12. FIG. Also, the missing feature point D13 is a missing feature point corresponding to the non-corresponding feature point F13 detected in the N+2 frame of the radar device 12 .

In this case, the hindrance factor determination unit 624 determines the fourth area (the partial area indicated by (4) in the figure) and the seventh area (the partial area indicated by (4) in the figure) and the seventh area (the It may be determined that there is a hindrance factor with respect to the partial area indicated by (7) in the middle. Further, the hindrance factor determination unit 624 may determine that a hindrance factor exists in a region in which missing feature points are continuously detected at the same position for a specified time or more. For example, when the missing feature points D11-1 and D11-2 detected in the N frame are continuously detected in the N+1 frame and the N+2 frame, the hindrance factor determination unit 624 determines that the missing feature points D11-1 and D11 It may be determined that there is an obstacle to object detection in the fourth area and the seventh area where -2 is recorded.

In addition, the hindrance factor determination unit 624 sums up the number of detected missing feature points for each partial area within the detection range in time series, and determines whether the partial area in which the total number is equal to or greater than the threshold value hinders object detection. You may judge that a factor exists. For example, FIG. 9 shows an example in which nine partial areas from first to ninth (areas (1) to (9) in the figure) are set within the detection range. Note that the method of dividing the partial regions in FIG. 9 is just an example, and the number, size, shape, etc. of the partial regions may be determined arbitrarily.

In the example of the recording surface M1, in the fourth area, one missing feature point D11-1 is recorded at the processing timing of the N frame, and one missing feature point D12-1 is recorded at the processing timing of the N+1 frame, Another missing feature point D13 is recorded at the processing timing of the N+2 frame. In this case, if the threshold is 2, the hindrance factor determination unit 624, with regard to object detection in the fourth region, determines that the hindrance factor determination unit 624, at the processing timings of the N+1 frame and the N+2 frame where the number of missing feature points in the region is 2 or more, Determine that the cause exists. Also, if the threshold is 3, the obstruction factor determination unit 624 determines that an obstruction factor exists at the processing timing of the N+2 frame when the number of missing feature points in the region is 3 or more.

On the other hand, in the seventh area of the recording surface M1, one missing feature point D11-2 is recorded at the processing timing of the N frame, and two missing feature points D12-2 and D12-3 are recorded at the processing timing of the N+1 frame. are recorded, and no new missing feature points are recorded at the processing timing of the N+2 frame. In this case, the number of missing feature points in the region becomes 2 or 3 or more at the processing timing of the N+1 frame. Therefore, in the example of FIG. 9, regardless of whether the threshold is 2 or 3, the impediment factor determination unit 624 determines whether the impediment factor exists at the processing timing of the N+1 frame with respect to object detection in the seventh region. It will be determined that

The object recognition device 16 of the embodiment configured as described above includes a first feature point extraction unit 621 that extracts the feature points of the detected object as the first feature points based on the detection result of the camera 10, the radar device 12 A second feature point extraction unit 622 that extracts the detected feature points of the object as second feature points based on the detection result of , and a point group of the first feature points of the camera 10 and the radar device 12, Of the second feature point group, which is the point group of the first feature point group and the second feature point group, the feature point group for which the corresponding feature point is missing (that is, the feature point group for which the missing feature point is present) ) is provided with an obstruction factor determination unit 624 that determines that an obstruction factor exists in detecting the object. With such a configuration, the object recognition device 16 of the embodiment compares object detection results obtained by a plurality of object detection devices, determines the presence or absence of a hindrance factor from the difference in the detection results, and determines whether there is a hindrance factor. can be located. Therefore, according to the object recognition device 16 of the embodiment, it is possible to more reliably suppress a decrease in accuracy in recognizing objects around the own vehicle.

In the above-described embodiment, water droplets adhering to the front window are assumed as an example of an obstructive factor, and the case where the heater device 300 is operated is described as an example of means for removing the water droplets. It is not limited to heater device 300 . For example, if dirt adhering to the outside of the front window is assumed to be a hindrance, a wiper device (not shown) may be operated as an example of means for removing the dirt.

Further, in the above embodiment, for the sake of simplicity, the case where missing feature points are recorded on a two-dimensional recording surface has been described. (referred to as “recording space”). In this case, the impediment factor determination unit 624 may be configured to divide the three-dimensional recording space into subspaces and determine the presence or absence of impediment factors based on the state of occurrence of missing feature points in each subspace.

The embodiment described above can be expressed as follows.
a storage device storing a program;
a hardware processor;
By the hardware processor executing the program,
Extracting feature points of the object detected by the first object detection device as first feature points based on the detection result of the first object detection device for detecting an object in front of the vehicle,
Based on the detection result of the second object detection device for detecting the object, the detection range of the second object detection device including part or all of the detection range of the first object detection device, extracting a feature point as a second feature point,
Extraction of a feature point group in which corresponding feature points are missing from a first feature point group that is a point group of the first feature points and a second feature point group that is a point group of the second feature points Determining that an obstacle to the detection of the object exists for the object detection device that has acquired the detection result.
A vehicle control device configured to:

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

REFERENCE SIGNS LIST 1 vehicle system 10 camera 12 radar device 14 LIDAR 20 communication device 30 HMI 40 vehicle sensor 50 navigation device 51 GNSS receiver 52 navigation HMI 53 Route determination unit 54 First map information 60 MPU 61 Recommended lane determination unit 62 Second map information 70 Driver monitor camera 80 Driving operator 82 Steering wheel 84 Steering grip Sensor 100 Automatic operation control device 120 First control unit 130 Recognition unit 140 Action plan generation unit 150 Mode determination unit 152 Driver state determination unit 154 Mode change processing unit 160 Second control unit 162 Acquisition unit 164 Speed control unit 166 Steering control unit 200 Driving force output device 210 Brake device 220 Steering device 300 Heater device 16 Object recognition Apparatus 610 Storage unit 620 Control unit 621 First feature point extraction unit 622 Second feature point extraction unit 623 Feature point superimposition unit 624 Inhibition factor determination unit 625 Heater control unit

Claims (9)

a first feature point extraction unit that extracts, as first feature points, feature points of an object detected by the first object detection device based on the detection result of the first object detection device that detects an object in front of the vehicle;
characteristics of the object based on the detection result of a second object detection device for detecting the object, the detection range including part or all of the detection range of the first object detection device; a second feature point extraction unit that extracts a point as a second feature point;
Among the first object detection device and the second object detection device, among the first feature point group that is the point group of the first feature points and the second feature point group that is the point group of the second feature points a determination unit that determines that an obstacle to detection of the object exists in the object detection device that has acquired the detection result related to the extraction of the feature point group from which the corresponding feature point is missing;
A detection device comprising: The determination unit determines that the obstruction factor exists when a first feature point or a second feature point having no corresponding feature point is continuously detected for a specified time or longer.
A sensing device according to claim 1 . The determination unit detects missing feature points, which are feature points that should be originally detected, in time series for first feature points or second feature points that do not have corresponding feature points, and detects the missing feature points detected. The number is totaled for each partial area obtained by dividing the detection range, and when the total number of missing feature points is equal to or greater than a threshold value, it is determined that the obstruction factor exists in the partial area from which the total number is obtained. do,
3. A sensing device according to claim 1 or 2. The first object detection device is a camera that detects an object in front of the vehicle from inside the vehicle through a front window,
A factor that hinders object detection by the first object detection device is water droplets adhering to the front window.
4. A sensing device according to any one of claims 1 to 3. The second object detection device is a radar device behind a radio wave transmission part provided in a part of the bumper,
A factor that hinders object detection by the second object detection device is an adhering matter on the radio wave transmitting portion.
5. A sensing device according to claim 4. The second object detection device is a lidar device provided on a bumper,
A factor that hinders object detection by the second object detection device is an adhering matter on the light emitting unit or the light receiving unit of the lidar device.
5. A sensing device according to claim 4. further comprising a heater control unit that operates a heater device that blows hot air onto the front window according to the determination result of the determination unit;
The heater control unit activates the heater device when the determination unit determines that the obstruction factor exists with respect to object detection by the first object detection device.
A sensing device according to any one of claims 4 to 6. the computer
Extracting feature points of the object detected by the first object detection device as first feature points based on the detection result of the first object detection device for detecting an object in front of the vehicle,
Based on the detection result of the second object detection device for detecting the object, the detection range of the second object detection device including part or all of the detection range of the first object detection device, extracting a feature point as a second feature point,
Among the first object detection device and the second object detection device, a first feature point group that is a point group of the first feature points and a second feature point group that is a point group of the second feature points Determining that there is a hindrance to the detection of the object for the object detection device that has acquired the detection result related to the extraction of the feature point group whose corresponding feature point is missing,
Detection method. to the computer,
extracting feature points of the object detected by the first object detection device as first feature points based on the detection result of the first object detection device for detecting an object in front of the vehicle;
Based on the detection result of the second object detection device for detecting the object, the detection range of the second object detection device including part or all of the detection range of the first object detection device, extracting the feature point as a second feature point,
Among the first object detection device and the second object detection device, a first feature point group that is a point group of the first feature points and a second feature point group that is a point group of the second feature points Determining that there is an obstacle to detection of the object with respect to the object detection device that has acquired the detection result related to the extraction of the feature point group of which the corresponding feature point is missing,
program.

Images