JP7028838B2 - Peripheral recognition device, peripheral recognition method, and program - Google Patents

Description

The present invention relates to a peripheral recognition device, a peripheral recognition method, and a program.

Conventionally, a radar means for detecting the position information of an object, an image pickup means for detecting the position size information of the object, and an object detected by the radar means when the detection result by the radar means and the image pickup means satisfy a predetermined fusion condition. It is provided with a fusion processing means for generating a fusion target having position information and size information by fusing the objects detected by the imaging means as the same object, and the radar means at the time of determining the update of the fusion condition for the generated fusion target. If the detection result by the imaging means does not satisfy the fusion condition with respect to the detection result by, a temporary fusion object whose size information is interpolated by estimation based on the size information of the fusion object acquired in the past is generated. The invention of the object detection device for updating the fusion target is disclosed (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2007-226680

The above-mentioned conventional technology performs sensor fusion between a camera and a radar when detecting a three-dimensional object such as another vehicle, and appropriately determines whether or not the recognition result of the road lane marking by the camera is adopted by the detection result of the other sensor. It's not something to do.

The present invention has been made in consideration of such circumstances, and is capable of appropriately determining whether or not to accept or reject the recognition result of the road lane marking by the camera based on the detection result of another sensor. One of the purposes is to provide a recognition method and a program.

The peripheral recognition device, the peripheral recognition method, and the program according to the present invention have the following configurations.
(1): The peripheral recognition device according to one aspect of the present invention is an image that recognizes the position of a road lane marking with respect to the vehicle based on an image acquired by a camera mounted on the vehicle and capturing the periphery of the vehicle. A group of target positions lined up linearly extends from a plurality of target positions mounted on the vehicle and the detection range around the vehicle and acquired by an object detection device different from the camera. A specific unit that specifies the first extension direction, which is a direction, and a determination unit that determines a reference track that is used as a reference for traveling control of the vehicle, and is recognized by the first extension direction and the image analysis unit. It is provided with a determination unit for determining the reference track based on the road lane marking recognized by the image analysis unit when the second extension direction, which is the extension direction of the road lane marking, matches. be.

(2): In the aspect of (1) above, by applying the position of the vehicle to the map information, the map-based recognition that recognizes the third extension direction, which is the extension direction of the road on which the vehicle exists with respect to the vehicle. The determination unit is further provided, and the determination unit is described when the first extension direction and the second extension direction match, and the second extension direction and the third extension direction do not match. The reference track is determined based on the road marking line recognized by the image analysis unit.

(3): In the aspect of (2) above, in the determination unit, the first extending direction and the second extending direction do not match, and the second extending direction and the third extending direction are present. When the direction does not match, the reference track is determined based on the road lane marking recognized by the map-based recognition unit.

(4): In any of the above aspects (1) to (3), it is presumed that the specific portion extends from the plurality of target positions to a predetermined length or more and is stationary. The target position corresponding to the target is specified as a group of target positions arranged in a line.

(5): In any of the above aspects (1) to (4), the specific unit performs a thinning process on the target positions lined up in a line among the plurality of target positions by a predetermined method. By going and reducing the number, the position of the target group arranged in the line is specified.

(6): In any of the above embodiments (1) to (5), the specific portion performs a first-order approximation to a group of target positions arranged in a line, thereby performing the first extension direction. Is to specify.

(7): In any of the above embodiments (1) to (6), the determination unit is of the vehicle among a group of target positions lined up in a line, which are recognized on the left and right sides of the vehicle. When the first extending direction specified based on the target position of the linearly arranged group on the side close to the traveling direction matches the second extending direction, the image analysis unit recognizes the first extending direction. The reference track is determined based on the road marking line.

(8): In any of the above aspects (1) to (7), the determination unit has a large number among the group of target positions lined up in a line, which are recognized on the left and right sides of the vehicle. The road marking line recognized by the image analysis unit when the first extending direction specified based on the target position of the group of targets arranged in the line and the second extending direction match. The reference track is determined based on the above.

(9): In the peripheral recognition method according to another aspect of the present invention, the position of the road lane marking with respect to the vehicle is based on an image acquired by a camera mounted on the vehicle and capturing the periphery of the vehicle. Is mounted on the vehicle, the area around the vehicle is set as the detection range, and a group of target positions arranged in a line are extended from a plurality of target positions acquired by an object detection device different from the camera. When the first extension direction, which is the direction, is specified and the first extension direction and the second extension direction, which is the extension direction of the road lane marking recognized by the image analysis unit, match, the said Based on the road lane marking recognized based on the image, the reference track as the reference for the traveling control of the vehicle is determined.

(10): The program according to another aspect of the present invention recognizes the position of the road lane marking with respect to the vehicle based on an image mounted on the vehicle and acquired by a camera that images the periphery of the vehicle on a computer. The detection range is the periphery of the vehicle, which is mounted on the vehicle, and is in the extending direction of a group of target positions arranged in a line from a plurality of target positions acquired by an object detection device different from the camera. When a certain first extension direction is specified and the first extension direction and the second extension direction, which is the extension direction of the road lane marking recognized by the image analysis unit, match, the image is displayed. Based on the road lane markings recognized based on the above, the reference track to be used as the reference for the traveling control of the vehicle is determined.

According to the above aspects (1) to (10), it is possible to appropriately determine whether or not to accept or reject the recognition result of the road lane marking by the camera based on the detection result of another sensor.

According to the above aspect (2) or (3), in addition to the comparison between the camera image and the target position information, the road section by the camera is used to determine the reference track based on the comparison between the map information and the camera image. It is possible to more appropriately determine whether or not to accept the line recognition result.

According to the aspect (4) above, it can be compared with a camera image limited to a stationary object along a road.

According to the aspect (5) above, the load of processing after thinning can be reduced.

According to the aspect (6) above, the first extension direction can be calculated with a low load in the whole.

According to the aspect (7) above, in order to compare the first extension direction specified based on the target position along the traveling direction of the vehicle with the second extension direction, the recognition result of the road lane marking by the camera. Can be decided more appropriately.

According to the aspect (8) above, the first extension specified based on the target position having a large number among the target positions unevenly distributed on either the left or right side of the vehicle (that is, the target position on the side closer to the vehicle). Since the current direction is compared with the second extension direction, it is possible to more appropriately determine whether or not to accept or reject the recognition result of the road lane marking by the camera.

It is a block diagram of the vehicle system using the peripheral recognition device which concerns on embodiment. It is a functional block diagram of the 1st control unit and the 2nd control unit of the automatic operation control device which concerns on 1st Embodiment. It is a figure for demonstrating the function of the image analysis part and the object target extension direction identification part. It is a figure which shows an example of the target position obtained by irradiating light with a certain layer, and the flag given to them. It is a flowchart which shows an example of the flow of the process executed by the reference track determination part. It is a figure for demonstrating the scene which determines the reference track based on a camera image. It is a figure for demonstrating the scene which determines the reference track based on a camera image. It is a figure for demonstrating the scene which determines the reference track based on a camera image. It is a figure for demonstrating the scene which determines a reference track based on map information. It is a flowchart which shows an example of the processing flow by an image analysis part, a target extension direction identification part, and a reference track determination part. It is a figure for demonstrating the process of narrowing down a target position. It is a figure for demonstrating the process of generating a vector. It is a figure which shows an example of the hardware composition of the automatic operation control device of an embodiment.

Hereinafter, embodiments of the peripheral recognition device, the peripheral recognition method, and the program of the present invention will be described with reference to the drawings.

[overall structure]
FIG. 1 is a configuration diagram of a vehicle system 1 using a peripheral recognition device according to an embodiment. The vehicle on which the vehicle system 1 is mounted is, for example, a vehicle such as a two-wheeled vehicle, a three-wheeled vehicle, or a four-wheeled vehicle, and the drive source thereof is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates by using the electric power generated by the generator connected to the internal combustion engine or the electric power generated by the secondary battery or the fuel cell. The peripheral recognition device may be a part of a vehicle control device that controls the running of the vehicle, or may be a device separate from the vehicle control device. In the following description, it is assumed that the peripheral recognition device is a part of the vehicle control device.

The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, and an MPU. (Map Positioning Unit) 60, a driving controller 80, an automated driving control device (Automated Driving Control Device) 100, a traveling driving force output device 200, a braking device 210, and a steering device 220. These devices and devices are connected to each other by a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. The configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted or another configuration may be added.

The camera 10 is, for example, a digital camera using a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary position of the vehicle on which the vehicle system 1 is mounted (hereinafter referred to as the own vehicle M). When photographing the front, the camera 10 is attached to the upper part of the front windshield, the back surface of the rear-view mirror, and the like. The camera 10 periodically and repeatedly images the periphery of the own vehicle M, for example. The camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves around the own vehicle M, and also detects radio waves (reflected waves) reflected by the object to detect at least the position (distance and direction) of the object. The radar device 12 is attached to an arbitrary position on the own vehicle M. The radar device 12 may detect the position and velocity of the object by the FM-CW (Frequency Modulated Continuous Wave) method.

The LIDAR (Light Detection and Ranging) 14 irradiates the periphery of the own vehicle M with light and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time from light emission to light reception. The emitted light is, for example, a pulsed laser beam. The LIDAR 14 is attached to an arbitrary position on the own vehicle M.

The LIDAR 14 can change the irradiation direction of light for both the elevation angle or the depression angle (hereinafter, the irradiation direction φ in the vertical direction) and the azimuth angle (the irradiation direction θ in the horizontal direction). For example, the LIDAR 14 scans while fixing the irradiation direction φ and changing the irradiation direction θ, then changing the irradiation direction φ in the vertical direction, fixing the irradiation direction φ at the changed angle, and changing the irradiation direction θ. The operation of scanning is repeated. Hereinafter, the irradiation direction φ is referred to as a “layer”. In LIDAR 14, for example, one scan (cycle) performed while fixing the layers and changing the irradiation direction θ is performed for all layers as one scan. The layers are set, for example, from L1 to Ln with a finite number (n is a natural number). The layer change is performed discontinuously with respect to the angle, for example, L0 → L4 → L2 → L5 → L1 ... So that the light emitted in the previous cycle does not interfere with the detection in this cycle. Not limited to this, the layer may be changed continuously with respect to the angle. The LIDAR 14 outputs, for example, a data set (rider data) having {φ, θ, d, p} as one unit to the object recognition device 16. d is the distance and p is the intensity of the reflected light.

The object recognition device 16 performs sensor fusion processing on the detection results of a part or all of the camera 10, the radar device 12, and the LIDAR 14, and recognizes the position, type, speed, and the like of the object. The object recognition device 16 outputs the recognition result to the automatic operation control device 100. Further, the object recognition device 16 outputs the detection results of the camera 10, the radar device 12, and the LIDAR 14 to the automatic operation control device 100 as they are or by performing a predetermined processing process. For example, the object recognition device 16 performs a process of adding a flag to the target position data output by the LIDAR 14. This will be described later.

The communication device 20 communicates with another vehicle existing in the vicinity of the own vehicle M by using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or wirelessly. Communicates with various server devices via the base station.

The HMI 30 presents various information to the occupants of the own vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys and the like.

The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the own vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, an orientation sensor that detects the direction of the own vehicle M, and the like.

The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a routing unit 53. The navigation device 50 holds the first map information 54 in a storage device such as an HDD (Hard Disk Drive) or a flash memory. The GNSS receiver 51 identifies the position of the own vehicle M based on the signal received from the GNSS satellite. The position of the own vehicle M may be specified or complemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, and the like. The navigation HMI 52 may be partially or wholly shared with the above-mentioned HMI 30. The route determination unit 53, for example, has a route from the position of the own vehicle M (or an arbitrary position input) specified by the GNSS receiver 51 to the destination input by the occupant using the navigation HMI 52 (hereinafter,). The route on the map) is determined with reference to the first map information 54. The first map information 54 is, for example, information in which a road shape is expressed by a link indicating a road and a node connected by the link. The route on the map is output to MPU60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized by, for example, the function of a terminal device such as a smartphone or a tablet terminal owned by an occupant. The navigation device 50 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire a route equivalent to the route on the map from the navigation server.

The MPU 60 includes, for example, a recommended lane determination unit 61, and holds the second map information 62 in a storage device such as an HDD or a flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (for example, divides the route into 100 [m] units with respect to the vehicle traveling direction), and refers to the second map information 62. Determine the recommended lane for each block. The recommended lane determination unit 61 determines which lane to drive from the left. When a branch point exists on the route on the map, the recommended lane determination unit 61 determines the recommended lane so that the own vehicle M can travel on a reasonable route to proceed to the branch destination.

The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of the lane, information on the boundary of the lane, and the like. Further, the second map information 62 may include road information, traffic regulation information, address information (address / zip code), facility information, telephone number information, and the like. The second map information 62 may be updated at any time by the communication device 20 communicating with another device.

The driving controller 80 includes, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a deformed steering wheel, a joystick, and other controls. A sensor for detecting the amount of operation or the presence or absence of operation is attached to the operation controller 80, and the detection result is the automatic operation control device 100, or the traveling driving force output device 200, the brake device 210, and the steering device. It is output to a part or all of 220.

The automatic operation control device 100 includes, for example, a first control unit 120 and a second control unit 190. The first control unit 120 and the second control unit 190 are each realized by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). In addition, some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). It may be realized by the part; including circuitry), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD or a flash memory of the automatic operation control device 100, or may be detachable such as a DVD or a CD-ROM. It is stored in the storage medium, and the storage medium (non-transient storage medium) may be installed in the HDD or the flash memory of the automatic operation control device 100 by being attached to the device in the drive device.

FIG. 2 is a functional configuration diagram of the first control unit 120 and the second control unit 190 of the automatic operation control device 100 according to the first embodiment. The first control unit 120 includes, for example, a recognition unit 130 and an action plan generation unit 180. The first control unit 120, for example, realizes a function by AI (Artificial Intelligence) and a function by a model given in advance in parallel. For example, the function of "recognizing an intersection" is executed in parallel with the recognition of an intersection by deep learning and the like, and the recognition based on a predetermined condition (there is a signal that can be pattern matched, a road sign, etc.). It may be realized by scoring and comprehensively evaluating. This ensures the reliability of autonomous driving.

Each unit of the recognition unit 130 and the action plan generation unit 180 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit parts) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit). It may be realized by (including circuits), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or a removable storage such as a DVD or a CD-ROM. It is stored in a medium (non-transient storage medium), and may be installed in the storage device by mounting the storage medium in the drive device.

The recognition unit 130 includes, for example, an object recognition unit 140, an image analysis unit 150, a target extension direction specifying unit 152, a map-based recognition unit 154, a reference track determination unit 156, and a vehicle position recognition unit 160. To prepare for. Among these, those including at least the image analysis unit 150, the target extension direction specifying unit 152, the map-based recognition unit 154, the reference track determination unit 156, and the own vehicle position recognition unit 160 are "peripheral recognition". This is an example of "device".

The object recognition unit 140 determines the position, speed, acceleration, and the like of the object around the own vehicle M based on the information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16. recognize. Objects are other vehicles, pedestrians, bicycles, and other obstacles. The position of the object is recognized as, for example, a position on absolute coordinates with the representative point of the own vehicle M (center of gravity, center of drive axis, etc.) as the origin and the central axis of the own vehicle M as the X axis, and is used for control. .. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by a represented area. The "state" of an object may include the object's acceleration, jerk, or "behavioral state" (eg, whether it is changing lanes or is about to change lanes).

[Recognition of reference track]
The functions of the image analysis unit 150 and the object target extension direction specifying unit 152 will be described with reference to FIG. FIG. 3 is a diagram for explaining the functions of the image analysis unit 150 and the target extension direction specifying unit 152. In the figure, RM is a representative point of the own vehicle _M , and the XY axis represents the coordinate axis of the above-mentioned absolute coordinates.

The image analysis unit 150 recognizes the position of the road lane marking Rl with respect to the own vehicle M based on the image acquired by the camera 10. The road lane marking Rl is a white line or a yellow line drawn on the road in the shape of a solid line or a broken line, and divides the lane. For example, the image analysis unit 150 converts an image plane into a real plane represented by an absolute coordinate system, and a pixel having a large brightness difference (or color difference) with an adjacent pixel (or an adjacent pixel group) on the absolute coordinate system. The edge point Ep which is (or a pixel group) is extracted, narrowed down to the edge points Ep which are linearly arranged by a method such as linear approximation, and the predetermined width Wd which is the width of a general road lane marking extends over a predetermined length. A row of a set of edge points _Ep extending along the line LEp is recognized as the outline of the road marking line. Then, the image analysis unit 150 recognizes the region partitioned by the contour as the road marking line Rl. Instead, the image analysis unit 150 may directly recognize the road lane marking Rl on the image plane, or considers the row LEp of the edge point _Ep alone as one contour of the road lane marking Rl. It may be (not necessarily narrowed down to the row LEp of a set of edge points _Ep having a predetermined width Wd). The image analysis unit 150 specifies the extension direction _DRl (second extension direction) of the recognized road lane marking Rl for each road lane marking Rl. In the figure, the extension directions _DRl (1), _DRl (2), and DRl (3) corresponding to each of the road marking lines Rl (1), Rl (2), and _Rl (3) are recognized. It is shown that.

The target extension direction specifying unit 152 sets a plurality of target positions (for example, represented by irradiation angles and distances) acquired by LIDAR 14, which is an example of an object detection device different from the camera 10, on an absolute coordinate system. Based on the information converted to the position of (for example, called point cloud data), the following processing is performed. The target extension direction specifying unit 152 specifies the extension direction _DTg (first extension direction) of a group of target positions _GTg arranged in a line for each of the left and right structures St of the own vehicle M. .. The group of target position _GTg to be specified by the target extension direction specifying unit 152 is the position of each part of the stationary structure St existing parallel to the road. The structure St is, for example, a curb, a guardrail, a side wall, or the like. The figure shows that the extending directions _DTg (L) and _DTg (R) corresponding to the structures St (L) and St (R) are recognized.

At the target position acquired by the target extension direction specifying unit 152, for example, a flag (FS) representing a road surface, a flag (DY) representing a moving object, and a flag representing a stationary object (a flag representing a stationary object) are used by the target recognition device 16. SA), a flag (UK) indicating that it is unknown, and the like are added. The target recognition device 16 performs plane extraction processing on point cloud data by a robust estimation method such as RANSAC (Random Sample Consensus), and the target position represents a road surface (no object exists on the road surface). When it is determined whether or not to represent the road surface, a flag (FS) is added to the target position. Further, the target recognition device 16 tracks the target position in time series using a technique such as a Kalman filter, and identifies the target position representing the same object in time series, thereby flagging the target position ( DY), flag (SA), or flag (UK) is given. FIG. 4 is a diagram showing an example of a target position obtained when light is irradiated on a certain layer and a flag assigned to them. As shown, a flag (SA) is used for the target position that is a part of the structures St (L) and St (R), and a flag (SA) is used for the target position that is a part of the road surface. A flag (DY) is given to the target position where the FS) is a part of a moving body such as another vehicle m1 or m2. For example, the target extension direction specifying unit 152 narrows down the target position acquired from the target recognition device 16 to the target position to which the flag (SA) or the flag (UK) is attached, and then, as described above. A process for specifying the extension direction _DTg is performed. The detailed method of obtaining the extension direction _DTg will be described later. A part or all of the functions of the object target extending direction specifying unit 152 described above may be mounted on the object recognition device 16, or a part or all of the functions of the object recognition device 16 may be mounted on the object recognition device 16 in the object extension direction. The specific unit 152 may be provided.

The map-based recognition unit 154 recognizes the position of the road lane marking with respect to the own vehicle M by applying the position of the own vehicle M to the map information (for example, the second map information 62). The map-based recognition unit 154 collates the position of the own vehicle M specified by the navigation device 50, the image captured by the camera 10, the output of the orientation sensor included in the vehicle sensor 40, and the like with the map information, and the own vehicle M collates with the map information. Recognize which road, which lane, and which direction you are driving on the map. For example, the map-based recognition unit 154 indicates the direction of the own vehicle M in the ground coordinate system (coordinate system centered on latitude and longitude) and the extension of the road (lane) in which the own vehicle M is recognized in the map information. Based on the angle difference from the current direction, the extension direction D _Mp (third extension direction) of the road (lane) in the map information is recognized.

The reference track determination unit 156 determines a reference track to be used as a reference for the travel control of the own vehicle M by the action plan generation unit 180. FIG. 5 is a flowchart showing an example of the flow of processing executed by the reference track determination unit 156. In parallel with the processing of this flowchart, the extension direction _DTg (first extension direction), the extension direction _DRl (second extension direction), and the extension direction _DMp (third extension direction). The process of obtaining the above is repeatedly executed, and in the process of this flowchart, it is assumed that the process is performed using the information obtained most recently.

First, the reference track determination unit 156 sets both the first flag and the second flag to False (step S100). The first flag is a flag in which True is set when the extension direction _DTg (first extension direction) and _DRl (second extension direction) do not match, and False is set when they match. .. The second flag is a flag in which True is set when the extension direction _DRl (second extension direction) and D _Mp (third extension direction) do not match, and False is set when they match. .. These are set to True or False in the subsequent processing, but False is set as the default value. Matching means, for example, that the angle difference is within a threshold value (for example, a value of about several degrees).

Next, the reference track determination unit 156 determines whether or not the extension direction _DTg (first extension direction) and the extension direction _DRl (second extension direction) do not match (step S102).

When it is determined that the extension direction _DTg (first extension direction) and the extension direction _DRl (second extension direction) do not match, the reference track determination unit 156 changes the first flag to True (step). S104). When the extending direction _DTg (first extending direction) and the extending direction _DRl (second extending direction) match, or when the extending direction _DTg (first extending direction) is not determined. , The reference track determination unit 156 keeps the first flag as False. The case where the extension direction _DTg (first extension direction) is not determined is, for example, the case where a group of target positions _GTg satisfying the conditions cannot be specified.

In the process of step S104, the reference track determination unit 156 has two extension directions _DTg (first extension direction) that are assumed to exist on the left and right of the own vehicle M, and one or more extension directions _DRl . Comparing each with (second extending direction), if they partially match, the extending direction _DTg (first extending direction) and the extending direction _DRl (second extending direction) match. Then it is determined. However, if only the extension direction _DTg (first extension direction) on the left side of the own vehicle M and the extension direction _DRl (second extension direction) on the right side of the own vehicle M match, the reference runway The determination unit 156 may determine that these do not match. In the example of FIG. 3, when any one of the following conditions is satisfied, the reference track determination unit 156 has an extension direction _DTg (first extension direction) and an extension direction _DRl (second extension direction). ) Matches.
(Condition 1) The extending direction _DTg (L) and the extending direction _DRl (1) match.
(Condition 2) The extending direction _DTg (R) matches the extending direction _DRl (2) or _DRl (3).

In the above, when the reference track determination unit 156 "partially matches (either left or right)", the extension direction _DTg (first extension direction) and the extension direction _DRl (second). It was determined that the extension direction) was "matched", but instead of this, if the reference track determination unit 156 does not match both the left and right sides, the extension direction _DTg (first extension direction) And the extension direction _DRl (second extension direction) may be determined to be "mismatch".

Next, the reference track determination unit 156 determines whether or not the extension direction _DRl (second extension direction) and the extension direction D _Mp (third extension direction) do not match (step S106). In this step, when there are a plurality of extending directions _DRl (second extending direction), some extending directions _DRl (second extending direction) are extending directions D _Mp (third extending direction). If it matches the direction), it may be determined as "matching", or if all the extending directions _DRl (second extending direction) match the extending direction D _Mp (third extending direction). It may be determined that "matches" only with. When it is determined that the extension direction _DRl (second extension direction) and the extension direction D _Mp (third extension direction) do not match, the reference track determination unit 156 changes the second flag to True (step). S108). When the extension direction _DRl (second extension direction) and the extension direction D _Mp (third extension direction) match, or when the extension direction D _Mp (third extension direction) is not obtained. , The reference track determination unit 156 keeps the second flag as False. The case where the extension direction D _Mp (third extension direction) is not obtained is, for example, the case where there is no map information corresponding to the position where the own vehicle M exists.

Next, the reference track determination unit 156 determines whether or not both the first flag and the second flag are False (step S110). When both the first flag and the second flag are False, that is, when the first extending direction and the second extending direction match, and when the second extending direction and the third extending direction match, the reference track is determined. The unit 156 determines the reference track based on the map information (step S112).

When a negative determination result is obtained in step S110, the reference track determination unit 156 determines whether or not the first flag is False and the second flag is True (step S114). When the first flag is False and the second flag is True, that is, the first extending direction and the second extending direction match, and the second extending direction and the third extending direction do not match. In this case, the reference track determination unit 156 determines the reference track based on the image (camera image) of the camera 10 (step S116).

When a negative determination result is obtained in step S114, the reference track determination unit 156 determines whether or not the first flag is True and the second flag is False (step S118). When the first flag is True and the second flag is False, that is, the first extension direction and the second extension direction do not match, and the second extension direction and the third extension direction match. If so, the reference track determination unit 156 determines the reference track based on the map information (step S112).

When a negative determination result is obtained in step S118, the reference track determination unit 156 determines whether or not the first flag is True and the second flag is True (step S120). When the first flag is True and the second flag is True, that is, the first extension direction and the second extension direction do not match, and the second extension direction and the third extension direction also match. If not, the reference track determination unit 156 determines the reference track based on the map information (step S112). If a negative determination result is obtained in step S120, the reference track determination unit 156 outputs error information to the action plan generation unit 180 (step S122). In this case, the action plan generation unit 180 outputs information requesting the driver to drive manually, or performs a process of decelerating and stopping the own vehicle M.

6 to 8 are diagrams for explaining a scene in which a reference track is determined based on a camera image. In the example shown in FIG. 6, the extending direction _DTg (L) and the extending direction _DRl (1) match, and the extending direction _DTg (R) and the extending direction _DRl (2) and _DRl (3). ) Matches. Since the road lane marking Rl (2) is closer to the own vehicle M than the road lane marking Rl (3), in this case, the reference track determination unit 156 is the position of the road lane marking Rl (1) and Rl (2). The reference track RR is determined based on. For example, the reference track determination unit 156 uses a region defined by the right end of the road marking line Rl (1) and the left end of the road marking line Rl (2) as the reference track RR.

In the example shown in FIG. 7, the extending direction _DTg (L) and the extending direction _DRl (1) match, but the road marking line Rl (2) is faint, so the extending direction _DRl (2). ) Has not been identified, and only the extending direction _DTg (R) and the extending direction _DRl (3) match. However, it is recognized that there are two lanes from the distance between the road marking lines Rl (1) and Rl (3). In this case, the reference track determination unit 156 determines the reference track RR based on the positions of the road marking lines Rl (1) and Rl (3). For example, the reference track determination unit 156 divides the area partitioned by the right end of the road marking line Rl (1) and the left end of the road marking line Rl (3) into two in the road width direction and divides the reference track. Set the RR.

In the example shown in FIG. 8, the extending direction _DTg (L) and the extending direction _DRl (1) match, but due to the influence of the shadow cast on the road, the extending direction _DRl (2). And the extension direction _DRl (3) is erroneously recognized in a direction different from the extension direction of the original road lane marking Rl. As a result, both the extending direction _DRl (2) and the extending direction _DRl (3) do not match the extending direction _DTg (R). In this case, the reference track determination unit 156 determines the reference track RR based on the position of the road lane marking Rl (1). For example, the reference track determination unit 156 uses the right end of the road lane marking Rl (1) as the left end and the region having a general lane width as the reference track RR.

FIG. 9 is a diagram for explaining a scene in which a reference track is determined based on map information. In the example shown in FIG. 9, the extension direction _DRl (1), the extension direction _DRl (2), and the extension direction _DRl (3) are the original road sections due to the influence of the shadow cast on the road. It is erroneously recognized in a direction different from the extending direction of the line Rl. As a result, the extending direction _DRl (1) and the extending direction _DTg (R) do not match, and both the extending direction _DRl (2) and the extending direction _DRl (3) are the extending direction _DTg . It does not match (R). In this case, the reference track determination unit 156 acquires information from the map information via the map-based recognition unit 154, for example, "the traveling lane is a straight line and matches the traveling direction of the own vehicle M". The reference lane RR is determined based on the position and the traveling direction of the own vehicle M.

The own vehicle position recognition unit 160 recognizes the position and attitude of the own vehicle M with respect to the reference track RR. The own vehicle position recognition unit 160 determines, for example, the deviation of the representative point of the own vehicle M from the center of the reference runway RR and the angle formed by the traveling direction of the own vehicle M with respect to the center line of the reference runway RR. It is recognized as the position and attitude of the own vehicle M with respect to the RR. Instead of this, the own vehicle position recognition unit 160 may recognize the position of the representative point of the own vehicle M with respect to any side end of the reference track RR as the position of the own vehicle M with respect to the traveling lane.

[Action plan generation]
In principle, the action plan generation unit 180 travels on the reference track RR determined by the reference track determination unit 156, and the own vehicle M automatically responds to the surrounding conditions of the own vehicle M. Generates a target track to be driven in the future (regardless of the driver's operation). The target trajectory contains, for example, a velocity element. For example, the target track is expressed as an arrangement of points (track points) to be reached by the own vehicle M in order. The track point is a point to be reached by the own vehicle M for each predetermined mileage (for example, about several [m]) along the road, and separately, for a predetermined sampling time (for example, about 0 comma number [sec]). ) Target velocity and target acceleration are generated as part of the target trajectory. Further, the track point may be a position to be reached by the own vehicle M at the sampling time for each predetermined sampling time. In this case, the information of the target velocity and the target acceleration is expressed by the interval of the orbital points. When the own vehicle M is made to travel on the reference track RR, the action plan generation unit 180 generates a target track on the center line in the width direction of the reference track RR, and outputs information on the target track to the second control unit 190. Further, the action plan generation unit 180 first determines whether or not it can be avoided in the reference lane RR when an obstacle exists in the traveling direction of the own vehicle M, and if it can be avoided, it is in the reference lane RR. Generate a target track for avoidance and switch to lane change control when avoidance is not possible. These are examples of "using the vehicle as a standard for driving control".

The action plan generation unit 180 may set an event for automatic driving when generating a target trajectory. Autonomous driving events include constant speed driving events, low speed following driving events, lane change events, branching events, merging events, takeover events, and the like. The action plan generation unit 180 generates a target trajectory according to the activated event.

The action plan generation unit 180 defines the automatic driving level while the automatic driving is being executed. The automatic driving level represents the degree of duty of care imposed on the driver, and is defined step by step depending on, for example, whether or not there is a duty of forward monitoring and whether or not there is a duty of gripping the steering wheel. The state where there is an obligation to monitor forward is called eyes-on, and the state where there is no obligation to monitor forward is called eyes-off. Further, a state in which the steering wheel is obliged to be gripped is referred to as hands-on, and a state in which the steering wheel is not obliged to be gripped is referred to as hands-off.

In Eyes-on, the action plan generation unit 180 is captured by a camera (not shown) that captures the interior of the vehicle M, and the driver analyzes an image including an occupant (driver) seated in the driver's seat. It is determined whether or not the front of the own vehicle M is visually recognized. If it is determined that the driver is not visually recognizing the front of the own vehicle M, the action plan generation unit 180 issues a warning to the driver using the HMI 30, and if the driver is still not visually recognizing the front of the own vehicle M, the action plan generation unit 180 is used. The automatic operation is stopped, or the own vehicle M is decelerated and stopped. In the case of eyes-off, the action plan generation unit 180 does not perform such processing.

In the hands-on, the action plan generation unit 180 refers to the detection result of the grip sensor (pressure sensor, capacitance sensor, etc.) attached to the steering wheel and determines whether or not the driver is gripping the steering wheel. .. If it is determined that the driver is not gripping the steering wheel, the action plan generator 180 issues a warning to the driver using the HMI 30, and if the driver still does not grip the steering wheel, the automatic driving is stopped. , The own vehicle M is decelerated and stopped. In the case of hands-off, the action plan generation unit 180 does not perform such processing.

In general, it is considered that the duty of care imposed on the driver is smaller (the level of automatic driving is higher) in the eyes-off than in the hands-off. Therefore, there are three possible automatic driving levels, for example, <eyes-off hands-off>, <eyes-on-hands-off>, and <eyes-on-hands-on>, in order from the one with the highest automatic driving level. It should be noted that these three are just examples, and the automatic operation level can be defined in any way. For example, higher levels of autonomous driving may be defined that allow the driver to sleep.

The action plan generation unit 180 determines the automatic driving level, for example, with the processing process of the reference track determination unit 156 as one element. Specifically, the action plan generation unit 180 is premised on satisfying a scene condition such as traveling straight on a motorway, and when both the first flag and the second flag described above are False, <eyes off. Automatic operation to <eyes-on-hands-off> when one of the first flag and the second flag is False and the other is True, and <eyes-on-hands-on> when both the first flag and the second flag are True. Set the level. Further, as described above, when the left and right extending directions of the own vehicle M match both (the scene of FIG. 6 or 7), the action plan generation unit 180 matches only one of the left and right (FIG. 8). The automatic driving level may be higher than that of the scene). Thereby, the automatic driving control device 100 can appropriately determine the automatic driving level based on the recognition degree of the road condition.

The second control unit 190 sets the traveling driving force output device 200, the brake device 210, and the steering device 220 so that the own vehicle M passes the target trajectory generated by the action plan generation unit 180 at the scheduled time. Control.

The second control unit 190 includes, for example, an acquisition unit 192, a speed control unit 194, and a steering control unit 196. The acquisition unit 192 acquires information on the target trajectory (orbit point) generated by the action plan generation unit 180 and stores it in a memory (not shown). The speed control unit 194 controls the traveling driving force output device 200 or the brake device 210 based on the speed element associated with the target trajectory stored in the memory. The steering control unit 196 controls the steering device 220 according to the degree of bending of the target trajectory stored in the memory. The processing of the speed control unit 194 and the steering control unit 196 is realized by, for example, a combination of feedforward control and feedback control. As an example, the steering control unit 196 executes a combination of feedforward control according to the curvature of the road in front of the own vehicle M and feedback control based on the deviation from the target track.

The traveling driving force output device 200 outputs a traveling driving force (torque) for the vehicle to travel to the drive wheels. The traveling driving force output device 200 includes, for example, a combination of an internal combustion engine, a motor, a transmission, and the like, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to the information input from the second control unit 190 or the information input from the operation controller 80.

The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to the information input from the second control unit 190 or the information input from the operation controller 80 so that the brake torque corresponding to the braking operation is output to each wheel. The brake device 210 may include a mechanism for transmitting the hydraulic pressure generated by the operation of the brake pedal included in the operation operator 80 to the cylinder via the master cylinder as a backup. The brake device 210 is not limited to the configuration described above, and is an electronically controlled hydraulic brake device that controls the actuator according to the information input from the second control unit 190 to transmit the hydraulic pressure of the master cylinder to the cylinder. May be good.

The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, exerts a force on the rack and pinion mechanism to change the direction of the steering wheel. The steering ECU drives the electric motor according to the information input from the second control unit 190 or the information input from the operation controller 80, and changes the direction of the steering wheel.

[More detailed processing example]
Hereinafter, more detailed processing contents for determining the reference track will be described. FIG. 10 is a flowchart showing an example of the processing flow by the image analysis unit 150, the target extension direction specifying unit 152, and the reference track determination unit 156.

First, the target extension direction specifying unit 152 narrows down the target positions to which the flag (SA) or the flag (UK) is attached to the two groups of target positions (step S200). FIG. 11 is a diagram for explaining the process of narrowing down the target position. As shown in the figure, when the frequency of appearance of the target position acquired by the target extension direction specifying unit 152 is summarized in the Y-axis direction, the left-right direction of the own vehicle M (here, one is defined as positive and the other is defined as negative). In each of the above), a peak corresponding to the position of the structure St appears. The target extension direction specifying unit 152 narrows down each peak to two groups of target positions within a range of, for example, plus or minus 1σ. This allows outliers to be appropriately excluded.

Returning to FIG. 10, the target extension direction specifying unit 152 thins out the target positions so that the distance in the X direction is equal to or larger than the specified value for the target positions of each group narrowed down to the target positions of the two groups. (Step S202). Next, the target extension direction specifying unit 152 approximates a linear equation by applying a least squares method to the target positions of each group after thinning (first-order approximation; step S204). Next, the target extension direction specifying unit 152 regenerates the target positions at equal intervals starting from a point where the distance from the own vehicle M = zero on the approximated linear equation (step S206). By doing so, it is possible to reduce the processing load as compared with the case of performing processing similar to, for example, a cubic equation. Since numerical rigor is not required so much in the process of determining the match between the road lane marking Rl and the structure St, it is possible to make a global decision while reducing the processing load by performing a first-order approximation. Become.

Next, each of the image analysis unit 150 and the target extension direction specifying unit 152 has the target position regenerated in step S206 and the edge point sequence which is the outline of the road marking line recognized from the camera image. The shorter one is selected (step S208), and vectors representing the first extending direction and the second extending direction are generated according to the shorter one (step S210). FIG. 12 is a diagram for explaining a process of generating a vector. In the figure, → _DTg is a vector indicating the first extending direction (including the concept of length), and → _DRl is a vector indicating the second extending direction (including the concept of length). As shown in the figure, each vector is generated according to the regenerated target position Tg * and the regenerated target position Tg * having the shorter extending length of the edge point Ep.

Then, the reference track determination unit 156 calculates the cosine similarity between the vector indicating the first extension direction and the vector indicating the second extension direction, and determines whether the first extension direction and the second extension direction match. It is determined whether or not (step S212). The process of step S212 corresponds to the process of step S102 of FIG. As described above, the fact that the extending directions match each other means that, for example, the angle difference is within a threshold value (for example, a value of several degrees), and when the judgment is made using the cosine similarity, the cosine similarity is determined. When it is equal to or more than the reference value corresponding to the above threshold value, it may be determined that the extending directions match each other. The reason why the lengths of the two vectors are aligned in step S210 is to obtain the cosine similarity.

In the above embodiment, as the first extension direction, both the extension directions _DTg (L) and _DTg (R) are used to determine the reference track, but only one of them is used as the reference. You may decide the track. For example, the reference track determination unit 156 determines whether or not the extension direction _DTg (L) and the extension direction _DTg (R) match (the definition may be as described above), and if they match, the original. The one with the most target positions may be selected, and if they do not match, the one closer to the traveling direction of the own vehicle M may be selected. In this case, the reference track determination unit 156 determines whether or not each of the one or more extending directions _DRl , which is the second extending direction, matches the selected extending direction _DTg . The "closer to the traveling direction" means, for example, the one that intersects first when the virtual line is set in the traveling direction from the representative point of the own vehicle M.

[Hardware configuration]
FIG. 13 is a diagram showing an example of the hardware configuration of the automatic operation control device 100 of the embodiment. As shown in the figure, the automatic operation control device 100 includes a communication controller 100-1, a CPU 100-2, a RAM (Random Access Memory) 100-3 used as a working memory, a ROM (Read Only Memory) for storing a boot program, and the like. The configuration is such that 100-4, a storage device 100-5 such as a flash memory or an HDD (Hard Disk Drive), a drive device 100-6, and the like are connected to each other by an internal bus or a dedicated communication line. The communication controller 100-1 communicates with a component other than the automatic operation control device 100. The storage device 100-5 stores a program 100-5a executed by the CPU 100-2. This program is expanded to the RAM 100-3 by a DMA (Direct Memory Access) controller (not shown) or the like, and is executed by the CPU 100-2. As a result, a part or all of the recognition unit 130, the action plan generation unit 180, and the second control unit 190 are realized.

According to the embodiment described above, image analysis for recognizing the position of the road lane marking with respect to the vehicle based on the image acquired by the camera (10) mounted on the vehicle (own vehicle M) and capturing the periphery of the vehicle. A group of target positions lined up linearly from a plurality of target positions acquired by an object detection device (for example, LIDAR14) different from the camera, with the detection range set around the vehicle and the unit (150) mounted on the vehicle. A specific unit that specifies the first extension direction, which is the extension direction of the vehicle (target extension direction specification unit 152), and a determination unit that determines the reference track (RR) that is the reference for vehicle travel control (reference track determination). In section 156), when the first extension direction and the second extension direction, which is the extension direction of the road lane marking recognized by the image analysis section, match, the image analysis section recognizes the first extension direction. By providing a determination unit for determining the reference track based on the road lane marking, it is possible to appropriately adopt or reject the recognition result of the road lane marking by the camera based on the detection result of another sensor.

The embodiment described above can be expressed as follows.
A storage device that stores the program and
Equipped with a hardware processor,
The hardware processor executes a program stored in the storage device to execute the program.
Based on the image mounted on the vehicle and acquired by the camera that images the surroundings of the vehicle, the position of the road lane marking with respect to the vehicle is recognized.
It is the extension direction of a group of target positions arranged in a line from a plurality of target positions mounted on the vehicle and having a detection range around the vehicle and acquired by an object detection device different from the camera. 1 Specify the extension direction,
Based on the road lane marking recognized by the image analysis unit when the first extension direction and the second extension direction which is the extension direction of the road lane marking recognized by the image analysis unit match. A peripheral recognition device configured to determine a reference track as a reference for driving control of the vehicle.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

10 Camera 12 Radar device 14 LIDAR
16 Object recognition device 30 HMI
54 1st map information 62 2nd map information 100 Automatic operation control device 120 1st control unit 130 Recognition unit 140 Object recognition unit 150 Image analysis unit 152 Object extension direction identification unit 154 Map base recognition unit 156 Reference track determination unit 160 Vehicle position recognition unit 190 2nd control unit

Claims (9)

An image analysis unit that recognizes the position of the road lane marking with respect to the vehicle based on an image mounted on the vehicle and acquired by a camera that images the surroundings of the vehicle.
It is the extension direction of a group of target positions arranged in a line from a plurality of target positions mounted on the vehicle and having a detection range around the vehicle and acquired by an object detection device different from the camera. 1 Specific part that specifies the extension direction and
By applying the position of the vehicle to the map information, a map-based recognition unit that recognizes a third extension direction, which is the extension direction of the road on which the vehicle exists with respect to the vehicle, and a map-based recognition unit.
A determination unit that determines a reference track to be used as a reference for vehicle travel control, and a first extension direction and a second extension direction that is an extension direction of a road lane marking recognized by the image analysis unit. When the second extension direction and the third extension direction do not match, the determination unit that determines the reference track based on the road marking line recognized by the image analysis unit, and the determination unit.
Peripheral recognition device equipped with. When the first extending direction and the second extending direction do not match, and the second extending direction and the third extending direction do not match, the determination unit is the map-based recognition unit. Determine the reference track based on the road lane markings recognized by
The peripheral recognition device according to claim 1 . The specific portion is a group of objects in which the target positions corresponding to the targets presumed to extend beyond a predetermined length and are stationary from the plurality of target positions are arranged in a line. Specify as a target position,
The peripheral recognition device according to claim 1 or 2 . The specific unit identifies a group of target positions arranged in a line by thinning out the target positions arranged in a line among the plurality of target positions by a predetermined method to reduce the number of the target positions. do,
The peripheral recognition device according to any one of claims 1 to 3 . The specific portion specifies the first extending direction by performing a linear approximation with respect to a group of target positions arranged in a line.
The peripheral recognition device according to any one of claims 1 to 4 . The determination unit is based on the linearly arranged target position on the side closer to the traveling direction of the vehicle among the linearly arranged target positions recognized on the left and right sides of the vehicle. When the specified first extending direction and the second extending direction match, the reference track is determined based on the road marking line recognized by the image analysis unit.
The peripheral recognition device according to any one of claims 1 to 5 . The determination unit is specified based on the group of target positions lined up in a line, whichever is larger in number, recognized on the left and right sides of the vehicle. When the first extending direction and the second extending direction match, the reference track is determined based on the road lane marking recognized by the image analysis unit.
The peripheral recognition device according to any one of claims 1 to 6 . The computer
Based on the image mounted on the vehicle and acquired by the camera that images the surroundings of the vehicle, the position of the road lane marking with respect to the vehicle is recognized.
It is the extension direction of a group of target positions arranged in a line from a plurality of target positions mounted on the vehicle and having a detection range around the vehicle and acquired by an object detection device different from the camera. 1 Specify the extension direction,
By applying the position of the vehicle to the map information, the third extending direction, which is the extending direction of the road on which the vehicle is located, is recognized with respect to the vehicle.
When the first extending direction and the second extending direction which is the extending direction of the recognized road lane marking match, and the second extending direction and the third extending direction do not match. , A reference track to be used as a reference for driving control of the vehicle is determined based on the recognized road lane marking.
Peripheral recognition method. On the computer
Based on the image mounted on the vehicle and acquired by the camera that images the surroundings of the vehicle, the position of the road lane marking with respect to the vehicle is recognized.
It is the extension direction of a group of target positions arranged in a line from a plurality of target positions mounted on the vehicle and having a detection range around the vehicle and acquired by an object detection device different from the camera. 1 Let the extension direction be specified,
By applying the position of the vehicle to the map information, the third extending direction, which is the extending direction of the road on which the vehicle is located, is recognized with respect to the vehicle.
When the first extending direction and the second extending direction which is the extending direction of the recognized road lane marking match, and the second extending direction and the third extending direction do not match. , The reference track to be used as the reference for the traveling control of the vehicle is determined based on the recognized road lane marking.
program.

Images