JP7165630B2 - Recognition system, vehicle control system, recognition method, and program - Google Patents

Description

The present invention relates to recognition systems, vehicle control systems, recognition methods, and programs.

Conventionally, a road shape recognition unit that recognizes the road shape, a travel route creation unit that creates a travel route using the recognized road shape, and a vehicle travel control device that realizes automatic travel according to the travel route. The recognizing unit includes: a coordinate information acquiring unit that acquires a plurality of pieces of coordinate information in which plane coordinates and height information are associated; and a shape specifying unit that specifies the shape of the road by statistically processing the extracted coordinates of interest (Patent Document 1).

JP 2010-250743 A

In the conventional technology, portions where the height difference is equal to or greater than a predetermined value are recognized as road shoulders or ditches, and other portions are recognized as the road surface. However, with this method, there are cases where a part of the road surface is not recognized as the road surface due to the curvature of the road surface itself, or small obstacles are overlooked by increasing the predetermined value.

SUMMARY OF THE INVENTION The present invention has been made in consideration of such circumstances, and one of its objects is to provide a recognition system, a vehicle control system, a recognition method, and a program capable of more accurately specifying a road surface. do.

A recognition system, a vehicle control system, a recognition method, and a program according to the present invention employ the following configuration.
(1): A recognition system according to an aspect of the present invention is a recognition system mounted on a vehicle, and includes a detection unit that detects the position of an object existing around the vehicle, and a detection result of the detection unit. a specifying unit that specifies a road surface in the vicinity of the vehicle based on, the specifying unit uses a predetermined algorithm for each individual region obtained by subdividing the detection result of the detecting unit on a two-dimensional plane It is determined whether or not the road surface is flat, and the determination results for each of the individual areas are aggregated to specify the road surface around the vehicle.

(2): In the aspect of (1) above, the detection unit is a lidar.

(3): In the aspect of (2) above, the detection unit irradiates a laser around the vehicle while changing the elevation angle or depression angle and the azimuth angle, and the specific unit includes the elevation angle, the depression angle, and the azimuth angle. , and the position of an object represented at least by a distance projected onto a two-dimensional plane.

(4): In any one of the aspects (1) to (3) above, the specifying unit varies the size of the individual area based on the distance from the vehicle on the two-dimensional plane.

(5): In any one of the above aspects (1) to (4), the specifying unit acquires information indicating the distribution of objects around the vehicle, and based on the acquired information indicating the distribution of objects, to change the size of the individual area.

(6): In any one of the above aspects (1) to (5), the specifying unit acquires information on the type of road on which the vehicle exists, and the acquired information on the type of road is a specific type. , the size of the individual area is increased compared to the case where the acquired information on the type of road does not indicate a specific type of road.

(7): The recognition system according to any one of the aspects (1) to (6) above, and the portion corresponding to the road surface specified by the specifying unit is excluded from the detection result of the detection unit in the recognition system. and a cruise control device that performs cruise control of the vehicle based on the information.

(8): In a recognition method according to another aspect of the present invention, a computer mounted on a vehicle acquires a detection result of a detection unit that detects the position of an object existing around the vehicle, and based on the detection result, a road surface in the vicinity of the vehicle is identified, and when the identification is made, it is determined whether or not the road surface is flat using a predetermined algorithm for each individual area obtained by subdividing the detection result on a two-dimensional plane. , the road surface around the vehicle is specified by aggregating the determination results for each of the individual areas.

(9): A program according to another aspect of the present invention causes a computer mounted on a vehicle to acquire a detection result of a detection unit that detects the position of an object existing around the vehicle, and based on the detection result, specifying a road surface in the vicinity of the vehicle, and determining whether or not the road surface is flat by using a predetermined algorithm for each individual area obtained by subdividing the detection result on a two-dimensional plane when specifying the road surface; The road surface around the vehicle is identified by summarizing the determination results for each of the individual areas.

According to the aspects (1) to (9) above, the road surface can be specified more accurately.

1 is a diagram showing a state in which a vehicle is equipped with a recognition system and a vehicle control system; FIG. 1 is a configuration diagram of an object recognition device; FIG. It is a figure which shows an example of point cloud data. It is a figure which shows the grid which is set. FIG. 10 is a diagram showing point cloud data from which the coordinates of a portion determined to be a road surface are removed by the method of the comparative example; It is a figure which shows the point cloud data which removed the coordinate of the part determined as the road surface by the method of embodiment. It is a figure which shows the point cloud data which removed the coordinate of the part determined as the road surface by the method of embodiment. 4 is a flow chart showing an example of the flow of processing executed by the recognition system;

Embodiments of a recognition system, a vehicle control system, a recognition method, and a program according to the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing how a vehicle M is equipped with a recognition system and a vehicle control system. The vehicle M includes, for example, a lidar (Light Detection and Ranging: LIDAR) 10 (an example of a “detection unit”), a camera 20, a radar device 30, an object recognition device 50, and a travel control device 100. be. A combination of the rider 10 and the object recognition device 50 is an example of a "recognition system", and a combination of the traveling control device 100 is an example of a "vehicle control system". A detector other than the lidar may be used as the detector.

The lidar 10 detects the distance to an object by irradiating light, detecting reflected light, and measuring the time from irradiation to detection. The lidar 10 can change the irradiation direction of light with respect to both the elevation angle or depression angle (hereinafter referred to as vertical irradiation direction φ) and the azimuth angle (horizontal irradiation direction θ). For example, the lidar 10 performs scanning while fixing the irradiation direction φ and changing the irradiation direction θ, then changes the irradiation direction φ in the vertical direction, fixes the irradiation direction φ at the changed angle, and changes the irradiation direction θ. The operation of scanning while scanning is repeated. Hereinafter, the irradiation direction φ will be referred to as a “layer”, one scan performed while fixing the layer and changing the irradiation direction θ will be referred to as a “cycle”, and scanning for all layers will be referred to as “1 called "scan". Layers are set by a finite number, for example, from L1 to Ln (n is a natural number). The layers are changed discontinuously with respect to angle, for example, L0→L4→L2→L5→L1 . In addition, it is not limited to this, and the layer may be changed continuously with respect to the angle.

The rider 10 outputs a data set (rider data) having {φ, θ, d, p} as one unit, for example, to the object recognition device 50 . d is the distance and p is the intensity of the reflected light. The object recognition device 50 is installed at an arbitrary location on the vehicle M. As shown in FIG. In FIG. 1, the rider 10 is installed on the roof of the vehicle M so that the irradiation direction θ can be changed by 360 degrees. A rider that can change the irradiation direction θ by 180 degrees around the front of the vehicle M, and a rider that is provided at the rear of the vehicle M and can change the irradiation direction θ by 180 degrees around the rear of the vehicle M. may be mounted on the

Camera 20 is installed in the arbitrary positions which can picturize the circumference of vehicles M (especially the front or the back). For example, the camera 20 is installed on top of the front windshield. The camera 20 is a digital camera equipped with an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and repeatedly images the surroundings of the vehicle M at predetermined intervals.

The radar device 30 radiates radio waves such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by an object to detect at least the position (distance and direction) of the object. The radar device 30 is attached to any location of the vehicle M. As shown in FIG. For example, the radar device 30 is attached inside the front grill of the vehicle M.

FIG. 2 is a configuration diagram of the object recognition device 50. As shown in FIG. The object recognition device 50 includes, for example, a lidar data processing unit 60, a camera image processing unit 70, a radar data processing unit 80, and a sensor fusion unit 90. The lidar data processing unit 60 includes, for example, a point cloud data generation unit 61, an information acquisition unit 62, a road surface identification unit 63 (an example of a “specification unit”), a non-road surface object extraction unit 64, and a road marking line. and a recognition unit 65 . The road surface identification unit 63 includes, for example, a grid setting unit 63A and a plane extraction processing unit 63B. These components are implemented by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Some or all of these components are hardware (circuit part; circuitry) or by cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as a HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage such as a DVD or CD-ROM. It may be stored in a medium (non-transitory storage medium) and installed by loading the storage medium into a drive device.

The point cloud data generator 61 generates point cloud data based on the rider data. The point cloud data in this embodiment is obtained by projecting the position of an object recognized from lidar data in a three-dimensional space onto a position on a two-dimensional plane as seen from above. FIG. 3 is a diagram showing an example of point cloud data. A two-dimensional plane defining the point cloud data (a two-dimensional plane represented by the X-axis and the Y-axis in the drawing) is a relative two-dimensional plane viewed from the lidar 10, for example. Although not shown in the figure, height information (displacement in the direction orthogonal to the X-axis and Y-axis) is given to each coordinate of the point cloud data. The height information is calculated by the point cloud data generation unit 61 based on the continuity and scattering between coordinates, the difference in illumination angles between layers, and the like.

The information acquisition unit 62 acquires various types of information used when the grid setting unit 63A sets the grid.

For example, the information acquisition unit 62 acquires information indicating the distribution of objects around the vehicle M. FIG. The distribution of objects around the vehicle M means, for example, the number of vehicles, the number of pedestrians, the number of bicycles, the number of traffic lights, pedestrian crossings, and intersections within the recognizable range of the recognition system. Get an indexed value (congestion index). The congestion index indicates a higher value, for example, as the above elements exist at higher density. The information acquisition unit 62 may calculate the congestion index by itself, or may acquire it from the camera image processing unit 70, the radar data processing unit 80, or the like.

The information acquisition unit 62 may also acquire information about the type of road on which the vehicle M is present. The information acquisition unit 62 may acquire the information on the type of road from a navigation device (not shown) mounted on the vehicle M, or derive it from the result of the camera image processing unit 70 recognizing the road sign from the camera image. You may

A grid setting unit 63A of the road surface specifying unit 63 virtually sets a plurality of grids, which are individual regions obtained by dividing a two-dimensional plane defining point cloud data. FIG. 4 is a diagram showing grids G to be set. The grid setting unit 63A, for example, sets the grid G in a rectangular shape (either square or rectangular). The grid setting unit 63A may set grids G of the same size, or may vary the size of the grids G based on the distance from the vehicle M on the two-dimensional plane. For example, the grid setting unit 63A may increase the size of the grid G with increasing distance from the vehicle M, as illustrated. The grid setting unit 63A does not need to set the grid G for areas that do not require recognition (for example, areas beyond guardrails and areas outside roads such as buildings) obtained by referring to past recognition results of the recognition system. The grid setting unit 63A may set the grid G with arbitrary polygons such as triangles and hexagons (two or more types of polygons may be included), or may set the grid G with irregular shapes. may

Also, the grid setting unit 63A may determine the size of the grid G based on the congestion index. For example, the grid setting unit 63A may reduce the size of the grid G as the congestion index increases.

Also, the grid setting unit 63A may determine the size of the grid G based on the type of road. For example, when the type of road is a specific type such as an expressway or an expressway with few traffic participants other than vehicles, the grid setting unit 63A sets the size of the grid G more than when the road is not of the specific type. You can make it bigger.

The plane extraction processing unit 63B performs plane extraction processing by a robust estimation method such as RANSAC (Random Sample Consensus) on the point cloud data included in the grid G for each grid G, and the grid G is a road surface ( It is determined whether or not there is no object on it), and the determination result is associated with each of the grids G. Note that the plane extraction processing unit 63B may perform other types of plane extraction processing instead of RANSAC.

RANSAC is performed, for example, by the following procedures. First, from the data set, randomly select more than the number of samples required for model determination (not all), derive a temporary model from the selected samples using the least squares method, etc., and try to fit the temporary model to the data. , if there are not so many outliers, add them to the model candidates. This process is repeated several times, and the model candidate that best matches the entire data set is set as the correct model. In this embodiment, the plane extraction processing unit 63B determines that the grid G whose correct model is a plane is a road surface.

The non-road surface object extraction unit 64 analyzes the point cloud data for the grids G other than the grid G determined to be the road surface by the plane extraction processing unit 63B, and extracts the outline of the object existing on the grid G, Based on the contour, the position of the object corresponding to the contour is recognized. Alternatively, the non-road surface object extraction unit 64 extracts the contour of the object based on the data corresponding to the grid G other than the grid G determined to be the road surface by the plane extraction processing unit 63B among the lidar data. , the position of the object corresponding to that contour may be recognized.

The road marking line recognition unit 65 focuses on the intensity p of the reflected light in the lidar data, and recognizes the portion where the rate of change of the intensity p caused by the color difference between the road surface and the road marking lines such as white lines and yellow lines is high. It is recognized as the contour of the road division line. Thereby, the road marking line recognizing unit 65 recognizes the position of the road marking line such as the white line.

The processing results of the non-road surface object extraction unit 64 and the road marking line recognition unit 65 are output to the sensor fusion unit 90 . The processing results of the camera image processing unit 70 and the radar data processing unit 80 are also input to the sensor fusion unit 90 .

The camera image processing unit 70 performs various image processing on the camera image acquired from the camera 20, and recognizes the position, size, type, etc. of an object existing around the vehicle M. The image processing performed by the camera image processing unit 70 includes a process of inputting a camera image into a trained model obtained by machine learning, and a process of extracting edge points and recognizing an object from a contour line connecting the edge points. It's okay.

The radar data processing unit 80 performs various object extraction processes on the radar data acquired from the radar device 30, and recognizes the position, size, type, etc. of objects existing around the vehicle M. FIG. The radar data processing unit 80, for example, estimates the material of the object based on the intensity of the reflected wave from the object, thereby estimating the type of the object.

The sensor fusion unit 90 integrates the processing results input from each of the lidar data processing unit 60, the camera image processing unit 70, and the radar data processing unit 80, determines the positions of objects and road markings, and operates the travel control device. Output to 100. The processing of the sensor fusion unit 90 may include, for example, processing of obtaining a logical sum, a logical product, a weighted sum, etc. for each processing result.

By performing the processing as described above, the recognition system can more accurately identify the road surface. FIG. 5 is a diagram showing point cloud data from which the coordinates of the portion determined as the road surface are removed by the technique of the comparative example. The method of the comparative example is a method of applying RANSAC to the entire lidar data (without dividing it into grids G) and removing the coordinates of the area determined to be the road surface. As illustrated, in the method of the comparative example, many coordinates remain even in areas A1 and A2 corresponding to the road surface. In particular, the area A1 is an uphill viewed from the vehicle M, and is difficult to recognize as a road surface when RANSAC is applied to the entire data. In addition, since a typical road structure is high in the center of the road and becomes lower toward the edge, in the method of the comparative example, it is determined that the road is not a road surface based on the height difference between the center and the edge of the road. There is a possibility that Furthermore, since the road has fine depressions and the like, there is a possibility that the road surface may be partially recognized as not being the road surface.

On the other hand, FIGS. 6 and 7 are diagrams showing point cloud data from which the coordinates of the portion determined as the road surface are removed by the technique of the embodiment. FIG. 6 shows the results when one side of the square grid G is set to X1, and FIG. 7 shows the results when one piece of the square grid G is set to X2 (X1>X2). . As shown in these figures, according to the method of the embodiment, many coordinates are removed in the areas A1 and A2 corresponding to the road surface, so that an object that does not actually exist is recognized as an obstacle. less likely to be lost. It should be noted that if one side of the grid G is made smaller (i.e., if the size of the grid G is made smaller), the accuracy of determining the road surface can be improved. Therefore, they are in a trade-off relationship. In this regard, by increasing the size of the grid G as it becomes farther from the vehicle M, it is possible to perform processing with a low load for distant areas where the influence of erroneous recognition is small, thereby achieving a good balance between recognition accuracy and processing load. can be processed. In addition, by reducing the size of the grid G as the congestion index increases, processing is performed with priority given to recognition accuracy in locations such as urban areas where there are many traffic participants, and processing is performed with priority given to reducing the processing load in other locations. , it is possible to perform processing with a good balance between recognition accuracy and processing load. In addition, when the type of road is a specific type, by increasing the size of the grid G compared to when the road type is not a specific type, processing that prioritizes the reduction of the processing load is performed in places where there are few traffic participants. In other places, the recognition accuracy is prioritized, so that the recognition accuracy and the processing load are well balanced.

The travel control device 100 is, for example, an automatic driving control device that controls both acceleration/deceleration and steering of the vehicle M. Based on the positions of objects, white lines, etc. output by the object recognition device 50, the travel control device 100 automatically causes the vehicle M to travel within the set lane so as not to come into contact with objects, It automatically controls lane changes, overtaking, branching, merging, and stopping. Alternatively, the cruise control device 100 may be a driving support device or the like that automatically stops when an object approaches. In this way, the traveling control device 100 detects the position of the object (on the identified road surface) recognized by the non-road surface object extraction unit 64 for the grids G other than the grid G determined to be the road surface by the plane extraction processing unit 63B. Based on the information excluding the corresponding portion) output through the sensor fusion unit 90, the traveling control of the vehicle M is performed.

FIG. 8 is a flow chart showing an example of the flow of processing executed by the recognition system. The rider 10 detects an object and repeatedly outputs rider data to the rider data processing unit 60 (step S100).

The rider data processing unit 60 waits until the rider data for one scan is obtained (step S102). Once the lidar data for one scan is acquired, the process proceeds to step S106.

On the other hand, the information acquisition unit 62 operates, for example, asynchronously with the components of the rider data processing unit 60 other than the information acquisition unit 62, acquires information for setting the grid G, and sends the information to the road surface identification unit 63. provided (step S104).

The point cloud data generator 61 generates point cloud data from the rider data (step S106). The grid setting unit 63A sets the grid G based on the information provided from the information acquisition unit 62 (step S108).

The plane extraction processing unit 63B identifies whether or not each grid G is a road surface (step S110). The non-road surface object extraction unit 64 performs object recognition on the non-road surface grid G that has not been identified as a road surface (step S112). Then, the rider data processing unit 60 outputs the processing results of the non-road surface object extraction unit 64 and the road marking line recognition unit 65 to the sensor fusion unit 90 (step S114). This completes the routine of the flow chart of FIG.

According to the recognition system of the embodiment described above, the detector (rider 10) detects the position of an object existing around the vehicle (M), and the road surface around the vehicle is detected based on the detection result of the detector. and a specifying unit (road surface specifying unit 63) that specifies a predetermined algorithm (RANSAC) for each individual area (grid G) obtained by subdividing the detection result of the detecting unit on a two-dimensional plane. is used to determine whether the road surface is flat, and the determination results for each individual area are aggregated to specify the road surface around the vehicle. Therefore, the road surface can be specified more accurately.

Note that the recognition system may not include some or all of the camera 20, the radar device 30, the camera image processing section 70, the radar data processing section 80, and the sensor fusion section 90. FIG. For example, the recognition system may include the rider 10 and the rider data processing unit 60 , and output the processing results of the non-road surface object extraction unit 64 and the road marking line recognition unit 65 to the traveling control device 100 .

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

10 Rider 50 Object recognition device 60 Rider data processing unit 61 Point cloud data generation unit 62 Information acquisition unit 63 Road surface identification unit 63A Grid setting unit 63B Plane extraction processing unit 64 Non-road surface object extraction unit 65 Road marking line recognition unit 100 Travel Control device

Claims (6)

A recognition system mounted on a vehicle, comprising:
a detection unit that detects the position of an object existing around the vehicle;
a specifying unit that specifies a road surface in the vicinity of the vehicle based on the detection result of the detecting unit;
The identifying unit uses a predetermined algorithm to determine whether each individual area obtained by subdividing the detection result of the detecting unit on a two-dimensional plane is a plane or not, and aggregates the determination results for each of the individual areas. to identify the road surface in the vicinity of the vehicle,
The predetermined algorithm randomly selects a predetermined number of samples from a data set that is the detection result of the detection unit for each of the individual regions, derives model candidates based on the selected samples, and applies the model candidates to the data. Among the model candidates whose number of outliers is less than a specified number, the model candidate that best matches the entire data set is set as the correct model. is an algorithm that
The detection unit is a lidar that irradiates a laser around the vehicle while changing the elevation angle or the depression angle and the azimuth angle,
The specifying unit is
For each individual region obtained by subdividing the point cloud data obtained by projecting the position of the object represented at least by the angle of elevation or depression, the azimuth angle, and the distance onto a two-dimensional plane, it is determined whether it is a plane using the above-mentioned predetermined algorithm. judge,
Varying the size of the individual regions based on the distance from the vehicle in the two-dimensional plane,
The individual region is not set for the recognition-unnecessary region obtained by referring to the past recognition result of the recognition system.
recognition system. The specifying unit acquires information indicating the distribution of objects around the vehicle, and changes the size of the individual region based on the acquired information indicating the distribution of objects.
2. The recognition system of claim 1 . The identifying unit obtains information about a type of road on which the vehicle is present, and if the obtained information about a road type indicates a road of a specific type, the identifying unit obtains information about a road type of the specific type. 3. Recognition system according to claim 1 or 2 , wherein the size of said individual area is increased compared to when no road is shown. a recognition system according to any one of claims 1 to 3 ;
a travel control device that performs travel control of the vehicle based on information obtained by excluding the portion corresponding to the road surface specified by the specifying unit from the detection result of the detection unit in the recognition system;
vehicle control system. A computer installed in the vehicle
Acquire the detection results of the detection unit that detects the positions of objects existing around the vehicle,
identifying a road surface in the vicinity of the vehicle based on the detection result;
When specifying the
Determining whether each individual region obtained by subdividing the detection result on a two-dimensional plane is a plane using a predetermined algorithm,
Aggregating the determination results for each of the individual regions to specify a road surface in the vicinity of the vehicle;
The predetermined algorithm randomly selects a predetermined number of samples from a data set that is the detection result of the detection unit for each of the individual regions, derives model candidates based on the selected samples, and applies the model candidates to the data. Among the model candidates whose number of outliers is less than a specified number, the model candidate that best matches the entire data set is set as the correct model. is an algorithm that
The detection unit is a lidar that irradiates a laser around the vehicle while changing the elevation angle or the depression angle and the azimuth angle,
When specifying the
For each individual region obtained by subdividing the point cloud data obtained by projecting the position of the object represented at least by the angle of elevation or depression, the azimuth angle, and the distance onto a two-dimensional plane, it is determined whether it is a plane using the above-mentioned predetermined algorithm. judge,
Varying the size of the individual regions based on the distance from the vehicle in the two-dimensional plane,
The individual region is not set for the recognition unnecessary region obtained by referring to the past recognition result of the computer.
recognition method. computer on board the vehicle,
Acquiring the detection result of a detection unit that detects the position of an object existing around the vehicle,
specifying a road surface in the vicinity of the vehicle based on the detection result;
When specifying the
Determining whether each individual region obtained by subdividing the detection result on a two-dimensional plane is a plane using a predetermined algorithm,
Aggregating the determination results for each of the individual areas to specify a road surface around the vehicle;
The predetermined algorithm randomly selects a predetermined number of samples from a data set that is the detection result of the detection unit for each of the individual regions, derives model candidates based on the selected samples, and applies the model candidates to the data. Among the model candidates whose number of outliers is less than a specified number, the model candidate that best matches the entire data set is set as the correct model. is an algorithm that
The detection unit is a lidar that irradiates a laser around the vehicle while changing the elevation angle or the depression angle and the azimuth angle,
When specifying the
For each individual region obtained by subdividing the point cloud data obtained by projecting the position of the object represented at least by the angle of elevation or depression, the azimuth angle, and the distance onto a two-dimensional plane, it is determined whether it is a plane using the above-mentioned predetermined algorithm. let me judge
making the sizes of the individual regions different based on the distance from the vehicle in the two-dimensional plane;
The individual regions are not set for recognition unnecessary regions obtained by referring to past recognition results of the computer;
program.

Images