JP7290240B2 - Object recognition device - Google Patents

Description

The present invention relates to a device for recognizing objects.

Features such as signs and buildings are extracted based on three-dimensional point cloud data obtained by measuring roads, signs, buildings, and the like with a laser scanner or the like. While the operator looks at the screen, determining, extracting, and assigning attributes to each feature is highly reliable, but has the problem of being complicated and requiring a long extraction time.

Non-Patent Document 1 discloses a process of extracting features from three-dimensional point cloud data and assigning attributes to them based on CAD data corresponding to the three-dimensional point cloud data.

Patent Literature 1 discloses an apparatus that recognizes a feature by template matching based on figures obtained by projecting three-dimensional point cloud data onto a plurality of horizontal planes.

With these, features can be recognized based on the three-dimensional point cloud data.

It is also proposed to extract features from 3D point cloud data by machine learning. In order to realize such a system, a large amount of learning data is required. Non-Patent Document 2 discloses a method of generating three-dimensional point cloud data as learning data based on CAD data.

JP 2017-167092

Kenji Nakamura et al., "Study on feature extraction technology for point cloud data based on completed floor plans," Journal of the Japan Society of Civil Engineers (Civil Engineering Information Science) Vol.73 No.2, I_424-I_432, 2017 Kenta Fukano et al., "Learning data generation method for terrestrial classification based on mobile measurement data" 2014 Japan Society for Precision Engineering Autumn Meeting Proceedings

However, in the method of Non-Patent Document 1, three-dimensional point cloud data corresponding to features are extracted based on CAD data and color image data corresponding to the three-dimensional point cloud data. Therefore, there is a problem that the features cannot be extracted when CAD data or color image data corresponding to the three-dimensional point cloud data do not exist.

In addition, there is also the problem that features that did not exist when the CAD data was created and features that were removed after the CAD data was created cannot be extracted correctly.

Further, in the method of Patent Document 1, recognition is performed using a plurality of horizontal cross sections, but there is a problem that accurate recognition cannot always be performed depending on the inclination and direction of the target. In addition, there is a problem that the feature of the object cannot be accurately captured and the recognition is not accurate.

Furthermore, even if deep learning or machine learning is performed to extract features from 3D point cloud data, a large amount of learning data has to be prepared. Although Non-Patent Document 2 is one method for solving this problem, it generates learning data based on CAD data, and cannot be used when CAD data does not exist.

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above-described problems and to provide an apparatus capable of appropriately extracting features without CAD data.

Some independently applicable features of the invention are set forth below.

(1)(2) A recognition device according to the present invention is a recognition device for distinguishing and recognizing objects, and comprises data acquisition means for acquiring three-dimensional point cloud data obtained by measuring the object; Based on the three-dimensional area containing the three-dimensional point cloud data, the distribution characteristic calculation means for calculating the distribution characteristics of the point cloud in the three-dimensional point cloud data as a numerical value, and the object estimating means for estimating

Therefore, the object can be estimated according to the distribution characteristics of the object.

(3) In the recognition device according to the present invention, the three-dimensional point cloud data is obtained by measuring roads and features, and the area generating means includes a three-dimensional point cloud including the features with reference to the road surface corresponding to the features. It is characterized by defining the original region.

Therefore, it is possible to form a three-dimensional area that similarly encloses even the same type of feature in different locations regardless of its orientation and angle, thereby improving the recognition accuracy.

(4) The recognition device according to the present invention is characterized in that the distribution characteristic calculation means calculates the X-direction length, Y-direction length, or Z-direction length of the three-dimensional area as the distribution characteristic.

Therefore, it is possible to estimate horizontally long and vertically long objects.

(5) In the recognition device according to the present invention, the distribution characteristic calculating means calculates the distribution characteristic by calculating the variance of the number of points included in each section when the three-dimensional area is divided into strips at least along the X axis or the Y axis. It is characterized by calculating as

It is possible to estimate an object with a lot of shape changes and an object with little change.

(6) The recognition device according to the present invention is characterized in that the distribution characteristic calculation means divides at least the three-dimensional area into upper and lower parts, and calculates the ratio of the number of points included in the upper part and the lower part as the distribution characteristic. .

Therefore, it is possible to estimate an object that spreads upwards and spreads downwards.

(7) The recognition device according to the present invention is characterized in that the distribution characteristic calculation means calculates, as the distribution characteristic, at least the ratio of the X-direction length or Y-direction length of the three-dimensional area to the Z-direction length. .

Therefore, it is possible to estimate horizontally long and vertically long objects.

(8) In the recognition device according to the present invention, the distribution characteristic calculation means distributes the amount of change in the X-direction width or the Y-direction width of the three-dimensional point cloud data at least at positions at predetermined intervals in the Z direction of the three-dimensional area. It is characterized in that it is calculated as a characteristic.

Therefore, an object with protruding parts can be estimated.

(9) In the recognition device according to the present invention, the distribution characteristic calculating means calculates, as the distribution characteristic, the density of points in a section having the largest number of points when the three-dimensional area is divided into strips at least along the Z axis. It is characterized by

Therefore, it is possible to estimate the object of man-made structures and natural objects.

(10) The recognition device according to the present invention is characterized in that the distribution characteristic calculation means calculates at least the inclination of the normal vector of the entire three-dimensional point cloud data with respect to the Z-axis as the distribution characteristic.

Therefore, it is possible to estimate a notation object and a planar object.

(11) In the recognition device according to the present invention, the distribution characteristic calculation means divides the three-dimensional area into strips along at least the Z axis, and the inclination of the normal vector of the point cloud data of each section with respect to the Z axis is It is characterized by calculating variations as distribution characteristics.

Therefore, it is possible to estimate a notation object and a planar object.

(12) In the recognition device according to the present invention, the distribution characteristic calculating means calculates, as the distribution characteristic, an amount of cumulative change in the area of the point cloud data in a cross section obtained by dividing the three-dimensional region into strips at least along the Z axis. is characterized by

Therefore, the object can be estimated based on the index indicating the shape change.

(13) The recognition device according to the present invention is characterized in that the distribution characteristic calculating means calculates at least a matching rate by template matching as the distribution characteristic.

Therefore, estimation can be performed including the results of template matching.

(14) The recognition device according to the present invention is characterized in that the estimating means performs estimation by performing machine learning based on the distribution characteristics.

Therefore, more accurate estimation can be performed.

(15) In the recognition device according to the present invention, the estimating means groups the three-dimensional point cloud data based on the positions where the three-dimensional point cloud data exist, and performs machine learning or deep learning for each group. It is characterized by estimating the object by

Therefore, accurate estimation can be performed even if the types of objects to be recognized increase.

(16) The recognition device according to the present invention is characterized in that the estimating means estimates the object taking into account the reflection intensity or color data at each point, or both.

Therefore, object estimation based on reflectance or color data or both can be made.

(17) A method for producing a recognition device according to the present invention is a method for producing a recognition device for distinguishing and identifying a target object based on three-dimensional point cloud data obtained by measuring the target object. A three-dimensional area containing the measured three-dimensional point cloud data is formed, and based on the three-dimensional area, the distribution characteristics of the point cloud in the three-dimensional point cloud data are calculated as numerical values, and the above recognition processing is realized by learning. It is characterized in that the calculated numerical value of the distribution characteristic is given as learning data to the apparatus for learning.

Therefore, the object can be estimated according to the distribution characteristics of the object.

(18)(19) A recognition device according to the present invention is a recognition device for recognizing a roadway covering object from measured 3D point cloud data, and is acquisition means for acquiring the measured 3D point cloud data. road extraction means for extracting a roadway object from the three-dimensional point cloud data; roadway upper object extraction means for extracting an object superimposed on the extracted roadway object; and setting the covering object to the roadway object. a roadway covering object extracting means for forming cross sections on planes perpendicular to the lane direction at predetermined intervals, and estimating that the covering object is a roadway covering object if the covering object has substantially the same shape in each cross section. .

Therefore, the roadway covering object can be properly estimated.

(20) The recognition device according to the present invention is characterized by including at least culverts, sheds, shelters or tunnels as roadway covering objects.

Therefore, these objects can be properly estimated.

(21) The recognition device according to the present invention is characterized in that the lane direction is acquired as the running direction at the time of measurement.

Therefore, the lane direction can be set correctly.

(22)(23) A recognition device according to the present invention is a recognition device for recognizing a retaining wall object from measured 3D point cloud data, and acquires 3D point cloud data obtained by measuring the object. Data acquisition means, roadway extraction means for extracting a roadway object from the three-dimensional point cloud data, and point cloud data forming an approximate plane contacting an edge of the roadway object, and an angle between the approximate plane and the roadway is determined. Retaining wall estimating means for estimating that the approximate plane is a retaining wall if the angle is greater than or equal to a predetermined angle.

Therefore, the retaining wall object can be properly estimated.

"Data acquisition means" corresponds to step S21 in the embodiment.

"Distribution characteristic calculation means" corresponds to step S274 in the embodiments.

In the embodiments, the "estimating means" corresponds to step S275.

"Program" is a concept that includes not only programs that can be directly executed by the CPU, but also programs in source format, compressed programs, encrypted programs, and the like.

It is a figure which shows the outline|summary of learning of the recognition apparatus by one Embodiment of this invention. It is the hardware configuration of the recognition device. 4 is a flow chart of learning processing of the recognition program 48. FIG. FIG. 4 is a diagram showing objects and boundary boxes; It is a figure which shows an example of a feature-value. It is a figure for demonstrating calculation of the feature-values F1, F2, and F3. It is a figure for demonstrating calculation of the feature-values F4 and F5. It is a figure for demonstrating calculation of the feature-value F6. It is a figure for demonstrating calculation of the feature-values F7 and F8. FIG. 10 is a diagram for explaining calculation of feature amounts F9 to F12; It is a figure for demonstrating calculation of the feature-value F13. It is a figure for demonstrating calculation of the feature-value F14. It is a figure for demonstrating calculation of the feature-value F15. It is a figure which shows the relationship between the produced|generated feature-value and three-dimensional point-group data. It is a figure which shows the production|generation process of a decision tree. FIG. 4 is a diagram for explaining selection of identifiers in a decision tree; FIG. FIG. 4 is a diagram for explaining generation of a decision tree; FIG. 11 is an overall configuration diagram of a recognition device according to a second embodiment; 4 is a flowchart of recognition processing of a recognition program 48; An example of measured 3D point cloud data 4 is a flow chart of ground point extraction. It is a figure for demonstrating ground point extraction. It is a figure which shows the state which set the boundary box. 10 is a flow chart of estimation processing of a tunnel or the like (road covering object). FIG. 4 is a diagram for explaining estimation processing of a tunnel or the like (road covering object); It is a flowchart of presumed treatment of a retaining wall. It is a figure for demonstrating the estimation process of a retaining wall. FIG. 4 is a diagram showing criteria for grouping based on position and height; It is a figure for demonstrating extraction of a sidewalk. 6 is a flowchart of estimation processing; 4 is a flowchart of estimation processing by random forest;

1. 1st embodiment
1.1 Overall Configuration FIG. 1 shows a learning method for a recognition device according to an embodiment of the present invention. This allows the machine learning program to learn and generate a recognition device that recognizes the type of object represented by the three-dimensional point cloud data. As the machine learning program, a program that performs machine learning by random forest can be used, but other machine learning programs may be used.

First, the 3D point cloud data 2 of the object is obtained. This is obtained by separating each object from the measured three-dimensional point cloud data. Next, with reference to the cuboid 4 containing the three-dimensional point cloud data 2 of the object, the distribution characteristics of the point cloud of the three-dimensional point cloud data are calculated as numerical values.

A machine learning program is trained using the distribution characteristics calculated as described above. This makes it possible to generate a recognition device that recognizes the type of object based on the three-dimensional point cloud data of the object.

1.2 Hardware configuration Figure 2 shows the hardware configuration of the recognition device. A memory 32 , a display 34 , a communication circuit 36 , a hard disk 38 , a DVD-ROM drive 40 , a keyboard/mouse 42 and a recording medium drive 44 are connected to the CPU 30 . The communication circuit 36 is for connecting to the Internet. The recording medium drive 44 is for taking in the three-dimensional point cloud data recorded on the portable recording medium 52 .

The hard disk 38 stores an operating system 46 and a recognition program 48 . The recognition program 48 recognizes the type of object indicated by the given three-dimensional point group data by learning by machine learning. The recognition program 48 cooperates with the operating system 46 to achieve its functions. These programs are recorded on the DVD-ROM 50 and installed on the hard disk 38 via the DVD-ROM drive 40 .

1.3 Learning Processing FIG. 3 shows a flowchart of learning data generation processing of the recognition program 48 . In this process, learning data is generated based on three-dimensional point cloud data for each object as learning source data.

The CPU 30 acquires three-dimensional point cloud data of many objects, which are original data for learning (step S1). FIG. 4A shows an example of three-dimensional point cloud data 2 of an object. This three-dimensional point cloud data is assigned the type of the object as an attribute. For example, the object in FIG. 4A is given a type of "traffic light". As learning source data, it is preferable to prepare a large number of three-dimensional point cloud data of several types of objects.

Next, the CPU 30 generates a rectangular parallelepiped (boundary box) containing each of the three-dimensional point cloud data of these objects (step S3). In this embodiment, a rectangular parallelepiped with the minimum volume containing the three-dimensional point cloud data is generated as the boundary box, but a boundary box having a predetermined dimension larger than this may be generated.

In this embodiment, a boundary box is generated, but if a boundary box has already been assigned to the 3D point cloud data of the object, it may be used as it is. FIG. 4B shows an example of a boundary box 4 formed with respect to the 3D point cloud data 2 of the object.

In this embodiment, the boundary box 4 is generated based on the road surface (the surface defined by the traveling direction of the road and the crossing direction of the road) at the position corresponding to the object. As a result, as shown in FIG. 7C, different reference planes of the boundary box 4 are generated according to the traveling direction 5 and the transverse direction 7 of the road. Since the features are usually installed along the direction of travel of the road and the inclination in the transverse direction, the boundary box is arranged so that the bottom surface of the boundary box is parallel to the road surface. By generating 4, it is possible to perform learning that clarifies the difference depending on the type of feature.

The road surface at the position corresponding to the object can be calculated, for example, as the position near the road perpendicular line when the perpendicular line in the traveling direction of the road intersects the center of the bottom surface of the object.

After the boundary box is generated based on the absolute coordinates, the boundary box may be rotated so that the bottom surface of the boundary box is parallel to the corresponding road surface.

Next, the CPU 30 calculates the distribution characteristics of the three-dimensional point cloud data using the boundary box 4 as a reference (step S4). FIG. 5 shows an example of distribution characteristics. Here, feature quantities F1 to F16+n are calculated as distribution characteristics.

FIG. 6 schematically shows feature amounts F1 to F3. In the figure, the Z-axis is the height direction, the Y-axis is the extension direction of the road (the traveling direction of the car when measuring 3D point cloud data), and the plane at the same height as the extension direction of the road is perpendicular to the Y-axis. is the X-axis.

The feature quantity F1 is the width of the boundary box 4 in the X-axis direction (referred to as the lateral width). The feature amount F2 is the Y-axis direction width (height) of the boundary box 4 . The feature quantity F3 is the height in the Z-axis direction.

These feature amounts F1 to F3 are features for specifying horizontally long and vertically long features such as lighting and guardrails.

Calculation of the feature amount F4 will be described with reference to FIG. As shown in FIG. 7A, the boundary box 4 is divided into strips on the YZ plane at predetermined intervals along the X axis. As a result, sections D1, D2, . . . Dn having a predetermined interval width are obtained. The number of points in each interval D1, D2 . . . Dn is counted and its histogram is calculated as shown in FIG. 7B. A feature quantity F4 is its standard deviation.

The feature quantity F5 is obtained by dividing the boundary box 4 into strips on the XZ plane at predetermined intervals along the Y axis and obtaining the standard deviation of the histogram in the same manner as described above.

These feature amounts F4 and F5 are features for specifying a feature such as a lighting pole that does not change in shape, or a feature that has a lot of shape changes such as illumination.

Calculation of the feature amount F6 will be described with reference to FIG. As shown in FIG. 8, when the boundary box 4 is divided vertically into two, the ratio of the number of points included in the upper half to the number of points included in the lower half is calculated. The feature quantity F6 is this ratio (the number of points in the upper half/the number of points in the lower half).

The feature amount F6 is a feature for specifying a feature such as a tree in which the number of points greatly changes between the trunk and the leaf.

Calculation of the feature amounts F7 and F8 will be described with reference to FIG. The feature amount F7 is the ratio of the horizontal width (the wider one of the X width and the Y width) to the height of the boundary box. The feature amount F8 is the ratio of the vertical width (the narrower of the X width and the Y width) to the height of the boundary box.

The feature amounts F7 and F8 are features for specifying horizontally long and vertically long features such as lighting and guardrails.

Calculation of the feature amounts F9 to F12 will be described with reference to FIG. As shown in FIG. 10, the boundary box is divided into strips on the XY plane at predetermined intervals along the Z axis. As a result, sections with a predetermined interval width are obtained. The maximum value for each section of the horizontal width of the point cloud (the width in the wider direction of the X width and the Y width of the boundary box 4) is defined as the section horizontal width. The widest section width in all sections is set as the maximum point cloud width, and the narrowest section width is set as the minimum point cloud width. The feature quantity F9 is calculated as (maximum horizontal width of point cloud - minimum horizontal width of point cloud).

The maximum value of the vertical width of the point group in each section (the width in the narrower direction of the X width and the Y width of the boundary box 4) is defined as the section vertical width. Let the widest section vertical width be the maximum point cloud vertical width in all sections, and let the narrowest section vertical width be the minimum point cloud vertical width. The feature quantity F10 is calculated as (maximum point cloud vertical width−minimum point cloud vertical width).

The maximum value of the reflection intensity at each point is calculated for each section and taken as the section maximum reflection intensity. The number of sections in which the section maximum reflection intensity is greater than or equal to a predetermined value is counted. The feature quantity F11 is the number of this section. It should be noted that the predetermined value is preferably a value that can distinguish a highly reflective portion such as a road sign from other portions.

Calculate the point density for each section. The point density of the section with the highest point density is the feature quantity F12.

The feature quantities F9 and F10 are features for identifying features with protruding portions such as signs. The feature amount F11 is a feature for identifying a feature having a portion with high reflection intensity, such as a sign. The feature quantity F12 is a feature for distinguishing an artificial structure (lighting etc.) from a natural object (tree etc.) based on the density of points where points are concentrated.

Calculation of the feature amount F13 will be described with reference to FIG. Calculate the approximate straight line of the 3D point cloud data. A normal vector V of this approximate straight line is obtained, and an angle between the normal vector V and the Z axis is calculated. The feature amount F13 is this angle.

Calculation of the feature amount F14 will be described with reference to FIG. As shown in FIG. 12, the boundary box is divided into strips on the XY plane at predetermined intervals along the Z axis. As a result, sections with a predetermined interval width are obtained. An approximate straight line of the three-dimensional point cloud data is calculated for each section, and its normal vectors V1 to Vn are calculated. The angle between these normal vectors V1 to Vn and the Z axis is calculated. A standard deviation of angles in each interval is calculated. The feature quantity F14 is this standard deviation.

The feature amounts F13 and F14 are features for specifying a columnar feature or a planar feature by calculating a vector.

Calculation of the feature amount F15 will be described with reference to FIG. As shown in FIG. 12, the boundary box is divided into strips on the XY plane at predetermined intervals along the Z axis. A cross-sectional area of point cloud data on each XY plane is calculated. The change in the cross-sectional area on the XY plane is observed from the bottom to calculate the cumulative change amount. The feature amount F15 is this cumulative change amount.

The feature amount F15 is an index indicating how much the shape changes.

In this embodiment, in addition to the above, the matching rate when template matching (shape comparison) is performed is also used as a feature amount. Prepare 3D point cloud data as a reference for each feature to be recognized, and the degree of similarity between the original data for learning (3D point cloud data) and the reference 3D point cloud data for each feature (Concordance rate) is calculated. This match rate is the feature quantity F16 to F16+n. In this example, since n reference three-dimensional point cloud data (of features) are prepared, n features are calculated from feature amounts F16 to F16+n.

For example, templates for traffic islands, dividers, plants and snow forests, signposts/display boards, lighting poles, road reflectors, traffic lights, guardrails, utility poles, pole cones, delineators, railroad crossings, facilities/buildings, fences, etc. can be prepared and the rate of matching with these can be calculated and used.

After calculating the distribution characteristics of the learning source data (three-dimensional point cloud data of the object) as described above, the CPU 30 assigns the feature name (traffic light, etc.) of the object as an attribute (step S5). Therefore, feature amounts F1 to F16+n and feature names are generated as learning data for one piece of learning original data.

The CPU 30 repeats this process for all original learning data to generate a large amount of learning data (steps S2 and S6). FIG. 14 shows an example of learning data generated in this way. A calculated feature amount is recorded for each piece of original data for learning (three-dimensional point cloud data D1, D2, . . . Dn). In addition, since not only the original data for learning of all attributes are given, but also a large number of original data for learning are given for objects of the same attribute, a large amount of learning data based on these are also generated. .

Next, the CPU 30 performs learning based on the generated learning data. In this embodiment, machine learning by random forest is performed as described below. The flowchart is shown in FIG.

The CPU 30 randomly selects a predetermined number of three-dimensional point cloud data from the three-dimensional point cloud data D1, D2 . . . Dn shown in FIG. 14 to generate m data sets (step S11). Therefore, m data sets as shown in the lower part of FIG. 14 are generated.

Next, the CPU 30 randomly selects a predetermined number p of features to be used for creating a decision tree from the features F1 to F16+n (step S13). The CPU 30 generates a decision tree for separating and determining the object type based on the selected p features for the target data set (step S14).

A random forest technique can be used to generate the decision tree. The outline is as follows.

Based on the data set of interest, the CPU 30 generates an identifier that can appropriately separate the types of objects contained therein, using the p features selected in step S13.

For example, assume that there is three-dimensional point cloud data as shown in FIG. CPU 30 varies the threshold for each feature to consider the degree of separation.

For example, if the threshold value is set to 9 using the feature F1 and the three-dimensional point cloud data is separated into two, the values equal to or greater than the threshold value 9 are D6 and D7, and the values less than the threshold value 9 are D1 to D5. The separation is not well done for buildings D5, D6 and D7.

If the threshold is set to 9 using the feature F3 and the three-dimensional point cloud data is separated into two, the threshold 9 or more is D6, and the threshold less than 9 is D1 to D5 and D7. Again, the separation is not good for buildings D5, D6 and D7.

Using the feature F3 and setting the threshold to 5, the three-dimensional point cloud data is separated into two. is D4. The former includes only lighting and buildings, while the latter includes only signs and posts. Therefore, in the above case, the identifier "feature F3 is 5 or more" is determined as the most preferable identifier.

Specifically, for each node of the binary tree, the feature F and the threshold that minimize the entropy when the three-dimensional point cloud data is evaluated for each feature F and the threshold are determined as identifiers.

Furthermore, for D1, D3, and D4 where F3 is less than 5, the threshold is varied for each feature to find the identifiers that provide the best separation. Similarly, for D2, D5, D6 and D7 where F3 is greater than or equal to 5, find the appropriate identifiers. Thus, a decision tree as shown in FIG. 17 is generated.

The depth of the hierarchy of the decision tree is, as shown in FIG. 17, until the objects cannot be further separated by the classification name, or until the predetermined hierarchy depth is reached.

After generating the decision tree for the target data set in this way, the decision tree is generated for the next data set. The feature set used at this time is randomly selected and different from the previous one (step S13). By repeating this process (steps S12 and S15), decision trees are generated for all data sets. That is, multiple decision trees are generated.

Each decision tree makes it possible to estimate which type of object the 3D point cloud data having the feature is by giving the feature of the 3D point cloud data.

As described above, the learning of the recognition device is completed. When estimating which type of object the given three-dimensional point cloud data is, this recognition device calculates the features F1 to F16+n of the three-dimensional point cloud data, and uses them as the above decision trees. give to The type of object of the given 3D point cloud data can be estimated by taking the majority of the estimation results from each decision tree.

1.4 Miscellaneous
(1) In the above embodiment, the matching rate by template matching is used as learning data as a distribution characteristic. However, the matching rate by template matching may not be used.

(2) In the above embodiment, when determining the identifier, the threshold is varied for each feature to find the optimum threshold. However, the threshold may be determined and fixed in advance for each feature.

(3) In the above embodiment, three-dimensional point cloud data has been described. However, it can be similarly applied to two-dimensional point cloud data.

(4) In the above embodiment, the feature amount is calculated based on the three-dimensional point cloud data obtained by measurement and used as learning data. However, the feature amount may be calculated based on the three-dimensional point cloud data obtained by measurement and processed such as by changing the point density, and used as learning data. Further, instead of the measured data, the feature amount may be calculated based on three-dimensional point cloud data generated from three-dimensional CAD data or the like, and used as learning data.

(5) In the above embodiment, a feature object has been described. However, it can also be applied to three-dimensional point cloud data obtained by measuring other objects such as automobiles.

(6) In the above embodiment, the boundary box is generated based on the road surface (so that the road surface and the bottom surface of the boundary box match). However, the boundary box may be generated based on absolute coordinates. Even in this case, the distribution of the 3D point cloud data can be extracted as a feature based on the boundary box, and the recognition accuracy can be improved compared to the case where the boundary box is not used.

(7) In the above embodiment, a rectangular parallelepiped containing three-dimensional point cloud data is generated as a boundary box. However, polygonal prisms, cylinders, or other shaped three-dimensional regions may be generated as boundary boxes.

(8) The above embodiments and modifications can be implemented in combination with other embodiments as long as they do not contradict their essence.

2. Second embodiment
2.1 Overall Configuration FIG. 18 shows a recognition device according to an embodiment of the present invention. A three-dimensional point cloud data acquisition means 10 acquires three-dimensional point cloud data obtained by measuring a recognition object. The distribution characteristic calculation means 12 calculates, as a numerical value, the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the cuboid containing the three-dimensional point cloud data. The estimating means 14 estimates the object based on the calculated numerical values of the distribution characteristics.

The estimating means 14 may be formed by machine learning such as random forest, or may be formed by other ensemble learning methods. Alternatively, it may be formed by deep learning. In addition, inference may be made with a deterministic logical structure (such as a single decision tree).

2.2 Hardware Configuration The hardware configuration is the same as in the first embodiment.

2.3 Estimation Processing FIG. 19 shows a flowchart of estimation processing of the recognition program 48 . In this embodiment, a recognition device that performs machine learning mainly by random forest is used.

In the following, a process of recognizing which feature an object appearing in the measured three-dimensional point cloud data corresponds to will be described.

The CPU 30 expands the three-dimensional point group data recorded on the hard disk 38 to the memory 32 (step S21). The three-dimensional point cloud data 32 are those recorded in the portable recording medium 52 and captured in the hard disk 38 via the recording medium drive 44 . Further, in this embodiment, data acquired by a mobile mapping system (MMS) is used as the three-dimensional point group data 32 . In MMS, a laser scanner/GPS receiver is installed in an automobile or the like, and 3D shapes such as road surfaces and features can be obtained as 3D point cloud data while driving. Further, in this embodiment, the laser reflection intensity is recorded as an attribute of each point of the three-dimensional point cloud data. Furthermore, the travel locus of the automobile etc. is also recorded as data.

FIG. 20 shows an example of three-dimensional point cloud data. The shape of the surface of roads and features is data represented by three-dimensional point clouds.

Next, the CPU 30 extracts ground points from this three-dimensional point cloud data (step S22). FIG. 21 shows a flowchart of ground point extraction processing. The CPU 30 extracts ground points using a cross simulation technique (step S221). The cloth simulation method is as follows. Invert the elevation value of the 3D point cloud data. For example, if there is cross-sectional 3D point cloud data as shown in FIG. 22A, inverted 3D point cloud data as shown in FIG. 22B can be obtained.

Next, the CPU 30 simulates the reversed three-dimensional point group data as if a cloth was applied from above. The simulated cloth is shown in dashed lines in FIG. 22C. Subsequently, the CPU 30 extracts three-dimensional point cloud data with which the simulated cloth is in contact as ground points. CPU 30 then re-inverts the elevation values to obtain ground points as shown in FIG. 22D.

The ground points extracted in this way are generally accurate, but may include some features in the vicinity 60 where the features exist, as shown in FIG. 22D. Therefore, the normal direction of the line formed by each of the extracted ground points is calculated, and the portion where the normal direction is at a predetermined angle or more (for example, 30 degrees or more) with respect to the vertical direction is excluded from the ground points (step S222). ). As a result, ground points as shown in FIG. 22E can be obtained.

In this embodiment, cross simulation is used for ground point extraction, but ground points may be extracted by other methods such as the lowest point extraction method.

After extracting the ground points as described above, the CPU 30 removes the ground points from the three-dimensional point cloud data (step S23). As a result, three-dimensional point cloud data of only objects existing on the ground can be obtained.

Next, the CPU 30 organizes the three-dimensional point cloud data into objects for each cluster (step S23). The three-dimensional space is divided into grids, and if there are points in grids that are adjacent vertically, horizontally, and obliquely, they are combined into one to find the object. For this processing, for example, a technique of spatial labeling using connected components can be used. Then, a boundary box containing each object is generated. FIG. 23 shows an example of found objects and boundary boxes.

Next, the CPU 30 estimates the type of each object enclosed by the boundary box. First, culverts, sheds, shelters, and tunnels are estimated (step S24). FIG. 24 shows a flow chart thereof.

The CPU 30 extracts three-dimensional point cloud data covering the roadway (having the same XY coordinates as the three-dimensional point cloud data of the roadway but different Z coordinate values) (step S241). 29, the three-dimensional point cloud data of the roadway can be extracted as a plane (a point cloud whose height difference is equal to or less than a predetermined value) C including the travel locus D (operator uses a mouse or the like). (It may be extracted to a manually specified area by ).

Next, the CPU 30 sets cross sections perpendicular to the travel locus at predetermined intervals. For example, as shown in FIG. 25A, cross sections are set at points c1, c2, and c3. As shown in FIG. 25B, the CPU 30 obtains cross-sections of the object in each cross-section (step S243) and compares the matching degrees of these shapes. If the degree of matching of the shape of the object in each cross section is a predetermined value or more, and it can be determined that the same shape is continuous, this object is presumed to be "culvert, shed, shelter, tunnel" ( Steps S244, S245).

The above processing is performed for all the objects extracted in step S241 (steps S242 and S246). As described above, it is possible to estimate a tunnel or the like.

Next, the CPU 30 estimates the retaining wall (step S25). FIG. 26 shows the flowchart. The CPU 30 extracts an object (three-dimensional point cloud data) existing at the edge of the roadway (continuous to the edge of the roadway and extending in the direction along the travel locus) (step S251).

Next, the CPU 30 calculates an approximate plane of the extracted object and roadway by the method of least squares while excluding outliers by RANSAC (step S253). Next, CPU 30 calculates the angle formed by the approximate plane of the extracted object and the approximate plane of the roadway, and determines whether or not this angle is equal to or greater than a predetermined value (step S254). In this embodiment, as shown in FIG. 27, the angle between the normal vector A of the approximation plane of the roadway and the normal vector B of the approximation plane of the extraction object is calculated.

If the angle is greater than or equal to the predetermined value, CPU 30 estimates that the extracted object is a "retaining wall" (step S255).

The above processing is performed for all the objects extracted in step S251 (steps S252 and S256). Retaining walls can be estimated in the manner described above.

It should be noted that there are cases where the estimation of tunnels and retaining walls cannot be performed accurately. Therefore, the object estimated by the CPU 30 may be presented as a candidate on the display, and the operator may perform a confirmation input to determine the candidate.

Next, the CPU 30 estimates the types of the objects extracted in step S23 based on the positions and heights of the objects whose types have not yet been estimated (step S26). FIG. 28 shows the judgment criteria. In this example, it is presumed that the objects are divided into groups AE.

Traffic islands and dividers can be extracted as group A, as objects existing near the center of the roadway. Whether or not it exists near the center of the roadway can be determined, for example, by whether the XY center coordinates of the boundary box are located within 1/6 of the width of the roadway from the centerline of the roadway.

Pedestrian bridges can be extracted as group B as objects crossing over the roadway from the vicinity of the sidewalk. As shown in FIG. 29, the sidewalk can be extracted by the CPU 30 as a flat area W adjoining the roadway C with a level difference of a predetermined value or more in the cross section perpendicular to the travel locus of the 3D point cloud data. Alternatively, the operator may designate and extract with a mouse or the like while viewing the three-dimensional point cloud data.

Since only pedestrian bridges are included in group B, it can be estimated that the objects extracted as group B are pedestrian bridges.

Facilities/buildings can be extracted as group C objects that are separated from the road by a predetermined distance or more and whose boundary box volume is larger than a predetermined value.

Since group C includes only facilities/buildings, it can be assumed that the objects extracted as group C are facilities/buildings.

Guardrails, pole cones, and delineators can be extracted as group D as objects of a predetermined height or less existing near the roadway. Objects located outside the center of the roadway or within a predetermined distance (eg, 1 m) from the edge of the roadway and whose boundary box is lower than a predetermined height (eg, 1 m) can be extracted.

Plants and snow protection forests, signposts/display boards, lighting poles, road reflectors, traffic lights, utility poles, railroad crossings, and fences are group E, and can be extracted as objects existing near the roadway (excluding group D objects). can.

The CPU 30 classifies the objects into groups A to E according to the above criteria. As a result, pedestrian bridges (group B) and facilities/buildings (group C) can be estimated.

Next, the CPU 30 performs machine learning estimation for each of the objects of the groups A, D, and E (step S27). In this embodiment, a random forest is used for estimation.

Group A objects include islands and dividers. Therefore, in the processing shown in the first embodiment, the estimation processing is performed using a plurality of decision trees learned by a large number of objects of traffic islands and dividers.

First, the CPU 30 calculates the distribution characteristics of the three-dimensional point cloud data for the objects extracted as the group A based on the already set boundary box (step S274). In this embodiment, feature quantities F1 to F16+n as shown in FIG. 5 are calculated as distribution characteristics. The calculation process is as described in the first embodiment.

Next, the CPU 30 estimates the type of the object based on the calculated feature amounts F1 to F16+n (step S275). FIG. 31 shows a flowchart of estimation processing.

Based on each of the plurality of decision trees generated by learning, the CPU 30 gives necessary feature amounts and obtains an object type estimation result for each decision tree (steps S281 to S283).

After obtaining a plurality of estimation results in this manner, CPU 30 obtains the final estimation result by majority vote (step S284). Here, either a "traffic island" or a "separator" is obtained. The CPU 30 assigns the obtained estimation result as an attribute of the object.

An estimation result is obtained for one object as described above. The CPU 30 repeatedly performs processing on all objects belonging to group A to obtain an estimation result.

When the CPU 30 obtains estimation results for all objects in group A, the CPU 30 performs similar processing to estimate other groups. When estimating the objects of group D, a decision tree learned from the objects of "guardrail", "pole cone" and "sight guide" is used. When estimating the objects of group E, the objects of "planting and snow protection forest", "signpost/display board", "lighting pole", "road reflector", "traffic light", "telephone pole", "railroad crossing", and "fence" are learned. A decision tree is used.

As described above, objects can be extracted from the measured three-dimensional point cloud data and their types can be given as attributes.

2.4 Miscellaneous
(1) In the above embodiment, the matching rate by template matching is used as learning data as a distribution characteristic. However, the matching rate by template matching may not be used.

(2) In the above embodiment, when determining the identifier, the threshold is varied for each feature to find the optimum threshold. However, the threshold may be determined and fixed in advance for each feature.

(3) In the above embodiment, three-dimensional point cloud data has been described. However, it can be similarly applied to two-dimensional point cloud data.

(4) In the above embodiment, in step S284, the estimation results are integrated by majority vote. However, weighting may be applied to guide the final result. For example, the final result may be derived by giving different weights to decision tree inference results that include template matching features in their identifiers and decision tree inference results that do not (for example, the weight of the former is increased).

(6) In the above embodiment, three-dimensional point cloud data measured by MMS is used. However, three-dimensional point cloud data measured by a stationary laser scanner or the like may also be used.

(7) In the above embodiment, the traveling direction of the road is determined by the trajectory at the time of measurement. However, the determination may be made based on the extension direction of the portion extracted as the road, the direction of the recognized white line, or the like. Alternatively, it may be specified by the operator using a mouse or the like while checking the screen.

(8) In the above embodiment, the estimation means is constructed by machine learning based on random forest. However, the estimation means may be constructed by machine learning using other teacher data. Alternatively, the estimation means may be constructed using deep learning or logical reasoning.

(9) In the above embodiment, one computer constitutes the recognition device. However, it may be constructed as a server device that receives three-dimensional point cloud data from a terminal device and performs estimation.

(10) In the above embodiment, a feature object has been described. However, it can also be applied to three-dimensional point cloud data obtained by measuring other objects such as automobiles.

(11) In the above embodiment, a rectangular parallelepiped containing three-dimensional point cloud data is generated as a boundary box. However, other three-dimensional areas such as polygonal prisms and cylinders may be generated as boundary boxes.

(12) In the above embodiment, reflection intensity is given as an attribute. However, this may not be granted. Alternatively, RGB color information captured by a camera or the like may be given as attributes for learning, and estimation may be performed in consideration of these attributes as well. The attributes may be used for learning and estimation.

(13) The above embodiments and modifications can be implemented in combination with other embodiments as long as they do not contradict their essence.

Claims (23)

A recognition device for distinguishing and recognizing objects,
a data acquisition means for acquiring three-dimensional point cloud data obtained by measuring an object;
a region generating means for defining a three-dimensional region containing the three-dimensional point cloud data of the object;
Distribution characteristic calculation means for calculating numerically the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the three-dimensional region containing the three-dimensional point cloud data of the object;
an estimating means for estimating an object based on the calculated numerical value of the distribution characteristic;
In a recognizer comprising
The estimating means groups the three-dimensional point cloud data representing the object based on whether or not the three-dimensional point cloud data representing the object is at least in the central portion of the roadway, and divides the three-dimensional point cloud data representing the object into groups. Machine learning or deep learning is performed to form a learned model, and the object is estimated using the learned model corresponding to the group to which the three-dimensional point cloud data representing the object belongs. Device. A recognition program for realizing a recognition device for distinguishing and recognizing objects by a computer, the computer comprising:
a data acquisition means for acquiring three-dimensional point cloud data obtained by measuring an object;
a region generating means for defining a three-dimensional region containing the three-dimensional point cloud data of the object;
Distribution characteristic calculation means for calculating numerically the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the three-dimensional region containing the three-dimensional point cloud data of the object;
In a recognition program for functioning as an estimation means for estimating an object based on the calculated numerical value of the distribution characteristics,
The estimating means groups the three-dimensional point cloud data representing the object based on whether or not the three-dimensional point cloud data representing the object is at least in the central portion of the roadway, and divides the three-dimensional point cloud data representing the object into groups. Machine learning or deep learning is performed to form a learned model, and the object is estimated using the learned model corresponding to the group to which the three-dimensional point cloud data representing the object belongs. program. A recognition device for distinguishing and recognizing objects,
a data acquisition means for acquiring three-dimensional point cloud data obtained by measuring an object while traveling by a mobile matching system;
a region generating means for defining a three-dimensional region containing the three-dimensional point cloud data of the object;
Distribution characteristic calculation means for calculating numerically the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the three-dimensional region containing the three-dimensional point cloud data of the object;
an estimating means for estimating an object based on the calculated numerical value of the distribution characteristic;
In a recognizer comprising
The distribution characteristic calculation means also calculates a match rate by template matching using at least the shape of the three-dimensional point cloud data as a distribution characteristic,
The estimating means groups the three-dimensional point cloud data representing the object based on whether or not the three-dimensional point cloud data representing the object is at least in the central portion of the roadway, and divides the three-dimensional point cloud data representing the object into groups. Machine learning or deep learning is performed to form a learned model, and the object is estimated using the learned model corresponding to the group to which the three-dimensional point cloud data representing the object belongs. Device. A recognition program for realizing a recognition device for distinguishing and recognizing objects by a computer, the computer comprising:
a data acquisition means for acquiring three-dimensional point cloud data obtained by measuring an object while traveling by a mobile matching system;
a region generating means for defining a three-dimensional region containing the three-dimensional point cloud data of the object;
Distribution characteristic calculation means for calculating numerically the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the three-dimensional region containing the three-dimensional point cloud data of the object;
In a recognition program for functioning as an estimation means for estimating an object based on the calculated numerical value of the distribution characteristics,
The distribution characteristic calculation means also calculates a match rate by template matching using at least the shape of the three-dimensional point cloud data as a distribution characteristic,
The estimating means groups the three-dimensional point cloud data representing the object based on whether or not the three-dimensional point cloud data representing the object is at least in the central portion of the roadway, and divides the three-dimensional point cloud data representing the object into groups. Machine learning or deep learning is performed to form a learned model, and the object is estimated using the learned model corresponding to the group to which the three-dimensional point cloud data representing the object belongs. program. A recognition device for distinguishing and recognizing objects existing near a road,
a data acquisition means for acquiring three-dimensional point cloud data obtained by measuring an object;
a region generating means for defining a three-dimensional region containing the three-dimensional point cloud data of the object;
Distribution characteristic calculation means for calculating numerically the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the three-dimensional region containing the three-dimensional point cloud data of the object;
an estimating means for estimating an object based on the calculated numerical value of the distribution characteristic;
In a recognizer comprising
The area generating means defines a rectangular parallelepiped along the traveling direction of the road as a three-dimensional area,
The estimating means groups the three-dimensional point cloud data representing the object based on whether or not the three-dimensional point cloud data representing the object is at least in the central portion of the roadway, and divides the three-dimensional point cloud data representing the object into groups. Machine learning or deep learning is performed to form a learned model, and the object is estimated using the learned model corresponding to the group to which the three-dimensional point cloud data representing the object belongs. Device. A recognition program for realizing, by a computer, a recognition device for distinguishing and recognizing objects existing near a road, the computer comprising:
a data acquisition means for acquiring three-dimensional point cloud data obtained by measuring an object;
a region generating means for defining a three-dimensional region containing the three-dimensional point cloud data of the object;
Distribution characteristic calculation means for calculating numerically the distribution characteristic of the point cloud in the three-dimensional point cloud data with reference to the three-dimensional region containing the three-dimensional point cloud data of the object;
In a recognition program for functioning as an estimation means for estimating an object based on the calculated numerical value of the distribution characteristics,
The area generating means defines a rectangular parallelepiped along the traveling direction of the road as a three-dimensional area,
The estimating means groups the three-dimensional point cloud data representing the object based on whether or not the three-dimensional point cloud data representing the object is at least in the central portion of the roadway, and divides the three-dimensional point cloud data representing the object into groups. Machine learning or deep learning is performed to form a learned model, and the object is estimated using the learned model corresponding to the group to which the three-dimensional point cloud data representing the object belongs. program. In the device or program according to any one of claims 1 to 6,
The three-dimensional point cloud data is obtained by measuring roads and features,
A device or program, wherein the area generating means defines a three-dimensional area including a feature with reference to a road surface corresponding to the feature. In the device or program according to any one of claims 1 to 7,
The apparatus or program, wherein the distribution characteristic calculation means calculates the X-direction length, Y-direction length, or Z-direction length of the three-dimensional area as the distribution characteristic. In the device or program according to any one of claims 1 to 8,
The distribution characteristic calculating means calculates, as the distribution characteristic, the distribution of the number of points included in each section when the three-dimensional area is divided into strips at least along the X-axis or the Y-axis, or program. In the device or program according to any one of claims 1 to 9,
The apparatus or program, wherein the distribution characteristic calculation means divides at least the three-dimensional area into upper and lower parts, and calculates a ratio of the number of points included in the upper part and the lower part as the distribution characteristic. In the device or program according to any one of claims 1 to 10,
The apparatus or program, wherein the distribution characteristic calculating means calculates, as the distribution characteristic, at least the ratio of the length in the X direction or the length in the Y direction and the length in the Z direction of the three-dimensional area. In the device or program according to any one of claims 1 to 11,
The distribution characteristic calculating means calculates, as the distribution characteristic, at least the amount of change in the X-direction width or the Y-direction width of the three-dimensional point cloud data at positions at predetermined intervals in the Z direction of the three-dimensional area. or program. In the device or program according to any one of claims 1 to 12,
The apparatus or program, wherein the distribution characteristic calculation means calculates, as the distribution characteristic, the density of points in a section having the largest number of points when the three-dimensional area is divided into strips at least along the Z axis. In the device or program according to any one of claims 1 to 13,
The apparatus or program, wherein the distribution characteristic calculating means calculates, as the distribution characteristic, at least the inclination of the normal vector of the entire three-dimensional point cloud data with respect to the Z-axis. In the device or program according to any one of claims 1 to 14,
The distribution characteristic calculating means calculates, as the distribution characteristic, a variation in the inclination of the normal vector of the point cloud data of each section with respect to the Z axis when the three-dimensional area is divided into strips at least along the Z axis. A device or program that In the device or program according to any one of claims 1 to 15,
The apparatus or program, wherein the distribution characteristic calculating means calculates, as the distribution characteristic, an amount of cumulative change in the area of the point cloud data in a cross section obtained by dividing the three-dimensional region into strips at least along the Z axis. In the device or program according to any one of claims 1, 2, 5 to 16,
The apparatus or program, wherein the distribution characteristic calculation means calculates at least a matching rate by template matching as the distribution characteristic. In the device or program according to any one of claims 1 to 17,
A device or program, wherein the estimation means performs estimation by performing machine learning based on the distribution characteristics. In the device or program according to any one of claims 3 to 18,
The estimating means groups the 3D point cloud data based on the position where the 3D point cloud data exists, and performs machine learning or deep learning for each group to estimate the target object. A device or program that does In the device or program according to any one of claims 1 to 19,
The apparatus or program, wherein the estimation means estimates the object by considering reflection intensity and/or color data at each point. A method for producing, by a computer, a recognition device for distinguishing and identifying an object based on three-dimensional point cloud data obtained by measuring the object,
The computer forms a three-dimensional area containing three-dimensional point cloud data obtained by measuring the learning target,
The computer calculates the distribution characteristics of the point cloud in the three-dimensional point cloud data as numerical values based on the three-dimensional region, and provides the calculated numerical values of the distribution characteristics as learning data to the device that realizes the above-described recognition processing by learning. In the method of producing a recognition device by learning by
The three-dimensional point cloud data representing the target object is grouped based on whether or not the three-dimensional point cloud data representing the target object is at least in the central portion of the roadway, and the learning is performed for each group. A production method of a recognition device, wherein a learning model is formed for each group and a device for realizing the recognition processing is produced. A method for producing, by a computer, a recognition device for distinguishing and identifying an object based on three-dimensional point cloud data obtained by measuring the object,
The computer forms a three-dimensional area containing the three-dimensional point cloud data obtained by measuring the learning object while traveling by the mobile matching system,
The computer calculates the distribution characteristics of the point cloud in the three-dimensional point cloud data as numerical values based on the three-dimensional region, and provides the calculated numerical values of the distribution characteristics as learning data to the device that realizes the above-described recognition processing by learning. In the method of producing a recognition device by learning by
The distribution characteristics are calculated including the matching rate by template matching using the shape of the three-dimensional point cloud data.
The three-dimensional point cloud data representing the target object is grouped based on whether or not the three-dimensional point cloud data representing the target object is at least in the central portion of the roadway, and the learning is performed for each group. A production method of a recognition device, wherein a learning model is formed for each group and a device for realizing the recognition processing is produced. A method for producing, by a computer, a recognition device for distinguishing and identifying objects based on three-dimensional point cloud data obtained by measuring objects existing near a road, comprising:
The computer forms a three-dimensional area containing three-dimensional point cloud data obtained by measuring the learning target,
The computer calculates the distribution characteristics of the point cloud in the three-dimensional point cloud data as numerical values based on the three-dimensional region, and provides the calculated numerical values of the distribution characteristics as learning data to the device that realizes the above-described recognition processing by learning. In the method of producing a recognition device by learning by
Defined as a rectangular parallelepiped along the traveling direction of the road as the three-dimensional area,
The three-dimensional point cloud data representing the target object is grouped based on whether or not the three-dimensional point cloud data representing the target object is at least in the central portion of the roadway, and the learning is performed for each group. A production method of a recognition device, wherein a learning model is formed for each group and a device for realizing the recognition processing is produced.

Images