JP6826023B2 - Target identification device, program and method for identifying a target from a point cloud - Google Patents

Description

The present invention relates to a technique for identifying a target from a point cloud (a set of point data) that can include target information acquired by a distance measuring sensor such as a LiDAR (Light Detection And Ranging) device.

A LiDAR device is a type of ranging sensor that emits a laser pulse outside the visible spectrum, measures the time until the pulse is reflected by the target and returns, and measures the distance to the target.

Currently, this LiDAR device is used to scan 360 degrees around 360 degrees, for example, hundreds of times per second, and analyze a "point cloud," which is a collection of enormous three-dimensional point data generated by the returned laser pulse. By doing so, it is possible to detect almost all objects in the surrounding area. Therefore, the application of LiDAR devices is rapidly advancing at present as a main technology for realizing advanced "visual" functions in autonomous vehicles, drones, various robots, and the like.

Here, as a very important conventional technique for point cloud analysis, for example, in Patent Document 1, acquisition is performed using an unsupervised classification method using height components at points in xy plane tiles that do not overlap each other. A technology for detecting and removing ground (road surface, ground, etc.) and buildings from the point cloud is disclosed.

The technique then uses a supervised classification technique to detect objects that exist in the vertical direction. Furthermore, from the object cloud after division, the geometric shape, the height from the road surface, the horizontal distance to the center line of the road, the density, the strength, the angle with the normal direction, and the flatness. Is extracted and the type class of the target object is specified.

Cited Document 1 further, as a merit of this disclosed technique, reduces the number of points to be subjected to the human labeling process required for the learning process by removing the ground and the building, and further, Under certain circumstances, objects can be classified with high accuracy.

In addition, Non-Patent Document 1 discloses a technique for classifying an object into a ground, a low foreground, a high foreground, and a sparse area of points. Of these, the foreground point cloud with a low height is converted into a distance image by calculating the distance between the points constituting the target and the rating surface. In addition, the rating surface is determined using principal component analysis (PCA) and is used to calculate the vertical section of the target point cloud.

Furthermore, a convolutional neural network (CNN, Convolutional Neural Network) trained with a distance image is used for the classification of the target, and the labels of the classification result based on the appearance are automobiles, pedestrians, the front of the building, and the road. Adopting a group.

U.S. Patent Application Publication No. 2016/01549999

A. Borcs, B. Nagy and C. Benedek, "Instant Object Detection in Lidar Point Clouds", IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 7, July 2017, pp. 992-996

In the technique disclosed in Patent Document 1 described above, a large number of parameters such as the size of horizontal tiles are fixed and set in advance when designating an area suitable for feature extraction by detecting the ground and analyzing a distance image. Must. That is, in this technique, it is a major premise that the ground plane is divided with high accuracy or each object is completely separated.

However, in general, the density of the point cloud acquired from the LiDAR device varies depending on the distance from the sensor position of the device. Therefore, the value of the preset parameter in the technique of Patent Document 1 is a certain distance range. It can be suitable only for the analysis of the point cloud. Therefore, the trained classification model also has no robustness such that it can be applied to a different environment such as an urban area different from the training data.

On the other hand, the technique disclosed in Non-Patent Document 1 described above uses a vertical cross section of a classification target obtained from a point cloud of one frame for analysis without considering the sensor position of the LiDAR device. Therefore, it is highly possible that the projected image has some loss of apparent data of the object as seen from the LiDAR device side.

For example, when the target vehicle faces the sensor of the LiDAR device, the points generated by the reflection are gathered in the front of the vehicle. However, since the projection plane of the points is set on the side surface of the automobile, only a few points are projected. Therefore, the distance values must be divided in ascending order and projected so that the points do not overlap each other. Here, a window representing a point projected in the projected image is used, but even in the technique of Non-Patent Document 1, the size of this window is fixedly set in advance. As a result, there is a possibility that the set windows will overlap.

For example, in a projected image, target points close to the sensor position of the LiDAR device are densely packed, and windows are superimposed so that distance values (pixel values) tend to overlap. On the other hand, the target points far from the sensor position are sparse in the projected image and are too far apart from each other, and as a result, the uniformity and smoothness of the projected image may be deteriorated.

In addition, the target classification process performed on the point cloud having such a large variation in density requires higher processing accuracy in order to accurately evaluate the target surface from the point distribution.

Therefore, an object of the present invention is to provide an object identification device, a program, and a method capable of generating a distance image having a pixel value distribution more suitable for object identification.

According to the present invention, it is an object identification device that identifies an object from a point cloud that can include information on the object acquired by a distance measuring sensor.
A point cloud classification means for classifying the point cloud into a target point cloud which is a set of point data adjacent to each other based on the distance between the point data.
A distance image generated on the projected plane on which the target point data included in the target point group is projected, and is a size that is a monotonically increasing function of at least the distance between the projected plane and the distance measuring sensor at the projected position. A distance image generation means for generating a distance image in which a surface patch having the above surface patch is set and the pixel value in the surface patch is determined based on the distance between the target point data related to the surface patch and the projection surface .
An object identification device having an object identification means for identifying an object included in the generated distance image by using a predetermined classifier is provided.

In the object identification device according to the present invention, the distance image generating means is a vertical surface perpendicular to the vertical surface including the reference point data which is the target point data closest to the distance measuring sensor in the target point group and the distance measuring sensor. Therefore, it is also preferable to generate the distance image on the projection plane which is the vertical plane including the reference point data.

Further, it is also preferable that the distance image generating means sets a surface patch having a size that becomes a monotonically increasing function with respect to the distance between the projection surface and the distance measuring sensor and the resolution related to the distance measuring sensor.

Further, it is also preferable that the distance image generating means calculates the distance between the target point data related to the surface patch and the projected surface based on the information related to the distance and direction from the distance measuring sensor in the target point data.

Further, as one embodiment of the object identification device according to the present invention, the object identification device extracts a smooth point group satisfying a predetermined smoothing condition from the point group, and creates a plane perpendicular to the vertical direction from the smooth point group. It further has a ground removing means for extracting a constituent ground point group and removing the point cloud portion in the vicinity of the ground point group and the ground point group from the smooth point group.
It is also preferable that the point cloud classification means classifies the point cloud processed by the ground removing means into a target point cloud.

Further, the object identification device further has a smoothing processing means for smoothing the pixel value of the generated distance image, and the object identifying means includes an object included in the distance image after the smoothing process. It is also preferable to specify.

Further, in the object identification device according to the present invention, it is also preferable that the distance measuring sensor is a LiDAR device.

According to the present invention, it is also a program for operating a computer mounted on a device for identifying an object from a point cloud that can include information on the object acquired by a distance measuring sensor.
A point cloud classification means for classifying the point cloud into a target point cloud which is a set of point data adjacent to each other based on the distance between the point data.
A distance image generated on the projected plane on which the target point data included in the target point group is projected, and is a size that is a monotonically increasing function of at least the distance between the projected plane and the distance measuring sensor at the projected position. A distance image generation means for generating a distance image in which a surface patch having the above surface patch is set and the pixel value in the surface patch is determined based on the distance between the target point data related to the surface patch and the projection surface .
An object identification program that causes a computer to function as an object identification means for identifying an object included in the generated distance image by using a predetermined classifier is provided.

According to the present invention, further, the object identification method in the computer mounted on the device for identifying the object from the point cloud that can include the information of the object acquired by the distance measuring sensor.
A step of dividing the point cloud into a target point cloud, which is a set of point data adjacent to each other, based on the distance between the point data.
A distance image generated on the projected plane on which the target point data included in the target point group is projected, and is a size that is a monotonically increasing function of at least the distance between the projected plane and the distance measuring sensor at the projected position. A step of setting a surface patch having the above, and generating a distance image in which the pixel values in the surface patch are determined based on the distance between the target point data related to the surface patch and the projection surface .
An object identification method is provided that includes a step of identifying an object included in the generated distance image using a predetermined classifier.

According to the object identification device, program and method of the present invention, it is possible to generate a distance image having a pixel value distribution more suitable for identifying an object.

It is a schematic diagram which shows one Embodiment of the object identification system including the object identification apparatus by this invention. It is a schematic diagram which showed an Example of the processing in the ground removal part and the point cloud division part. It is a schematic diagram for demonstrating one Embodiment of the projection surface generation, surface patch setting processing which concerns on this invention. It is a schematic diagram for demonstrating one Embodiment of the surface patch setting process which concerns on this invention. It is a schematic diagram for demonstrating one Example of the distance image generated by the distance image generation part. It is a schematic diagram which shows the example of the distance image generated by applying the surface patch which concerns on this invention, and the comparative example of the distance image generated by using the conventional fixed size surface patch. It is a flowchart for demonstrating the outline in one Embodiment of the processing in a distance image generation unit and a smoothing processing unit.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a schematic diagram showing an embodiment of an object identification system including an object identification device according to the present invention.

The target identification system of the present embodiment shown in FIG. 1 is
(A) LiDAR2 installed on a moving body such as a car traveling on the road,
(B) Includes a target identification device 1 that is communicated with LiDAR 2 via a wired (cable or the like) or wireless (wireless LAN, short-range wireless communication, wireless operator access network, etc.).

Of these, LiDAR2 is a ranging sensor that emits a laser pulse outside the visible spectrum, measures the time until the pulse is reflected by the target and returns, and measures the distance to the target. Light Detection And Ranging) device. Here, the measurement targets are radiated from surrounding vehicles such as automobiles and bicycles, people such as pedestrians and runners, animals, grounds such as road surfaces and ground, buildings, transportation facilities, etc. Various things are applicable as long as they reflect electromagnetic waves.

Specifically, LiDAR2 rotates a head equipped with a sensor and outputs point data including reflection intensity (or a three-dimensional distance calculated from it) at each angular position (coordinates) around the entire circumference (0 to 359.9 °). It may be a device that outputs one frame (one file) for each lap.

Here, in the vertical direction, for example, the head radiates 10 to 20 lasers in a range of plus or minus 10 ° to 20 ° to measure each reflected wave, and acquires point data with an angular resolution α. Can be done. By the way, this number of lasers defines the angular resolution α. On the other hand, in the horizontal direction, point data may be acquired with an angular resolution β of, for example, 0.1 to 0.5 ° while rotating at a speed of, for example, 200 to 1500 rpm.

Further, the object identification device 1 is a device according to the present invention that identifies an object captured by LiDAR2. Specifically, a "point cloud", which is a point cloud (a set of point data) that can contain target information, is acquired from LiDAR2, and this "point cloud" is analyzed to generate a "distance image". The target included in the generated "distance image" is specified.

Specifically, the target identification device 1 uses the acquired "point cloud" as its feature (A) as a "target point cloud" which is a set of points adjacent to each other based on the distance between points (point data). Point cloud (point cloud) division 112 that divides into (target point cloud),
(B) The target point (target point data) included in the "target point cloud" is a projected "distance image", and is a monotonically increasing function of the distance from the projected position to at least the sensor position of LiDAR2. A "face patch" having a size is set, and a "distance image" is generated in which the pixel values in the "face patch" are determined based on the distance between the target point related to the "face patch" and (the distance image). Distance image generation unit 113 and
(C) It has an object identification unit 115 that identifies an object included in the generated "distance image" by using a predetermined classifier.

As described above, the target identification device 1 employs a distance-adaptive "plane patch" having a size that becomes a monotonically increasing function of the distance from the sensor position of LiDAR2. That is, when this distance is small and the points tend to be dense, a "face patch" with a smaller size is adopted to more accurately reflect the distance information of each dense point, and the "face patch" is used. Avoid the situation where the pixel values related to the distance overlap with each other. On the other hand, when this distance is large and the points tend to be sparse, a "face patch" having a larger size is adopted to more reliably interpolate the distance information of each sparse point.

By adopting the distance-adaptive "plane patch" in this way, it is possible to generate a distance image having uniformity and smoothness and a pixel value distribution more suitable for identifying and classifying objects. That is, for example, the object identification / classification process can be suitably performed by this distance image without relying on the RGB image.

In general, since the point density of the point cloud changes according to the distance from the sensor position, the size of the surface patch has been fixedly set even though it is a very important parameter. On the other hand, according to the present invention, since the distance-adaptive size is adjusted and adopted, a distance image that is more suitable for later analysis is generated.

By the way, the target identification device 1 does not project the entire acquired point cloud to generate a distance image, but generates a distance image for each “target point cloud”. Therefore, a suitable distance image is realized by the later target classification, which is provided with a surface patch having a size individually corresponding to the distance between each “target point cloud” and the sensor position of LiDAR2.

As a matter of course, the destination for the target identification device according to the present invention to acquire the point cloud information is not limited to the LiDAR device. For example, it is possible to acquire and analyze point group information from a radar (Radar, Radio detecting and ranging) device or a scanning sonar (Sound navigation and ranging) device.

Naturally, the target on which such a distance measuring sensor device is mounted / installed is not limited to automobiles. For example, we want to identify objects around us (especially objects that exist on the ground such as the ground and road surface) such as flying drones, various means of transportation, pedestrians / runners, and even outdoor / indoor fixed positions. If there is a need for, various devices will be installed / installed.

Similarly, according to the functional block diagram of FIG. 1, the target identification device 1 of the present embodiment has an input / output interface 101 that enables information to be exchanged with and from LiDAR 2, a point cloud storage unit 102, and a target point cloud storage. It has a unit 103, a distance image storage unit 104, a target information storage unit 105, a keyboard display (KB / DP) 106, and a processor memory.

Here, this processor memory stores one embodiment of the target identification program according to the present invention, has a computer function, and executes the target identification process by executing the target identification program. To do. For this reason, the target identification device 1 may be a target identification dedicated device or unit, but is equipped with, for example, a personal computer (PC), a notebook or tablet computer, or a smartphone equipped with the target identification program according to the present invention. Etc. are also possible.

Further, the processor memory includes a ground removal unit 111, a point cloud division unit 112, a distance image generation unit 113 including a projection surface generation unit 113a and a patch setting unit 113b, a smoothing processing unit 114, and a target identification unit 115. , And an input / output control unit 121. It should be noted that these functional components can be regarded as the functions of the target identification program stored in the processor memory. Further, the processing flow shown by connecting the functional components of the target identification device 1 in FIG. 1 with arrows is also understood as an embodiment of the target identification method according to the present invention.

Similarly, in the embodiment shown in FIG. 1, the input / output interface 101 acquires a signal including point cloud information for one head circumference (360 °) in one frame from LiDAR2, and uses this point cloud as the point cloud storage unit 102. Output to the ground removal unit 111 while appropriately buffering. At this time, it is also preferable that the point cloud is filtered by a known method in order to remove noise and sparsely existing outliers.

[Ground removal process]
The ground removal unit 111 extracts a smoothing point cloud satisfying a predetermined smoothing condition from the input point cloud, extracts a ground point cloud forming a plane perpendicular to the vertical direction from the smoothing point cloud, and (a) Of the smooth point cloud, the point cloud portion near the ground point cloud and (b) the ground point cloud are removed from the input point cloud.

Here, the input point cloud contains information related to various objects such as roads, sidewalks, safety zones in roads, automobiles, pedestrians, bicycles (persons on the road), poles, and the like. , It does not directly include information on the boundaries and edges of the objects that separate and divide these objects. Therefore, by removing the ground (plane) that exists so as to connect these targets in many parts of the point cloud from the input point cloud, it becomes easier to carry out the subsequent target classification process. is there.

Specifically, as one embodiment, it is also preferable to carry out the following ground removal treatment.
(A) First, the point cloud segment that does not satisfy the known predetermined smoothing condition is once excluded from the input point cloud. At this point, the point cloud contains a "ground", a "smooth object continuous with the ground (such as a slope or sidewalk)", and a "smooth object".
(B) Next, using the RANSAC (RANdom SAmple Consensus) method, in the point cloud remaining in (a) above, the "ground" forming a plane perpendicular to the vertical direction (z-axis direction) is detected.

(C) Further, from the point cloud segments remaining in (a) above, the point cloud segment whose distance from the detected "ground" is equal to or greater than a predetermined threshold value is judged as a "smooth object" and excluded. As a result, "ground" and "smooth object continuous with ground" remain.
(D) Finally, the point cloud segment remaining in (c) above is removed from the input point cloud.

By the way, the process of (a) above is described in, for example, T. Rabbania, FA van den Heuvelb, and G. Vosselmanc, “Segmentation of point clouds using smoothness constraint”, ISPRS, 36 (5), 2006. It can be carried out by detecting a planar region by a region growing method using smoothing conditions. As a result, even a flat surface portion having a slightly inclined or slightly discontinuous portion such as a sidewalk can be separated from the input point cloud.

Regarding RANSAC, which is the known method of (a) above, for example, M. Fisher and R. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography”, Comm. Of the It is described in ACM, 24 (6), 1981.

[Point cloud classification processing]
Similarly, in FIG. 1, the point cloud division unit 112 is a set of points (data) adjacent to each other based on the distance between points (data) in the point cloud subjected to the ground removal process by the ground removal unit 111. Divide into target point clouds.

As one embodiment, the point cloud division unit 112 may apply a predetermined distance condition to group points in the point cloud. Here, it is also preferable that the Euclidean distance is adopted as the distance between the points.

Specifically, first, a kd tree (kd-tree, k-dimensional tree) having a cluster with an empty list is generated, and a set of points that satisfy a predetermined distance condition and are close to each other are assigned to the same cluster. The point cloud may be classified into clusters, and a set of points belonging to each cluster may be used as each divided target point cloud. Here, the kd tree is a kind of known spatial division data structure for classifying points in the k-dimensional Euclidean space.

It is also preferable that the generated target point cloud is appropriately stored in the target point cloud storage unit 103, appropriately taken out, and used for the next processing.

FIG. 2 is a schematic view showing an embodiment of the processing in the ground removal unit 111 and the point cloud division unit 112 described above.

FIG. 2A schematically shows an example of the ground removal process by the ground removal unit 111 using a point cloud distribution map. According to the figure, the ground point cloud (“ground” and “smoothing object continuous with the ground”) is subtracted from the input point cloud to generate the point cloud after the ground removal process.

Next, the target point cloud shown in FIG. 2 (B) is generated by performing the point cloud classification process using the kd tree on the point cloud after this process. Here, in FIG. 2B, the point cloud in one closed broken line is one target point cloud.

Here, for each target point cloud generated in this way, a distance image is generated as described in detail later. This is a considerable reduction in the amount of calculation compared to projecting the entire point cloud to generate a distance image.

[Distance image generation processing]
Hereinafter, an embodiment of the projection surface generation / surface patch setting process performed by the distance image generation unit 113 (FIG. 1), which is the main unit of the target identification device 1, will be described.

FIG. 3 is a schematic diagram for explaining an embodiment of the projection surface generation / surface patch setting process according to the present invention. By the way, in the figure, the xyz coordinate system for the point cloud detected by LiDAR2 and the ij coordinate system in the projection plane are set. Regarding these coordinate systems, in the present embodiment, the z-axis and the j-axis are the axes in the vertical direction, and the xy plane is the horizontal plane, but the present invention is not limited to this.

FIGS. 3 (A) and 3 (B) schematically show the relationship between the sensor position s of LiDAR2 and the generated projection surface PS. As shown in the figure, the projection plane generation unit 113a of the distance image generation unit 113 (FIG. 1) is used for each target point cloud generated by the point cloud division unit 112.
(A) Of the points included in the target point cloud, the reference point (reference point data) P, which is the target point (target point data) closest to the sensor position s (x _s , y _s , z _s ) of LiDAR2. Vertical to the vertical plane including ₀ (x ₀ , y ₀ , z ₀ ) and the sensor position s of LiDAR2, and (b) the vertical plane including this reference point P ₀ (the plane including the z-axis). Set to the projection plane PS. That is, the projection plane PS is generated according to the reference point P ₀ of the target point cloud.

Further, in the present embodiment, the patch setting unit 113b of the distance image generation unit 113 (FIG. 1) is located at a position on the projection surface PS on which each target point is projected.
(A) The distance r ₀ between the sensor position s of LiDAR2 and the projection surface PS,
(B) A surface patch SP having a size that becomes a monotonically increasing function with respect to the resolution (angle resolution α and β) of LiDAR2 (related to the output point cloud) is set.

As shown in FIG. 3C, the distance image generation unit 113 sets the surface patch SP set as described above on the projection surface PS for each projection target point P _n , thereby creating a distance image. It will generate.

Here, the size setting of the surface patch SP will be described in detail later with reference to FIGS. 4A and 4B.

Further, the patch setting unit 113b, a pixel value _{g n (i n, j n} ) in the plane patch SP the target point according to the surface patch _{SP P n (x n, y} n, z n) and the projection surface PS Determined based on the distance d _n of. For example, the pixel value g _n can be the distance d _n (g _n = d _n ). In any case, the distance d _n is an intensity value that determines the pixel values of all the pixels existing in the surface patch SP.

Regarding the calculation of this distance d _n , instead of directly calculating the Euclidean distance between the target point P _n and the projection plane PS, the information related to the distance and direction of LiDAR2 from the sensor position s at the target point P _n is used. It is also preferable to calculate based on. Specifically, from the geometric relationship shown in Fig. 3 (B), the following equation (1) d _n = R _n * cos θ _n −r ₀
Can be used to derive the distance d _n related to the pixel value g _n with a smaller amount of calculation.

In the above formula (1) wherein each R _n and theta _n, a distance and direction angle in the xy plane (horizontal plane) of the sensor position s of LiDAR2 in a subject point P _n, point cloud obtained from LiDAR2 It is a value included in the information or easily calculated from the information.

Further, the distance r ₀ between the sensor position s of LiDAR 2 and the projection surface PS is the magnitude of the perpendicular vector R ₀ from the sensor position s to the projection surface PS, and is the xy plane between the sensor position s and the reference point P _0. It is also the distance within the (horizontal plane), and the following equation (2) r ₀ = ((x ₀ −x _s ) ² + (y ₀ −y _s ) ² ) ^0.5
It can also be calculated by.

Further, the position coordinates i _n and j _n of the pixel value g _n of the plane patch SP, when the origin of the ij coordinate system provided in the xy plane (horizontal plane), the following expressions (3) i _n = R _n * Sin θ _n
(4) j _n = z _n
Can be determined using.

FIG. 4 is a schematic view for explaining an embodiment of the surface patch setting process according to the present invention. Incidentally, in the same figure as in FIG. 3, the point cloud xyz coordinate system and the ij coordinate system in the projection plane are set.

According to the embodiment of FIG. 4A, a rectangular surface patch having a width in the i-axis direction of W _w and a height in the j-axis direction of W _h on the projection surface PS (ij coordinate system). The SP is set with its center as the target point projection position. As shown in the figure, the target points usually do not have enough density to be projected on all the pixels of the generated distance image. Therefore, the surface patch SP is set to interpolate the image area between these target points.

Further, the density of the target points changes depending on the distance from the sensor position of LiDAR2 and the resolution of LiDAR2 as described above. In order to accurately deal with this, in this embodiment, as shown in FIG. 4 (B), the width W _w and the height W _h are set to the following equation (5) W _w = r ₀ * tan β.
(6) W _h = r ₀ * tan α
The surface patch SP specified in is prepared. Here, β in the above equation (5) is the angular resolution of LiDAR2 in the horizontal direction (in the xy in-plane direction), and α in the above equation (6) is the vertical direction (z-axis direction) of LiDAR2. ) Is the angular resolution.

According to the above equations (5) and (6), the surface patch SP has a higher density as the distance from the sensor position of LiDAR2 is smaller and the resolution of LiDAR2 is higher (the smaller the α value and β value). It is set to a smaller size corresponding to the target point. As a result, it is possible to avoid a situation in which the surface patches overlap each other and the distance values (pixel values) overlap. On the other hand, the larger the distance from the sensor position of LiDAR2 and the lower the resolution of LiDAR2 (the larger the α value and β value), the larger the size is set corresponding to the target point where the density decreases. As a result, sufficient interpolation processing can be performed on the pixel values.

In this way, by applying the surface patch SP that is adaptive in terms of distance and resolution, it is possible to generate a suitable distance image having uniformity and smoothness.

The size of the surface patch SP is, of course, not limited to the above equations (5) and (6) even if it is a function of the distance r ₀ or the angular resolution. Various functional forms can be adopted as long as they are at least a monotonically increasing function of distance r ₀ and angular resolution.

It is also preferable that the distance image generated for each target point cloud is appropriately stored in the distance image storage unit 104, appropriately taken out, and used for the next processing.

Returning to FIG. 1, the smoothing processing unit 114 performs smoothing processing on the pixel values of the generated distance image. In particular,
(A) All pixel values of the distance image are standardized to values within a predetermined range, adjusted to a shape suitable for input to a classifier for later target classification, and then adjusted.
(B) It is also preferable to smooth the pixel value distribution and further reduce noise by using, for example, a Gaussian filter.

FIG. 5 is a schematic diagram for explaining an embodiment of the distance image generated by the distance image generation unit 113.

In this embodiment, as shown in FIG. 5A, a target point cloud is generated as a result of capturing the back surface of a vehicle traveling in front by LiDAR2, and a distance image is generated from this target point cloud. .. Here, in this embodiment, it can be seen that the target point cloud is grasped as a front view of the point distribution seen from LiDAR2.

On the other hand, as a comparative example, FIG. 5 (B) shows a point distribution plan view (top view) when the target point cloud is viewed from above. According to FIG. 5 (B), as compared with FIG. 5 (A), the distance between the points changes greatly depending on the surface shape of the target (automobile) and the laser reflection characteristics, and the state of the point distribution is the LiDAR2 position. It is far from the appearance of the object seen from. In addition, there is a region where a plurality of points overlap at the same position, and it can be seen that the information on the target surface is considerably lost.

On the other hand, the distance image generated by projecting the target point cloud of the present embodiment can express the same target shape as the corresponding RGB image shown in FIG. 5 (C).

As described above, in this embodiment, since the projection surface PS is set to face LiDAR2 as described above, the generated distance image reflects the state of the surface on the LiDAR2 side in the target, in other words, LiDAR2. The image fully reflects the apparent information of the target and the information on the surface of the target as seen from the position. As a result, the object identification / classification process to be executed after that can be performed more accurately by using the existing classification / identification framework that has learned such appearance.

FIG. 6 is a schematic view showing an example of a distance image generated by applying the surface patch according to the present invention and a comparative example of a distance image generated by using a conventional fixed-size surface patch.

FIG. 6 shows
(A) Examples of distance images (of a car, a running person, and a standing person) generated by applying a surface patch SP having a size corresponding to a distance and a resolution according to the present invention.
(B) Generated using a conventional surface patch with a fixed size (2 (pixels) x 2 (pixels), 5 x 5, 10 x 10, 15 x 15, 20 x 20) (automobile, running) A comparative example of a distance image (of a person and a standing person) is shown.

According to FIG. 6, in the comparative example, when the small surface patch is used, the pixel value jump (gap) is conspicuous in the distance image, while when the large surface patch is used, the surface patches are superimposed. There are some spots. On the other hand, in this embodiment, it is understood that a distance image having a uniform and smooth pixel value distribution without overlapping pixel values is realized.

FIG. 7 is a flowchart for explaining an outline of the processing in the distance image generation unit 113 and the smoothing processing unit 114 in one embodiment.

(S101) Acquire the target point cloud from the point cloud division unit 112.
(S102) Among the target point clouds, the point closest to the sensor position of LiDAR2 is set as the reference point.
(S103) A vertical plane perpendicular to the vertical plane including the set reference point and the sensor position of LiDAR2, and the vertical plane including the reference point is set as the projection plane PS.

(S104) A surface patch SP having a size calculated from the distance r ₀ between the projection surface PS and the sensor position of LiDAR2 and the angular resolutions α and β of LiDAR2 is generated.
(S105) The distance between each target point and the projection plane PS is calculated.
(S106) The calculated distance value is determined as the pixel value of the pixel in the surface patch SP, and a distance image is generated.
(S107) The pixel value of the distance image is standardized to a value within a predetermined range.
(S108) A smoothing process is performed on a distance image having a standardized pixel value.

[Target identification / classification processing]
Next, an embodiment of a process of identifying / specifying an object included in the distance image will be described using the distance image generated / adjusted by the above processing.

Returning to FIG. 1, the target identification unit 115 identifies the target included in the distance image after the smoothing process by using a predetermined classifier. An example of a classifier that can be used here is a known high-speed R-CNN (Regions with Convolutional Neural Network features). In general, R-CNN is a label in which a region candidate determination unit that determines a target region candidate by FCN (Fully Convolutional Network), a feature detection unit that calculates a feature quantity from a target region candidate by CNN, and a calculated feature quantity match. Includes a label classification unit that determines.

For high-speed R-CNN, for example, S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, Proc. Of NIPS, It is explained in detail in 2015.

The target identification unit 115 first performs a learning process with a data set of a target distance image and the target correct label in advance, and generates a trained identification model of R-CNN. The target to be learned here includes, for example (in the case of application to autonomous driving technology), a moving body (automobile, pedestrian, etc.) detected while driving on a road.

The target identification unit 115 then inputs the smoothed distance image acquired from the smoothing unit 114 into the trained discriminative model. As a result, as an output, a label specifying the type of the target (for example, a car, a pedestrian, etc.) included in the target point cloud related to the input distance image is obtained.

Here, according to R-CNN, the target position in the input distance image can be individually detected as a region candidate. Therefore, the target identification unit 115 can individually detect and classify each target even when one target point cloud contains information on a plurality of targets. For example, it is possible to output a classification result (label) as a pedestrian group even for a distance image including a plurality of pedestrians close to each other.

Further, the distance image processed by the target identification unit 115 is a target shape representation based on the appearance of the target, while the R-CNN identification model of the target identification unit 115 also learns the appearance of the distance image. It has become. As a result, in the distance image input as the classification target, some inconvenient phenomena such as occlusion between the targets, loss of the target part, an ambiguous part (connected blob) in which the targets are connected, and incomplete ground removal processing occur. Even if it does occur, it is possible to carry out robust identification / classification processing for those phenomena without being affected by them.

That is, since the identification / classification process in the target identification unit 115 does not perform identification / classification based on, for example, geometric features, information on the complete shape of the target is obtained in the distance image input as the classification target. It's not what you need.

The identification / classification result (and, in some cases, the smoothed distance image itself) generated as described above is incorporated into the application program (AP) 131 mounted on the device 1 (for example, for automatic driving). The output may be provided to various control devices and information processing devices outside the device 1 via the input / output interface 101. Further, for example, it is possible to receive a display instruction from the keyboard 106 by the user and display it on the display 106.

Further, it is also preferable that the generated identification / classification result is stored in the target information storage unit 105 every time it is generated or as appropriate, and is appropriately taken out and used. In particular, the classification result (label) here can be used for the learning process of the classifier in the target identification unit 115. As a result, the burden of manual labeling can be significantly reduced.

As described in detail above, according to the present invention, uniformity and smoothness are achieved by adopting a distance-adaptive "plane patch" having a size that is a monotonically increasing function of the distance from the distance measuring sensor position. It is possible to generate a distance image having a pixel value distribution more suitable for identifying and classifying an object.

Further, instead of projecting the entire acquired point cloud (point cloud) to generate a distance image, a distance image is generated for each target point cloud. As a result, a distance image suitable for later target classification is realized, which includes a surface patch having a size individually adapted to the distance between each target point cloud and the sensor position of LiDAR2.

Incidentally, the configuration and method of the present invention can be applied as a dynamic mapping technique for realizing advanced "visual" functions in the fields of, for example, autonomous vehicles, drones and various robots.

In the various embodiments of the present invention described above, various changes, modifications and omissions in the technical idea and the scope of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Target tracking device 101 Input / output interface 102 Point cloud storage unit 103 Target point cloud storage unit 104 Distance image storage unit 105 Target information storage unit 106 Keyboard display (KB / DP)
111 Ground removal unit 112 Point cloud division unit 113 Distance image generation unit 113a Projection surface generation unit 113b Patch setting unit 114 Smoothing processing unit 115 Target identification unit 121 Input / output control unit 131 Application program (AP)
2 LiDAR

Claims (9)

An object identification device that identifies an object from a point cloud that can contain information about the object acquired by a distance measuring sensor.
A point cloud classification means for classifying the point cloud into a target point cloud which is a set of point data adjacent to each other based on the distance between the point data.
The target point data included in the target point group is a distance image generated on the projected projection surface, and is a monotonically increasing function of at least the distance between the projection surface and the distance measuring sensor at the projected position. A distance image generation means that sets a surface patch having a size and generates a distance image in which the pixel values in the surface patch are determined based on the distance between the target point data related to the surface patch and the projection surface .
An object identification device having an object identification means for identifying an object included in the generated distance image by using a predetermined classifier. The distance image generating means is a vertical plane perpendicular to the vertical plane including the reference point data which is the target point data closest to the ranging sensor in the target point group and the ranging sensor, and the reference. The object identification device according to claim 1, wherein a distance image is generated on the projection surface which is a vertical plane including point data. Claim 1 is characterized in that the distance image generation means sets a surface patch having a size that becomes a monotonically increasing function with respect to the distance between the projection surface and the distance measurement sensor and the resolution related to the distance measurement sensor. Or the target identification device according to 2. The distance image generating means is characterized in that the distance between the target point data related to the surface patch and the projected surface is calculated based on the information related to the distance and direction from the distance measuring sensor in the target point data. The target identification device according to any one of claims 1 to 3. A smooth point cloud satisfying a predetermined smoothing condition is extracted from the point cloud, a ground point cloud constituting a plane perpendicular to the vertical direction is extracted from the smooth point cloud, and the ground point cloud is included in the smooth point cloud. Further has a ground removing means for removing the point cloud portion in the vicinity of the point cloud and the ground point cloud from the point cloud.
The target identification device according to any one of claims 1 to 4, wherein the point cloud classification means classifies the point cloud processed by the ground removing means into a target point group. Further having a smoothing processing means for performing smoothing processing on the pixel value of the generated distance image,
The target identification device according to any one of claims 1 to 5, wherein the target identification means identifies a target included in the distance image after smoothing. The target identification device according to any one of claims 1 to 6, wherein the range finder is a LiDAR (Light Detection and Ranging) device. A program that activates a computer mounted on a device that identifies an object from a point cloud that can contain information about the object acquired by a distance measuring sensor.
A point cloud classification means for classifying the point cloud into a target point cloud which is a set of point data adjacent to each other based on the distance between the point data.
The target point data included in the target point group is a distance image generated on the projected projection surface, and is a monotonically increasing function of at least the distance between the projection surface and the distance measuring sensor at the projected position. A distance image generation means that sets a surface patch having a size and generates a distance image in which the pixel values in the surface patch are determined based on the distance between the target point data related to the surface patch and the projection surface .
An object identification program characterized in that a computer functions as an object identification means for identifying an object included in the generated distance image by using a predetermined classifier. It is an object identification method in a computer mounted on a device for identifying an object from a point cloud that can include information on the object acquired by a distance measuring sensor.
A step of dividing the point cloud into a target point cloud, which is a set of point data adjacent to each other, based on the distance between the point data.
The target point data included in the target point group is a distance image generated on the projected projection surface, and is a monotonically increasing function of at least the distance between the projection surface and the distance measuring sensor at the projected position. A step of setting a surface patch having a size and generating a distance image in which the pixel values in the surface patch are determined based on the distance between the target point data related to the surface patch and the projection surface .
An object identification method comprising: a step of identifying an object included in the generated distance image by using a predetermined classifier.