JP2020061049A - Point group data structure - Google Patents

Abstract

To provide a point group data structure which can easily utilize processed point group data.SOLUTION: In a data structure of point group data, predetermined information is given to a constitution point corresponding to a shadow and the constitution point corresponding to the shadow can be processed with reduction of contribution. Therefore, when the processed point group data (total point group data) are used for estimation of a self-location, etc., it is possible not to use the constitution point corresponding to the shadow and to reduce degree of use. Since the constitution point corresponding to the shadow can be changed depending on time and date and weather, it is possible to facilitate usage of the processed point group data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to point cloud data structures.

  Conventionally, a method (SLAM; Simultaneous Localization and Mapping) of simultaneously performing self-position estimation and environment map creation in a moving body is known. As a kind of such SLAM, a method using a video camera instead of a laser device (VSLAM; Visual SLAM) has been proposed (for example, refer to Patent Document 1). In the VSLAM described in Patent Document 1, the cost is reduced by using a video camera.

JP, 2014-222550, A

  In the VSLAM described in Patent Document 1, feature points are extracted from each image (frame) captured by a video camera, and self-position estimation and environment map creation are performed based on the change of the feature points between the images. To do. The point cloud data included in the environment map created in this way may be utilized for self-position estimation and the like.

  The plurality of constituent points forming the point cloud data may include constituent points that change depending on, for example, date and time or weather. In such a case, since the generation timing of the point cloud data and the timing of self-position estimation are different, there is a discrepancy between the constituent points forming the point cloud data and the features detected during self-position estimation, It becomes difficult to utilize point cloud data.

  Therefore, an object of the present invention is to provide, as an example, a point cloud data structure that makes it easy to utilize the processed point cloud data.

  In order to solve the above problems and achieve the object, a point cloud data structure according to the present invention according to claim 1 has a collecting unit that collects a plurality of point cloud data, and a plurality of points collected by the collecting unit. A data structure of point cloud data generated based on imaging information, which is used by an information processing apparatus including: a processing unit that processes group data, wherein among the constituent points forming the point cloud data, Predetermined information is added to the corresponding constituent points, so that the processing unit can process the constituent points corresponding to the shadow with a reduced contribution degree.

It is a block diagram which shows the outline of the information processing apparatus which concerns on the Example of this invention. It is a top view showing an example of a frame of imaging information which the information processor acquires. It is a top view which shows an example of the point cloud data which the said information processing apparatus produces | generates.

  Embodiments of the present invention will be described below. The point cloud data structure according to the embodiment of the present invention is used by an information processing apparatus that includes a collection unit that collects a plurality of point cloud data and a processing unit that processes a plurality of point cloud data collected by the collection unit. And the data structure of the point cloud data generated based on the imaging information. Of the constituent points that make up the point cloud data, given information is added to the constituent points corresponding to the shadows, so that the processing unit can process the constituent points corresponding to the shadows with a reduced contribution degree. Has been done.

  According to the point cloud data structure of the present embodiment as described above, predetermined information is given to the constituent points corresponding to the shadows, and the constituent points corresponding to the shadows can be processed with a reduced degree of contribution. Thus, when the processed point cloud data is used for self-position estimation or the like, it is possible to prevent the constituent points corresponding to the shadow from being used or reduce the degree of use. Since the constituent points corresponding to the shadows may change depending on the date and time and the weather, it is possible to easily utilize the processed point cloud data.

  The predetermined information preferably includes information indicating that the constituent point corresponds to the shadow. As a result, the processing unit can process the component points corresponding to the shadows while reducing the contribution degree. In addition, it is preferable that the predetermined information further includes information about a feature that generates a shadow corresponding to the constituent point. In addition, the predetermined information may include information that the configuration point has lower reliability than other configuration points. The fact that the constituent point corresponding to the shadow has lower reliability than the other constituent points also includes that the reliability of the constituent point corresponding to the shadow is 0.

  Further, the point cloud data having the above point cloud data structure may be stored in a computer-readable recording medium. By doing so, the program can be distributed as a single unit in addition to being installed in the device, and the version can be easily upgraded. Further, the point cloud data having the above point cloud data structure may be stored in the storage device.

  Hereinafter, examples of the present invention will be specifically described. As shown in FIG. 1, the information processing unit 1 of the present embodiment includes a first information processing apparatus 1A, which is a VSLAM apparatus mounted on the moving body 10, and a second information processing apparatus mounted on the external server 100. 1B and. The first information processing apparatus 1A includes an information acquisition unit 2, a generation unit 3, and an output unit 4. The moving body 10 is, for example, a vehicle, and is equipped with an imaging unit 20, a map information storage unit 30, a self-position estimation unit 40, a travel information sensor 50, and a date / time information management unit 60.

  The second information processing device 1B includes a collection unit 11 and a generation unit 12. A communication unit 101 and a storage unit main body 102 are mounted on the external server 100. The external server 100 is configured to communicate with the plurality of first information processing apparatuses 1A.

  The imaging unit 20 is, for example, a video camera, and is arranged so as to image the front side of the moving body 10 in the traveling direction. The moving image captured by the image capturing unit 20 is composed of a plurality of image capturing information (frames) and is transmitted to the information acquisition unit 2. The image capturing section may be capable of continuously capturing still images at predetermined time intervals.

  The map information storage unit 30 is composed of, for example, a hard disk or a non-volatile memory, and stores existing map information. The map information stored in the map information storage unit 30 may include at least information about the feature. Note that information about a feature is not only information about the position and size of the feature, but also information about the attribute and type of the feature (such as the distinction between residential and commercial facilities and the type of commercial facility). Shall be included.

  The self-position estimating unit 40 estimates the current position (absolute position) of the moving body 10, and may be, for example, a GPS receiving unit that receives radio waves transmitted from a plurality of GPS (Global Positioning System) satellites. . The current position estimated by the self-position estimation unit 40 is transmitted to the information acquisition unit 2 as position information indicating the image pickup position of the image pickup information by the image pickup unit 20.

  The travel information sensor 50 is a sensor for measuring the displacement amount of the moving body 10, and for example, for measuring the azimuth displacement amount of the moving body 10 and the vehicle speed pulse acquisition unit that acquires the vehicle speed pulse of the moving body 10. The gyro sensor and the acceleration sensor for acquiring the acceleration of the moving body 10. The traveling direction of the moving body 10 is estimated based on the displacement amount acquired by the travel information sensor 50. The moving direction of the moving body 10 and the mounting direction of the image capturing unit 20 on the moving body 10 determine the image capturing direction information regarding the orientation of the image capturing unit 20 during image capturing. When the imaging unit 20 is directed to the front of the moving body, the traveling direction and the imaging azimuth substantially match. Therefore, the traveling direction sensor 50 transmits the imaging direction information to the information acquisition unit 2. The travel information sensor 50 may be at least capable of estimating the traveling direction of the moving body 10.

  The date / time information management unit 60 has a calendar function and a clock function. The timepiece function is preferably a radio timepiece function that receives standard radio waves and automatically corrects errors. The date and time information is transmitted from the date and time information management unit 60 to the information acquisition unit 2. The sun altitude information and the sun direction information can be specified based on the date information and the time information. Therefore, the information acquisition unit 2 that acquires the date information and the time information can be regarded as acquiring the sun altitude information and the sun azimuth information.

  The information acquisition unit 2 functions as an imaging information acquisition unit by acquiring the imaging information from the imaging unit 20 as described above, and the position information acquisition unit by acquiring the position information from the self-position estimation unit 40 as described above. Functions as a map information acquisition unit by acquiring map information from the map information storage unit 30 and functions as an imaging direction information acquisition unit by acquiring imaging direction information from the travel information sensor 50 as described above. However, by acquiring the sun altitude information and the sun direction information from the date / time information management unit 60 as described above, the sun information processing unit 60 functions as a sun information acquisition unit.

  The generation unit 3 extracts a feature point from the imaging information acquired by the information acquisition unit 2 to generate a two-dimensional feature point group, and further generates point group data based on the plurality of feature point groups. The point cloud data generated by the generation unit 3 may be, for example, three-dimensional point cloud data. It should be noted that the extraction of the feature points may use, for example, an ORB (Oriented fast and Rotated Brief) algorithm or another algorithm.

  The output unit 4 includes a circuit for communicating with a network such as the Internet and a public line, an antenna, and the like, and communicates with the communication unit 101 of the external server 100 to generate information (point cloud data) generated by the generation unit 3. To send. As a result, information is stored in the storage unit main body 102 of the external server 100.

  The collection unit 11 of the second information processing apparatus 1B acquires the point cloud data (individual point cloud data) generated by the first information processing apparatus 1A via the communication unit 101. The collection unit 11 collects individual point cloud data by acquiring individual point cloud data from a plurality of moving bodies or acquiring individual point cloud data from one moving body a plurality of times.

  The generation unit 12 generates comprehensive point cloud data based on the plurality of individual point cloud data collected by the collection unit 11. The total point cloud data may be data obtained by superposing a plurality of individual point cloud data, or may be data indicating an average position of constituent points of the plurality of individual point cloud data.

  The comprehensive point cloud data generated in this way is used, for example, for estimating the self-position of the moving body. That is, when a feature is detected by an optical sensor (so-called LIDAR; Laser Imaging Detection and Ranging) mounted on the moving body, the self-position of the moving body may be estimated by collating with the comprehensive point cloud data.

[Identification of Shadow in First Information Processing Device]
When generating the point cloud data, the generation unit 3 may specify and extract a constituent point corresponding to a shadow among the constituent points that form the point cloud data (that is, may function as an extraction unit). . An example of the detailed procedure at this time will be described below. First, the generation unit 3 specifies the shadow generation direction based on the sun altitude information, the sun azimuth information, and the imaging azimuth information. That is, if the position of the sun and the orientation of the imaging unit 20 are determined, the direction in which an edge of a feature such as a building that extends along the vertical direction forms a shadow at the time of imaging (shadow generation direction) should be estimated. You can That is, when there are constituent points arranged in the shadow generation direction, it can be estimated that these constituent points correspond to the shadows of the edges of the feature that extend along the vertical direction.

  Whether or not the constituent points arranged in the shadow generation direction correspond to the shadow is determined based on, for example, the number of constituent points arranged in a line or the length of a straight line formed by a plurality of constituent points. do it. That is, when the number of constituent points arranged in the shadow generation direction is small, or when the length of a straight line formed by a plurality of constituent points is short, these constituent points are accidentally arranged in the shadow generation direction, There is a high possibility that it does not correspond to the shadow.

  When the generating unit 3 extracts the constituent points arranged in the shadow generation direction and determines that these constituent points correspond to the shadow, these constituent points are removed from the point cloud data.

  Here, a specific example of a method of extracting constituent points arranged in the shadow generation direction will be described with reference to FIGS. 2 and 3, the roads and features other than the target are omitted as appropriate.

  As shown in FIG. 2, when a building B exists along the road R, the building B forms a shadow S on the ground according to the position of the sun and the orientation of the imaging unit. The building B has an edge B1 extending along the vertical direction, and the shadow S has a linear boundary line S1 corresponding to the edge B1. The feature points on the boundary line S1 are extracted in the frame of FIG. 2, and the feature points on the boundary line S1 are also extracted in the frames before and after this. Therefore, when the generation unit 3 generates the point cloud data as shown in FIG. 3, the point cloud data includes the constituent points PB corresponding to the building B and the constituent points P1 to P4 corresponding to the boundary line S1. included.

  The generation unit 3 specifies the shadow generation direction L1 based on the sun altitude information, the sun direction information, and the imaging direction information. In the illustrated example, the sun is located in the southwest, and the imaging unit 20 faces north-northwest. By specifying the shadow generation direction L1, the generation unit 3 can extract the constituent points P1 to P4 arranged in this direction.

  When the generating unit 3 extracts the constituent points arranged in the shadow generation direction and determines that these constituent points correspond to the shadow, the generating unit 3 does not remove the constituent points in the point cloud data and determines the predetermined points. Information may be added as a flag. The predetermined information given to the constituent point may be, for example, information that it is a shadow or information that there is a possibility of a shadow, and may reduce the reliability of the constituent point (0 Information (including the case of setting).

  Further, when extracting the component points arranged in the shadow generation direction as described above, the generation unit 3 is based on the position information acquired from the self-position estimation unit 40 and the map information acquired from the map information storage unit 30. The shape of the shadow of the feature included in the imaging range may be estimated. The shape of the shadow formed is determined by the shape of the edge of the feature that extends along the vertical direction. For example, if the feature along the road is a natural product or a building with a complicated shape, it is difficult to form a linear shadow due to the edge extending along the vertical direction. Even if there is, it can be determined that it is not a constituent point corresponding to the shadow of this feature.

  Further, when the map information includes the height information of the feature, the generation unit 3 may use this height information. That is, not only the shadow generation direction but also the length of the boundary line of the shadow can be estimated based on the height information, the sun altitude information, the sun direction information, and the imaging direction information. Therefore, even if the constituent points are arranged along the shadow generation direction, it is possible to determine that the region exceeding the length of the boundary does not correspond to the shadow.

  In the above example, the constituent points corresponding to the shadow of the edge of the feature extending along the vertical direction are extracted, but the constituent points corresponding to other shadows may be extracted. For example, the constituent points corresponding to the shadow may be extracted based on the complexity of the shape represented by the plurality of constituent points. As such a method, a method of estimating a fractal dimension to identify a shadow formed by vegetation leaves is exemplified.

  Further, the generation unit 3 may extract a feature point corresponding to a shadow from the feature points of each frame of the imaging information before generating the point cloud data by VSLAM. At this time, the generation unit 3 may generate the point cloud data by removing the characteristic points corresponding to the shadow. Further, information indicating that feature points have been removed may be added to the point cloud data.

[Data structure of point cloud data]
When the generation unit 3 adds predetermined information as a flag to the constituent points corresponding to the shadow as described above, this point cloud data has a data structure as described in detail below. The predetermined information given to the constituent point may be the information that it is a shadow as described above, the information that there is a possibility of a shadow, or the reliability of the constituent point. It may be information that the value is lowered (including setting it to 0).

  When generating the total point cloud data based on the plurality of individual point cloud data collected by the collection unit 11, the generation unit (processing unit) 12 of the second information processing apparatus 1B is provided with information (that is, shaded). It can be processed with a lower contribution of the (corresponding) constituent points. For example, when generating a total point cloud data by superimposing a plurality of individual point cloud data, when the position of a flagged constituent point included in a certain point cloud data is not located in another point cloud data , It can be treated as if there is no constituent point at this position.

  In addition, the information given to the constituent points is information about the features that generate shadows corresponding to the constituent points (for example, information that the constituent points correspond to the shadows generated by the building and vegetation information). (Information that corresponds to the generated shadow) may be included. Table 1 shows an example of the data structure of such point cloud data.

  That is, an ID is assigned to each constituent point, and position information, shadow flag information (that is, whether or not information is added), and a feature that generates a shadow corresponding to each constituent point are assigned to each constituent point. And information is added. Although the position information is two-dimensional coordinate information in the example of Table 1, the position information may be three-dimensional (in the xyz space) coordinate information.

  The shadow flag information indicates that the information is not attached when it is “0”, and indicates that the information is attached when it is “1”. When the shadow flag information is 0, the information about a feature that generates a shadow corresponding to a constituent point is blank, and when the shadow flag information is 1, it is information specifying the attribute of the feature. In the example of Table 1, the constituent points with ID = 2 and the constituent points with ID = 3 correspond to shadows generated by different buildings. It should be noted that although it has been specified that the constituent points correspond to the shadow, but if the feature generating the shadow has not been specified, the information about the feature may be blank.

[Generation of comprehensive point cloud data]
As described above, in the second information processing device 1B, the collection unit 11 collects the individual point cloud data, and the generation unit 12 generates the comprehensive point cloud data based on the plurality of individual point cloud data. At this time, the collection unit 11 adds not only the individual point cloud data but also the individual point cloud data to the additional information (the above-described shadow flag information and the configuration points Information about the feature that produces the corresponding shadow).

  The generation unit 12 determines the degree of contribution of the constituent points based on the acquired addition information, and generates total point cloud data. The contribution may be, for example, a value used for weighting when a plurality of individual point cloud data are overlaid or a position average is obtained when generating the total point cloud data.

  When the shadow flag information is acquired, the generation unit 12 may set the contribution degree of the constituent point to 0, that is, remove the constituent point to generate the total point cloud data.

  In addition, when the flag information is added to the constituent points at the same position in the plurality of individual point group data with different imaging conditions, the generation unit 12 does not reduce the contribution of these constituent points and the total point cloud data. May be generated. Constituent points determined to be shadows in individual point cloud data generated based on daytime imaging information, and constituent points determined to be shadows in individual point cloud data generated based on nighttime imaging information If and exist at the same position, these constituent points may not be shadows. Therefore, when a certain constituent point is erroneously determined to be a shadow and additional information is added, it is possible to prevent the degree of contribution of this constituent point from decreasing.

  Further, when determining the degree of contribution of the constituent points, the generation unit 12 may refer to not only the shadow flag information but also the information about the feature generating the shadow. The clearer the shape of the feature, the easier it is to accurately determine whether the constituent points correspond to the shadows. That is, even for the constituent points to which the shadow flag information is added, the determination accuracy differs depending on the features that generate shadows. Therefore, for example, by performing a process of reducing the contribution of the constituent points as the shape of the feature for generating a shadow is clearer, it is possible to generate the total point cloud data in consideration of the shadow determination accuracy.

  With the above configuration, by assigning predetermined information to the constituent points corresponding to the shadow among the constituent points forming the point cloud data, the constituent points corresponding to the shadow are used when the point cloud data is used for self-position estimation or the like. You can choose not to use or reduce the usage. Since the constituent points corresponding to the shadow can change depending on the date and time and the weather, it is possible to make the point cloud data easier to use.

  Further, by extracting and removing the characteristic points corresponding to the shadow in the imaging information to generate the point cloud data, it is possible to prevent the influence of the shadow when the point cloud data is used for self-position estimation or the like. it can. Since the constituent points corresponding to the shadow can change depending on the date and time and the weather, it is possible to make the point cloud data easier to use.

  Moreover, by outputting the point cloud data to the external server 100 by the output unit 4, the point cloud data can be utilized in the external server 100 that is the output destination.

  Further, in the data structure of the point cloud data, predetermined information is given to the constituent points corresponding to the shadows, and the constituent points corresponding to the shadows can be processed with a reduced degree of contribution. When using the point cloud data (combined point cloud data) for self-position estimation or the like, it is possible not to use the constituent points corresponding to the shadow or to reduce the degree of use. Since the constituent points corresponding to the shadows may change depending on the date and time and the weather, it is possible to easily utilize the processed point cloud data.

  In addition, by determining the degree of contribution of the constituent points based on the flag information given to the constituent points corresponding to the shadow, and generating the total point cloud data based on the plurality of individual point cloud data, , It is possible not to use the constituent points corresponding to the shadow, or to reduce the usage. Since the constituent points corresponding to the shadow can change depending on the date and time and the weather, it is possible to easily utilize the comprehensive point cloud data.

  The present invention is not limited to the above-described embodiments, but includes other configurations and the like that can achieve the object of the present invention, and the following modifications and the like are also included in the present invention.

  For example, in the above-described embodiment, the information acquisition unit 2 and the generation unit 3 are mounted on the moving body 10; however, the imaging information acquisition unit, the generation unit, and the extraction unit included in the first information processing apparatus. Does not have to be mounted on the moving body, and may be mounted on different devices or apparatuses. That is, each component of the information processing device may span a plurality of physically separated devices.

  Further, in the above-described embodiment, the collection unit 11 and the generation unit 12 are mounted on the external server 100, but the collection unit and the generation unit configuring the second information processing apparatus are mounted on the external server. It does not have to be provided, and may be installed in different devices and apparatuses. That is, each component of the information processing device may span a plurality of physically separated devices.

  Besides, the best configuration, method, and the like for carrying out the present invention have been disclosed in the above description, but the present invention is not limited thereto. That is, the present invention is mainly illustrated and described mainly with respect to a specific embodiment, but without departing from the technical idea and the scope of the object of the present invention, the shape with respect to the embodiment described above Those skilled in the art can make various modifications in terms of material, quantity, and other detailed configurations. Therefore, the description limiting the shapes and materials disclosed above is described as an example for facilitating the understanding of the present invention, and does not limit the present invention. The description with the names of members excluding some or all of the above limitations is included in the present invention.

1A First information processing device 2 Information acquisition unit (imaging information acquisition unit)
3 Generator (extractor)
4 Output Unit 10 Moving Object 20 Imaging Unit 1B Second Information Processing Device 11 Collecting Unit 12 Generating Unit

Claims (6)

A point cloud generated based on imaging information, which is used by an information processing apparatus including a collection unit that collects a plurality of point cloud data and a processing unit that processes a plurality of point cloud data collected by the collection unit The data structure of the data,
Among the constituent points forming the point cloud data, predetermined information is given to the constituent points corresponding to the shadow,
A point cloud data structure, wherein the processing unit is configured to be able to process a component point corresponding to the shadow with a reduced contribution degree.   The point cloud data structure according to claim 1, wherein the predetermined information includes information indicating that the constituent point corresponds to a shadow.   The data structure according to claim 2, wherein the predetermined information further includes information about a feature that generates a shadow corresponding to the constituent point.   The point cloud data structure according to claim 1, wherein the predetermined information includes information that the configuration point has lower reliability than other configuration points.   A computer-readable recording medium storing the point cloud data having the point cloud data structure according to any one of claims 1 to 4.   A storage device, which stores point cloud data having the point cloud data structure according to claim 1.

Images