JP6819758B1 - Point cloud data identity estimation device and point cloud data identity estimation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a point cloud data identity estimation device and a point cloud data identity estimation system for improving the accuracy of evaluation of identity of information related to point cloud data. SOLUTION: This is a point group data identity estimation device that estimates the identity of an object that is the source of two three-dimensional point group data, and is a first point group data and a second point composed of three-dimensional point group data. Input the point group data acquisition unit that acquires group data, the first neural network that inputs information about the first point group data and outputs the first point group data feature amount, and the information about the second point group data. Based on the second neural network that outputs the second point group data feature amount, the first point group data feature amount, and the second point group data feature amount, the first point group data and the second point group It has an identity evaluation unit that outputs an evaluation of data identity, and the first neural network and the second neural network share weights with each other. [Selection diagram] Fig. 1

Description

The present disclosure relates to a point cloud data identity estimation device and a point cloud data identity estimation system.

Conventionally, so-called "point cloud data" such as 3D point cloud data acquired by measuring the actual environment or converted data obtained by converting the 3D point cloud data into a different data format is used. A model creation technique for creating a three-dimensional model is known.

When creating a 3D model using the model creation technology, each "point cloud" has different measurement conditions such as measurement direction, measurement position, measurement distance, measurement timing, and type of measurement device when measuring the actual environment. It is important to properly align (evaluate the identity) of "information about data".

Japanese Unexamined Patent Publication No. 2019-101927

The present disclosure provides a point cloud data identity estimation device and a point cloud data identity estimation system that improve the accuracy of evaluation of identity of information related to point cloud data.

The first aspect of the present disclosure is a point cloud data identity estimation device that estimates the identity of an object that is the source of two three-dimensional point cloud data.
A point cloud data acquisition unit that acquires first point cloud data and second point cloud data consisting of three-dimensional point cloud data,
A first neural network that inputs information about the first point cloud data and outputs the first point cloud data features,
A second neural network that inputs information about the second point cloud data and outputs the second point cloud data features,
Based on the first point cloud data feature amount and the second point group data feature amount, an identity evaluation unit that outputs an evaluation regarding the identity of the first point cloud data and the second point group data is provided. Have and
The first neural network and the second neural network share weights with each other.

According to the first aspect of the present disclosure, it is possible to provide a point cloud data identity estimation device that improves the accuracy of evaluation of identity of information related to point cloud data.

Further, the second aspect of the present disclosure is the point cloud data identity estimation device according to the first aspect.
When information about two three-dimensional point group data created based on the same three-dimensional shapes is input to the first neural network and the second neural network as first teacher data, the identity evaluation unit The weights of the first neural network and the second neural network are updated while sharing each other so as to output an evaluation indicating that the identity is high.
When information about two three-dimensional point group data created based on three-dimensional shapes different from each other is input to the first neural network and the second neural network as second teacher data, the identity evaluation unit The weights of the first neural network and the second neural network are updated while sharing each other so as to output an evaluation indicating that the identity is low.

Further, the third aspect of the present disclosure is the point cloud data identity estimation device according to the second aspect.
The first teacher data is one in which at least one of the information about two three-dimensional point cloud data created based on the same three-dimensional shape is deleted, or at least one in which noise is added. is there.

Further, the fourth aspect of the present disclosure is the point cloud data identity estimation device according to the second aspect.
The first teacher data is information on two three-dimensional point cloud data created based on measurements of the same three-dimensional shapes under different conditions.

A fifth aspect of the present disclosure is the point cloud data identity estimation device according to the first aspect.
The information regarding the first point cloud data and the information regarding the second point cloud data are three-dimensional point cloud data, respectively.

Further, the sixth aspect of the present disclosure is the point cloud data identity estimation device according to the fifth aspect.
Three-dimensional point group data created based on a predetermined three-dimensional shape, and three-dimensional point group data created based on those in which the predetermined three-dimensional shape is not tilted or tilted less than a predetermined angle. , The first neural network and the second neural network so that the identity evaluation unit outputs an evaluation indicating that the identity is high when input to the first neural network and the second neural network as the third teacher data. Update while sharing the weights of the neural network with each other
The 3D point group data created based on the predetermined 3D shape and the 3D point group data created based on the predetermined 3D shape tilted by the predetermined angle or more are obtained. 4 The first neural network and the second neural network so that the identity evaluation unit outputs an evaluation indicating that the identity is low when the teacher data is input to the first neural network and the second neural network. Update while sharing network weights with each other.

Further, the seventh aspect of the present disclosure is the point cloud data identity estimation device according to the fifth aspect.
Three-dimensional point group data created based on a predetermined three-dimensional shape, and one having the same size as the predetermined three-dimensional shape or the predetermined three-dimensional shape is enlarged with an enlargement ratio or reduction ratio within a predetermined range. Or, when the three-dimensional point group data created based on the reduced data is input to the first neural network and the second neural network as the fifth teacher data, the identity evaluation unit has high identity. The weights of the first neural network and the second neural network are updated while sharing each other so as to output the indicated evaluation.
It is created based on the three-dimensional point group data created based on the predetermined three-dimensional shape, and the one obtained by enlarging or reducing the predetermined three-dimensional shape at an enlargement ratio or reduction ratio exceeding the predetermined range. When the three-dimensional point group data is input to the first neural network and the second neural network as the sixth teacher data, the identity evaluation unit outputs an evaluation indicating that the identity is low. The weights of the first neural network and the second neural network are updated while being shared with each other.

Further, the eighth aspect of the present disclosure is a measuring device that measures an object and creates three-dimensional point cloud data, and a point cloud that estimates the identity of the object that is the source of the two three-dimensional point cloud data. A point cloud data identity estimation system having a data identity estimation device.
The point cloud data identity estimation device is
A point cloud data acquisition unit that acquires first point cloud data and second point cloud data consisting of three-dimensional point cloud data,
A first neural network that inputs information about the first point cloud data and outputs the first point cloud data features,
A second neural network that inputs information about the second point cloud data and outputs the second point cloud data features,
Based on the first point cloud data feature amount and the second point group data feature amount, an identity evaluation unit that outputs an evaluation regarding the identity of the first point cloud data and the second point group data is provided. Have and
The first neural network and the second neural network share weights with each other.

According to the eighth aspect of the present disclosure, it is possible to provide a point cloud data identity estimation system that improves the accuracy of the evaluation of identity for information regarding point cloud data.

It is a figure which shows an example of the system configuration of the point cloud data identity estimation system. It is a figure which shows an example of the hardware composition of the point cloud data identity estimation apparatus. It is a figure which shows an example of the learning data. It is a figure which shows an example of the functional structure of a data expansion part. It is a figure which shows an example of the functional structure of a learning part. It is a flowchart which shows the flow of the learning process by a learning part. It is a figure which shows an example of the functional structure of an inference part. It is a flowchart which shows the flow of inference processing by an inference part. It is a figure which shows the specific example of the inference result.

Hereinafter, each embodiment will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit duplicate explanations.

[First Embodiment]
<System configuration of point cloud data identity estimation system>
First, the system configuration of the point cloud data identity estimation system according to the first embodiment will be described. FIG. 1 is a diagram showing an example of a system configuration of a point cloud data identity estimation system.

As shown in FIG. 1, the point cloud data identity estimation system 100 includes a first measurement device 110, a second measurement device 111, and a point cloud data identity estimation device 120. The first measuring device 110 and the second measuring device 111 and the point cloud data identity estimation device 120 are connected by wire or wirelessly.

The first measuring device 110 and the second measuring device 111 measure various objects such as the object A and the object B placed in the actual environment 130, respectively, and create three-dimensional point cloud data. .. The first measuring device 110 and the second measuring device 111 each transmit the created three-dimensional point cloud data to the point cloud data identity estimation device 120.

The first measuring device 110 and the second measuring device 111 are among various measurement conditions such as a measurement direction, a measurement position, a measurement distance, a measurement timing, and a type of device when measuring various objects. It is assumed that at least one of the measurement conditions is different.

Further, in the first embodiment, the point cloud data identity estimation system 100 will be described as having two measuring devices, the first measuring device 110 and the second measuring device 111, but the point cloud data is the same. The number of measuring devices included in the sex estimation system 100 may be one. Alternatively, the number of measuring devices included in the point cloud data identity estimation system 100 may be three or more.

The point cloud data identity estimation device 120 is a device that evaluates the identity of an object that is the source of two three-dimensional point cloud data.

The two 3D point cloud data are 3D point cloud data when an object is measured under different measurement conditions such as measurement direction, measurement position, measurement distance, measurement timing, and type of measurement device. Point to. Specifically, the first measuring device 110 and the second measuring device 111 each refer to three-dimensional point cloud data created by measuring an object placed in the actual environment 130.

Various programs are installed in the point cloud data identity estimation device, and by executing the program, the point cloud data identity estimation device 120 has a data acquisition unit 121, a data conversion unit 122, and a data expansion unit 123. , It functions as a learning unit 124 and an inference unit 125.

The data acquisition unit 121 acquires the three-dimensional point cloud data transmitted from the first measuring device 110 and the second measuring device 111. The three-dimensional point cloud data acquired by the data acquisition unit 121 includes three-dimensional point cloud data acquired for the purpose of using as learning data and three-dimensional point cloud data acquired for the purpose of using as inference data. Is included.

The data conversion unit 122 converts the data format of the three-dimensional point cloud data acquired by the data acquisition unit 121, and generates data in a data format different from the three-dimensional point cloud data. The data generated by the data conversion unit 122 includes, for example, data in various data formats such as mesh data (polygon data), surface data, and CAD data. In the first embodiment, the data in these various data formats including the three-dimensional point cloud data are collectively referred to as "information about the point cloud data". However, in the following, for the sake of simplification of the description, the data format conversion function of the data conversion unit 122 is not operated, and the information regarding the point cloud data = the three-dimensional point cloud data itself will be described.

Further, the data conversion unit 122 has a dividing function of dividing the three-dimensional point cloud data into a plurality of local three-dimensional point cloud data representing a part of the three-dimensional shape of the object.

Further, the data conversion unit 122 uses a plurality of local three-dimensional point cloud data obtained by dividing the three-dimensional point cloud data acquired for the purpose of using it as learning data as training data for learning. It has a storage function for storing in the data storage unit 126.

Specifically, the data conversion unit 122 has the same object among the two three-dimensional point cloud data acquired by measuring the same object by the first and second measuring devices 110 and 111. Two local three-dimensional point cloud data representing a part of the three-dimensional shape of the above are stored in association with each other. At this time, the data conversion unit 122 also stores information indicating that the data is created based on the same three-dimensional shapes.

Further, the data conversion unit 122 has a part of the two three-dimensional point cloud data acquired by measuring the same object by the first and second measuring devices 110 and 111, which are different from each other. Two local three-dimensional point cloud data representing a three-dimensional shape are stored in association with each other. At this time, the data conversion unit 122 also stores information indicating that the data is created based on three-dimensional shapes different from each other.

Further, the data conversion unit 122 has a part of two three-dimensional point cloud data of each object among the two three-dimensional point cloud data acquired by measuring different objects by the first and second measuring devices 110 and 111. Two local three-dimensional point cloud data representing the shape are stored in association with each other. At this time, the data conversion unit 122 also stores information indicating that the data is created based on three-dimensional shapes different from each other.

Further, in the data conversion unit 122, a plurality of local three-dimensional point cloud data obtained by dividing the three-dimensional point cloud data acquired for the purpose of using the data for inference is stored in the inference data storage unit 127. It has a storage function to store.

Specifically, the data conversion unit 122
A plurality of local 3D point cloud data obtained by dividing the 3D point cloud data acquired by the measurement by the first measuring device 110, and
A plurality of local 3D point cloud data obtained by dividing the 3D point cloud data acquired by the measurement by the second measuring device 111, and
Are stored in association with each other.

The data expansion unit 123 expands the learning data by performing various modification processes on the plurality of local three-dimensional point cloud data stored in the learning data storage unit 126 as the learning data.

The learning unit 124 has a deep learning framework. The learning unit 124 reads out a plurality of local three-dimensional point cloud data stored in the learning data storage unit 126, and reads the data.
-A deep neural network (DNN: Deep Neural Network, for example, PointNet, which is a DNN for 3D point cloud data) is executed by using a set of two local 3D point cloud data associated with each other as input data.
-Weights included in the deep neural network (for example, PointNet, which is a DNN for 3D point cloud data), using information indicating whether or not the information is created based on the same 3D shape as a correct label. Update parameters,
Perform learning process.

The learning unit 124 reflects the weight parameter adjusted by performing the learning process on the inference unit 125.

The inference unit 125, like the learning unit 124, has a deep learning framework for three-dimensional point cloud data.

The inference unit 125 reads out a plurality of local three-dimensional point cloud data stored in the inference data storage unit 127, and reads the data.
-A deep neural network (for example, PointNet, which is a DNN for 3D point cloud data) is executed by using a set of two local 3D point cloud data associated with each other as input data.
-Infer whether or not the three-dimensional shapes that are the basis of the two local three-dimensional point cloud data associated with each other are the same.
Performs inference processing.

<Hardware configuration of point cloud data identity estimation device>
Next, the hardware configuration of the point cloud data identity estimation device 120 will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the point cloud data identity estimation device.

As shown in FIG. 2, the point cloud data identity estimation device 120 includes a processor 201, a memory 202, an auxiliary storage device 203, a display device 204, an operation device 205, and an I / F (Interface) device 206. , Drive device 207. The hardware of the point cloud data identity estimation device 120 is connected to each other via the bus 208.

The processor 201 has various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 201 reads various programs onto the memory 202 and executes them.

The memory 202 has a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 201 and the memory 202 form a so-called computer, and the processor 201 realizes the above function (see FIG. 1) by executing various programs read on the memory 202.

The auxiliary storage device 203 stores various programs and various data used when the various programs are executed by the processor 201. The learning data storage unit 126 and the inference data storage unit 127 are realized in the auxiliary storage device 203.

The display device 204 is a display device that displays the internal state of the point cloud data identity estimation device 120. The operation device 205 is an input device used by the user of the point cloud data identity estimation device 120 to input various instructions to the point cloud data identity estimation device 120. The I / F device 206 is a connecting device for connecting to the first measuring device 110 and the second measuring device 111 and receiving the three-dimensional point cloud data.

The drive device 207 is a device for setting the recording medium 210. The recording medium 210 referred to here includes a medium such as a CD-ROM, a flexible disk, a magneto-optical disk, or the like that optically, electrically, or magnetically records information. Further, the recording medium 210 may include a semiconductor memory or the like for electrically recording information such as a ROM or a flash memory.

The various programs installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 210 in the drive device 207 and reading the various programs recorded in the recording medium 210 by the drive device 207. Will be done. Alternatively, the various programs installed in the auxiliary storage device 203 may be installed by being downloaded via a network (not shown).

<Specific examples of learning data>
Next, a specific example of the learning data stored in the learning data storage unit 126 will be described. FIG. 3 is a diagram showing an example of learning data. As shown in FIG. 3, the learning data 300 has "first point cloud data", "second point cloud data", and "correct answer label" as information items.

In the "first point cloud data", a plurality of local three-dimensional point cloud data obtained by dividing the three-dimensional point cloud data 310 acquired by measuring the object by the first measuring device 110 is stored.

The example of FIG. 3 shows specific examples of the local three-dimensional point cloud data 311 and 312 as a plurality of local three-dimensional point cloud data. As shown in FIG. 3, the local three-dimensional point cloud data 311 and 312 include the X coordinate, Y coordinate, Z coordinate, R value, G value, and B value of each point. Further, in the case of the example of FIG. 3, the local three-dimensional point cloud data 311 and 312 include a point cloud of 2048 points.

In the "second point cloud data", a plurality of local three-dimensional point cloud data obtained by dividing the three-dimensional point cloud data 320 acquired by measuring the object by the second measuring device 111 is stored.

The example of FIG. 3 shows a specific example of the local three-dimensional point cloud data 321 as a plurality of local three-dimensional point cloud data. As shown in FIG. 3, the local three-dimensional point cloud data 321 includes the X coordinate, Y coordinate, Z coordinate, R value, G value, and B value of each point. Further, in the case of the example of FIG. 3, the local three-dimensional point cloud data 321 includes a point cloud of 2048 points.

"Correct label"
・ Local 3D point cloud data of "1st point cloud data" and
・ Local 3D point cloud data of "2nd point cloud data" and
As information indicating whether or not the two local three-dimensional point cloud data associated with each other are created based on the same three-dimensional shape, either "identical" or "non-identical". Is stored.

For example, since the local 3D point cloud data 311 and the local 3D point cloud data 321 are created based on the same 3D shape, "same" is stored in the "correct answer label". Will be done. These are examples of the first teacher data.

On the other hand, since the local 3D point cloud data 312 and the local 3D point cloud data 321 are created based on different 3D shapes, "non-identical" is stored in the "correct answer label". To. These are examples of the second teacher data.

<Details of the functional configuration of the data expansion unit>
Next, the details of the functional configuration of the data expansion unit 123 will be described. FIG. 4 is a diagram showing an example of the functional configuration of the data expansion unit.

As shown in FIG. 4, the data expansion unit 123 includes a read unit 401, a density change unit 402, a defect unit 403, a noise addition unit 404, a bias addition unit 405, an angle change unit 406, an enlargement / reduction unit 407, and a shift unit 408. ..

The reading unit 401 reads predetermined teacher data from the learning data 300 stored in the learning data storage unit 126. The example of FIG. 4 shows a state in which the teacher data 410 including the local three-dimensional point cloud data 311 and 321 is read out as predetermined teacher data from the learning data storage unit 126. The predetermined teacher data read by the reading unit 401 is teacher data in which information indicating that the "correct answer label" is the same is stored.

The density change unit 402 is a local 3D point cloud data of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. The density of each point is changed by thinning each point at regular intervals. The density changing unit 402 stores the teacher data 421 including the changed local three-dimensional point cloud data in the learning data storage unit 126.

The missing portion 403 is each of the local three-dimensional point cloud data of one of the two local three-dimensional point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. Randomly delete points. In addition, the defect unit 403 stores the teacher data 422 including the local three-dimensional point cloud data after the defect in the learning data storage unit 126.

The noise addition unit 404 is a local 3D point cloud data of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. Add noise (point) to an arbitrary position. Further, the noise addition unit 404 stores the teacher data 423 including the local three-dimensional point cloud data after the noise is added in the learning data storage unit 126.

The bias addition unit 405 is the local 3D point cloud data of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. A bias value is added to the R value, G value, and B value of each point (that is, the color is changed). Further, the bias adding unit 405 stores the teacher data 424 including the local three-dimensional point cloud data after adding the bias value in the learning data storage unit 126.

The angle changing unit 406 selects the local 3D point cloud data of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. , Tilt more than the specified angle. Further, the angle changing unit 406 stores the teacher data 425 including the local three-dimensional point cloud data after being tilted by a predetermined angle or more in the learning data storage unit 126. The teacher data 425 is an example of the fourth teacher data.

Further, the angle changing unit 406 changes the "correct answer label" from "identical" to "non-identical" for the teacher data 425 including the local three-dimensional point cloud data after being tilted by a predetermined angle or more. When one of the two local three-dimensional point cloud data is tilted by a predetermined angle or more, the two local three-dimensional point cloud data are said to have been created based on different three-dimensional shapes. Because it can be regarded.

The angle changing unit 406 uses the local 3D point cloud of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. The data may not be tilted or tilted less than a predetermined angle. Further, the angle changing unit 406 may store the teacher data 425 including the local three-dimensional point cloud data after not tilting or tilting less than a predetermined angle in the learning data storage unit 126. The teacher data 425 in this case is an example of the third teacher data. Further, regarding the teacher data 425 in this case, the angle changing unit 406 does not change the "correct answer label".

The scaling unit 407 uses the local 3D point cloud data of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. Enlarge or reduce at an enlargement or reduction rate that exceeds a predetermined range. In addition, the scaling unit 407 stores the teacher data 426 including the local three-dimensional point cloud data after being enlarged or reduced at a magnification or reduction rate exceeding a predetermined range in the learning data storage unit 126. The teacher data 426 is an example of the sixth teacher data.

Further, in the scaling unit 407, the "correct label" is changed from "same" to "non" for the teacher data 426 including the local three-dimensional point cloud data after being enlarged or reduced at a magnification or reduction rate exceeding a predetermined range. Change to "same". When one of the two local 3D point cloud data is enlarged or reduced at a magnification or reduction rate exceeding a predetermined range, the two local 3D point cloud data are based on different 3D shapes. This is because it can be regarded as being created in.

The scaling unit 407 keeps the two local three-dimensional point cloud data included in the teacher data 410 at the same size, or reduces or reduces the scaling factor of one of the local three-dimensional point cloud data within a predetermined range. It may be enlarged or reduced at a rate. Further, the scaling unit 407 stores the training data 426 including the local three-dimensional point cloud data of the same size or the local three-dimensional point cloud data after being enlarged or reduced at the enlargement ratio or reduction ratio within a predetermined range. It may be stored in the unit 126. The teacher data 426 in this case is an example of the fifth teacher data. Further, regarding the teacher data 426 in this case, the scaling unit 407 does not change the "correct answer label".

The shift unit 408 is the center of the local 3D point cloud data of one of the two local 3D point cloud data included in the "first point cloud data" and the "second point cloud data" of the teacher data 410. Shift the position. Further, the shift unit 408 stores the teacher data 427 including the local three-dimensional point cloud data after shifting the center position in the learning data storage unit 126.

In this way, by expanding the learning data 300 by the data expansion unit 123, two local three-dimensional point clouds having different measurement conditions such as measurement direction, measurement position, measurement distance, measurement timing, and type of measurement device are different. A set of data can be generated in a pseudo manner. Then, by using the expanded learning data 300, according to the first embodiment, it is possible to improve the accuracy of the evaluation of the identity of two local three-dimensional point cloud data having different measurement conditions. ..

<Details of the functional configuration of the learning department>
Next, the details of the functional configuration of the learning unit 124 will be described. FIG. 5 is a diagram showing an example of the functional configuration of the learning unit. As shown in FIG. 5, the learning unit 124 infers whether or not the two local three-dimensional point cloud data are created based on the same three-dimensional shape (evaluate the identity). To use the Siamese network structure.

Specifically, as shown in FIG. 5, the learning unit 124 includes a first point cloud data input unit 501, a first DNN unit 502, a second point cloud data input unit 511, a second DNN unit 512, and an identity evaluation unit 521. Has. Further, the learning unit 124 has a comparison unit 531.

The first point cloud data input unit 501 reads out the local three-dimensional point cloud data stored in the "first point cloud data" from the learning data 300 stored in the learning data storage unit 126, and the first DNN unit 502. Enter in. Further, the second point cloud data input unit 511 reads out the local three-dimensional point cloud data stored in the "second point cloud data" from the learning data 300 stored in the learning data storage unit 126, and reads the second DNN. Input to unit 512.

The local 3D point cloud data input by the 1st point cloud data input unit 501 and the local 3D point cloud data input by the 2nd point cloud data input unit 511 correspond to each other in the training data 300. It is the attached local three-dimensional point cloud data.

The first DNN unit 502 is an example of the first neural network, and for example, PointNet, which is a DNN for three-dimensional point cloud data, is used. The first DNN unit 502 operates by inputting local three-dimensional point cloud data by the first point cloud data input unit 501, and outputs the first point cloud data feature amount (see the following equation (1)).

The above equation (1) represents that the first point cloud data feature amount including 1024 feature amounts is output from the local three-dimensional point cloud data including the 2048 point cloud.

The second DNN unit 512 is an example of the second neural network, and for example, PointNet, which is a DNN for three-dimensional point cloud data, is used. The second DNN unit 512 operates by inputting local three-dimensional point cloud data by the second point cloud data input unit 511, and outputs the second point cloud data feature amount (see the following equation (2)).

The above equation (2) represents that the second point cloud data feature amount including the 1024 feature amount is output from the local three-dimensional point cloud data including the 2048 point cloud.

The identity evaluation unit 521 processes the first point cloud data feature amount output from the first DNN unit 502 and the second point cloud data feature amount output from the second DNN unit 512. As a result, the identity evaluation unit 521 is data for inferring whether or not the two local three-dimensional point cloud data are created based on the same three-dimensional shape (two local 3). (Evaluation of identity of 3D point cloud data) is output.

Specifically, in the identity evaluation unit 521, the Euclidean distance is based on the first point cloud data feature amount pn _α and the second point cloud data feature amount pn _β output from the first DNN unit 502 and the second DNN unit 512. The distance D _w is calculated and output to the comparison unit 531 (see the following equation (3)).

However, in the above equation (3), the first point cloud data feature amount pn _α and the second point group data feature amount pn _β satisfy the following equations (4-1) and (4-2).

That is, it can be said that the Euclidean distance D _W represents an evaluation regarding the identity of two local three-dimensional point cloud data. If the Euclidean distance D _W is small, it can be said that an evaluation indicating that the identity is high is output, and if the Euclidean distance D _W is large, an evaluation indicating that the identity is low is output. it can.

The comparison unit 531 indicates that the Euclidean distance D _w output by the identity evaluation unit 521 is the same (or non-identical) stored in the "correct answer label" read from the learning data storage unit 126. The loss function L _c is calculated based on the information (see equation (5) below).

However, in the above equation (5), "0" is input to C if the correct label is "same", and "1" is input if the correct label is "non-identical". Further, in the above equation (5), m is a margin.

The comparison unit 531 updates the weight parameters of the first DNN unit 502 and the second DNN unit 512 by back-propagating the calculated loss function L _c .

Incidentally, the weight parameters of the 1DNN portion 502 and the weight parameters of the 2DNN portion 512 are shared with each other, when the loss function _{L c} is reversely propagated, and the 1DNN unit 502 and the 2DNN portion 512, the same value Will be updated to the weight parameter of.

In this way, at the time of learning, the loss function L _c is back-propagated and updated while sharing the weight parameter between the first DNN unit 502 and the second DNN unit 512.
-Weight parameters can be learned so that point cloud data features suitable for outputting an evaluation of the identity of two local 3D point cloud data are output.
-In the training data 300, even if the characteristics of the local 3D point cloud data stored in the "first point cloud data" and the "second point cloud data" are biased, the weight parameter with high generalization performance is high. Can learn,
It will have the effect. As a result, according to the first embodiment, the accuracy of the evaluation of identity can be improved.

<Flow of learning process>
Next, the flow of the learning process by the learning unit 124 will be described. FIG. 6 is a flowchart showing the flow of learning processing by the learning unit.

In step S601, the first point cloud data input unit 501 and the second point cloud data input unit 511 read out a plurality of sets of two local three-dimensional point cloud data associated with each other from the learning data storage unit 126.

In step S602, the first DNN unit 502 and the second DNN unit 512 operate by inputting a set of two read local three-dimensional point cloud data, and the first point cloud data feature amount pn _α and the second point The group data feature amount pn _β is output.

In step S603, the identity evaluation unit 521 calculates the Euclidean distance D _w based on the first point cloud data feature amount pn _α and the second point group data feature amount pn _β .

In step S604, the comparison unit 531 calculates the loss function L _c based on the calculated Euclidean distance D _w and the correct label C read from the learning data storage unit 126. Further, the comparison unit 531 updates while sharing the weight parameters of the first DNN unit 502 and the second DNN unit 512 based on the calculated loss function L _c .

In step S605, the comparison unit 531 determines whether or not to end the learning process. If it is determined in step S605 that the learning process is not completed (if No in step S605), the process returns to step S602. As a result, the first point cloud data input unit 501 and the second point cloud data input unit 511 are a set of two local three-dimensional point cloud data associated with each other and have not yet been used in the learning process. Is input to the first DNN unit 502 and the second DNN unit 512.

On the other hand, when it is determined in step S605 that the learning process is finished (in the case of Yes in step S605), the learning process is finished and the updated weight parameter is reflected in the inference unit 125.

<Functional configuration of inference unit>
Next, the details of the functional configuration of the inference unit 125 will be described. FIG. 7 is a diagram showing an example of the functional configuration of the inference unit. Similar to the learning unit 124, the inference unit 125 infers whether or not the two local 3D point cloud data are created based on the same 3D shape (evaluate the identity). To take advantage of the Siamese network structure.

Specifically, as shown in FIG. 7, the inference unit 125 includes a first point cloud data input unit 701, a first DNN unit 702, a second point cloud data input unit 711, a second DNN unit 712, and an identity evaluation unit 721. Has. Further, the inference unit 125 has an output unit 731.

The first point group data input unit 701 reads out the local three-dimensional point group data stored in the "first point group data" from the inference data stored in the inference data storage unit 127, and causes the first DNN unit 702 to read the local three-dimensional point group data. input. Further, the second point group data input unit 711 reads out the local three-dimensional point group data stored in the "second point group data" from the inference data stored in the inference data storage unit 127, and the second DNN unit. Enter in 712.

The inference data stored in the inference data storage unit 127 has the same configuration as the learning data 300 in FIG. 3, except that there is no "correct answer label" as an information item. To do.

The local 3D point cloud data input by the 1st point cloud data input unit 701 and the local 3D point cloud data input by the 2nd point cloud data input unit 711 are associated with each other in the inference data. It is the local three-dimensional point cloud data obtained.

The first DNN unit 702 is an example of the first neural network, operates by inputting local three-dimensional point cloud data by the first point cloud data input unit 701, and outputs the first point cloud data feature amount pn _α . ..

The second DNN unit 712 is an example of the second neural network, operates by inputting local three-dimensional point cloud data by the second point cloud data input unit 711, and outputs the second point cloud data feature amount pn _β . .. The first DNN unit 702 and the second DNN unit 712 share weight parameters with each other.

The identity evaluation unit 721 processes the first point cloud data feature amount output from the first DNN unit 702 and the second point cloud data feature amount output from the second DNN unit 712. As a result, the identity evaluation unit 721 data for inferring whether or not the two local three-dimensional point cloud data are created based on the same three-dimensional shape (two local 3). (Evaluation of identity of 3D point cloud data) is output.

Specifically, in the identity evaluation unit 721, the first point cloud data feature amount pn _α output from the first DNN unit 702 and the second point cloud data feature amount pn _β output from the second DNN unit 712 are set. Based on this, the Euclidean distance D _w is calculated and output to the output unit 731. As described above, if the Euclidean distance D _W is small, it can be said that an evaluation indicating that the identity is high is output, and if the Euclidean distance D _W is large, an evaluation indicating that the identity is low is output. It can be said that it was done.

The output unit 731 determines whether or not the Euclidean distance D _w output from the identity evaluation unit 721 is equal to or greater than a predetermined threshold value (see the following equations (6-1) and (6-2)).

When it is determined that the Euclidean distance D _w is larger than the predetermined threshold value t _d (the above equation (6-1)), the output unit 731 outputs "non-identical" as an evaluation of identity. On the other hand, when it is determined that the Euclidean distance D _w is equal to or less than the predetermined threshold value t _d (the above equation (6-2)), the output unit 731 outputs "identical" as an evaluation of the sameness.

In this way, by using the first DNN unit 702 and the second DNN unit 712 that share the weight parameters with each other at the time of inference, according to the inference unit 125, for example,
-It is possible to output a point cloud data feature amount suitable for outputting an evaluation regarding the identity of two local three-dimensional point cloud data.
-The same data can be output as an evaluation of identity regardless of whether the two local three-dimensional point cloud data are input to the first DNN unit 702 or the second DNN unit 712.
It will have the effect. As a result, according to the first embodiment, the accuracy of the evaluation of identity can be improved.

<Flow of inference processing>
Next, the flow of the inference process by the inference unit 125 will be described. FIG. 8 is a flowchart showing the flow of inference processing by the inference unit.

In step S801, the first point cloud data input unit 701 and the second point cloud data input unit 711 read out a set of two local three-dimensional point cloud data associated with each other from the inference data storage unit 127.

In step S802, the first DNN unit 702 and the second DNN unit 712 operate by inputting a set of two read local three-dimensional point cloud data, and the first point cloud data feature amount pn _α and the second point The group data feature amount pn _β is output.

In step S803, the identity evaluation unit 721 calculates the Euclidean distance D _w based on the first point cloud data feature amount pn _α and the second point group data feature amount pn _β .

In step S804, the output unit 731 outputs an evaluation of identity (“identical” or “non-identical”) based on the Euclidean distance D _w output from the identity evaluation unit 721.

<Specific example of inference result>
Next, a specific example of the inference result by the inference unit 125 will be described. FIG. 9 is a diagram showing a specific example of the inference result.

Of these, FIG. 9A shows an outline of the inference data used in the inference processing. In FIG. 9A, "type of object" indicates the type of object measured by the first measuring device 110 and the second measuring device 111.

Further, in FIG. 9A, the "number of sets in which two local three-dimensional point cloud data are the same" includes among a plurality of local three-dimensional point cloud data included in the corresponding "object type". The number of pairs of two local 3D point cloud data created based on the same 3D shape is stored. Further, for "the number of pairs in which two local three-dimensional point cloud data are not the same", three-dimensional shapes different from each other among a plurality of local three-dimensional point cloud data included in the corresponding "object type" are used. The number of pairs of two local 3D point cloud data created based on this is stored.

On the other hand, FIG. 9B shows the precision of the inference result inferred by the inference unit 125 for each "type of object" using the inference data shown in FIG. 9A as input data. As shown in FIG. 9B, according to the inference unit 125, it is possible to realize a high precision rate regardless of the "type of object".

<Summary>
As is clear from the above description, in the point cloud data identity estimation system according to the first embodiment, the point cloud data identity estimation device is used.
-It has a first point cloud data acquisition unit and a second point cloud data acquisition unit that acquire first point cloud data and second point cloud data composed of three-dimensional point cloud data.
It has a first DNN unit that inputs the first point cloud data and outputs the first point cloud data feature amount, and a second DNN unit that inputs the second point cloud data and outputs the second point cloud data feature amount. ..
-It has an identity evaluation unit that outputs an evaluation regarding the identity of the first point cloud data and the second point cloud data based on the first point cloud data feature amount and the second point group data feature amount.
-The first DNN part and the second DNN part share weight parameters with each other.

Thereby, according to the first embodiment, it is possible to provide a point cloud data identity estimation device and a point cloud data identity estimation system that improve the accuracy of the evaluation of the identity of the three-dimensional point cloud data.

[Second Embodiment]
In the first embodiment, the information regarding the point cloud data has been described as being the three-dimensional point cloud data itself. However, the information regarding the point cloud data is not limited to the three-dimensional point cloud data itself, and may be mesh data (polygon data), surface data, CAD data, or the like.

Further, in the first embodiment, the association method in the case where two local three-dimensional point cloud data created based on different three-dimensional shapes are associated and stored as learning data 300 is not mentioned. However, when associating two local three-dimensional point cloud data, for example, the association may be limited to the three-dimensional point cloud data in which each three-dimensional shape is located within a predetermined distance.

Further, in the first embodiment, among the parts included in the data expansion part 123,
・ When the angle changing unit 406 changes the angle, and
-When the scaling unit 407 changes its size
According to the degree of change, the teacher data whose "correct label" is "same" and the teacher data whose "correct label" is "non-identical" are generated.

However, for each part other than the angle changing part 406 and the scaling part 407, similarly, depending on the degree of change, the teacher data whose "correct label" is "same" and the teacher whose "correct label" is "non-identical" It may be configured to generate data.

Further, in the first embodiment, the data expansion unit 123 has described a case where the teacher data having the "correct answer label" of "same" is read out to expand the learning data 300. However, the teacher data read when the data expansion unit 123 expands the learning data 300 is not limited to this, and the teacher data whose "correct answer label" is "non-identical" may be read.

As a result, for example, when different individuals having the same three-dimensional shape are different only in color or different only in material, they are erroneously "identical" as an evaluation of identity. It is possible to avoid the situation where "" is output.

Further, in the first embodiment, the case where each unit included in the data expansion unit 123 expands the learning data 300 has been described. However, the method for expanding the learning data 300 is not limited to this, and the learning data 300 may be expanded by artificially changing various measurement conditions.

For example, instead of operating the density changing unit 402 or the scaling unit 407, the measurement distance from the first measuring device 110 to the object is changed, or the type of the first measuring device 110 is changed for learning. The data 300 may be expanded.

Further, for example, instead of operating the angle changing unit 406, the learning data 300 may be expanded by tilting the angle of the first measuring device 110 and changing the measuring direction. Further, for example, instead of operating the shift unit 408, the learning data 300 may be expanded by shifting the measurement position of the first measuring device 110. Further, for example, instead of operating the noise addition unit 404 or the bias addition unit 405, the brightness of the illumination in the actual environment 130 is changed, or the measurement timing of the first measurement device 110 is changed for learning. The data 300 may be expanded.

Further, in the first embodiment, the identity evaluation unit 521 outputs the Euclidean distance D _w calculated based on the first point cloud data feature amount and the second point group data feature amount as an evaluation of identity. The case of doing so was explained. However, the output method of the evaluation regarding the identity by the identity evaluation unit 521 is not limited to this, and the data calculated by using another function may be output as the evaluation regarding the identity. Alternatively, the data calculated by a method other than the function (for example, using NN) may be output as an evaluation of identity.

Further, in the first embodiment, the point cloud data identity estimation device 120 has been described as having a learning unit 124 and an inference unit 125. However, the learning unit 124 and the inference unit 125 may be configured by separate devices.

Although the embodiments have been described above, it will be understood that various modifications of the embodiments and details are possible without departing from the purpose and scope of the claims.

100: Point group data identity estimation system 110, 111: First and second measuring devices 120: Point group data identity estimation device 121: Data acquisition unit 122: Data conversion unit 123: Data expansion unit 124: Learning unit 125 : Reasoning unit 300: Learning data 310, 320: Three-dimensional point group data 311, 312, 321: Local three-dimensional point group data 401: Reading unit 402: Density changing unit 403: Missing unit 404: Noise addition unit 405: Bias Addition unit 406: Angle change unit 407: Expansion / contraction unit 408: Shift unit 501: First point group data input unit 502: First DNN unit 511: Second point group data input unit 512: Second DNN unit 521: Identity evaluation unit 531 : Comparison unit 701: First point group data input unit 702: First DNN unit 711: Second point group data input unit 712: Second DNN unit 721: Identity evaluation unit 731: Output unit

Claims (8)

It is a point cloud data identity estimation device that estimates the identity of an object that is the source of two three-dimensional point cloud data.
A point cloud data acquisition unit that acquires first point cloud data and second point cloud data consisting of three-dimensional point cloud data,
A first neural network that inputs information about the first point cloud data and outputs the first point cloud data features,
A second neural network that inputs information about the second point cloud data and outputs the second point cloud data features,
Based on the first point cloud data feature amount and the second point group data feature amount, an identity evaluation unit that outputs an evaluation regarding the identity of the first point cloud data and the second point group data is provided. Have and
A point cloud data identity estimation device in which the first neural network and the second neural network share weights with each other. When information about two three-dimensional point group data created based on the same three-dimensional shapes is input to the first neural network and the second neural network as first teacher data, the identity evaluation unit The weights of the first neural network and the second neural network are updated while sharing each other so as to output an evaluation indicating that the identity is high.
When information about two three-dimensional point group data created based on three-dimensional shapes different from each other is input to the first neural network and the second neural network as second teacher data, the identity evaluation unit The point group data identity estimation device according to claim 1, wherein the weights of the first neural network and the second neural network are updated while being shared with each other so as to output an evaluation indicating low identity. The first teacher data is one in which at least one of the information about two three-dimensional point cloud data created based on the same three-dimensional shape is deleted, or at least one in which noise is added. The point cloud data identity estimation device according to claim 2. The point cloud data identity according to claim 2, wherein the first teacher data is information on two three-dimensional point cloud data created based on measurements of the same three-dimensional shapes under different conditions. Estimator. The point cloud data identity estimation device according to claim 1, wherein the information regarding the first point cloud data and the information regarding the second point cloud data are three-dimensional point cloud data, respectively. Three-dimensional point group data created based on a predetermined three-dimensional shape, and three-dimensional point group data created based on those in which the predetermined three-dimensional shape is not tilted or tilted less than a predetermined angle. , The first neural network and the second neural network so that the identity evaluation unit outputs an evaluation indicating that the identity is high when input to the first neural network and the second neural network as the third teacher data. Update while sharing the weights of the neural network with each other
The 3D point group data created based on the predetermined 3D shape and the 3D point group data created based on the predetermined 3D shape tilted by the predetermined angle or more are obtained. 4 The first neural network and the second neural network so that the identity evaluation unit outputs an evaluation indicating that the identity is low when input to the first neural network and the second neural network as teacher data. The point group data identity estimation device according to claim 5, wherein the weights of the networks are updated while being shared with each other. Three-dimensional point group data created based on a predetermined three-dimensional shape, and one having the same size as the predetermined three-dimensional shape or the predetermined three-dimensional shape is enlarged with an enlargement ratio or reduction ratio within a predetermined range. Or, when the three-dimensional point group data created based on the reduced data is input to the first neural network and the second neural network as the fifth teacher data, the identity evaluation unit has high identity. The weights of the first neural network and the second neural network are updated while sharing each other so as to output the indicated evaluation.
It is created based on the three-dimensional point group data created based on the predetermined three-dimensional shape, and the one obtained by enlarging or reducing the predetermined three-dimensional shape at an enlargement ratio or reduction ratio exceeding the predetermined range. When the three-dimensional point group data is input to the first neural network and the second neural network as the sixth teacher data, the identity evaluation unit outputs an evaluation indicating that the identity is low. The point group data identity estimation device according to claim 5, wherein the weights of the first neural network and the second neural network are updated while being shared with each other. A measuring device that measures an object and creates 3D point cloud data,
A point cloud data identity estimation system having a point cloud data identity estimation device that estimates the identity of an object that is the source of two three-dimensional point cloud data.
The point cloud data identity estimation device is
A point cloud data acquisition unit that acquires first point cloud data and second point cloud data consisting of three-dimensional point cloud data,
A first neural network that inputs information about the first point cloud data and outputs the first point cloud data features,
A second neural network that inputs information about the second point cloud data and outputs the second point cloud data features,
Based on the first point cloud data feature amount and the second point group data feature amount, an identity evaluation unit that outputs an evaluation regarding the identity of the first point cloud data and the second point group data is provided. Have and
A point cloud data identity estimation system in which the first neural network and the second neural network share weights with each other.