TW201926258A - Three-dimensional point cloud tracking apparatus and method by recurrent neural network - Google Patents

Abstract

The present disclosure illustrates a three-dimensional point cloud tracking apparatus and method by recurrent neural network. The three-dimensional point cloud tracking apparatus and method would track the three-dimensional point cloud of the entire environment and model the entire environment by using recurrent neural network. This invention can be used to reconstruct the three-dimensional point cloud of the entire environment in the current moment, but also can be used to predict the three-dimensional point cloud of the entire environment in the follow-up moment.

Description

Three-dimensional point cloud tracking device and method using recurrent neural network

The present invention relates to a point cloud tracking device and method, and more particularly to a three-dimensional (3D) point cloud tracking device and method using a Recurrent Neural Network (RNN).

"Point cloud" refers to the form of data obtained through a three-dimensional laser scanner. Nowadays, the three-dimensional laser scanner can also be called "LiDAR", which is mainly used to sense the reflected laser beam to quickly obtain a large number of points densely on the surface of the scanned object, and because These points can all contain three-dimensional coordinates, so the light can establish a three-dimensional point cloud about the scanned object to describe the surface shape of the scanned object.

Therefore, in recent years, Ganda has been used in self-driving systems or road sensing systems as a means of avoiding obstacles or tracking vehicles. However, when the scanned object is obscured or is in a dead angle of light, the prior art cannot establish a three-dimensional point cloud about the scanned object, and thus loses the above use. In view of this, there is a need in the art for a way to reconstruct and predict a three-dimensional point cloud.

It is an object of the present invention to provide a three-dimensional point cloud tracking apparatus and method using a recurrent neural network, and in order to cope with a complex environment having multiple moving targets, the present invention uses a three-dimensional point cloud of the entire environment as a tracking object. That is, the present invention is a three-dimensional point cloud used to reconstruct and predict the entire environment.

To achieve the above objective, an embodiment of the present invention provides a three-dimensional point cloud tracking device using a recurrent neural network. The three-dimensional point cloud tracking device includes an input/output interface, a storage, and a processor. The input/output interface is used to receive different observed 3D point clouds of the environment at different times, and the observed 3D point clouds are obtained by scanning at least one light. The storage is used to store at least one memory three-dimensional point cloud about the environment. The processor is electrically connected to the input/output interface and the storage respectively for receiving the observed three-dimensional point cloud and the memory three-dimensional point cloud, and the processor utilizes when the environment receives the three-dimensional point cloud at the first moment. At least one recursive neural network model is used to perform an environmental reconstruction operation on the observed three-dimensional point cloud and the memory three-dimensional point cloud to obtain a reconstructed three-dimensional point cloud of the environment at the first moment, and then recursive neural network model is used to The three-dimensional point cloud and the blank three-dimensional point cloud are used to perform environmental prediction operations to obtain a predicted three-dimensional point cloud of the environment at the second moment. The second moment is later than the first moment.

In addition, the embodiment of the present invention further provides a three-dimensional point cloud tracking method using a recurrent neural network, which is implemented in the foregoing three-dimensional point cloud tracking device. The three-dimensional point cloud tracking method includes the following steps. The input/output interface receives different observed three-dimensional point clouds of the environment at different times, wherein the observed three-dimensional point clouds are obtained by scanning at least one light. The storage stores at least one memory three-dimensional point cloud of the environment. Having the processor receive the observed 3D point cloud and the memory 3D point cloud, and when receiving the 3D point cloud in the environment at the first moment, let the processor utilize at least one recursive neural network model to observe the 3D point cloud And the memory 3D point cloud performs the environment reconstruction operation to obtain the reconstructed 3D point cloud of the environment at the first moment, and the processor reuses the recurrent neural network model to perform the environment on the memory 3D point cloud and the blank 3D point cloud. The prediction operation is performed to obtain a predicted three-dimensional point cloud of the environment at the second moment. The second moment is later than the first moment.

The detailed description of the present invention and the accompanying drawings are to be understood by the claims The scope is subject to any restrictions.

In the following, the invention will be described in detail by way of illustration of various embodiments of the invention. However, the inventive concept may be embodied in many different forms and should not be construed as being limited to the illustrative embodiments set forth herein. In addition, the same reference numerals may be used in the drawings to represent similar elements.

Please refer to FIG. 1. FIG. 1 is a functional block diagram of a three-dimensional point cloud tracking device using a recurrent neural network according to an embodiment of the present invention. The three-dimensional point cloud tracking device 1 includes an input/output interface 11, a processor 13, and a storage 15. It should be noted that the above components may be implemented by a pure hardware circuit, or by hardware or software, or the software, but the invention is not limited thereto. In addition, the above components may be integrated or separately provided, and the present invention is not limited thereto. In summary, the present invention does not limit the specific implementation of the three-dimensional point cloud tracking device 1.

In this embodiment, the input/output interface 11 is used to receive different observed three-dimensional point clouds S at different times in the environment (not shown), and the observed three-dimensional point cloud S is composed of at least one light (not shown) ) was scanned. Since the scanning principle of the optical signal is well known to those of ordinary skill in the art, the details of the above-described observation of the three-dimensional point cloud S will not be further described herein. In addition, the storage 15 is for storing at least one memory three-dimensional point cloud M related to the environment. The specific content of the memory three-dimensional point cloud M will be described in detail below through other embodiments, and thus will not be further described herein. The processor 13 is electrically connected to the input/output interface 11 and the storage device 15, and is configured to receive the observed three-dimensional point cloud S and the memory three-dimensional point cloud M.

Please refer to FIG. 2 together. FIG. 2 is a diagram for explaining the specific operation mode of the processor 13 in FIG. As shown in FIG. 2, when the observed three-dimensional point cloud S(t) is received at a certain time (eg, time t), the processor 13 uses at least one recurrent neural network model 17 to observe the observation. The three-dimensional point cloud S(t) and the memory three-dimensional point cloud M perform an environment reconstruction operation to obtain a reconstructed three-dimensional point cloud R(t) of the environment at the first moment, and reuse the recurrent neural network model 17 to memorize the memory. The three-dimensional point cloud M and a blank three-dimensional point cloud (not shown) perform an environment prediction operation to obtain a predicted three-dimensional point cloud P(t+1) of the environment at a second time (for example, time t+1). In this regard, it should be understood that the second time is later than the first time.

However, for convenience of the following description, the first and second timings of the present embodiment are described by using only the time t and the time t+1 first, but are not intended to limit the present invention. Similarly, for convenience of the following description, the recursive neural network model 17 of FIG. 2 is also described by using only an example of a number of 1, but it is not intended to limit the present invention. That is to say, the recursive neural network model 17 for performing the environmental reconstruction operation or the environmental prediction operation in Fig. 2 may refer to the same recurrent neural network model 17, or a different recurrent neural network model 17, but The present invention is not limited thereto.

In addition, the specific operation mode for performing the environment reconstruction operation or the environment prediction operation in the recurrent neural network model 17 of FIG. 2 will be described in detail below by other embodiments, so that it is no longer More details. It should be noted that although FIG. 2 only uses the recursive neural network model 17 again to obtain the predicted three-dimensional point cloud P(t+1) of the environment at time t+1, the present invention does not For the limit. That is, as long as the processing time and the computing power are sufficient, the embodiment of the present invention can follow the predicted next time (ie, the time interval between the second and the first time). It is decided whether or not to use the recursive neural network model 17 that is recursed several times, and then the predicted three-dimensional point cloud of the environment at other second moments (for example, time t+2) is obtained.

In summary, according to the teachings of the above, it should be understood by those of ordinary skill in the art, because the present invention is particularly a three-dimensional point cloud of the entire environment as a tracking object, so when the environment is at the current time (for example, the first At time t), when there is a partial 3D point cloud in which a moving object is obscured and the environment is not obtained, the present invention estimates that this can be utilized by using a memory three-dimensional point cloud stored in the past. The three-dimensional point cloud information of the moving object is further supplemented by the above-mentioned partial area three-dimensional point cloud which is not currently known. In other words, correctly reconstruct the 3D point cloud of this environment at the current moment.

On the other hand, for the above-mentioned shaded moving object, since the prior art generally can only predict the future change of the moving object in a manner of moving at a constant speed, when the moving object moves in a non-equal speed manner, or When the moving object is obscured for too long, the prior art will easily lose track of the moving object. However, since the present invention utilizes a recursive neural network model to encode a three-dimensional point cloud of the entire environment, even a moving object in the environment moves in a complicated manner or is in a time when the moving object is obscured. When the length is too long, the present invention will be able to still predict the three-dimensional point cloud of this environment at a subsequent moment. That is, accurately tracking future changes in moving objects within this environment.

Next, please refer to FIG. 3A together. FIG. 3A is a detailed operation mode for explaining the environment reconstruction operation in the recurrent neural network model 17 of FIG. 2 . It is worth mentioning that, in order to facilitate the following description, the memory three-dimensional point cloud M in FIG. 3A is described by using an example of a number of two. That is to say, the memory three-dimensional point cloud M may include the first memory three-dimensional point cloud M1 and the second memory three-dimensional point cloud M2, but it is not intended to limit the present invention.

As shown in FIG. 3A, the recurrent neural network model 17 first performs a first sparse convolution operation on the observed three-dimensional point cloud S(t) to obtain a three-dimensional operation after the first sparse convolution SP1. Point cloud Q1(t). Then, the recursive neural network model 17 performs a second sparse convolution operation on the computed three-dimensional point cloud Q1(t) and the first memory three-dimensional point cloud M1 to obtain an operation three-dimensional point cloud after the second sparse convolution SP2. Q2(t), and the first memory three-dimensional point cloud M1 is updated by computing the three-dimensional point cloud Q2(t). Finally, the recurrent neural network model 17 performs a third sparse convolution operation on the computed three-dimensional point cloud Q2(t) and the second memory three-dimensional point cloud M2 to obtain an environment at this time t (ie, the first moment). The 3D point cloud R(t) is reconstructed, and the second memory 3D point cloud M2 is updated by reconstructing the 3D point cloud R(t).

As can be seen from the above, since FIG. 3A is an operational characteristic in which sparse convolution can be used, the three-dimensional point cloud tracking device 1 of the present embodiment is capable of processing complex three-dimensional point cloud information with reasonable time and computational power. However, since the arithmetic principle of the sparse convolution is well known to those of ordinary skill in the art, the details of the sparse convolutions SP1 to SP3 will not be further described herein. It should be noted that the three-layer sparse convolution mode (that is, the sparse convolutions SP1 to SP3) used in FIG. 3A is merely an example, and is not intended to limit the present invention. In other words, those of ordinary skill in the art should be able to perform different levels of sparse convolution design depending on actual needs or applications.

In addition, in one of the applications, the recursive neural network model 17 is capable of using only part of the sparse convolution operation result for the next layer of the sparse convolution operation or output, and the other part of the sparse convolution operation result for updating. Memory 3D point cloud. For example, after the sparse convolution SP2 of FIG. 3A, the recursive neural network model 17 can use only part of the computational three-dimensional point cloud Q2(t) for the next layer of sparse convolution operations (ie, sparse convolution). SP3), and another part of the operational three-dimensional point cloud Q2(t) is used to update the first memory three-dimensional point cloud M1.

However, since the number of convolution kernels of the operation three-dimensional point cloud Q2(t) after the second sparse convolution SP2 can be divided into several features (or channels), the operation of the above two parts is three-dimensional point cloud Q2 (t ) can refer to data that contains different channels. That is to say, the data of the above three parts of the operation three-dimensional point cloud Q2(t) may be completely unoverlapping. In summary, the present invention does not limit the specific implementation when performing a sparse convolution operation or updating a memory three-dimensional point cloud.

On the other hand, if it is considered that the first or second memory three-dimensional point cloud M1, M2 is not only completely replaceable by computing the three-dimensional point cloud Q2(t) or reconstructing the three-dimensional point cloud R(t), In one application, the processor 13 is further operable to define at least one weighted custom function f , at least one first sparse convolution kernel K1, and at least one second sparse convolution kernel K2. It is worth mentioning that the weight custom function f , the first sparse convolution kernel K1 and the second sparse convolution kernel K2 may be defined after the completion of a training mode by the three-dimensional point cloud tracking device 1, but the present invention This is not a limitation.

For example, the training mode may be a recursive neural network model 17 to perform an environmental reconstruction operation on a known three-dimensional point cloud (not shown) to obtain a reconstructed three-dimensional point cloud, and may borrow The characteristic parameters in the recursive neural network model 17 are further determined by comparing the error relationship between the known three-dimensional point cloud and the reconstructed three-dimensional point cloud (for example, the weight custom function f , the first sparse volume) The product K1 and the second sparse convolution kernel K2, etc., or the convolution kernel parameters in the sparse convolutions SP1 to SP3).

In addition, in the training mode, the recurrent neural network model 17 can also increase the sparsity of the sparse convolution by using a linear hinge loss and adding a L1 penalty. Since the principles of the training mode are well known to those of ordinary skill in the art, the above description will be merely illustrative, and the following description will not be repeated. In summary, the present invention does not limit the specific implementation manner when the three-dimensional point cloud tracking device 1 performs the training mode. Therefore, those skilled in the art should be able to perform related designs according to actual needs or applications.

Further, in a preferred implementation of updating the first memory three-dimensional point cloud M1 by computing the three-dimensional point cloud Q2(t), the recursive neural network model 17 uses the weighted custom function f to be used by the first memory. In the three-dimensional point cloud M1, the three-dimensional point cloud Q2(t), the first sparse convolution kernel K1, and the second sparse convolution kernel K2, a weight vector p is determined, and the first memory three-dimensional point cloud M1 is updated to the first The result of memorizing a three-dimensional point cloud M1, computing a three-dimensional point cloud Q2(t), and a weight vector p into a weighting equation.

Similarly, in a preferred implementation of updating the second memory three-dimensional point cloud M2 by reconstructing the three-dimensional point cloud R(t), the recursive neural network model 17 uses the weighted custom function f to obtain the second memory three-dimensional The point cloud M2, the reconstructed three-dimensional point cloud R(t), the first sparse convolution kernel K1 and the second sparse convolution kernel K2, determine the weight vector p, and update the second memory three-dimensional point cloud M2 to the second memory three-dimensional The result of the point cloud M2, the reconstruction of the three-dimensional point cloud R(t) and the weight vector p into the aforementioned weight equation.

Therefore, it should be understood that in the above two preferred embodiments for updating the first and second memory three-dimensional point clouds M1, M2, the weighting function f , the first one that can be used in each preferred embodiment And the second sparse convolution kernels K1, K2 may be different from each other. In summary, the present invention also does not limit the specific implementation of the weight custom function f , the first and second sparse convolution kernels K1, K2. Next, please refer to FIG. 3B together. FIG. 3B is a specific operation mode of a preferred embodiment for updating the first or second memory three-dimensional point cloud in the environment reconstruction operation of FIG. 3A. In FIG. 3B, the weight equation is p*C1+(1-p)*C2, and the weight vector p is expressed as: p= f (C1*K1+C2*K2), and C1 and C2 are respectively the first. Memorize the 3D point cloud M1 with the computational 3D point cloud Q2(t), or the second memory 3D point cloud M2 and reconstruct the 3D point cloud R(t).

Based on the teachings above, it should also be understood that the component of the weight vector p is between 0 and 1. That is to say, assuming that the weight vector p is all zero, the recursive neural network model 17 will use only the computational three-dimensional point cloud Q2(t) or reconstruct the three-dimensional point cloud R(t) (ie, C2). To replace the current first or second memory three-dimensional point cloud M1, M2, and so on, assuming that the weight vector p is all one, the recursive neural network model 17 will only use the original first or The second memory three-dimensional point cloud M1, M2 (ie, C1) is maintained as the current first or second memory three-dimensional point cloud M1, M2, and is not utilized to calculate the three-dimensional point cloud Q2 (t) or reconstruct three-dimensional The point cloud R(t) updates the current first or second memory three-dimensional point cloud M1, M2. In summary, the specific implementation of updating the first or second memory three-dimensional point cloud M1, M2 used in FIG. 3B is merely an example, and is not intended to limit the present invention.

Furthermore, it can be seen from the above that the memory three-dimensional point cloud M (that is, including the first and second memory three-dimensional point clouds M1, M2) stored in the storage 15 in FIG. 1 is not just the input/output interface 11 These observations of the three-dimensional point cloud S at different times are received, but the data results of the observations of the three-dimensional point cloud S after several convolution and update processes (for example, FIG. 3A). That is to say, the memory three-dimensional point cloud M in FIG. 2 is the data result after several times of convolution and update processing of the observed three-dimensional point cloud S(t-1) (not shown) received in the past. Therefore, in one of the applications, the memory three-dimensional point cloud M can be generated only after the three-dimensional point cloud tracking device 1 starts detecting the environment, instead of being stored in the storage 15 in advance. In addition, assuming that the three-dimensional point cloud S(t) is the first observation data, the memory three-dimensional point cloud M stored in the memory 15 can be processed by the blank three-dimensional point cloud after several times of convolution and update processing. produce. In summary, the present invention also does not limit the specific implementation of the memory three-dimensional point cloud M.

Furthermore, please refer to FIG. 3C together. FIG. 3C is a detailed operation mode for explaining the environment prediction operation in the recurrent neural network model 17 of FIG. 2 . As shown in FIG. 3C, the recurrent neural network model 17 first performs a fourth sparse convolution operation on a blank three-dimensional point cloud (not shown) and the first memory three-dimensional point cloud M1 to obtain a fourth thinned volume. Computation of the 3D point cloud Q3(t) after SP4. Then, the recurrent neural network model 17 performs a fifth sparse convolution operation on the computed three-dimensional point cloud Q3(t) and the second memory three-dimensional point cloud M2 to obtain an environment at time t+1 (ie, second The predicted three-dimensional point cloud P(t+1) under time). Since some of the technical principles in FIG. 3C are the same as those in FIG. 3A, no further details are provided herein. In summary, the specific implementation of the environment prediction operation used in FIG. 3C is merely an example here, and is not intended to limit the present invention.

Finally, in order to further explain the operational flow of the three-dimensional point cloud tracking device 1, the present invention further provides an embodiment of its three-dimensional point cloud tracking method. Please refer to FIG. 4. FIG. 4 is a schematic flowchart diagram of a three-dimensional point cloud tracking method using a recurrent neural network according to an embodiment of the present invention. The three-dimensional point cloud tracking method of FIG. 4 can be executed in the three-dimensional point cloud tracking device 1 of FIG. 1 , but the present invention does not limit the three-dimensional point cloud tracking method of FIG. 4 to only perform the three-dimensional point cloud tracking of FIG. 1 . In device 1. In addition, the detailed step flow is as described in the foregoing embodiments, and is merely summarized herein and will not be redundant.

As shown in FIG. 4, first, in step S410, the input/output interface is configured to receive different observed three-dimensional point clouds of the environment at different times, wherein the observed three-dimensional point clouds are obtained by scanning at least one light. Next, in step S420, the storage is caused to store at least one memory three-dimensional point cloud of the environment. Next, in step S430, the processor is caused to receive the observed three-dimensional point cloud and the memory three-dimensional point cloud, and when the observed three-dimensional point cloud is received at the first time, the process proceeds to step S440 to step S450.

In step S440, the processor is configured to perform an environment reconstruction operation on the observed three-dimensional point cloud and the memory three-dimensional point cloud by using at least one recurrent neural network model to obtain a reconstructed three-dimensional point cloud of the environment at the first moment, and In step S450, the processor uses the recurrent neural network model to perform an environment prediction operation on the memory three-dimensional point cloud and the blank three-dimensional point cloud to obtain a predicted three-dimensional point cloud of the environment at the second moment. The second moment is later than the first moment.

In light of the above teachings, those of ordinary skill in the art will appreciate that steps S410, S420, and S430 should be performed in parallel without conflicting steps. In addition, in order to further explain the implementation details regarding step S440, the present invention further provides an embodiment of step S440 thereof. Please refer to FIG. 5A. FIG. 5A is a schematic flowchart of an environment reconstruction operation using a recurrent neural network model in the three-dimensional point cloud tracking method of FIG. 4. The steps in FIG. 5A that are the same as those in FIG. 4 are denoted by the same reference numerals, and thus the details thereof will not be described in detail.

In the embodiment of FIG. 5A, step S440 may further include steps S441 to S445. First, in step S441, the recursive neural network model first performs a first sparse convolution operation on the observed three-dimensional point cloud to obtain a first computed three-dimensional point cloud after the first sparse convolution. Next, in step S443, the recursive neural network model performs a second sparse convolution operation on the first computed three-dimensional point cloud and the first memory three-dimensional point cloud stored in the storage to obtain the second sparsely convolved convolution The second computing three-dimensional point cloud, and updating the first memory three-dimensional point cloud with the second computing three-dimensional point cloud.

Then, in step S445, the recursive neural network model performs a third sparse convolution operation on the second computed three-dimensional point cloud and the second memory three-dimensional point cloud stored in the storage to obtain the environment at the first moment. The 3D point cloud is reconstructed, and the second memory 3D point cloud is updated by reconstructing the 3D point cloud. It is to be noted that the embodiment adopted in FIG. 5A is merely for exemplification, and is not intended to limit the present invention. In addition, a preferred embodiment of updating the first or second memory three-dimensional point cloud in FIG. 5A can be referred to FIG. 3B, and thus no further details are provided herein.

In addition, in order to further explain the implementation details regarding step S450, the present invention further provides an embodiment of step S450 thereof. Please refer to FIG. 5B. FIG. 5B is a schematic flowchart of the environment prediction operation using the recurrent neural network model in the three-dimensional point cloud tracking method of FIG. 4. The process steps in FIG. 5B that are the same as those in FIG. 4 are denoted by the same reference numerals, and thus the details thereof will not be described in detail.

In the embodiment of FIG. 5B, step S450 may further include steps S451 to S453. First, in step S451, the recursive neural network model first performs a fourth sparse convolution operation on a blank three-dimensional point cloud and a first memory three-dimensional point cloud stored in the storage to obtain a fourth sparse convolution. The third operation of the 3D point cloud. Then, in step S453, the recursive neural network model performs a fifth sparse convolution operation on the second computed three-dimensional point cloud and the second memory three-dimensional point cloud stored in the storage to obtain the environment at the second moment. The prediction of 3D point clouds. Since the detailed step flow is as described in the foregoing embodiment, the details thereof will not be described in detail herein.

In summary, the 3D point cloud tracking device and method using the recurrent neural network provided by the embodiment of the present invention can be used not only to reconstruct a 3D point cloud of the entire environment, but also to predict the entire environment at a subsequent time. The 3D point cloud below. In particular, since the present invention uses a three-dimensional point cloud of the entire environment as a tracking object, in the process of reconstruction, the present invention can utilize past point cloud information to mask the current moving object of the environment. The partial region point cloud that cannot be detected is supplemented to thereby correctly reconstruct the three-dimensional point cloud of the environment at the current moment, and in the process of prediction, the present invention can be utilized by using a recurrent neural network model. The entire environment is modeled (ie, coded) in order to predict the 3D point cloud of the environment at subsequent times and thereby accurately track future changes in a moving object within the environment. In addition, the present invention can also apply the operational characteristics of sparse convolution, so the present invention can effectively process complex three-dimensional point cloud information in a reasonable time and computing power, thereby implementing the above reconstruction and prediction. The best results.

The above description is only an embodiment of the present invention, and is not intended to limit the scope of the invention.

1‧‧‧3D point cloud tracking device

11‧‧‧Input/Output Interface

13‧‧‧ Processor

15‧‧‧Storage

S, S (t) ‧ ‧ observation three-dimensional point cloud

M‧‧‧ memory three-dimensional point cloud

17‧‧‧Recurrent neural network model

R(t)‧‧‧Reconstruct 3D point cloud

P(t+1)‧‧‧ forecast 3D point cloud

M1‧‧‧ first memory 3D point cloud

M2‧‧‧Second memory 3D point cloud

SP1～SP5‧‧‧Sparse Convolution

Q1(t)~Q3(t)‧‧‧ computing three-dimensional point cloud

C1, C2‧‧‧3D point cloud

K1‧‧‧first sparse convolution kernel

K2‧‧‧Second sparse convolution kernel

f ‧‧‧weight custom function

P‧‧‧weight vector

S410～S450, S441～S445, S451～S453‧‧‧ Process steps

FIG. 1 is a functional block diagram of a three-dimensional point cloud tracking device using a recurrent neural network according to an embodiment of the present invention. 2 is a schematic diagram of the operation of a processor in the three-dimensional point cloud tracking device of FIG. 1. FIG. 3A is a schematic diagram of the operation of performing an environment reconstruction operation in the recurrent neural network model of FIG. 2. FIG. FIG. 3B is a schematic diagram of the operation of a preferred embodiment of updating the first or second memory three-dimensional point cloud in the environment reconstruction operation of FIG. 3A. FIG. 3C is a schematic diagram of the operation of the environment prediction operation in the recurrent neural network model of FIG. 2. FIG. 4 is a schematic flow chart of a three-dimensional point cloud tracking method using a recurrent neural network according to an embodiment of the present invention. FIG. 5A is a schematic flow chart of an environment reconstruction operation using a recurrent neural network model in the three-dimensional point cloud tracking method of FIG. 4. FIG. FIG. 5B is a schematic flow chart of the environment prediction operation using the recurrent neural network model in the three-dimensional point cloud tracking method of FIG. 4. FIG.

Claims (14)

A three-dimensional point cloud tracking device using a recurrent neural network, comprising: an input/output interface for receiving different observed three-dimensional point clouds of an environment at different times, wherein the three-dimensional point cloud systems are at least one light Obtaining a memory for storing at least one memory three-dimensional point cloud of the environment; and a processor electrically connected to the input/output interface and the storage for receiving the observed three-dimensional points a cloud and the at least one memory three-dimensional point cloud, and when receiving the observed three-dimensional point cloud of the environment at a first time, the processor utilizes at least one recurrent neural network model to observe the three-dimensional point cloud and Performing an environment reconstruction operation on the at least one memory three-dimensional point cloud to obtain a reconstructed three-dimensional point cloud of the environment at the first moment, and reusing the recursive neural network model to the at least one memory three-dimensional point cloud and one The blank three-dimensional point cloud performs an environment prediction operation to obtain a predicted three-dimensional point cloud of the environment at a second moment, wherein the second moment is later than the first moment. The three-dimensional point cloud tracking device of claim 1, wherein the at least one memory three-dimensional point cloud comprises a first memory three-dimensional point cloud and a second memory three-dimensional point cloud, and using the recurrent neural network model, Performing the environment reconstruction operation on the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud to obtain the reconstructed three-dimensional point cloud in the environment at the first moment, comprising: performing the observed three-dimensional point cloud a first sparse convolution operation to obtain a first computing three-dimensional point cloud; performing a second sparse convolution operation on the first computing three-dimensional point cloud and the first memory three-dimensional point cloud to obtain a second computing three-dimensional Point cloud, and updating the first memory three-dimensional point cloud with the second computing three-dimensional point cloud; and performing a third sparse convolution operation on the second computing three-dimensional point cloud and the second memory three-dimensional point cloud to obtain the The environment reconstructs the three-dimensional point cloud at the first moment, and updates the second memory three-dimensional point cloud with the reconstructed three-dimensional point cloud. The three-dimensional point cloud tracking device of claim 2, wherein the recursive neural network model is used to perform an environment prediction operation on the at least one memory three-dimensional point cloud and the blank three-dimensional point cloud to obtain the environment. The step of predicting the three-dimensional point cloud in the second moment includes: performing a fourth sparse convolution operation on the blank three-dimensional point cloud and the first memory three-dimensional point cloud to obtain a third computing three-dimensional point cloud; And performing a fifth sparse convolution operation on the third computing three-dimensional point cloud and the second memory three-dimensional point cloud to obtain the predicted three-dimensional point cloud of the environment at the second moment. The three-dimensional point cloud tracking device of claim 3, wherein the processor is further configured to define at least one weight custom function, at least one first sparse convolution kernel, and at least one second sparse convolution kernel, and And the step of updating the first memory three-dimensional point cloud by using the second computing three-dimensional point cloud, further comprising: using the weight custom function to generate the first memory three-dimensional point cloud, the second computing three-dimensional point cloud, the first a sparse convolution kernel and the second sparse convolution kernel, determining a weight vector, and updating the first memory three-dimensional point cloud to the first memory three-dimensional point cloud, the second computing three-dimensional point cloud, and the weight vector The result of substituting a weight equation. The third-dimensional point cloud tracking device of claim 4, wherein, in the step of updating the second memory three-dimensional point cloud with the reconstructed three-dimensional point cloud, the method further comprises: using the weight custom function to be used by the second Determining the weight vector in the memory three-dimensional point cloud, the reconstructed three-dimensional point cloud, the first sparse convolution kernel, and the second sparse convolution kernel, and updating the second memory three-dimensional point cloud to the second memory three-dimensional point The cloud, the reconstructed three-dimensional point cloud, and the result of the weight vector being substituted into the weighting equation. The three-dimensional point cloud tracking device of claim 5, wherein the weight self-determination function, the first sparse convolution kernel, and the second sparse convolution kernel are performed through the three-dimensional point cloud tracking device. The mode is defined after the mode, and the component of the weight vector is between 0 and 1. The three-dimensional point cloud tracking device of claim 6, wherein the weighting equation is p*C1+(1-p)*C2, where p is the weight vector, and C1 and C2 are respectively the first memory three-dimensional point The cloud and the second computing three-dimensional point cloud, or the second memory three-dimensional point cloud and the reconstructed three-dimensional point cloud. A three-dimensional point cloud tracking method using a recurrent neural network is implemented in a three-dimensional point cloud tracking device, the three-dimensional point cloud tracking device comprising an input/output interface, a storage and a processor, and the three-dimensional point cloud tracking method The method includes: causing the input/output interface to receive a different observed three-dimensional point cloud of an environment at different times, wherein the observed three-dimensional point cloud is obtained by scanning at least one light; and causing the storage to store at least one of the environment Memorizing the three-dimensional point cloud; and causing the processor to receive the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud, and when the environment receives the observed three-dimensional point cloud at a first time, the processor is Performing an environment reconstruction operation on the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud by using at least one recurrent neural network model to obtain a reconstructed three-dimensional point cloud of the environment at the first moment, and causing the processing Using the recursive neural network model, the environment prediction operation is performed on the at least one memory three-dimensional point cloud and a blank three-dimensional point cloud to obtain the environment. He predicted a 3D point cloud in two moments, wherein the first time point in the time line of the second night. The method of claim 3, wherein the at least one memory three-dimensional point cloud comprises a first memory three-dimensional point cloud and a second memory three-dimensional point cloud, and using the recurrent neural network model, Performing the environment reconstruction operation on the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud to obtain the reconstructed three-dimensional point cloud in the environment at the first moment, comprising: performing the observed three-dimensional point cloud a first sparse convolution operation to obtain a first computing three-dimensional point cloud; performing a second sparse convolution operation on the first computing three-dimensional point cloud and the first memory three-dimensional point cloud to obtain a second computing three-dimensional Point cloud, and updating the first memory three-dimensional point cloud with the second computing three-dimensional point cloud; and performing a third sparse convolution operation on the second computing three-dimensional point cloud and the second memory three-dimensional point cloud to obtain the The environment reconstructs the three-dimensional point cloud at the first moment, and updates the second memory three-dimensional point cloud with the reconstructed three-dimensional point cloud. The method of claim 3, wherein the recursive neural network model is used to perform an environment prediction operation on the at least one memory three-dimensional point cloud and the blank three-dimensional point cloud to obtain the environment. The step of predicting the three-dimensional point cloud in the second moment includes: performing a fourth sparse convolution operation on the blank three-dimensional point cloud and the first memory three-dimensional point cloud to obtain a third computing three-dimensional point cloud; And performing a fifth sparse convolution operation on the third computing three-dimensional point cloud and the second memory three-dimensional point cloud to obtain the predicted three-dimensional point cloud of the environment at the second moment. The method of claim 3, wherein the processor is further configured to define at least one weight custom function, at least one first sparse convolution kernel, and at least one second sparse convolution kernel, and And the step of updating the first memory three-dimensional point cloud by using the second computing three-dimensional point cloud, further comprising: using the weight custom function to generate the first memory three-dimensional point cloud, the second computing three-dimensional point cloud, the first a sparse convolution kernel and the second sparse convolution kernel, determining a weight vector, and updating the first memory three-dimensional point cloud to the first memory three-dimensional point cloud, the second computing three-dimensional point cloud, and the weight vector The result of substituting a weight equation. The method of claim 3, wherein in the step of updating the second three-dimensional point cloud with the reconstructed three-dimensional point cloud, the method further comprises: using the weighting function to obtain the second Determining the weight vector in the memory three-dimensional point cloud, the reconstructed three-dimensional point cloud, the first sparse convolution kernel, and the second sparse convolution kernel, and updating the second memory three-dimensional point cloud to the second memory three-dimensional point The cloud, the reconstructed three-dimensional point cloud, and the result of the weight vector being substituted into the weighting equation. The three-dimensional point cloud tracking method of claim 12, wherein the weight self-determination function, the first sparse convolution kernel, and the second sparse convolution kernel are performed through the three-dimensional point cloud tracking device. The mode is defined after the mode, and the component of the weight vector is between 0 and 1. The method of claim 3, wherein the weighting equation is p*C1+(1-p)*C2, where p is the weight vector, and C1 and C2 are the first memory three-dimensional point, respectively. The cloud and the second computing three-dimensional point cloud, or the second memory three-dimensional point cloud and the reconstructed three-dimensional point cloud.