CN114543797A

CN114543797A - Pose prediction method and apparatus, device, and medium

Info

Publication number: CN114543797A
Application number: CN202210152194.1A
Authority: CN
Inventors: 陈浩然; 刘浩敏
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2022-02-18
Filing date: 2022-02-18
Publication date: 2022-05-27
Also published as: WO2023155434A1

Abstract

The application discloses a pose prediction method, a pose prediction device, equipment and a medium, wherein the pose prediction method comprises the following steps: acquiring a plurality of pose information and a plurality of inertial measurement data of a target object in a first preset time period; determining first certainty degree representation information of each pose information based on a plurality of inertial measurement data; and predicting the target pose information of the target object in a second preset time period by using the pose information, the inertia measurement data and the first certainty factor representation information of each pose information. By the aid of the scheme, accuracy of pose prediction can be improved.

Description

Pose prediction method and apparatus, device, and medium

Technical Field

The application relates to the technical field of computer vision, in particular to a pose prediction method, a pose prediction device, pose prediction equipment and pose prediction media.

Background

With the development of computer vision and the gradual maturity of hardware, Augmented Reality (AR) technology is rapidly developing. Augmented reality technology can blend virtual objects into the real world and allow users to interact with it. In order to better integrate a virtual object into the real world, the current position of the AR device needs to be accurately determined, and the current position of the AR device is determined mainly by using inertial measurement data in a period of time to predict the pose of the AR device in a future period of time.

Disclosure of Invention

The application at least provides a pose prediction method, a pose prediction device, pose prediction equipment and pose prediction media.

The application provides a pose prediction method, which comprises the following steps: acquiring a plurality of pose information and a plurality of inertial measurement data of a target object in a first preset time period; determining first certainty degree representation information of each pose information based on a plurality of inertial measurement data; and predicting target pose information of the target object in a second preset time period by using the pose information, the inertial measurement data and the first certainty degree representation information of each pose information.

Therefore, by determining the first certainty factor representing information of each pose information by using a plurality of pieces of inertial measurement data of the target object in the first preset time period, the determination factor representing information can be used for representing the certainty factor of each pose information, so that the predicted target pose information is more accurate according to the plurality of pose information, the plurality of pieces of inertial measurement data and the first certainty factor representing information of each pose information of the target object in the first preset time period.

The method for predicting the target pose information of the target object in the second preset time period by using the pose information, the inertial measurement data and the first certainty factor representation information of each pose information comprises the following steps: inputting the pose information, the inertial measurement data and the first certainty degree representation information of each pose information into a target pose prediction network, and predicting target pose information of a target object in a second preset time period; the target pose prediction network comprises an encoding layer and a decoding layer.

Therefore, the target pose information of the target object in the second preset time period is acquired by using the target pose prediction network, so that the prediction process is convenient and quick.

The method for predicting the target pose of the target object in the second preset time period by inputting the pose information, the inertial measurement data and the first certainty degree representation information of each pose information into a target pose prediction network comprises the following steps: the method comprises the steps that a coding layer is utilized to code a plurality of pieces of inertia measurement data to obtain a first vector, a plurality of pieces of position information and first determining table characteristic information of the position information are coded to obtain a second vector, and the first vector and the second vector are used for representing motion parameters of a target object in a first preset time period; and decoding by using a decoding layer based on the first vector and the second vector to obtain target pose information.

Therefore, the inertial measurement data, the pose information and the first certainty factor representation information of each pose information are encoded by using the target pose prediction network, so that the predicted target pose information is more accurate.

The target pose prediction network further comprises a merging layer; predicting target pose information of the target object in a second preset time period by using the pose information, the inertial measurement data and the first certainty factor representation information of each pose information, and further comprising: splicing the first vector, the second vector, the position and attitude information and the inertia measurement data by utilizing a merging layer to obtain a third vector; decoding by using a decoding layer based on the first vector and the second vector to obtain target pose information, wherein the target pose information comprises: and decoding the third vector by using a decoding layer to obtain target pose information.

Therefore, the third vector is obtained by splicing the first vector, the second vector, the position and posture information and the inertia measurement data, so that the third vector can contain more layers of motion information, and the determined target position and posture information is more accurate.

Wherein, the method further comprises: inputting the plurality of position posture information, the plurality of inertial measurement data and first certainty degree representation information of the position posture information into a reference position posture prediction network to obtain reference position posture information, wherein the network structures of the reference position posture prediction network and the target position posture prediction network are the same; acquiring real pose information in a second preset time period, and adjusting parameters in the reference pose prediction network by utilizing first loss between the real pose information and the reference pose information; acquiring a second loss between the real pose information and the target pose information; in response to the first loss being less than the second loss, parameters in the target pose prediction network are updated with parameters in the reference pose prediction network.

Therefore, by updating the parameters in the target pose prediction network by using the parameters in the reference pose prediction network under the condition that the loss of the reference pose prediction network is less than the second loss, the stability and the generalization capability of the target pose prediction network can be guaranteed.

The target pose prediction network is further configured to obtain second certainty factor representing information of the target pose information, and the reference pose prediction network is further configured to obtain reference certainty factor representing information of the reference pose information, and obtain a first loss mode or a second loss mode, where the method includes: taking the first loss as a target loss, the reference pose information as pose information to be processed, and the reference certainty factor representation information as certainty factor representation information of the pose information to be processed; or taking the second loss as the target loss, the target pose information as pose information to be processed, and the second certainty factor characterizing information as certainty factor characterizing information of the pose information to be processed; acquiring a certainty matrix representation corresponding to the pose information to be processed, acquiring a mean square loss corresponding to the vector representation of the pose difference, and acquiring a seventh product between the certainty characteristic information of the pose information to be processed and a second preset value, wherein the certainty matrix representation is a matrix representation of the opposite number of the certainty characteristic information of the pose information to be processed, and the pose difference represents the difference between the real pose information and the pose information to be processed; and acquiring an eighth product of the certainty matrix representation and the mean square loss, and adding the eighth product and the seventh product to obtain the target loss.

Therefore, the target loss is determined by combining the certainty of the pose information to be processed and the corresponding pose difference, so that the determined target loss is more accurate.

The method for acquiring the pose information of the target object in the first preset time period comprises the following steps: acquiring a visual pose sequence and an inertia pose sequence of a target object in a first preset time period, wherein the visual pose sequence comprises the visual pose of the target object at the shooting time of a plurality of positioning images, and the inertia pose sequence comprises the inertia pose of the target object at the measurement time of a plurality of inertia measurement data; and obtaining a relative pose sequence related to the inertia pose sequence based on the vision pose sequence and the inertia pose sequence, wherein the relative pose sequence comprises pose information corresponding to each measurement moment, and the pose information corresponding to each measurement moment represents the relative pose between the inertia pose at the measurement moment and the associated vision pose.

Therefore, the relative pose sequence is determined according to the visual pose sequence and the inertia pose sequence, and the situation that the accuracy of predicting the target pose information is low due to the error of each inertia pose in the inertia pose sequence can be reduced.

Wherein obtaining a relative pose sequence for the inertial pose sequence based on the visual pose sequence and the inertial pose sequence comprises: respectively taking each measuring moment as a target measuring moment; taking the inertial pose at the target measurement time as a target inertial pose, and selecting a visual pose associated with the target measurement time as a target visual pose, wherein the shooting time corresponding to the target visual pose is earlier than the target measurement time; and acquiring the relative pose between the target inertia pose and the target vision pose as pose information corresponding to the target measurement time.

Therefore, each measuring moment is taken as a target measuring moment, and then the visual pose associated with the target measuring moment is selected as the target visual pose, so that the corresponding pose information is determined, and the determined pose information is more accurate.

The plurality of pose information comprises pose information corresponding to the measurement time of each inertial measurement datum, the pose information corresponding to the measurement time represents the relative pose between the inertial pose at the measurement time and the associated visual pose, and the first certainty table characteristic information comprises first uncertainty; determining first certainty degree representation information of each pose information based on a plurality of inertial measurement data, wherein the first certainty degree representation information comprises the following steps: respectively taking each measuring moment as a current measuring moment, and taking a measuring moment adjacent to the current measuring moment as a reference measuring moment; and obtaining a first uncertainty of the pose information corresponding to the current measurement time based on the inertia information of the reference measurement time and the relative pose information of the current measurement time and the reference measurement time, wherein the inertia information of the reference measurement time comprises inertia measurement data of the reference measurement time and at least one inertia pose component, and the inertia pose component is the component of the inertia pose of the reference measurement time.

Therefore, the acquisition of the first uncertainty of the pose information at the current measurement time is realized by based on the inertia information at the reference measurement time and the relative pose information between the current measurement time and the reference measurement time.

The method for obtaining the first uncertainty of the pose information corresponding to the current measurement time based on the inertia information of the reference measurement time and the relative pose information of the current measurement time and the reference measurement time comprises the following steps: obtaining a second uncertainty of the inertial pose of the current measurement time based on the inertial information of the reference measurement time and the reference uncertainty, wherein the reference uncertainty is the second uncertainty of the inertial pose of the reference measurement time; obtaining a first error parameter corresponding to the current measuring time based on the relative pose information of the current measuring time and the reference measuring time, wherein the first error parameter represents the transmission degree of the pose error from the inertial pose of the current measuring time to the pose information of the current measuring time; and obtaining the first uncertainty of the pose information of the current measuring moment based on the second uncertainty and the first error parameter corresponding to the current measuring moment.

Therefore, the second uncertainty of the inertia pose at the current measuring time is determined according to the reference uncertainty and the inertia information of the reference measuring time, the first error parameter corresponding to the current measuring time is determined according to the relative pose information of the current measuring time and the reference measuring time, and then the first uncertainty of the pose information at the current measuring time is determined according to the second uncertainty of the inertia pose at the current measuring time and the first error parameter corresponding to the current measuring time.

Wherein, based on the inertial information of the reference measurement time and the reference uncertainty, obtaining a second uncertainty of the inertial pose of the current measurement time, comprises: determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertia information of the reference measurement time, wherein the second error parameter represents the contribution degree of the inertia pose error of the reference measurement time to the inertia pose error of the current measurement time, and the third error parameter represents the contribution degree of the inertia measurement noise of the reference measurement time to the inertia pose error of the current measurement time; and obtaining a second uncertainty of the inertial pose of the current measurement moment by using the second error parameter and the third error parameter corresponding to the reference measurement moment and the reference uncertainty.

Therefore, the corresponding second error parameter and the third error parameter are determined by referring to the inertial information at the measurement time, so that the second uncertainty of the inertial pose at the current measurement time can be acquired by combining the reference uncertainty.

The method for obtaining the second uncertainty of the inertial pose at the current measurement moment by using the second error parameter and the third error parameter corresponding to the reference measurement moment and the reference uncertainty comprises the following steps: multiplying the second error parameter, the reference uncertainty and the first transposition to obtain a first product, and multiplying the third error parameter, a preset uncertainty and a second transposition to obtain a second product, wherein the preset uncertainty is used as the uncertainty corresponding to the error of the inertia measurement data at each measurement moment, the first transposition is the transposition of the second error parameter, and the second transposition is the transposition of the third error parameter; adding the first product and the second product to obtain a second uncertainty of the inertial pose at the current measurement moment; and/or the inertial measurement data at the reference measurement moment comprise the angular velocity and the acceleration at the reference measurement moment, at least one inertial pose component at the reference measurement moment comprises the inertial rotation at the reference measurement moment, and the second error parameter and the third error parameter are matrixes; determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertial information of the reference measurement time, including: obtaining a first element of a second error parameter based on matrix representation of a third product of the angular velocity and the measurement time difference at the reference measurement time, obtaining a fourth product between the inertia rotation and the oblique symmetry matrix of the acceleration at the reference measurement time, obtaining a second element of the second error parameter based on the product of the fourth product and the measurement time difference, and obtaining a third element of the second error parameter based on the product of the fourth product and the square of the measurement time difference; obtaining a second error parameter based on the first element, the second element and the third element, wherein the measurement time difference is a time difference between adjacent measurement moments, and the first element, the second element and the third element are positioned in the same column; obtaining a fourth element of a third error parameter based on a product of the first preset matrix of the third product and the measurement time difference, obtaining a fifth element of the third error parameter based on a product of the inertial rotation at the parameter measurement time and the measurement time difference, and obtaining a sixth element of the third error parameter based on a product of the inertial rotation at the reference measurement time and the square of the measurement time difference; and obtaining a third error parameter based on a fourth element, a fifth element and a sixth element, wherein the fourth element and the fifth element are in different columns, and the fifth element and the sixth element are in the same column.

Therefore, the corresponding second error parameter and the third error parameter are determined by referring to the inertial information of the measurement moment, so that the second uncertainty of the inertial pose of the current measurement moment can be acquired by combining the reference uncertainty. In addition, because certain errors inevitably exist in the measurement process of the angular velocity and the acceleration, the second error parameter and the third error parameter are determined by referring to the angular velocity and the acceleration at the measurement moment, so that the determined second error parameter and the determined third error parameter are more accurate.

The relative pose information of the current measurement time and the reference measurement time is the inertial rotation change between the reference measurement time and the current measurement time; obtaining a first error parameter corresponding to the current measurement time based on the relative pose information of the current measurement time and the reference measurement time, and the method comprises the following steps: acquiring a second preset matrix represented by the vector of the inertial rotation change, and taking the product of the inverse matrix of the second preset matrix and the inertial rotation change as a seventh element of the first error parameter; based on the seventh element, a first error parameter is determined.

Therefore, the vector of the inertial rotation change represents the corresponding second preset matrix, and the first error parameter is determined.

Wherein, the method further comprises: predicting second certainty factor representation information corresponding to the target pose information of the target object in a second preset time period by using the plurality of pose information, the plurality of inertial measurement data and the first certainty factor representation information of each pose information; and correcting the target pose information based on the second certainty factor representation information to obtain corrected target pose information.

Therefore, second certainty factor representing information corresponding to the target pose information is predicted by utilizing the plurality of pose information, the plurality of inertial measurement data and the first certainty factor representing information of the each pose information, so that the target pose can be corrected according to the second certainty factor representing information, and the corrected target pose information is more accurate.

Wherein, based on the second certainty factor representation information, the target pose information is corrected to obtain corrected target pose information, and the method comprises the following steps: acquiring historical pose information of the target object, wherein the historical pose information comprises corrected target pose information obtained in the last pose prediction process; and correcting the target pose information by combining the historical pose information and the second certainty factor representation information to obtain corrected target pose information of the pose prediction process.

Therefore, the historical pose information and the second certainty factor representation information are combined to correct the pose of the target, so that the corrected pose of the target is more accurate.

The second certainty factor representing information comprises a third uncertainty which is used for representing an error of the target pose information, the historical pose information comprises historical rotation information and historical translation information, the target pose information comprises target rotation information and target translation information, and the target pose information is corrected by combining the historical pose information and the second certainty factor representing information to obtain corrected target pose information of the pose prediction process, and the method comprises the following steps: acquiring a difference value between the first preset value and the third uncertainty as a reference difference value, acquiring a fifth product between the reference difference value and the historical translation information and a sixth product between the third uncertainty and the target translation information, and taking the sum of the fifth product and the sixth product as corrected target translation information; and performing interpolation processing by taking the historical rotation information as a starting point, the target rotation information as an end point and the reference difference value as an interpolation parameter to obtain corrected target rotation information.

Therefore, the target rotation information and the target translation information are corrected, so that the determined target pose information is more accurate.

After the target pose information of the target object in the second preset time period is predicted, the method further comprises the following steps: obtaining a target pose of the target object in a second preset time period based on the pose of the target object in the first preset time period and target pose information, wherein the target pose information comprises a relative pose of the second preset time period relative to the first preset time period; and/or determining the relative pose of the virtual object and the target object in a second preset time period based on the target pose information of the target object in the second preset time period, and determining the display position of the virtual object on the display screen of the target object based on the relative pose of the virtual object and the target object.

Therefore, the target pose of the target object in the second preset time period can be determined through the target pose information. In addition, after the target pose information of the target object in the second preset time period is obtained, the method can be applied to the technical field of augmented reality, and the accuracy of the real position of the virtual object on the display picture of the target object is improved.

The application provides a position appearance prediction unit includes: the data acquisition module is used for acquiring a plurality of position and posture information and a plurality of inertial measurement data of the target object within a first preset time period; the preprocessing module is used for determining first certainty factor representation information of each pose information based on a plurality of inertial measurement data; and the prediction module is used for predicting the target pose information of the target object in a second preset time period by utilizing the plurality of pose information, the plurality of inertial measurement data and the first certainty degree representation information of each pose information.

The application provides an electronic device comprising a memory and a processor, wherein the processor is used for executing program instructions stored in the memory so as to realize the pose prediction method.

Wherein, the electronic equipment is AR glasses.

The present application provides a computer-readable storage medium having stored thereon program instructions that, when executed by a processor, implement the above pose prediction method.

According to the scheme, the first certainty factor representing information of each pose information is determined by utilizing a plurality of inertial measurement data of the target object in the first preset time period, the determined characterization information can be used for representing the certainty factor of each pose information, and therefore the predicted target pose information is more accurate according to the plurality of pose information, the plurality of inertial measurement data and the first certainty factor representing information of each pose information of the target object in the first preset time period.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a pose prediction method of the present application;

fig. 2 is a partial sub-flowchart diagram illustrating step S12 in an embodiment of the pose prediction method of the present application;

FIG. 3 is another schematic flow chart diagram illustrating an embodiment of the pose prediction method of the present application;

FIG. 4 is a schematic flowchart illustrating a process of correcting pose information of an object according to an embodiment of the present disclosure;

FIG. 5 is a schematic flowchart illustrating a process of obtaining training samples according to an embodiment of the pose prediction method of the present application;

FIG. 6 is a schematic structural diagram of an embodiment of the pose prediction apparatus of the present application;

FIG. 7 is a schematic structural diagram of an embodiment of an electronic device of the present application;

FIG. 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The following describes in detail the embodiments of the present application with reference to the drawings attached hereto.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The term "and/or" herein is merely an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of a pose prediction method according to the present application.

Specifically, the pose prediction method provided by the embodiment of the present disclosure may include the following steps:

step S11: the method comprises the steps of obtaining a plurality of pose information and a plurality of inertia measurement data of a target object in a first preset time period.

In the embodiments of the present disclosure, several means one or more. The target object may be an execution device of the pose prediction method in the embodiment of the disclosure, for example, the target object may be an AR device such as AR glasses, an intelligent vehicle-mounted device, an intelligent mobile phone, and other intelligent terminals. Illustratively, a plurality of pose information of the target object in the first preset time period may be obtained by a visual inertial positioning manner, or may be obtained by a positioning device with a fixed relative pose with respect to the target object.

The inertial measurement data may be measured by an inertial measurement device fixed in relative pose with respect to the target object, and the inertial measurement device may be an IMU.

Step S12: and determining first certainty degree representation information of each pose information based on the plurality of inertial measurement data.

Generally speaking, a certain offset may exist in an inertial measurement device, which causes a certain error to exist in measured inertial measurement data, so that several acquired pose information may be inaccurate. And determining first certainty factor representation information of each pose information through the inertial measurement data, so that the first certainty factor representation information can be used as a credibility reference of each pose information.

Step S13: and predicting target pose information of the target object in a second preset time period by using the pose information, the inertial measurement data and the first certainty degree representation information of each pose information.

Wherein the start point of the second preset time period is later than the start point of the first preset time period. For example, the start point of the second preset time period may be the end point of the first preset time period.

In some disclosed embodiments, the manner of acquiring the pose information of the target object within the first preset time period may be:

and acquiring a visual pose sequence and an inertial pose sequence of the target object in a first preset time period. The visual pose sequence comprises visual poses of the target object at the shooting moments of the plurality of positioning images. The inertial pose sequence includes inertial poses of the target object at measurement times of the plurality of inertial measurement data. The positioning image may be an image of the target object captured from the environment at a certain time, and the image may be used to position the target object. Illustratively, the positioning image may be combined with inertial measurement data to achieve visual inertial positioning.

In some application scenarios, the frame rate for capturing the positioning images may be lower than the measurement frequency of the inertial measurement data. That is, there may be a time difference between the capturing time of each positioning image and the measurement time of each inertial measurement data. Therefore, the number of visual poses included in the visual pose sequence in the first preset time period may be less than the number of inertial poses included in the inertial pose sequence in the time period.

And obtaining a relative pose sequence related to the inertia pose sequence based on the visual pose sequence and the inertia pose sequence. And the relative pose sequence comprises pose information corresponding to each measuring moment. The pose information corresponding to each measurement time represents the relative pose between the inertial pose and the associated visual pose at the measurement time. That is, several pieces of pose information of the target object obtained in step S11 in the first preset time period may be pose information corresponding to the target object at each measurement time.

By determining the relative pose sequence according to the visual pose sequence and the inertial pose sequence, the situation that the accuracy of predicting target pose information is low due to errors of all inertial poses in the inertial pose sequence can be reduced.

In some disclosed embodiments, based on the visual pose sequence and the inertial pose sequence, the manner of obtaining a relative pose sequence for the inertial pose sequence may be:

and taking each measuring moment as a target measuring moment respectively. And then, taking the inertial measurement pose at the target measurement time as a target inertial pose, and selecting a visual pose associated with the target measurement time as a target visual pose. And the shooting time corresponding to the target visual pose is earlier than the target measurement time. Optionally, the shooting time corresponding to the visual pose associated with the target measurement time, that is, the shooting time corresponding to the target visual pose, is the shooting time with the smallest time difference with the target measurement time among several shooting times in the visual pose sequence, where the shooting time is earlier than the target measurement time.

And acquiring the relative pose between the target inertia pose and the target vision pose as pose information corresponding to the target measurement time. As described above, the plurality of pose information includes pose information corresponding to the measurement time of the inertial measurement data. And the pose information corresponding to the measurement time represents the relative pose between the inertial pose at the measurement time and the associated visual pose. Wherein the first certainty factor characterizing information comprises a first uncertainty. And if the first uncertainty corresponding to the pose information is larger, the more unreliable the pose information is.

By taking each measuring moment as a target measuring moment and then selecting the visual pose associated with the target measuring moment as a target visual pose, corresponding pose information is determined, and the determined pose information is more accurate.

Wherein the pose information includes relative translation and relative rotation. The inertial pose includes an inertial translation representing a translation of the target object with respect to the world coordinate system origin at a measurement time corresponding to the measurement pose, and an inertial rotation representing a rotation (rotation may be understood as an orientation) of the target object with respect to the world coordinate system origin at the measurement time. The visual pose includes a visual translation representing a translation of the target object relative to the world coordinate system origin at the shooting time corresponding to the visual pose, and a visual rotation representing a rotation (rotation may be understood as orientation) of the target object relative to the world coordinate system origin at the shooting time corresponding to the visual pose. The origin of the world coordinate system can be regarded as the position of the target object when the target object takes the first frame positioning image.

Specifically, the manner of acquiring the relative pose between the target inertial pose and the target visual pose may be:

and taking the difference between the inertial translation of the target inertial pose and the visual translation of the target visual pose as the relative translation in the pose information. And multiplying the transpose of the inertial rotation of the target visual pose by the inertial rotation of the target inertial pose to obtain the relative rotation.

Specifically, a relative pose sequence P is acquired^*See formulas (1) to (3):

P^*＝{(log(ΔR_i)，(ΔT_i))} (1)；

ΔT_i＝T_i-T_vi (3)；

wherein, P^*Representing a sequence of relative poses, the sequence of relative poses being a set of several relative poses, Δ R_iRepresenting the relative rotation, R, corresponding to the target measurement instant i_iRepresenting the inertial rotation of the inertial pose of the target at the target measurement time i,

a transposition, Δ T, of a visual rotation representing the visual pose of the target associated with the target measurement instant i_iRepresenting the relative translation, T, corresponding to the target measurement instant i_iInertial translation of the inertial pose of the target representing the target measurement time i, T_viA visual translation representing the target visual pose associated with target measurement time i.

As described above, the plurality of pose information includes pose information corresponding to the measurement time of the inertial measurement data. And the pose information corresponding to the measurement time represents the relative pose between the inertial pose at the measurement time and the associated visual pose. Wherein the first certainty degree characterizing information is a first uncertainty degree. And if the first uncertainty corresponding to the pose information is larger, the more unreliable the pose information is.

In some application scenarios, if the target object is an execution device such as AR glasses that has a high requirement on battery life, the AR glasses need to have a persistent environmental sensing and positioning capability, so the camera must be always in a powered-on state, which occupies the maximum power consumption. Typical AR glasses have a battery life of 2-3 hours, which greatly limits their applications. In order to reduce hardware requirements and power consumption as much as possible, the accuracy of the predicted target pose can be improved through the scheme of the application, so that long-time pose prediction can be carried out, the pose can reach 120ms through experiments, in addition, the relative pose sequence is used for carrying out the pose prediction instead of directly using the visual pose for carrying out the prediction, the corresponding first certainty factor representing information is also determined for each relative pose in the relative pose sequence, and therefore, a better prediction result can be obtained under the condition that the frame rate of a camera is reduced.

Referring to fig. 2, fig. 2 is a partial sub-flow diagram illustrating step S12 according to an embodiment of the pose prediction method. As shown in fig. 2, the step S12 may include the following steps:

step S121: and respectively taking each measuring moment as a current measuring moment, and taking the measuring moment adjacent to the current measuring moment as a reference measuring moment.

The corresponding first uncertainty can be determined for each pose information obtained. Therefore, each measurement time can be used as the current measurement time, and the measurement time adjacent to the current measurement time can be used as the reference measurement time. And taking the measurement time adjacent to the current measurement time and earlier than the current measurement time as the reference measurement time.

Step S122: and obtaining a first uncertainty of the pose information corresponding to the current measuring time based on the inertia information of the reference measuring time and the relative pose information of the current measuring time and the reference measuring time.

The inertial information of the reference measurement time comprises inertial measurement data of the reference measurement time and at least one inertial pose component. And the inertia pose component is the component of the inertia pose at the reference measurement moment. As described above, the inertial pose includes an inertial translation and an inertial rotation, and the inertial pose component may be the inertial translation at the reference measurement time or the inertial rotation at the reference measurement time.

The first uncertainty of the pose information of the current measuring moment is acquired based on the inertia information of the reference measuring moment and the relative pose information of the current measuring moment and the reference measuring moment.

Specifically, step S122 may include the steps of:

and obtaining a second uncertainty of the inertial pose of the current measurement moment based on the inertial information of the reference measurement moment and the reference uncertainty. The inertial information of the reference measurement instant comprises inertial measurement data of the reference measurement instant. Wherein the reference uncertainty is a second uncertainty of the inertial pose at the reference measurement time. Wherein the second uncertainty of the inertial pose at the first reference measurement time may be a preset value. The first reference measurement instant may be considered to be the measurement instant of the inertial pose closest to the world coordinate system origin.

And obtaining a first error parameter corresponding to the current measuring time based on the relative pose information of the current measuring time and the reference measuring time. The first error parameter represents the degree of transmission of the pose error from the inertial pose at the current measurement time to the pose information at the current measurement time. The degree of transmission may be considered the degree of influence.

And then, obtaining the first uncertainty of the pose information of the current measuring moment based on the second uncertainty and the first error parameter corresponding to the current measuring moment.

The second uncertainty of the inertia pose at the current measuring time is determined according to the reference uncertainty and the inertia information of the reference measuring time, the first error parameter corresponding to the current measuring time is determined according to the relative pose information of the current measuring time and the reference measuring time, and then the first uncertainty of the pose information at the current measuring time is determined accurately according to the second uncertainty of the inertia pose at the current measuring time and the first error parameter corresponding to the current measuring time.

In some disclosed embodiments, the manner of obtaining the second uncertainty of the inertial pose at the current measurement time based on the inertial information at the reference measurement time and the reference uncertainty may be:

and determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertia information of the reference measurement time. And the second error parameter represents the contribution degree of the inertial pose error at the reference measurement time to the inertial pose error at the current measurement time. And the third error parameter represents the contribution degree of the inertial measurement noise at the reference measurement moment to the inertial pose error at the current measurement moment.

And then, obtaining a second uncertainty of the inertial pose of the current measurement moment by using a second error parameter and a third error parameter corresponding to the reference measurement moment and the reference uncertainty. Specifically, the second error parameter, the reference uncertainty, and the first transpose are multiplied to obtain a first product, and the third error parameter, the preset uncertainty, and the second transpose are multiplied to obtain a second product. And then, adding the first product and the second product to obtain a second uncertainty of the inertial pose at the current measurement time. The preset uncertainty is used as the uncertainty corresponding to the error of the inertial measurement data at each measurement moment, namely the error of the inertial measurement data obtained at each measurement moment is actually unknown, and the situation that the accuracy of the second uncertainty is reduced due to error estimation of the error can be reduced by determining the corresponding uncertainty for the error of the inertial measurement data obtained at a single time. The first transpose is a transpose of the second error parameter and the second transpose is a transpose of the third error parameter.

Specifically, a second uncertainty C of the inertial pose at the current measurement time is calculated_i+1See formula (4):

wherein, C_i+1And representing a second uncertainty of the inertial pose at the current measurement time, i +1 representing the current measurement time, and i representing the reference measurement time. A. the_iRepresenting a second error parameter, C_iIndicating a reference uncertainty,

Denotes the first transpose, B_iRepresenting a third error parameter, Q representing a predetermined uncertainty,

Representing a second transpose.

And determining the corresponding second error parameter and the third error parameter by referring to the inertia information at the measurement moment, so that the second uncertainty of the inertia pose at the current measurement moment can be acquired by combining the reference uncertainty.

In some disclosed embodiments, the inertial measurement data for the reference measurement time includes angular velocity and acceleration for the reference measurement time. Wherein the at least one inertial pose component of the reference measurement time comprises an inertial rotation of the reference measurement time. Wherein the second error parameter and the third error parameter are matrices.

Based on the inertia information of the reference measurement time, the manner of determining the second error parameter and the third error parameter corresponding to the reference measurement time may be:

the first element of the second error parameter is derived based on a matrix representation of a third product of the angular velocity at the reference measurement instant and the measurement time difference. And acquiring a fourth product between the inertial rotation at the reference measurement moment and the oblique symmetric matrix of the acceleration, and obtaining a second element of the second error parameter based on the product of the fourth product and the measurement time difference. And deriving a third element of the second error parameter based on a product between the fourth product and a square of the measurement time difference. And obtaining a second error parameter based on the first element, the second element and the third element. Wherein the measurement time difference is a time difference between adjacent measurement moments. Wherein the first element, the second element and the third element are located in the same column. Illustratively, a second error parameter A is obtained_iSee formula (5):

wherein the content of the first and second substances,

show that

Mapping from the three-dimensional vector to a matrix, I denotes the identity matrix,

and represents the angular velocity at the reference measurement time, which is the data measured by the gyroscope. Δ t represents a measurement time difference, and "-" represents a negative sign.

Representing the inertial rotation at the reference measurement instant,

indicating the acceleration at the reference measurement instant,

a diagonally symmetric matrix representing acceleration. Δ t²Representing the square of the difference in measurement time. As can be seen from this, it is,

is a first element that is a first element,

is a second element that is a function of the first element,

is the third element.

And obtaining a fourth element of the third error based on a product of the first preset matrix of the third product and the measurement time difference. A fifth element of the third error parameter is derived based on a product of the inertial rotation at the reference measurement time and the measurement time difference. And deriving a sixth element of the third error parameter based on a product between the inertial rotation at the reference measurement time and a square of the measurement time difference. Then, a third error parameter is obtained based on the fourth element, the fifth element, and the sixth element. Wherein the fourth element and the fifth element are in different columns, and the fifth element and the sixth element are in the same column. Illustratively, a third error parameter B is obtained_iSee formula (6):

wherein the first preset matrix may be a right jacobian matrix. J in formula (6)_rIs the right jacobian matrix of SO 3. Wherein the content of the first and second substances,

is a fourth element,

Is a fifth element,

Is the sixth element.

In some disclosed embodiments, the relative pose information of the current measurement time and the reference measurement time is an inertial rotation change between the reference measurement time and the current measurement time. In particular, the inertial rotation variation is in particular the difference of the inertial rotation at the current measurement instant and the inertial rotation at the reference measurement instant. The manner of obtaining the inertial rotation difference value at two moments can be referred to in the above formula (2), and is not described herein again.

The method for obtaining the first error parameter corresponding to the current measurement time based on the relative pose information of the current measurement time and the reference measurement time may be:

and acquiring a second preset matrix represented by the inertial rotation change, and taking the product of the inverse matrix of the second preset matrix and the inertial rotation change as a seventh element of the first error parameter. Wherein the second preset matrix may be a right jacobian matrix. Then, based on the seventh element, the first error parameter is determined. Specifically, a first error parameter D is obtained_i+1See equation (7):

wherein the content of the first and second substances,

a seventh element is represented as a function of,

representing the inertial rotation variation between the reference measurement instant and the current measurement instant.

In some disclosed embodiments, the manner of obtaining the first uncertainty of the pose information at the current measurement time based on the second uncertainty and the first error parameter corresponding to the current measurement time may be:

and multiplying the product of the first error parameter and the second uncertainty by the transposition of the first error parameter to obtain a covariance matrix corresponding to the pose information of the current measurement moment. And then combining elements on the diagonal line of the covariance matrix to obtain a first uncertainty corresponding to the pose information.

Acquiring a covariance matrix corresponding to pose information at the current measurement time

See equation (8):

wherein the content of the first and second substances,

covariance matrix representing pose information correspondence at current measurement time, D_i+1Representing a first error parameter, C_i+1A second degree of uncertainty is represented as a function of,

representing the transpose of the first error parameter. By the method, the first uncertainty corresponding to the position information can be obtained.

And determining the corresponding second error parameter and the third error parameter by referring to the inertia information at the measurement moment, so that the second uncertainty of the inertia pose at the current measurement moment can be acquired by combining the reference uncertainty. In addition, because certain errors inevitably exist in the measurement process of the angular velocity and the acceleration, the second error parameter and the third error parameter are determined by referring to the angular velocity and the acceleration at the measurement moment, so that the determined second error parameter and the determined third error parameter are more accurate.

In addition, the vector of the inertial rotation change represents a corresponding right jacobian matrix, and the first error parameter is determined.

In some disclosed embodiments, the step S13 may include the following steps:

and inputting the pose information, the inertial measurement data and the first certainty degree representation information of each pose information into a target pose prediction network, and predicting target pose information of the target object in a second preset time period. The target pose prediction network comprises an encoding layer and a decoding layer.

Specifically, a first vector is obtained by encoding a plurality of pieces of inertial measurement data by using an encoding layer, and a second vector is obtained by encoding a plurality of pieces of attitude information and first determination table characteristic information of the attitude information. The first vector and the second vector are used for representing the motion parameters of the target object in a first preset time period. In particular, the first vector and the second vector may be used to represent a motion pattern and a trend of the target object within a first preset time period. The movement pattern may be a preset pattern.

And then, decoding by using a decoding layer based on the first vector and the second vector to obtain target pose information. Target pose information of the target object in a second preset time period is acquired by using a target pose prediction network, so that the prediction process is convenient and fast.

In some disclosed embodiments, the target pose prediction network further comprises a merge layer.

The step S13 further includes the following steps: and splicing the first vector, the second vector, the position and attitude information and the inertia measurement data by utilizing the merging layer to obtain a third vector. And then, decoding the third vector by using a decoding layer to obtain target pose information.

In some disclosed embodiments, the number of inertial measurement data may be preprocessed before being encoded with the encoding layer. Wherein the preprocessing may include subtracting the gravity component from the acceleration to eliminate the influence of gravity. Specifically, the formula for preprocessing the acceleration may be as follows, formula (9) and formula (10):

wherein I denotes a number of inertial measurement data after preprocessing, w_iRepresenting the measured angular velocity at the measurement instant i,

representing the acceleration at the measurement instant i after the preprocessing, a_iRepresenting the measured acceleration at the measurement instant i,

representing the transpose of the inertial rotation at the measurement instant i and g representing the gravitational component.

The first vector, the second vector, the position and posture information and the inertia measurement data are spliced to obtain a third vector, so that the third vector can contain more layers of motion information, and the determined target position and posture information is more accurate.

In order to better understand the process of predicting the target pose information of the target object in the second preset time period based on the pose information, the inertial measurement data and the first certainty factor representing information of each pose information through the target pose prediction network, please refer to fig. 3 at the same time, and fig. 3 is another schematic flow chart of an embodiment of the pose prediction method.

As shown in fig. 3, after obtaining the obtained plurality of pieces of inertial measurement data, the plurality of pieces of inertial measurement data are preprocessed, and then the preprocessed inertial measurement data are input to an inertial measurement data encoding layer, which may be an LSTM network, to obtain a first vector. And inputting the pose information and the first uncertainty corresponding to each pose information into a pose information coding layer to obtain a second vector. Wherein, the pose information coding layer can also be an LSTM network. And then the merging layer merges the latest pose information, the latest inertia measurement data, the first vector and the second vector to obtain a third vector. The latest pose information and the latest inertia measurement data can be a plurality of pose information and a plurality of inertia measurement data in a third preset time period, wherein the latest pose information and the latest inertia measurement data in the first preset time period are a plurality of pose information and a plurality of inertia data in the first preset time period, the third preset time period is in the first preset time period, and the time interval between the starting time point of the third preset time period and the ending time point of the first preset time period is less than the set time interval. And finally, inputting the merged third vector into a first decoding layer to obtain target pose information. Wherein the first decoding layer comprises a plurality of full connection layers FC. In some other disclosed embodiments, the target pose prediction network further includes a second decoding layer, and the third vector is input to the second decoding layer to obtain certainty characterizing information of the target pose information. The certainty-characterizing information can be an uncertainty. The uncertainty of the target pose information may be used to correct the target pose information, and the following specific manner is referred to for correcting the target pose information.

In some disclosed embodiments, the pose prediction method may further include the steps of:

and predicting second certainty degree representation information corresponding to the target pose information of the target object in a second preset time period by using the plurality of position and posture information, the plurality of inertial measurement data and the first certainty degree representation information of the position and posture information. Wherein the second certainty characterizing information may be a third uncertainty. Wherein the third uncertainty is an error used to represent the target pose information. Specifically, the manner of obtaining the second certainty factor characterizing information may refer to the above-mentioned manner of inputting the third vector into the second decoding layer of the target pose prediction network, so as to obtain the certainty factor characterizing information of the target pose information.

And then, based on the second certainty factor representation information, correcting the target pose information to obtain the corrected target pose information.

Second certainty factor representing information corresponding to the target pose information is predicted by utilizing the plurality of pose information, the plurality of inertial measurement data and the first certainty factor representing information of the pose information, so that the target pose can be corrected according to the second certainty factor representing information, and the corrected target pose information is more accurate.

Referring to fig. 4, fig. 4 is a schematic flowchart illustrating a process of correcting target pose information according to an embodiment of the pose prediction method. As shown in fig. 4, the manner of correcting the target pose information based on the second certainty factor characterizing information to obtain the corrected target pose information may include the following steps:

step S21: and acquiring historical pose information of the target object, wherein the historical pose information comprises corrected target pose information obtained in the last pose prediction process.

Illustratively, if an execution apparatus that executes the pose prediction method provided by the embodiments of the present disclosure executes the pose prediction method every 120ms, the last pose prediction process refers to a process that executed the pose prediction method 120ms ago.

The target pose information of the target object in the second preset time period may be the relative pose information of the target object in the second preset time period. The target pose information comprises target rotation information and target translation information. The historical pose information includes historical rotation information and historical translation information.

Step S22: and correcting the target pose information by combining the historical pose information and the second certainty factor representation information to obtain corrected target pose information of the pose prediction process.

Specifically, a difference between the first preset value and the third uncertainty is acquired as a reference difference. And acquiring a fifth product between the reference difference and the historical translation information, a sixth product between the third uncertainty and the target translation information, and taking the sum of the fifth product and the sixth product as the corrected target translation information. And performing interpolation processing by taking the historical rotation information as a starting point, the target rotation information as a key point and the reference difference value as a difference value parameter to obtain corrected target rotation information. The interpolation processing mode may be spherical interpolation, and specifically may be spherical linear interpolation.

Specifically, please refer to formula (11) for the method of correcting the target translation information, and refer to formula (12) for the method of correcting the target rotation information:

wherein, the left side of the equation (11) is the middle one

And representing the corrected target translation information, wherein the target translation information can be used as historical translation information in the next pose prediction process. To the right of the equation: sigma represents a third uncertainty, wherein the sigma is specifically normalized to [0, 1 ] for the third uncertainty of the target pose prediction network output]And then obtaining the compound. A "1" is a first preset value,

the target translation information before correction is shown,

historical translation information is represented. Therefore, the smaller the third uncertainty is, the closer the corrected target translation information is to the target translation information before correction, and otherwise, the closer the corrected target translation information is to the historical translation information.

Wherein, the formula (12) is intermediateLeft side of the

And representing the corrected target rotation information which can be used as historical rotation information in the next pose prediction process. To the right of the equation: slerp denotes a spherical linear interpolation process,

it is the historical rotation information that is represented,

target rotation information before correction is shown, and interpolation parameters are shown by 1-sigma.

The historical pose information and the second certainty factor representation information are combined to correct the pose of the target, so that the corrected pose of the target is more accurate. In addition, the target rotation information and the target translation information are corrected, so that the determined target pose information is more accurate.

After step S13 is executed, the pose prediction method may further include the steps of:

and obtaining the target pose of the target object in a second preset time period based on the pose of the target object in the first preset time period and the target pose information. And the target pose information comprises the relative pose of the second preset time period relative to the first preset time period. Specifically, the target pose information includes a relative pose between an end time of the second preset time period with respect to an end time of the first preset time period. In the embodiment of the present disclosure, the ending time of the first preset time period is the same as the starting time of the second preset time period.

Specifically, the pose of the target object within the first preset time period may be the pose of the target object at the end time of the first preset time period. And obtaining the pose of the target object at the termination time of the second preset time period based on the pose of the target object at the termination time of the first preset time period and the target pose information in the second preset time period. Specifically, the pose of the target object at the termination time of the first preset time period is added to the target pose information, so that the pose of the target object at the termination time of the second preset time period is obtained. Then, the pose of each time in the second preset time period is determined based on the pose of the target object at the end time of the first preset time period and the pose of the target object at the end time of the second preset time period. The target object moves at a constant speed within a second preset time period, and the pose of the target object at each time is determined according to the ratio of the time difference between each time and the starting time of the second preset time period to the second preset time period.

In some disclosed embodiments, after performing step S13, the pose prediction method may further include the steps of:

and determining the relative pose of the virtual object and the target object in a second preset time period based on the target pose information of the target object in the second preset time period, and determining the display position of the virtual object on the display picture of the target object based on the relative pose of the virtual object and the target object. Wherein the virtual object includes visible display information or effects, such as visible text, graphics, models, animation images, and the like.

Specifically, the target pose of the target object at each moment in a second preset time period is determined based on the target pose information of the target object in the second preset time period. The target pose includes target rotation and target translation. Because the pose of the virtual object in the virtual space is known, the relative pose between the pose of the virtual object in the virtual space and the pose of the target object in the world coordinate system can be determined, and the virtual object is projected to the display position on the display picture of the target object based on the relative pose. In some application scenarios, the augmented reality mode may be a visual perspective technology, for example, when the execution device is a smartphone, a virtual object may be overlaid on a video stream captured by the smartphone when an AR application is run on the smartphone. In some application scenarios, the augmented reality method may also use an optical perspective technology, for example, the execution device is an optical perspective AR glasses, and the virtual object may be directly mixed with the physical world observed by the human eye. Specific visual perspective techniques and optical perspective techniques can be referred to in general technology, and will not be described herein too much.

And determining the target pose of the target object in a second preset time period through the target pose information. In addition, after the target pose information of the target object in the second preset time period is obtained, the method can be applied to the technical field of augmented reality, and the accuracy of the real position of the virtual object on the display picture of the target object is improved.

and inputting the plurality of position posture information, the plurality of inertial measurement data and the first certainty degree representation information of the position posture information into a reference position posture prediction network to obtain reference position posture information. The network structures of the reference pose prediction network and the target pose prediction network are the same. Because the network structures of the reference pose prediction network and the target pose prediction network are the same, the reference pose prediction network is not described in detail herein.

And then, acquiring real pose information in a second preset time period, and adjusting parameters in the reference pose prediction network by utilizing a first loss between the real pose information and the reference pose information. The method for obtaining the real pose information in the second preset time period may be based on visual inertial positioning or using other positioning devices to obtain the real pose of the target object in the second preset time period, where the real pose includes a real pose corresponding to the end time of the second preset time period. And then determining real pose information between the starting time of the second preset time period and the ending time of the second preset time period according to the real pose of the target object at the ending time of the second preset time period and the pose of the target object at the starting time of the second preset time period. The real pose information and the target pose information are relative poses between the two moments. The time for acquiring the real pose information in the second preset time period is acquired after the end time of the second preset time period, and the pose prediction method predicts the relative pose of the target object in the time period before the end time of the second preset time period so as to acquire the pose of the target object at each time in the time period.

And acquiring a second loss between the real pose information and the target pose information. In response to the first loss being less than the second loss, parameters in the target pose prediction network are updated with parameters in the reference pose prediction network. Specifically, parameters in the target pose prediction network are updated using parameters of the reference pose prediction network adjusted for the first loss.

And updating parameters in the target pose prediction network by using the parameters in the reference pose prediction network under the condition that the loss of the reference pose prediction network is less than the second loss, so that the stability and the generalization capability of the target pose prediction network can be ensured.

As described above, the target motion prediction network is further configured to obtain second certainty factor representing information of the target pose information, and the reference pose prediction network is further configured to obtain reference certainty factor representing information of the reference pose information. Wherein the second certainty degree characterizing information includes a third uncertainty, and the reference certainty degree characterizing information includes a reference uncertainty.

The manner of obtaining the first loss or the manner of obtaining the second loss may be:

and taking the first loss as the target loss, the reference pose information as the pose information to be processed, and the reference certainty factor representation information as the certainty factor representation information of the pose information to be processed. Or taking the second loss as the target loss, the target pose information as the pose information to be processed, and the second certainty factor representing information as the certainty factor representing information of the pose information to be processed.

And then, acquiring a certainty matrix representation corresponding to the pose information to be processed. I.e. a matrix representation of the third or reference uncertainty is obtained. And acquiring a seventh product between the certainty characteristic information of the pose information to be processed and a second preset value. And the certainty matrix is represented as a matrix representation of the opposite number of certainty characteristic information of the pose information to be processed. Wherein the pose difference represents a difference between the true pose information and the pose information to be processed.

And acquiring an eighth product of the certainty matrix representation and the mean square loss, and adding the eighth product and the seventh product to obtain the target loss.

Specifically, the formula for obtaining the target loss can be seen in formula (13):

wherein L is_finalRepresenting target loss, s representing the certainty factor representing information of the pose information to be processed, in particular to uncertainty corresponding to the pose information to be processed, exp (-s) used for mapping (-s) to matrix representation,

for true pose information, F_θ(I and P) are pose information to be processed,

for mapping the pose difference to a vector representation,

the vector representing the pose difference represents the corresponding mean square loss, and the second preset value is 12.

And determining the target loss by combining the certainty of the pose information to be processed and the corresponding pose difference, so that the determined target loss is more accurate.

In some disclosed embodiments, if the target pose prediction network and the reference pose prediction network do not output corresponding certainty characterizing information, the target loss is directly the mean square error of the vector representation of the pose difference.

In some disclosed embodiments, prior to performing pose prediction using the target pose prediction network, the pose prediction method further includes a pre-training process for the target pose prediction network. The target pose prediction network and training process mainly comprises the following steps:

in order to simulate the real prediction situation to the maximum extent, a training sample for training the target pose prediction network is obtained through a plurality of real pose data and a plurality of associated inertial measurement data. Specifically, please refer to fig. 5 for a manner of obtaining a training sample, and fig. 5 is a schematic flowchart illustrating a process of obtaining the training sample according to an embodiment of the pose prediction method of the present application. As shown in FIG. 5, the first row represents a sample of image data, where each image data corresponds to a visual pose. The second row represents a sequence of inertial measurement data samples, including inertial measurement data at a plurality of time instances. The third row represents the true pose aligned with the inertial measurement instant. The fourth row represents training samples, i.e., inertial pose sequence samples, for subsequent use in acquiring relative pose sequence samples. Therefore, the number of the visual poses is less than that of the inertial measurement data, the inertial measurement time of the real poses is aligned with that of the inertial measurement data, and the inertial pose sequence sample is aligned with the real poses.

And the real pose synchronized with the image data is the corresponding visual pose. For the training sample, only the real pose synchronized with the visual pose is used as the inertial pose, and the rest inertial poses are obtained by transmitting the visual poses associated with each inertial pose and the corresponding inertial measurement data, so as to ensure that the obtained inertial poses can keep original noise.

And then determining relative pose sequence samples by using the inertial pose sequence samples and the visual pose sequence samples, determining the uncertainty of each relative pose in the relative pose sequence by using the inertial measurement data sequence samples to obtain uncertainty sequence samples, and taking the relative pose sequence samples and the uncertainty sequence samples as input to obtain training pose information output by the model and the uncertainty corresponding to the training pose information. And then, determining real pose information based on the real pose sequence, and calculating loss according to the real pose information, so as to adjust parameters in the target pose prediction network.

In addition, by predicting the uncertainty of the target pose information, the reliability of the prediction can be determined, and useful smoothing can be performed. In addition, the inherent ability of deep learning is utilized, and the pose prediction with higher precision and longer time can be carried out. In addition, the characteristic of long-time prediction can be realized by using the algorithm, and the high requirement of equipment on algorithm delay is reduced.

In some application scenes, the pose prediction scheme provided by the embodiment of the disclosure can accurately predict human behaviors in an AR scene, so that a smooth picture is rendered, and user experience is improved. In addition, the frame rate of the equipment camera can be reduced by utilizing the characteristic that the algorithm can predict under the condition of high noise, so that the service life of the battery of the AR equipment is prolonged.

The executing subject of the pose prediction method may be a pose prediction apparatus, for example, the pose prediction apparatus may be an electronic device (e.g., an AR device such as AR glasses) or a server or other processing device, and the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like. In some possible implementations, the pose prediction method may be implemented by a processor invoking computer readable instructions stored in a memory.

Referring to fig. 6, fig. 6 is a schematic structural diagram of an embodiment of the pose prediction apparatus according to the present application. The pose prediction apparatus 40 includes a data acquisition module 41, a preprocessing module 42, and a prediction module 43. The data acquisition module 41 is configured to acquire a plurality of posture information and a plurality of inertial measurement data of the target object within a first preset time period; the preprocessing module 42 is configured to determine first certainty factor representing information of each pose information based on the plurality of inertial measurement data; and the predicting module 43 is configured to predict target pose information of the target object within a second preset time period by using the plurality of pose information, the plurality of inertial measurement data, and the first certainty factor representing information of each pose information.

In some disclosed embodiments, the predicting module 43 predicts the target pose information of the target object within the second preset time period by using the pose information, the inertial measurement data, and the first certainty factor representing information of each pose information, including: inputting the pose information, the inertial measurement data and the first certainty degree representation information of each pose information into a target pose prediction network, and predicting target pose information of a target object in a second preset time period; the target pose prediction network comprises an encoding layer and a decoding layer.

According to the scheme, the target pose information of the target object in the second preset time period is obtained by using the target pose prediction network, so that the prediction process is convenient and fast.

In some disclosed embodiments, the predicting module 43 inputs the pose information, the inertial measurement data, and the first certainty factor representing information of each pose information into the target pose prediction network, and predicts the target pose information of the target object within a second preset time period, including: the method comprises the steps that a coding layer is utilized to code a plurality of pieces of inertia measurement data to obtain a first vector, a plurality of pieces of position information and first determining table characteristic information of the position information are coded to obtain a second vector, and the first vector and the second vector are used for representing motion parameters of a target object in a first preset time period; and decoding by using a decoding layer based on the first vector and the second vector to obtain target pose information.

According to the scheme, the target pose prediction network is used for coding the inertial measurement data, the pose information and the first certainty characterizing information of each pose information, so that the predicted target pose information is more accurate.

In some disclosed embodiments, the target pose prediction network further comprises a merge layer; the predicting module 43 predicts the target pose information of the target object within a second preset time period by using the pose information, the inertial measurement data, and the first certainty factor representing information of each pose information, and further includes: splicing the first vector, the second vector, the position information and the inertia measurement data by utilizing a merging layer to obtain a third vector; decoding by using a decoding layer based on the first vector and the second vector to obtain target pose information, wherein the target pose information comprises: and decoding the third vector by using a decoding layer to obtain target pose information.

According to the scheme, the first vector, the second vector, the position and posture information and the inertia measurement data are spliced to obtain the third vector, so that the third vector can contain motion information of more layers, and the determined target position and posture information is more accurate.

In some disclosed embodiments, the pose prediction apparatus 40 further comprises a training module (not shown) configured to: inputting the plurality of position posture information, the plurality of inertial measurement data and first certainty degree representation information of the position posture information into a reference position posture prediction network to obtain reference position posture information, wherein the network structures of the reference position posture prediction network and the target position posture prediction network are the same; acquiring real pose information in a second preset time period, and adjusting parameters in the reference pose prediction network by utilizing first loss between the real pose information and the reference pose information; acquiring a second loss between the real pose information and the target pose information; in response to the first loss being less than the second loss, parameters in the target pose prediction network are updated with parameters in the reference pose prediction network.

According to the scheme, the parameters in the target pose prediction network are updated by using the parameters in the reference pose prediction network under the condition that the loss of the reference pose prediction network is less than the second loss, so that the stability and the generalization capability of the target pose prediction network can be guaranteed.

In some disclosed embodiments, the target pose prediction network is further configured to obtain second certainty characterizing information of the target pose information, the reference pose prediction network is further configured to obtain reference certainty characterizing information of the reference pose information, and the training module obtains the first loss manner or obtains the second loss manner, including: taking the first loss as a target loss, the reference pose information as pose information to be processed, and the reference certainty factor representation information as certainty factor representation information of the pose information to be processed; or taking the second loss as the target loss, the target pose information as the pose information to be processed and the second certainty factor representing information as the certainty factor representing information of the pose information to be processed; acquiring a certainty matrix representation corresponding to the pose information to be processed, acquiring a mean square loss corresponding to the vector representation of the pose difference, and acquiring a seventh product between the certainty characteristic information of the pose information to be processed and a second preset value, wherein the certainty matrix representation is a matrix representation of the opposite number of the certainty characteristic information of the pose information to be processed, and the pose difference represents the difference between the real pose information and the pose information to be processed; and acquiring an eighth product of the certainty matrix representation and the mean square loss, and adding the eighth product and the seventh product to obtain the target loss.

According to the scheme, the target loss is determined by combining the certainty of the pose information to be processed and the corresponding pose difference, so that the determined target loss is more accurate.

In some disclosed embodiments, the acquiring, by the data acquiring module 41, a number of pose information of the target object within a first preset time period includes: acquiring a visual pose sequence and an inertia pose sequence of a target object in a first preset time period, wherein the visual pose sequence comprises the visual pose of the target object at the shooting time of a plurality of positioning images, and the inertia pose sequence comprises the inertia pose of the target object at the measurement time of a plurality of inertia measurement data; and obtaining a relative pose sequence related to the inertia pose sequence based on the vision pose sequence and the inertia pose sequence, wherein the relative pose sequence comprises pose information corresponding to each measurement moment, and the pose information corresponding to each measurement moment represents the relative pose between the inertia pose at the measurement moment and the associated vision pose.

According to the scheme, the relative pose sequence is determined according to the visual pose sequence and the inertia pose sequence, and the situation that the accuracy of the predicted target pose information is low due to errors of all inertia poses in the inertia pose sequence can be reduced.

In some disclosed embodiments, the data obtaining module 41 obtains a relative pose sequence with respect to the inertial pose sequence based on the visual pose sequence and the inertial pose sequence, including: respectively taking each measuring moment as a target measuring moment; taking the inertial pose at the target measurement time as a target inertial pose, and selecting a visual pose associated with the target measurement time as a target visual pose, wherein the shooting time corresponding to the target visual pose is earlier than the target measurement time; and acquiring the relative pose between the inertial pose of the target and the visual pose of the target as pose information corresponding to the target measurement time.

According to the scheme, each measuring moment is used as a target measuring moment, and then the visual pose associated with the target measuring moment is selected as the target visual pose, so that the corresponding pose information is determined, and the determined pose information is more accurate.

In some disclosed embodiments, the pose information includes pose information corresponding to a measurement time of each inertial measurement data, the pose information corresponding to the measurement time represents a relative pose between the inertial pose at the measurement time and the associated visual pose, and the first certainty table characterization information includes a first uncertainty; the preprocessing module 42 determines first certainty characterizing information of each pose information based on a plurality of inertial measurement data, including: respectively taking each measuring moment as a current measuring moment, and taking a measuring moment adjacent to the current measuring moment as a reference measuring moment; and obtaining a first uncertainty of the pose information corresponding to the current measurement time based on the inertia information of the reference measurement time and the relative pose information of the current measurement time and the reference measurement time, wherein the inertia information of the reference measurement time comprises inertia measurement data of the reference measurement time and at least one inertia pose component, and the inertia pose component is the component of the inertia pose of the reference measurement time.

According to the scheme, the first uncertainty of the pose information of the current measuring time is acquired based on the inertia information of the reference measuring time and the relative pose information of the current measuring time and the reference measuring time.

In some disclosed embodiments, the obtaining, by the preprocessing module 42, a first uncertainty of the pose information corresponding to the current measurement time based on the inertia information of the reference measurement time and the relative pose information of the current measurement time and the reference measurement time includes: obtaining a second uncertainty of the inertial pose of the current measurement time based on the inertial information of the reference measurement time and the reference uncertainty, wherein the reference uncertainty is the second uncertainty of the inertial pose of the reference measurement time; obtaining a first error parameter corresponding to the current measuring time based on the relative pose information of the current measuring time and the reference measuring time, wherein the first error parameter represents the transmission degree of the pose error from the inertial pose of the current measuring time to the pose information of the current measuring time; and obtaining the first uncertainty of the pose information of the current measuring moment based on the second uncertainty and the first error parameter corresponding to the current measuring moment.

According to the scheme, the second uncertainty of the inertia pose at the current measuring time is determined according to the reference uncertainty and the inertia information of the reference measuring time, the first error parameter corresponding to the current measuring time is determined according to the relative pose information of the current measuring time and the reference measuring time, and then the first uncertainty of the pose information at the current measuring time is determined accurately according to the second uncertainty of the inertia pose at the current measuring time and the first error parameter corresponding to the current measuring time.

In some disclosed embodiments, the preprocessing module 42 obtains a second uncertainty of the inertial pose at the current measurement time based on the inertial information at the reference measurement time and the reference uncertainty, including: determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertia information of the reference measurement time, wherein the second error parameter represents the contribution degree of the inertia pose error of the reference measurement time to the inertia pose error of the current measurement time, and the third error parameter represents the contribution degree of the inertia measurement noise of the reference measurement time to the inertia pose error of the current measurement time; and obtaining a second uncertainty of the inertial pose of the current measurement moment by using the second error parameter and the third error parameter corresponding to the reference measurement moment and the reference uncertainty.

According to the scheme, the corresponding second error parameter and the third error parameter are determined by referring to the inertia information at the measurement time, so that the second uncertainty of the inertia pose at the current measurement time can be acquired by combining the reference uncertainty.

In some disclosed embodiments, the preprocessing module 42 obtains the second uncertainty of the inertial pose at the current measurement time by using the second error parameter and the third error parameter corresponding to the reference measurement time, and the reference uncertainty, and includes: multiplying the second error parameter, the reference uncertainty and the first transposition to obtain a first product, and multiplying the third error parameter, a preset uncertainty and a second transposition to obtain a second product, wherein the preset uncertainty is used as the uncertainty corresponding to the error of the inertial measurement data at each measurement moment, the first transposition is the transposition of the second error parameter, and the second transposition is the transposition of the third error parameter; adding the first product and the second product to obtain a second uncertainty of the inertial pose at the current measurement moment; and/or the inertial measurement data at the reference measurement time comprise angular velocity and acceleration at the reference measurement time, at least one inertial pose component at the reference measurement time comprises inertial rotation at the reference measurement time, and the second error parameter and the third error parameter are matrixes; determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertial information of the reference measurement time, including: obtaining a first element of a second error parameter based on matrix representation of a third product of the angular velocity and the measurement time difference at the reference measurement time, obtaining a fourth product between the inertial rotation and the oblique symmetric matrix of the acceleration at the reference measurement time, obtaining a second element of the second error parameter based on the product of the fourth product and the measurement time difference, and obtaining a third element of the second error parameter based on the product of the fourth product and the square of the measurement time difference; obtaining a second error parameter based on the first element, the second element and the third element, wherein the measurement time difference is a time difference between adjacent measurement moments, and the first element, the second element and the third element are positioned in the same column; obtaining a fourth element of a third error parameter based on a product of the first preset matrix of the third product and the measurement time difference, obtaining a fifth element of the third error parameter based on a product of the inertial rotation at the parameter measurement time and the measurement time difference, and obtaining a sixth element of the third error parameter based on a product of the inertial rotation at the reference measurement time and the square of the measurement time difference; and obtaining a third error parameter based on a fourth element, a fifth element and a sixth element, wherein the fourth element and the fifth element are in different columns, and the fifth element and the sixth element are in the same column.

According to the scheme, the corresponding second error parameter and the third error parameter are determined by referring to the inertia information at the measurement time, so that the second uncertainty of the inertia pose at the current measurement time can be acquired by combining the reference uncertainty. In addition, because certain errors inevitably exist in the measurement process of the angular velocity and the acceleration, the second error parameter and the third error parameter are determined by referring to the angular velocity and the acceleration at the measurement moment, so that the determined second error parameter and the determined third error parameter are more accurate.

In some disclosed embodiments, the relative pose information of the current measurement time and the reference measurement time is the inertial rotation change between the reference measurement time and the current measurement time; the preprocessing module 42 obtains a first error parameter corresponding to the current measurement time based on the relative pose information of the current measurement time and the reference measurement time, and includes: acquiring a second preset matrix represented by the vector of the inertial rotation change, and taking the product of the inverse matrix of the second preset matrix and the inertial rotation change as a seventh element of the first error parameter; based on the seventh element, a first error parameter is determined.

According to the scheme, the corresponding second preset matrix can be represented by the vector of the inertial rotation change, and the first error parameter is determined.

In some disclosed embodiments, prediction module 43 is further configured to: predicting second certainty factor representation information corresponding to the target pose information of the target object in a second preset time period by using the plurality of pose information, the plurality of inertial measurement data and the first certainty factor representation information of each pose information; and correcting the target pose information based on the second certainty factor representation information to obtain corrected target pose information.

According to the scheme, the second certainty factor representing information corresponding to the target pose information is predicted by utilizing the plurality of pose information, the plurality of inertial measurement data and the first certainty factor representing information of the each pose information, so that the target pose can be corrected according to the second certainty factor representing information, and the corrected target pose information is more accurate.

In some disclosed embodiments, the predicting module 43 corrects the target pose information based on the second certainty factor characterizing information to obtain corrected target pose information, including: acquiring historical pose information of the target object, wherein the historical pose information comprises corrected target pose information obtained in the last pose prediction process; and correcting the target pose information by combining the historical pose information and the second certainty factor representation information to obtain corrected target pose information of the pose prediction process.

According to the scheme, the historical pose information and the second certainty degree representation information are combined to correct the target pose, so that the corrected target pose is more accurate.

In some disclosed embodiments, the second certainty characterizing information includes a third uncertainty, the third uncertainty is used to represent an error of the target pose information, the historical pose information includes historical rotation information and historical translation information, the target pose information includes target rotation information and target translation information, and the predicting module 43 combines the historical pose information and the second certainty characterizing information to correct the target pose information to obtain corrected target pose information of the pose prediction process, including: acquiring a difference value between the first preset value and the third uncertainty as a reference difference value, acquiring a fifth product between the reference difference value and the historical translation information and a sixth product between the third uncertainty and the target translation information, and taking the sum of the fifth product and the sixth product as corrected target translation information; and performing interpolation processing by taking the historical rotation information as a starting point, the target rotation information as an end point and the reference difference value as an interpolation parameter to obtain corrected target rotation information.

According to the scheme, the target rotation information and the target translation information are corrected, so that the determined target pose information is more accurate.

In some disclosed embodiments, the pose prediction apparatus 40 further comprises an application module (not shown), after predicting the target pose information of the target object within a second preset time period, the application module is configured to: obtaining a target pose of the target object in a second preset time period based on the pose of the target object in the first preset time period and target pose information, wherein the target pose information comprises a relative pose of the second preset time period relative to the first preset time period; and/or determining the relative pose of the virtual object and the target object in a second preset time period based on the target pose information of the target object in the second preset time period, and determining the display position of the virtual object on the display screen of the target object based on the relative pose of the virtual object and the target object.

According to the scheme, the target pose of the target object in the second preset time period can be determined through the target pose information. In addition, after the target pose information of the target object in the second preset time period is obtained, the method can be applied to the technical field of augmented reality, and the accuracy of the real position of the virtual object on the display picture of the target object is improved.

Referring to fig. 7, fig. 7 is a schematic structural diagram of an embodiment of an electronic device according to the present application. The electronic device 50 comprises a memory 51 and a processor 52, the processor 52 being configured to execute program instructions stored in the memory 51 to implement the steps in any of the above-described pose prediction method embodiments. In one particular implementation scenario, electronic device 50 may include, but is not limited to: the electronic device 50 may further include a mobile device such as a laptop or a tablet, which is not limited herein.

Specifically, the processor 52 is configured to control itself and the memory 51 to implement the steps in any of the above-described embodiments of the pose prediction method. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip having signal processing capabilities. The Processor 52 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 52 may be commonly implemented by an integrated circuit chip.

In some disclosed embodiments, the electronic device 50 may be AR glasses. In other disclosed embodiments, the electronic device may also be other AR devices or smart car devices, smart phones, and other smart terminals.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application. The computer readable storage medium 60 stores program instructions 61, and the program instructions 61, when executed by the processor, implement the steps in any of the above-described embodiments of the pose prediction method.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

The foregoing description of the various embodiments is intended to highlight various differences between the embodiments, and the same or similar parts may be referred to each other, and for brevity, will not be described again herein.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely one type of logical division, and an actual implementation may have another division, for example, a unit or a component may be combined or integrated with another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electrical, mechanical or other form.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit. The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. A pose prediction method, comprising:

acquiring a plurality of pose information and a plurality of inertial measurement data of a target object in a first preset time period;

determining first certainty degree representation information of the pose information based on the inertial measurement data;

and predicting target pose information of the target object in a second preset time period by using the pose information, the inertial measurement data and the first certainty degree representation information of each pose information.

2. The method of claim 1,

the predicting the target pose information of the target object in a second preset time period by using the pose information, the inertial measurement data and the first certainty factor representation information of each pose information comprises: inputting the pose information, the inertial measurement data and the first certainty degree representation information of each pose information into a target pose prediction network, and predicting target pose information of the target object in a second preset time period; the target pose prediction network includes an encoding layer and a decoding layer.

3. The method according to claim 2, wherein the inputting the pose information, the inertial measurement data, and the first certainty factor characterizing information of each pose information into a target pose prediction network to predict target pose information of the target object within a second preset time period comprises:

encoding the plurality of pieces of inertial measurement data by using the encoding layer to obtain a first vector, and encoding the plurality of pieces of position posture information and first determination degree characteristic information of each piece of position posture information to obtain a second vector, wherein the first vector and the second vector are used for representing motion parameters of the target object in the first preset time period;

and decoding by utilizing the decoding layer based on the first vector and the second vector to obtain the target pose information.

4. The method of claim 3, wherein the target pose prediction network further comprises a merge layer; the predicting the target pose information of the target object in a second preset time period by using the pose information, the inertial measurement data and the first certainty degree representation information of each pose information further comprises:

splicing the first vector, the second vector, the position information and the inertia measurement data by utilizing the merging layer to obtain a third vector;

the decoding by using the decoding layer based on the first vector and the second vector to obtain the target pose information includes:

and decoding the third vector by using the decoding layer to obtain the target pose information.

5. The method according to claim 3 or 4, characterized in that the method further comprises:

inputting the position and posture information, the inertial measurement data and the first certainty degree representation information of the position and posture information into a reference position and posture prediction network to obtain reference position and posture information, wherein the network structures of the reference position and posture prediction network and the target position and posture prediction network are the same;

acquiring real pose information in the second preset time period, and adjusting parameters in the reference pose prediction network by utilizing a first loss between the real pose information and the reference pose information;

acquiring a second loss between the real pose information and the target pose information;

in response to the first loss being less than the second loss, updating parameters in the target pose prediction network with parameters in the reference pose prediction network.

6. The method of claim 5, wherein the target pose prediction network is further configured to obtain second certainty information indicative of the target pose information, and wherein the reference pose prediction network is further configured to obtain reference certainty information indicative of the reference pose information, the manner in which the first loss is obtained, or the manner in which the second loss is obtained, comprises:

taking the first loss as a target loss, the reference pose information as pose information to be processed, and the reference certainty factor representation information as certainty factor representation information of the pose information to be processed; or taking the second loss as a target loss, the target pose information as the pose information to be processed, and the second certainty factor representing information as certainty factor representing information of the pose information to be processed;

acquiring a certainty matrix representation corresponding to the pose information to be processed, acquiring a mean square loss corresponding to a vector representation of a pose difference, and acquiring a seventh product between certainty characteristic information of the pose information to be processed and a second preset value, wherein the certainty matrix representation is a matrix representation of the opposite number of the certainty characteristic information of the pose information to be processed, and the pose difference represents the difference between the real pose information and the pose information to be processed;

7. The method according to any one of claims 1-6, wherein the acquiring pose information of the target object within a first preset time period comprises:

acquiring a visual pose sequence and an inertial pose sequence of the target object in the first preset time period, wherein the visual pose sequence comprises the visual pose of the target object at the shooting time of a plurality of positioning images, and the inertial pose sequence comprises the inertial pose of the target object at the measurement time of a plurality of pieces of inertial measurement data;

obtaining a relative pose sequence related to the inertial pose sequence based on the visual pose sequence and the inertial pose sequence, wherein the relative pose sequence comprises pose information corresponding to each measurement time, and the pose information corresponding to each measurement time represents a relative pose between the inertial pose at the measurement time and the associated visual pose.

8. The method of claim 7, wherein the deriving a sequence of relative poses for the sequence of inertial poses based on the sequence of visual poses and the sequence of inertial poses comprises:

taking each measuring time as a target measuring time;

taking the inertial pose at the target measurement time as a target inertial pose, and selecting a visual pose associated with the target measurement time as a target visual pose, wherein the shooting time corresponding to the target visual pose is earlier than the target measurement time;

and acquiring a relative pose between the target inertia pose and the target vision pose as pose information corresponding to the target measurement time.

9. The method of any of claims 1 to 8, wherein the number of pose information includes pose information corresponding to a measurement time of each of the inertial measurement data, the measurement time corresponding pose information indicating a relative pose between the inertial pose at the measurement time and an associated visual pose, the first certainty table characteristic information including a first uncertainty;

the determining first certainty degree representation information of each pose information based on the plurality of inertial measurement data includes:

taking each measuring moment as a current measuring moment, and taking a measuring moment adjacent to the current measuring moment as a reference measuring moment;

and obtaining a first uncertainty of the pose information corresponding to the current measurement time based on the inertia information of the reference measurement time and the relative pose information of the current measurement time and the reference measurement time, wherein the inertia information of the reference measurement time comprises inertia measurement data of the reference measurement time and at least one inertia pose component, and the inertia pose component is the inertia pose component of the reference measurement time.

10. The method of claim 9, wherein the obtaining a first uncertainty of the pose information corresponding to the current measurement time based on the inertial information of the reference measurement time and the relative pose information of the current measurement time and the reference measurement time comprises:

obtaining a second uncertainty of the inertial pose of the current measurement time based on the inertial information of the reference measurement time and a reference uncertainty, wherein the reference uncertainty is the second uncertainty of the inertial pose of the reference measurement time; and

obtaining a first error parameter corresponding to the current measurement time based on the relative pose information of the current measurement time and the reference measurement time, wherein the first error parameter represents the transmission degree of the pose error from the inertial pose of the current measurement time to the pose information of the current measurement time;

and obtaining the first uncertainty of the pose information of the current measuring moment based on the second uncertainty and the first error parameter corresponding to the current measuring moment.

11. The method of claim 10, wherein deriving a second uncertainty of the inertial pose of the current measurement time based on the inertial information of the reference measurement time and a reference uncertainty comprises:

determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertia information of the reference measurement time, wherein the second error parameter represents the degree of contribution of the inertia pose error of the reference measurement time to the inertia pose error of the current measurement time, and the third error parameter represents the degree of contribution of the inertia measurement noise of the reference measurement time to the inertia pose error of the current measurement time;

and obtaining a second uncertainty of the inertial pose of the current measurement moment by using the second error parameter and the third error parameter corresponding to the reference measurement moment and the reference uncertainty.

12. The method according to claim 11, wherein the obtaining of the second uncertainty of the inertial pose at the current measurement time by using the second error parameter and the third error parameter corresponding to the reference measurement time and the reference uncertainty comprises:

multiplying the second error parameter, the reference uncertainty and a first transposition to obtain a first product, and multiplying the third error parameter, a preset uncertainty and a second transposition to obtain a second product, wherein the preset uncertainty is used as the uncertainty corresponding to the error of the inertial measurement data at each measurement time, the first transposition is the transposition of the second error parameter, and the second transposition is the transposition of the third error parameter;

adding the first product and the second product to obtain a second uncertainty of the inertial pose of the current measurement moment;

and/or the inertial measurement data of the reference measurement time comprise the angular velocity and the acceleration of the reference measurement time, at least one inertial pose component of the reference measurement time comprises the inertial rotation of the reference measurement time, and the second error parameter and the third error parameter are matrixes; the determining a second error parameter and a third error parameter corresponding to the reference measurement time based on the inertial information of the reference measurement time includes:

obtaining a first element of the second error parameter based on a matrix representation of a third product of the angular velocity at the reference measurement time and the measurement time difference, obtaining a fourth product between the inertial rotation at the reference measurement time and the oblique symmetry matrix of the acceleration, obtaining a second element of the second error parameter based on the product of the fourth product and the measurement time difference, and obtaining a third element of the second error parameter based on the product of the fourth product and the square of the measurement time difference; obtaining the second error parameter based on the first element, the second element and the third element, where the measurement time difference is a time difference between adjacent measurement times, and the first element, the second element and the third element are located in the same column;

obtaining a fourth element of the third error parameter based on a product of a first preset matrix of the third product and the measurement time difference, obtaining a fifth element of the third error parameter based on a product of inertial rotation at the parameter measurement time and the measurement time difference, and obtaining a sixth element of the third error parameter based on a product of the inertial rotation at the reference measurement time and a square of the measurement time difference; and obtaining the third error parameter based on the fourth element, the fifth element and the sixth element, wherein the fourth element and the fifth element are in different columns, and the fifth element and the sixth element are in the same column.

13. The method according to any one of claims 10 to 12, characterized in that the relative pose information of the current measurement time and the reference measurement time is an inertial rotation change between the reference measurement time and the current measurement time; the obtaining of the first error parameter corresponding to the current measurement time based on the relative pose information of the current measurement time and the reference measurement time includes:

acquiring a second preset matrix represented by the vector of the inertial rotation change, and taking the product of the inverse matrix of the second preset matrix and the inertial rotation change as a seventh element of the first error parameter;

determining the first error parameter based on the seventh element.

14. The method according to any one of claims 1 to 13, further comprising:

predicting second certainty degree representation information corresponding to the target pose information of the target object in a second preset time period by using the position pose information, the inertial measurement data and the first certainty degree representation information of the pose information;

and correcting the target pose information based on the second certainty factor representation information to obtain corrected target pose information.

15. The method of claim 14, wherein the modifying the object pose information based on the second certainty characterizing information to obtain modified object pose information comprises:

acquiring historical pose information of the target object, wherein the historical pose information comprises corrected target pose information obtained in the last pose prediction process;

and correcting the target pose information by combining the historical pose information and the second certainty factor representation information to obtain corrected target pose information of the pose prediction process.

16. The method of claim 15, wherein the second certainty-characterizing information includes a third uncertainty that represents an error in the object pose information, the historical pose information including historical rotation information and historical translation information, the object pose information including object rotation information and object translation information,

the correcting the target pose information by combining the historical pose information and the second certainty factor representation information to obtain corrected target pose information of the pose prediction process includes:

acquiring a difference value between a first preset value and the third uncertainty as a reference difference value, acquiring a fifth product between the reference difference value and the historical translation information and a sixth product between the third uncertainty and the target translation information, and taking the sum of the fifth product and the sixth product as corrected target translation information;

and performing interpolation processing by using the historical rotation information as a starting point, the target rotation information as an end point and the reference difference value as an interpolation parameter to obtain corrected target rotation information.

17. The method according to any one of claims 1 to 16, wherein after predicting target pose information of the target object within a second preset time period, the method further comprises:

obtaining a target pose of the target object in the second preset time period based on the pose of the target object in the first preset time period and the target pose information, wherein the target pose information comprises a relative pose of the second preset time period to the first preset time period; and/or the presence of a gas in the gas,

and determining the relative pose of the virtual object and the target object in the second preset time period based on the target pose information of the target object in the second preset time period, and determining the display position of the virtual object on the display screen of the target object based on the relative pose of the virtual object and the target object.

18. A pose prediction apparatus, comprising:

the data acquisition module is used for acquiring a plurality of position and posture information and a plurality of inertial measurement data of the target object within a first preset time period;

the preprocessing module is used for determining first certainty degree representation information of the pose information based on the plurality of inertial measurement data;

and the predicting module is used for predicting the target pose information of the target object in a second preset time period by utilizing the pose information, the inertial measurement data and the first certainty degree representation information of the pose information.

19. An electronic device comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method of any of claims 1 to 17.

20. The electronic device of claim 19, wherein the electronic device is AR glasses.

21. A computer readable storage medium having stored thereon program instructions, which when executed by a processor implement the method of any of claims 1 to 17.