CN113899364A

CN113899364A - Positioning method and device, equipment and storage medium

Info

Publication number: CN113899364A
Application number: CN202111154872.XA
Authority: CN
Inventors: 陈丹鹏; 王楠
Original assignee: Shenzhen TetrasAI Technology Co Ltd
Current assignee: Shenzhen TetrasAI Technology Co Ltd
Priority date: 2021-09-29
Filing date: 2021-09-29
Publication date: 2022-01-07
Anticipated expiration: 2041-09-29
Also published as: WO2023050634A1; TW202314593A; CN113899364B

Abstract

The application discloses a positioning method, a positioning device, equipment and a storage medium, wherein the positioning method comprises the following steps: acquiring a plurality of inertia measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame; positioning processing is carried out on the basis of the plurality of inertial measurement data, and pose change information between the first historical image frame and the target image frame is obtained; determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor comprises a plurality of inertial measurement data. According to the scheme, the positioning accuracy can be improved.

Description

Positioning method and device, equipment and storage medium

Technical Field

The present application relates to the field of positioning, and in particular, to a positioning method, apparatus, device, and storage medium.

Background

At present, the positioning mode mainly comprises visual inertial positioning. The visual inertial positioning method mainly comprises the steps of constructing a three-dimensional map through image information between images shot by equipment, and then determining the position of the equipment. The positioning mode depends on the external environment, and visual positioning cannot work well under the conditions of dynamics, severe illumination change, weak texture, long shot, shielding and the like. In particular, in this case, the image captured by the device has less information extracted, and the positioning cannot be performed well.

Disclosure of Invention

The application at least provides a positioning method, a positioning device, equipment and a storage medium.

The application provides a positioning method, which comprises the following steps: acquiring a plurality of inertia measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame; positioning processing is carried out on the basis of the plurality of inertial measurement data, and pose change information between the first historical image frame and the target image frame is obtained; determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor comprises a plurality of inertial measurement data.

Therefore, by acquiring inertial measurement data between the first historical image frame and the target image frame and performing positioning processing according to the inertial measurement data, the pose change information between the two frames of images can be acquired, and the pose information of the target image frame can be further acquired according to the pose change information. In addition, after the pose change information is obtained, the pose of the target image frame is determined by combining the reference factors, and therefore more accurate pose can be obtained. In the process, the pose change is calculated by mainly utilizing the inertial measurement data so as to realize positioning, so that the adverse effect on positioning in the aspect of vision is reduced, and the condition of low positioning accuracy caused by factors such as external environment and the like is reduced.

Wherein the at least one reference factor further comprises image information about the target image frame and the first history image frame.

Therefore, the pose of the target image frame is obtained by combining the image information besides the pose change information and the inertial measurement data, so that the positioning accuracy can be improved.

Wherein determining the pose of the target image frame based on the pose change information and the at least one reference factor comprises: determining a first pose of the target image frame based on the pose change information; constructing a total energy relation corresponding to the positioning by using the pose change information and at least one reference factor, wherein the total energy relation corresponding to the positioning is used for determining the pose deviation of the target image frame to be optimized; and optimizing the first pose by using the total energy relation corresponding to the positioning to obtain the pose of the target image frame.

Therefore, the total energy relation is constructed by utilizing the pose change information and at least one reference factor, and the pose deviation is determined through the total energy relation, so that the first pose of the target image frame is optimized.

The method for constructing the total energy relationship corresponding to the positioning by using the pose change information and at least one reference factor comprises the following steps: respectively determining a measurement energy relation corresponding to each reference factor by using each reference factor, and determining a motion prior energy relation by using pose change information; and constructing a total energy relation corresponding to the positioning based on the motion prior energy relation and the measurement energy relation corresponding to each reference factor.

Therefore, a corresponding measurement energy relation is established for each reference factor, and the pose deviation determined by the acquired total energy relation is more accurate by combining the motion prior energy relation.

Wherein the determining the measurement energy relationship corresponding to each reference factor by using each reference factor respectively comprises: acquiring pre-integral information of a plurality of inertial measurement data; and constructing an inertia measurement energy relation by using pre-integration information of a plurality of inertia measurement data.

Therefore, by combining the inertia measurement energy relation constructed by the pre-integration information with the motion prior energy relation, the determined pose deviation is more accurate compared with a single motion prior energy relation.

The pose change information comprises at least one pose change amount, and the at least one pose change amount comprises a position change amount and/or a posture change amount; determining a motion prior energy relationship using pose change information, comprising: determining a motion prior energy relationship by utilizing a plurality of parameters corresponding to at least one positioning, wherein the at least one positioning comprises the positioning, and the plurality of parameters corresponding to each positioning comprise pose variation obtained in the positioning process, position difference between the initial measurement time and the final measurement time of a plurality of inertial measurement data adopted in the positioning process, and initial pose information corresponding to the initial measurement time.

Therefore, the motion prior energy relationship is determined by acquiring the pose variation of historical positioning and the position difference of a plurality of inertial measurement data during the measurement period, so that the constructed motion prior energy relationship is more accurate, and the optimized pose is more accurate.

The pose change information also comprises definition representation information of the pose change amount; determining a motion prior energy relationship by using a plurality of parameters corresponding to at least one positioning, including: obtaining the weight of corresponding positioning based on the certainty factor representation information obtained in each positioning process; determining a motion prior energy relation by using the weight of at least one positioning and a plurality of parameters; and/or, the at least one positioning is a plurality of times of positioning, and the motion prior energy relation is determined by utilizing a plurality of parameters corresponding to the at least one positioning, and the method comprises the following steps: removing the positioning meeting the removal condition from the multiple positioning; the removing condition is that the pose variation corresponding to positioning and a preset processing result between the determination degree representation information of the pose variation meet a first preset requirement; and determining the prior energy relationship of the motion by utilizing a plurality of parameters corresponding to the eliminated residual times of positioning.

Therefore, the corresponding weight is obtained through the certainty degree characterization information, so that the constructed motion prior energy relationship is more accurate. In addition, the parameters are screened to eliminate abnormal parameters, so that the acquired motion prior energy relationship is more accurate.

Wherein the initial attitude information comprises yaw angle information; and/or positioning the corresponding first historical image frame and the target image frame each time, wherein the image frame which is shot earliest is a starting image frame, and the image frame which is shot latest is an ending image frame; before determining the motion prior energy relationship using a number of parameters corresponding to at least one of the locations, the method further comprises: and determining initial attitude information, a position of an initial measurement moment and a position of an end measurement moment corresponding to the target positioning based on pre-integral information of a plurality of inertial measurement data corresponding to the target positioning, the position of a starting image frame and the position of an end image frame.

Therefore, the motion prior energy is constructed by using the yaw angle information, so that the constructed motion prior energy relation is more accurate. In addition, the positions of the initial measurement time and the ending measurement time of the inertial measurement data are determined through the pre-integral information and the poses of the related image frames, the correction of the positions of the initial measurement time and the ending measurement time is realized, and the accuracy of the motion prior energy relation is further improved.

Wherein the reference factors further include image information about the target image frame and the first historical image frame, and the determining of the measured energy relationship corresponding to the reference factors by using each reference factor respectively includes: determining a visual measurement energy relation corresponding to the image information by using the image information; before determining the vision measurement energy relationship corresponding to the image information by using the image information, the method further comprises: matching the feature points of a plurality of reference image frames to obtain a feature point matching result, wherein the plurality of reference images comprise a first historical image frame and a target image frame in at least one positioning process, and the at least one positioning process comprises the current positioning; determining a vision measurement energy relation corresponding to the image information by using the image information, wherein the method comprises the following steps: determining at least one pair of matched image frames from a plurality of reference image frames based on the feature point matching result, wherein each pair of matched image frames has a matched feature point pair; and determining the visual measurement energy relation by using the pose of each pair of matched image frames and the positions of the matched feature point pairs in the matched image frames.

Therefore, the vision measurement energy relationship is established through the two-dimensional point information, and the vision measurement energy relationship is not established through the three-dimensional point information, so that the condition that the vision measurement energy relationship is inaccurate due to the precision problem of the three-dimensional point is reduced, and the obtained vision measurement energy relationship is more accurate.

Before the total energy relationship corresponding to the current positioning is constructed based on the motion prior energy relationship and the measured energy relationship corresponding to each reference factor, the method further comprises the following steps: determining an optimized prior energy relation based on a total energy relation corresponding to historical positioning; based on the motion prior energy relationship and the measurement energy relationship corresponding to each reference factor, constructing a total energy relationship corresponding to the current positioning, comprising: and constructing a total energy relation corresponding to the positioning based on the motion prior energy relation, the optimized prior energy relation and the measured energy relation corresponding to each reference factor.

Therefore, the total energy relation is constructed by combining the optimization prior energy relation, so that the determined pose deviation is more accurate.

The total energy relation corresponding to the historical positioning is the total energy relation corresponding to the last positioning; and/or the pose deviation corresponding to the target image frame is determined by at least the pose of the target image frame, the poses corresponding to the first number of image frames before the target image frame and the inertia information corresponding to the target image frame; determining an optimized prior energy relationship based on a total energy relationship corresponding to historical positioning, comprising: updating to obtain a new pose deviation corresponding to a second historical image frame by using the pose of the second historical image frame, the poses corresponding to a second number of image frames before the second historical image frame, the pose of the target image frame and the inertia information corresponding to the second historical image frame, wherein the second historical image frame is the target image frame in the historical positioning, and the second number is smaller than the first number; and replacing the pose deviation in the total energy relation corresponding to the historical positioning with a new pose deviation to obtain an optimized prior energy relation.

Therefore, the pose of the earliest image frame of the first number of image frames before the second historical image frame is replaced by the pose of the target image frame to update the pose deviation corresponding to the second historical image frame, so that the determined optimization prior energy relation is associated with the pose of the target image frame, and the pose deviation of the target image frame determined by the energy relation is more accurate.

Wherein, the total energy relationship represents the relationship between the pose deviation and the total energy; optimizing the first pose by using the total energy relationship corresponding to the positioning to obtain the pose of the target image frame, wherein the pose comprises the following steps: determining the pose deviation enabling the total energy to meet a second preset requirement by utilizing the total energy relation corresponding to the positioning; optimizing a first pose of the target image frame based on the determined pose deviation; and/or the pose change information comprises at least one pose change amount; determining a first pose of the target image frame based on the pose change information, comprising: and determining the pose of the target image frame by using the pose variation quantity corresponding to the target image frame.

Therefore, the total energy relation meets the pose deviation of the second preset requirement, and the first pose of the target image frame is optimized based on the pose deviation, so that the pose of the final target image frame is more accurate. In addition, the pose of the target image frame can be determined through the pose variation, and the whole process is convenient and fast.

The positioning processing of the inertial measurement data is executed by a positioning model; and/or positioning based on a plurality of inertial measurement data to obtain pose change information between the first historical image frame and the target image frame, wherein the pose change information comprises: determining final motion state information obtained by the current positioning processing by using the inertia measurement data and reference motion state information, wherein the reference motion state information is the final motion state information corresponding to the historical positioning processing; and obtaining pose change information between the first historical image frame and the target image frame based on the final motion state information obtained by the positioning processing.

Therefore, the final motion state information obtained by the positioning processing at this time is more accurate by combining the final motion state information obtained by the historical positioning processing.

Wherein the positioning method is performed by a positioning system, and before determining the pose of the target image frame based on the pose change information and at least one reference factor, the method further comprises: judging whether parameters of a positioning system are initialized, wherein the parameters comprise at least one of a gravity direction and an inertia offset; in response to the parameters having been initialized, performing pose determination for the target image frame based on the pose change information and the at least one reference factor; and in response to the non-initialization of the parameters, selecting an initialization mode matched with the state corresponding to the target image frame, initializing the parameters of the positioning system, and determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the states comprise a motion state and a static state.

Therefore, the initialization mode matched with the state corresponding to the target image frame is selected under the condition that the positioning system is not initialized, and the parameters of the positioning system are initialized, so that the initialized parameters are more accurate.

Wherein the positioning method is performed by a positioning system, and after determining the pose of the target image frame based on the pose change information and at least one reference factor, the method further comprises: and optimizing parameters of the positioning system based on the pose of the target image frame, wherein the parameters comprise at least one of a gravity direction and an inertia offset.

Therefore, the parameters of the positioning system are optimized, so that the precision of the next positioning is higher.

The pose of the target image frame represents the pose of an object to be positioned at the shooting time of the target image frame, the target image frame and the first historical image frame are shot by a shooting device fixed relative to the object to be positioned, and the inertial measurement data are obtained by measuring an inertial measurement device fixed relative to the object to be positioned; and/or after acquiring a plurality of pieces of inertia measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame, the method further comprises the following steps: pre-processing the inertial measurement data, wherein the pre-processed inertial measurement data is used for positioning processing, the pre-processing including one or more of converting the inertial measurement data into a gravitational system, removing bias, removing gravity, and normalizing.

Therefore, the target image frame is obtained by shooting with the shooting device fixed relative to the object to be positioned, and the inertial measurement data is obtained with the inertial measurement device fixed relative to the object to be positioned. In addition, the obtained pose change information is more accurate by preprocessing the inertia measurement data.

The application provides a positioning device, includes: the data acquisition module is used for acquiring a plurality of inertia measurement data measured during the shooting period from the first historical image frame to the target image frame; the positioning processing module is used for performing positioning processing based on the plurality of inertial measurement data to obtain pose change information between the first historical image frame and the target image frame; and the pose determination module is used for determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor comprises a plurality of inertial measurement data.

The present application provides an electronic device comprising a memory and a processor for executing program instructions stored in the memory to implement the above-mentioned positioning method.

The present application provides a computer readable storage medium having stored thereon program instructions which, when executed by a processor, implement the above-described positioning method.

According to the scheme, the position and orientation change information between the two frames of images can be obtained by obtaining the inertia measurement data between the first historical image frame and the target image frame and carrying out positioning processing according to the inertia measurement data, and the position and orientation information of the target image frame can be obtained according to the position and orientation change information. In addition, after the pose change information is obtained, the pose of the target image frame is determined by combining the reference factors, and therefore more accurate pose can be obtained. In the process, the pose change is calculated by mainly utilizing the inertial measurement data so as to realize positioning, so that the adverse effect on positioning in the aspect of vision is reduced, and the condition of low positioning accuracy caused by factors such as external environment and the like is reduced.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a positioning method of the present application;

fig. 2 is a schematic flowchart illustrating step S13 in an embodiment of the positioning method of the present application;

FIG. 3 is another schematic flow chart diagram illustrating an embodiment of a positioning method of the present application;

FIG. 4 is a schematic structural diagram of an embodiment of a positioning apparatus of the present application;

FIG. 5 is a schematic structural diagram of an embodiment of an electronic device of the present application;

FIG. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The following describes in detail the embodiments of the present application with reference to the drawings attached hereto.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Referring to fig. 1, fig. 1 is a schematic flow chart of an embodiment of a positioning method according to the present application. Specifically, the method may include the steps of:

step S11: and acquiring a plurality of pieces of inertia measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame.

The target image frame and the first historical image frame can be obtained by an object to be positioned or shot by a camera assembly with a position relatively fixed with the object to be positioned, namely the pose condition of the camera assembly can represent the pose condition of the object to be positioned. Of course, the inertial measurement data may also be obtained by the object to be positioned, or by an inertial sensor whose position relative to the object to be positioned is fixed, that is, the pose condition of the inertial sensor may represent the pose condition of the object to be positioned.

The object to be positioned can be equipment or any animal body with life. For example, the object to be located may be a vehicle, a robot, a person, a kitten, a puppy, or the like. It will be appreciated that when the object to be located is a device, the camera assembly and inertial sensor described above may be components within the device or components external to the device.

The inertial measurement data refers to data measured by an inertial sensor. The number of the inertial sensors may be multiple, for example, the inertial sensors may be accelerometers, gyroscopes, and the like. The position between the object to be positioned and the inertial sensor is relatively fixed. That is, the inertial measurement data measured by the inertial sensor during the target time period may represent the inertial measurement data of the object to be positioned during the target time period. If a plurality of frame images are included between the target image frame and the first historical image frame, the inertial measurement data includes inertial measurement data between shooting times of adjacent image frames.

Step S12: and positioning based on the inertial measurement data to obtain pose change information between the first historical image frame and the target image frame.

The pose change information may include relative displacement between the first history image frame and the target image frame, and may also include relative rotation between the first history image frame and the target image frame.

In the embodiment of the present disclosure, the pose change information includes, for example, relative displacement and relative rotation.

Step S13: determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor comprises a plurality of inertial measurement data.

Specifically, the pose information of the first history image frame and the pose change information between the first history image frame and the target image frame can be used to determine the first pose corresponding to the target image frame. The pose of the target image frame can be regarded as the pose of an object to be positioned when the target image frame is shot.

And then, optimizing the first pose corresponding to the target image frame by using at least one reference factor to obtain the pose corresponding to the optimized target image frame.

Of course, if a plurality of image frames are included between the target image frame and the first historical image frame, the first pose may also be optimized by at least one reference factor, so as to determine the pose of the intermediate frame. The intermediate frame refers to one of the frames between the first history image frame and the target image frame.

In addition, in a general visual inertial positioning method, pre-integration needs to be calculated, so that parameters such as offset of an inertial sensor need to be accurately calibrated.

In some disclosed embodiments, the localization process of the plurality of inertial measurement data is performed by a localization model. The mode of acquiring the pose change information between the first history image frame and the target image frame may be:

and determining final motion state information obtained by the positioning processing by using the inertial measurement data and the reference motion state information. And then, obtaining pose change information between the first historical image frame and the target image frame based on the final motion state information obtained by the current positioning processing. The pose change information may be considered as a position change and/or a posture change of the object to be positioned between the shooting time of the first history image frame and the shooting time of the target image frame. The change in position between two points in time may in particular be a relative displacement between the two. The change in posture may specifically be a change in orientation of the object to be positioned.

The reference motion state information is final motion state information obtained through historical positioning processing. For example, if the current positioning process is a fourth positioning process performed on an object to be positioned, the historical positioning process may be a third positioning process performed on the same object to be positioned, or may be a combination of the second and third positioning processes, or may be a combination of all the previous positioning processes. The embodiment of the present disclosure selects the final motion state information obtained in the last positioning process as the reference motion state information in the current positioning process. The final motion state information obtained by the positioning processing can be used for deducing the motion of the object to be positioned between the shooting time point of the first historical image frame and the shooting time point of the target image frame. Similarly, the reference motion state information can be used to determine a motion between a photographing time point corresponding to the first history image frame to a photographing time point corresponding to the target image frame in the history positioning process. The final motion state information corresponding to the current positioning processing and the historical positioning processing may specifically include motion parameters of the object to be positioned at each time within the corresponding time period. For example, the motion parameters may include the acceleration and angular velocity of the object to be positioned at each time, or the motion velocity and motion direction of the object to be positioned at each time, etc. In some application scenarios, in the case where the position to be determined corresponds to a human being, the final motion state information may be considered as a local window representing the acceleration and angular velocity derived based on inertial measurement data, which may be similar to the acceleration and angular velocity of human motion, but which may be noisy due to various factors.

Wherein, the positioning processing of the plurality of inertial measurement data is executed by the positioning model. Specifically, the step of determining the final motion state information obtained by the current positioning process by using the inertial measurement data and the reference motion state information includes the following steps: based on the inertial measurement data, obtaining initial motion state information between the first historical image frame and the target image frame, and then fusing the reference motion state information and the initial motion state information to obtain the final motion state information. The initial motion state information may specifically include motion parameters of the object to be positioned at each time between the shooting time point of the first historical image frame and the shooting time point of the target image frame. For example, the initial motion state information may include an acceleration and an angular velocity of the object to be positioned at each time, or a motion velocity and a motion direction of the object to be positioned at each time, and the like. In particular, the initial motion state information may be in the form of a hidden state matrix for describing the motion of the object to be located within the target time period. Wherein the initial motion state information may be used to derive motion of the object to be located within the target time period. However, because the initial motion state obtained by the acquisition contains noise due to the influence of factors such as the offset of the sensor, if only the initial motion state information is used to deduce the motion of the object to be positioned in the target time period, a certain error may exist.

Specifically, the final motion state information and the reference motion state information may also be in the form of a hidden state matrix for describing the motion of the object to be positioned in the corresponding time period. The motion of the object to be positioned is considered to be continuous and regular, so that the initial motion state information used for representing the motion of the object to be positioned in the corresponding time period is fused with the reference motion state information used for representing the motion of the object to be positioned in the time period corresponding to the historical positioning processing, so that the more accurate final motion state information of the object to be positioned in the corresponding time period of the current positioning processing is estimated, and the motion of the object to be positioned in the corresponding time period can be estimated according to the final motion state information.

Wherein the positioning model comprises a first sub-network and a second sub-network and a third sub-network. Wherein the first sub-network may be a residual network, such as a ResNet18 network. The second sub-network may be a LSTM (Long Short-Term Memory network). The first sub-network may be configured to perform a step of obtaining initial motion state information between the first historical image frame and the target image frame based on the inertial measurement data. The second sub-network is used for executing the step of fusing the reference motion state information and the initial motion state information to obtain the final motion state information, and the third sub-network is used for executing the step of obtaining the pose change information between the first historical image frame and the target image frame based on the final motion state information obtained by the current positioning processing. The final motion state information obtained by the positioning processing is more accurate by combining the final motion state information corresponding to the last positioning processing.

In the embodiment of the present disclosure, after acquiring a plurality of pieces of inertial measurement data, the positioning method further includes the following steps:

and preprocessing the inertia measurement data. Wherein the preprocessed inertial measurement data is used for positioning processing. The pre-processing includes one or more of converting the inertial measurement data to be under a gravitational system, removing bias, removing gravity, and normalizing. The preprocessing further comprises fixed frame rate interpolation of the inertia measurement data and caching of the interpolated data. By preprocessing the inertia measurement data, the obtained pose change information is more accurate.

In some disclosed embodiments, the positioning method is performed by a positioning system. Before executing step S13, the method further includes the following steps: it is determined whether parameters of the positioning system have been initialized. Wherein the parameter includes at least one of a gravity direction and an inertial bias. In response to the parameters having been initialized, step S13 is performed. In response to the non-initialization of the parameters, an initialization mode matching the state corresponding to the target image frame is selected, the parameters of the positioning system are initialized, and then the above step S13 is performed. Wherein the states include a moving state and a stationary state. Before the initialization mode matched with the state corresponding to the target image frame is selected, the state corresponding to the target image frame is determined. The mode for determining the state corresponding to the target image frame may be that if the average displacement of the two-dimensional features tracked in the continuous frames of images on the image plane is lower than a first threshold and/or the standard deviation of the inertial measurement data is lower than a second threshold, the state corresponding to the target image frame is considered to be a static state, otherwise, the state corresponding to the target image frame is considered to be a motion state.

The initialization mode corresponding to the static state is static initialization, and the initialization mode corresponding to the moving state is moving initialization. Wherein the static initialization mode includes setting an initial translation to 0 and the initial local gravity is an average of accelerometer measurements between the last two frames of images. The initial rotation is aligned with the local initial gravitational force. The initial gyroscope bias is the average of the gyroscope measurements between the last two frames of images. The initial acceleration bias is set to 0. The last two frames refer to the target image frame and the frame preceding the target image frame.

The corresponding initialization of the motion state is to acquire the gesture without dimension only through visual positioning, and then align the inertial measurement data pre-integration result with the result of the visual positioning to recover the dimension, the velocity, the gravity and the inertial offset.

By selecting the initialization mode matched with the state corresponding to the target image frame under the condition that the positioning system is not initialized, the parameters of the positioning system are initialized, so that the initialized parameters are more accurate.

In some disclosed embodiments, please refer to fig. 2, and fig. 2 is a schematic flowchart illustrating step S13 in an embodiment of the positioning method of the present application. As shown in fig. 2, the step S13 includes the following steps:

step S131: and determining a first pose of the target image frame based on the pose change information.

The first pose of the target image frame represents the pose of the object to be positioned at the shooting moment of the target image frame. In the embodiment of the disclosure, the target image and the first historical image frame are obtained by shooting by a shooting device which fixes the object to be positioned relatively, and the inertial measurement data is obtained by measuring by an inertial measurement device which fixes the object to be positioned relatively. The target image frame is obtained by shooting through the shooting device fixed relative to the object to be positioned, and the inertial measurement data is obtained through the inertial measurement device fixed relative to the object to be positioned, so that any object can be positioned.

Specifically, the pose change information includes at least one pose change amount. The at least one pose variation includes a position variation and a pose variation. The manner of acquiring the first pose of the target image frame may be to determine the first pose of the target image frame by using the pose variation amount corresponding to the target image frame. Specifically, the pose of the first history image frame is known, and the first pose of the target image frame can be obtained based on the pose variation between the first history image frame and the target image frame. By determining the position variation and the posture variation, the determined first posture is more accurate.

Step S132: and constructing a total energy relation corresponding to the positioning by using the pose change information and at least one reference factor.

And determining the pose deviation of the target image frame to be optimized according to the total energy relation corresponding to the positioning.

Optionally, the measured energy relationship corresponding to each reference factor is determined by using each reference factor. And determining a motion prior energy relationship by using the pose change information. And finally, constructing a total energy relation corresponding to the positioning based on the motion prior energy relation and the measured energy relation corresponding to each reference factor.

In some disclosed embodiments, pre-integration information for a number of inertial measurement data is obtained. And then, constructing an inertia measurement energy relation by using pre-integration information of a plurality of inertia measurement data.

The method for constructing the relationship between the inertial measurement energies based on the pre-integration information of the inertial measurement data can be referred to generally known techniques.

Simply enumerating herein obtaining an inertial measurement energy relationship c_uThe method comprises the following steps:

wherein H_kIs the jacobian matrix corresponding to the last positioning process,

is a jacobian matrix of the pose of the positioning,

is the jacobian matrix of this positioning with respect to the inertial offset,

is the pose deviation corresponding to the last positioning,

is the pose deviation corresponding to the positioning at this time,

and the deviation corresponding to the inertial offset corresponding to the current positioning. Sigma_uIs a covariance matrix, r, corresponding to the relationship of the inertial measurement energies_uk+1Is the inertial sensor measurement residual. The manner of obtaining the parameters may refer to generally known techniques, and is not described herein.

The method for obtaining the motion prior energy relationship may be:

and determining the prior energy relation of the motion by utilizing a plurality of parameters corresponding to at least one positioning. The at least one positioning is a multiple positioning. The multiple times are twice or more. The at least one positioning comprises the positioning, and the plurality of parameters corresponding to each positioning comprise pose variation obtained in the positioning process, a position difference between the initial measurement time and the end measurement time corresponding to the positioning, and initial attitude information corresponding to the initial measurement time. The position difference between the starting measurement time and the ending measurement time refers to the difference between the position of the object to be positioned at the starting measurement time and the position of the object to be positioned at the ending measurement time. The starting attitude information of the starting measurement moment refers to the orientation of the object to be positioned at the starting measurement moment.

The motion prior energy relationship is determined by obtaining the pose variation of historical positioning and the position difference of a plurality of inertial measurement data during the measurement period, so that the constructed motion prior energy relationship is more accurate, and the optimized pose is more accurate.

Wherein the starting attitude information of the starting image frame comprises yaw angle information. Typically, the inertial measurement data is not presented in the form of yaw angle by converting the inertial measurement data to a quaternion form. If the yaw angle obtained by conversion into the XYZ coordinate axes is 90 °, the yaw angle information is acquired again by adjusting the coordinate system to the YXZ coordinate axes. The motion prior energy is constructed by using the yaw angle information, so that the constructed motion prior energy relation is more accurate.

Optionally, the pose change information further includes certainty degree representation information of the pose change amount. The certainty degree characterization information can be used for representing certainty degree and can also be used for representing uncertainty degree. And obtaining the weight of the corresponding secondary positioning based on the certainty characterizing information obtained in each positioning process. The weight of the positioning is determined and obtained based on a preset multiple of the corresponding certainty factor representing information of the positioning, wherein the preset multiple is a natural number. Specifically, the weight may be a covariance matrix transformed based on the certainty characterizing information of the preset multiple. Alternatively, when the certainty degree characterizing information is used to indicate the certainty degree, the preset multiple is generally less than or equal to 1, for example, the preset multiple may be 0.1 or the like,

of course, this is merely an example, and in other embodiments, the preset multiple may also be greater than 1. When the certainty factor representing information is used to represent the uncertainty, the preset multiple is generally greater than or equal to 1, for example, the preset multiple may be 10, etc., this is merely an example, and in other embodiments, the preset multiple may also be less than 1.

And determining the prior energy relation of the motion by using the weight of at least one positioning and a plurality of parameters. Optionally, the motion prior energy relationship of each positioning may be determined first, and then the motion prior energy relationship of each positioning is combined based on the weight of each positioning to obtain the final motion prior energy relationship. And obtaining corresponding weight through determining degree characterization information, so that the constructed motion prior energy relationship is more accurate.

Specifically, a motion prior energy relationship c is obtained_nThe method can be as follows:

wherein, the initial measurement time of a plurality of inertia measurement data corresponding to each positioning is time i, the end measurement time is time j, and p_iAnd p_jRespectively showing the positions of the objects to be positioned corresponding to the ith time and the jth time, and d_ijRepresenting the pose variation quantity between the ith moment and the jth moment of the positioning model output.

And indicating the yaw angle matrix of the target object corresponding to the ith moment. T denotes transposition. Sigma_ijRepresenting a covariance matrix (weight) corresponding to the positioning model, the covariance matrix being obtained from the characteristic information of the determination table. Specifically, the certainty characterizing information is a three-dimensional vector, and three elements in the three-dimensional vector are logarithms of diagonal elements of the covariance matrix. Alternatively, a preset multiple of the certainty-characterizing information may be used as the logarithm of the diagonal elements of the covariance matrix. The certainty factor representing information is used to represent uncertainty, and the certainty factor representing information may be original certainty factor representing information or representing information enlarged by a preset multiple.

And in the first history image frame and the target image frame corresponding to each positioning, the image frame which is shot earliest is a starting image frame, and the image frame which is shot latest is an ending image frame. In the embodiment of the present disclosure, the first history image frame is considered as a start image frame, and the target image frame is considered as an end image frame. And the pose change information comprises at least one pose change amount. The at least one pose change amount includes a position change amount and/or a pose change amount.

In order to reduce the problem of positioning accuracy reduction caused by timestamp errors, the embodiments of the present disclosure may further provide the following manner to obtain a more accurate motion prior energy relationship. Before obtaining the motion prior energy relation, the method further comprises the following steps: and determining initial attitude information, a position of an initial measurement moment and a position of an end measurement moment corresponding to the target positioning based on pre-integral information of a plurality of inertial measurement data corresponding to the target positioning, the position of a starting image frame and the position of an end image frame. Specifically, if the target location is the historical location, the pose of the start image frame and the pose of the end image frame in the target location may be the first pose before the optimization or the optimized pose. And when the target is positioned at this time, the pose of the initial image frame and the pose of the end image frame are the first pose before optimization. That is, if the target location is the last location, the initial attitude information, the position of the initial measurement time, and the position of the end measurement time in the last location may be determined by the pre-integration information of the plurality of inertial measurement data, the attitude of the initial image frame, and the attitude of the end image frame in the last location. Of course, in other disclosed embodiments, the pose of the object to be positioned at the start measurement time may be considered to be the same as the pose at the start image frame capture time, and the pose at the end measurement time may be considered to be the same as the pose at the end image frame capture time.

Specifically, the specific manner of determining the positions of the start attitude information, the start measurement time, and the end measurement time corresponding to the target positioning according to the pre-integration information, the pose of the start image frame, and the pose of the end image frame may be:

where m denotes a start image frame and n denotes an end image frame. i denotes the start measurement time and j denotes the end measurement time. p is a radical of_IiIndicating the position of the starting measuring moment, w indicating the world coordinate system, e.g.^Wp_IiThe position of the initial measurement moment in the world coordinate system is shown, and the other similar reasons are adopted. p is a radical of_IjPosition indicating the moment of ending measurement, R_IiRepresenting the starting pose information. p is a radical of_ImIndicating the position of the starting image frame, v_ImRepresenting the speed, Δ t, of the starting image frame_mRepresenting the time interval between the shooting time of the starting image frame and the starting measurement instant. R_ImIndicating the corresponding pose information of the starting image frame,

indicating the position pre-integrated using inertial measurement data between the starting image frame and the starting measurement instant, regardless of initial velocity and gravity. p is a radical of_IjIndicating the position of the moment of ending the measurement,

the attitude obtained by pre-integrating the inertia measurement data between the initial image frame and the initial measurement time is represented, and the different superscripts and subscripts of other same symbols in the three formulas can be referred to the above analysis, and are not described herein again. And selecting the image frame with the shooting time closest to the time i as a starting image frame and the image frame closest to the time j as an ending image frame from the image frames in a time stamp mode.

From this, the resulting motion prior energy relationship is obtained

Wherein the content of the first and second substances,

representing the pose deviation in the (k + 1) th fix. H denotes the corresponding jacobian matrix and r denotes the corresponding measurement residuals. Sigma_nAnd the covariance matrix corresponding to the final motion prior energy relation. The covariance matrix can be obtained from certainty characterizing information and pre-integration information output by the positioning model. For subscripts, see above.

The specific acquisition mode sigma_nCan be as follows:

therein, sigma_ijThe covariance matrix is determined based on preset multiples of certainty characterizing information output by the positioning model.

And

and the pose parts of the covariance matrixes corresponding to the pre-integration at the time j and the time i respectively. w denotes a world coordinate system and T denotes transposition. For subscripts, see above.

If pre-integration information is used in the calculation of the motion prior energy relationship, the finally obtained motion prior energy relationship can be considered to include the inertia measurement energy relationship, so that the inertia measurement energy relationship is not needed to be used any more to construct the total energy relationship. The positions of the initial measurement time and the ending measurement time of the plurality of inertial measurement data are determined through the pre-integral information and the pose of the related image frame, the correction of the positions of the initial measurement time and the ending measurement time is realized, and the accuracy of the motion prior energy relation is further improved.

In some disclosed embodiments, from a plurality of locations, locations satisfying a removal condition are culled. The removing condition is that the pose variation corresponding to positioning and a preset processing result between the determination degree representation information of the pose variation meet a first preset requirement. And then determining the prior energy relationship of the motion by utilizing a plurality of parameters corresponding to the eliminated remaining times of positioning. Specifically, abnormal positioning is removed by acquiring the mahalanobis distance corresponding to the pose variation. In particular, the removal conditions may be in

Greater than a threshold value. d represents the pose variation of the preliminary estimate of the filter,

representing the pose variation quantity output by the positioning model. Wherein, H and P represent a measured Jacobian matrix and corresponding state covariance in the motion prior energy relationship.

And representing a state covariance matrix of the positioning system, wherein the matrix is obtained by fusing a covariance matrix corresponding to the motion prior energy relation and the visual measurement energy relation. Of course, besides the elimination, the covariance matrix corresponding to the motion prior energy relationship may be amplified by several times to reduce the inaccuracy of the measured data, for example, by ten times. The parameters are screened to eliminate abnormal parameters, so that the acquired motion prior energy relationship is more accurate.

The total energy relationship constructed based on the inertial measurement energy relationship and the motion prior energy relationship may be:

wherein, here

Pose bias determined for k +1 fixes. H_uFor inertia measurement of the Jacobian matrix in the energy relationship, r_uFor measuring residual errors in the relationship of the inertially measured energies, H_nIs a Jacobian matrix in the prior energy relation of motion, r_nIs the measured residual in the motion prior energy relationship. Sigma_nIs a covariance matrix, sigma, corresponding to the prior energy relationship of the motion_zFor the covariance matrix corresponding to the inertial measurement energy relationship, it can be considered that the covariance can be a weight in the correspondence.

Of course, if the acquisition of the motion prior energy relationship is combined with the pre-integration information, the total energy relationship may not include the relevant parameters of the inertia measurement energy relationship.

In some disclosed embodiments, the at least one reference factor further comprises image information about the target image frame and the first historical image frame. Optionally, when a multi-frame image is included between the target image frame and the first history image frame, image information of each image frame between the target image frame and the first history image frame may be further included. Besides the pose change information and the inertial measurement data, the pose of the target image frame is obtained by combining the image information, so that the positioning precision can be improved.

The method for determining the measurement energy relationship corresponding to the reference factors by respectively using each reference factor comprises the following steps: and determining the vision measurement energy relation corresponding to the image information by using the image information. Specifically, before determining the vision measurement energy relationship corresponding to the image information by using the image information, the method further includes the following steps: and matching the characteristic points of the plurality of reference image frames to obtain a characteristic point matching result. The plurality of reference images comprise a first historical image frame and a target image frame in at least one positioning process. And at least one positioning comprises the current positioning. And the feature point matching result comprises a feature point set. If a feature point exists in both reference image frames, the feature point is added to the feature point set. Of course, in other embodiments, feature points are added to the feature point set only if they exist in three or more reference image frames at the same time. Wherein the feature points are two-dimensional feature points.

At least one pair of matching image frames is determined from the number of reference image frames based on the feature point matching result. Wherein there are pairs of matching feature points for each pair of matching image frames. And determining the visual measurement energy relation by using the pose of each pair of matched image frames and the positions of the matched feature point pairs in the matched image frames. If the matched image frame is not the target image frame in the historical positioning, that is, the matched image frame is the image frame between the first historical image frame and the target image frame in the current positioning or the target image frame, the pose of the matched image frame is the first pose, and if the matched image frame is the target image frame in the historical positioning, the pose of the matched image frame can be the first pose before optimization or the pose after optimization. The vision measurement energy relationship is established through the two-dimensional point information, and is not established through the three-dimensional point information, so that the condition that the vision measurement energy relationship is inaccurate due to the precision problem of the three-dimensional point is reduced, and the obtained vision measurement energy relationship is more accurate.

Specifically, the manner of obtaining the visual energy relationship is as follows:

wherein, F is a feature point set that can be tracked by each reference image frame, wherein if one feature point is observed by two or more reference image frames, the feature point can be added into the feature point set. And C is a camera state set capable of tracking the two-dimensional feature point set F.

Is the two-dimensional position of the k-th characteristic point on the j-th frame reference image frame. K is the projection matrix of the camera module, R_iAnd the rotation matrix of the camera assembly corresponding to the reference image frame of the ith frame is represented. T denotes transposition. Sigma_ZIs the corresponding covariance matrix. The i frame proposed in the present formula is a reference image frame photographed first, and the j frame is a reference image frame photographed later.

Before obtaining the total energy relationship, the method may further include the following steps: and determining an optimized prior energy relation based on the total energy relation corresponding to the historical positioning. The optimized prior energy relationship can be used to construct the total energy relationship. The total energy relation is constructed by combining the optimization prior energy relation, so that the determined pose deviation is more accurate. Specifically, the manner of obtaining the optimized prior energy relationship may be: and updating to obtain a new pose deviation corresponding to the second historical image frame by using the pose of the second historical image frame, the poses corresponding to the second number of image frames before the second historical image frame, the pose of the target image frame and the inertia information corresponding to the second historical image frame. And the second historical image frame is a target image frame in the historical positioning. The pose deviation corresponding to the target image frame is determined at least by the poses of the target image frame, the poses corresponding to the first number of image frames before the target image frame and the inertia information corresponding to the target image frame. Wherein the second number is smaller than the first number. The first number of image frames may be corresponding target image frames in the previous positioning processes. The poses of the first number of image frames can be the poses after being optimized in the history positioning process, and can also be the first poses before being optimized.

The first historical image frame used in the current positioning and the first historical image frame used in the last positioning are not the same frame. For example, if positioning is performed for the first time, the first history image frame may be a first captured frame, the first frame serves as an origin of a world coordinate system, the target image frame is a 3 rd captured frame, because after positioning is performed for the first time, poses of the 2 nd frame and the 3 rd frame are known, the first history image frame used for positioning performed for the second time may be the 2 nd frame or the 3 rd frame, if the first history image frame used for positioning performed for the second time is the 2 nd frame, the target image frame is the 4 th frame, and similarly, if the first history image frame used for positioning performed for the second time is the 3 rd frame, the target image frame may be the 5 th frame. For the second time the localization is performed, the second historical image frame is the 3 rd frame. Of course, in this positioning process, the pose of the second historical image frame used may be the pose after the last positioning optimization, and other disclosed embodiments may also be the pose before the optimization.

And then replacing the pose deviation in the total energy relation corresponding to the historical positioning with a new pose deviation to obtain an optimized prior energy relation. The inertial information corresponding to the target image frame may be an inertial bias. The inertial bias may specifically include an acceleration bias, an angular velocity bias, and the like. The pose of the first image frame of the first number of image frames before the second historical image frame is replaced by the pose of the target image frame to update the pose deviation corresponding to the second historical image frame, so that the determined optimization prior energy relation is associated with the pose of the target image frame, and the pose deviation of the target image frame determined by the energy relation is more accurate.

Wherein obtaining an optimized prior energy relationship

The method comprises the following steps:

wherein H_k+1The corresponding jacobian matrix is located for k +1 times.

The pose deviation r corresponding to the target image frame_k+1Is the corresponding measurement residual.

And then, constructing a total energy relation corresponding to the current positioning based on one or more of the motion prior energy relation, the vision measurement energy relation, the optimization prior energy relation and the inertia measurement energy relation.

In some disclosed embodiments, the total energy relationship corresponding to the current positioning is constructed based on a motion prior energy relationship and an inertia measurement energy relationship, in other disclosed embodiments, the total energy relationship corresponding to the current positioning is constructed based on the motion prior energy relationship and a visual measurement energy relationship, or the total energy relationship corresponding to the current positioning is constructed based on the motion prior energy relationship, the visual measurement energy relationship and the visual measurement energy relationship, or the total energy relationship corresponding to the current positioning is constructed based on the motion prior energy relationship, the visual measurement energy relationship, an optimized prior energy relationship and an inertia measurement energy relationship.

Step S133: and optimizing the first pose by using the total energy relation corresponding to the positioning to obtain the pose of the target image frame.

Wherein the total energy relationship represents a relationship between the pose deviation and the total energy. Specifically, the method for obtaining the pose of the target image frame by optimizing the first pose according to the total energy relationship corresponding to the current positioning may be: and determining the pose deviation of the total energy meeting the second preset requirement by using the total energy relation corresponding to the positioning. Wherein the second predetermined requirement may be a minimum total energy. And then optimizing the first pose of the target image frame based on the determined pose deviation. Specifically, the acquired pose deviation is summed with the first pose of the target image frame to obtain the optimized pose of the target image frame.

Thus, the total energy relationship is minimized to update the corresponding states of the respective image frames and the bias of the inertial sensor.

Specifically, the way to minimize the total energy relationship is:

wherein, here

Pose bias determined for k +1 fixes. H_k+1To optimize the Jacobian matrix in the prior energy relationship, r_k+1To optimize the measurement residual in the prior energy relationship. H_zFor visual measurement of the Jacobian matrix in the energy relationship, r_zFor the visual measurement of the measurement residual in the energy relationship, H_nIs a Jacobian matrix in the prior energy relation of motion, r_nIs the measured residual in the motion prior energy relationship. Sigma_nIs a covariance matrix, sigma, corresponding to the prior energy relationship of the motion_zAnd measuring the covariance matrix corresponding to the energy relation for the vision.

By the formula, the final product can be obtained by solving

And summing the pose and the first pose to obtain an optimized pose. The obtained bias of the inertial sensor can replace the original bias in the positioning system, so that the subsequent pose calculation is more accurate. Of course, if the parameters such as the offset and the gravity direction of the inertial sensor are required to be optimized, the parameters are included

It may also mean that the corresponding deviations for all parameters to be optimized are included, i.e. in this case,

the total deviation of the deviation corresponding to the parameters such as the pose deviation and each offset is included.

In other disclosed embodiments, after determining the pose of the target image frame, the parameters of the positioning system may be optimized based on the pose of the target image frame. Specifically, the parameter deviation required to be optimized in the positioning process is obtained by adjusting the total deviation including the pose deviation and the parameter deviation, and the obtained deviation is added to the corresponding parameter to obtain the optimized parameter. And finally, replacing the parameters in the positioning system by using the optimized parameters. Wherein the parameter includes at least one of a gravity direction and an inertial bias. In the next positioning process, the optimized parameters can be used for positioning. By optimizing the parameters of the positioning system, the precision of the next positioning is higher.

For better understanding of the technical solutions provided by the embodiments of the present disclosure, please refer to the following examples. Please refer to fig. 3, fig. 3 is another schematic flow chart of an embodiment of the positioning method of the present application. As shown in fig. 3, the positioning method provided by the embodiment of the present disclosure includes the following steps:

step S21: the camera assembly acquires a plurality of image frames.

Wherein, the image frames refer to the image frames from the first history image frame to the target image frame.

Step S22: and extracting and tracking features.

Specifically, feature extraction and tracking are performed on a plurality of image frames to obtain image information corresponding to the plurality of image frames.

Step S23: the inertial sensor acquires a plurality of inertial measurement data.

Wherein the plurality of inertial measurement data refer to inertial measurement data during a photographing period between a photographing time of the first history image frame and a photographing time of the target image frame.

The specific obtaining manner is as described above, and is not described herein again.

Step S24: and carrying out data caching.

Specifically, one or more of converting inertial measurement data to a gravitational system, removing bias, removing gravity, and normalizing is performed in a data cache.

Step S25: and inputting a positioning model.

Specifically, a number of inertial measurement data are input into the localization model. And the positioning model carries out positioning processing based on a plurality of inertial measurement data to obtain the pose change information of the positioning processing. The specific way of the positioning model for positioning the inertial measurement data is as described above, and is not described here again.

Step S26: a pre-integration is performed.

For a specific way of pre-integrating the inertial measurement data, reference may be made to general known technologies, and details thereof are not described herein.

Step S27: and judging whether the positioning system is initialized.

The specific determination method is as described above, and is not described herein again.

If the determination result is that the positioning system is not initialized, step S28 is executed, otherwise, step S29 is executed.

Step S28: a still initialization/motion initialization is performed.

The specific manner of performing the still initialization/motion initialization is as described above and will not be described herein.

Step S29: anomaly detection is performed.

The abnormal point detection is a step of removing the data satisfying the second preset condition in the foregoing, and therefore details are not described here again.

Step S30: and optimizing and updating.

Specifically, a total energy relation is constructed based on the image information, the pre-integration information and the pose change information, and the poses corresponding to the image frames are optimized and updated to obtain the final poses of the target image frames.

Step S31: and optimizing the parameters.

Specifically, the parameters of the positioning system that can be optimized may include an acceleration bias Ba, an angular velocity bias Bg, and other parameters R that can be optimized for the positioning system.

The above-mentioned steps are not performed in the exact order, and for example, step S21 and step S23 may be performed synchronously. Step S22 and step S24 and step S26 may also be executed in synchronization or the like.

In particular, the inputs to the positioning system may include the outputs of the camera assembly and the inertial sensor, i.e. all image frames between the first history image to the target image frame and several inertial measurement data. And respectively preprocessing each image frame and the inertia measurement data. The preprocessing mode of the image frame mainly comprises feature extraction and tracking. The preprocessing of the inertia measurement data can also be divided into two steps, namely, pre-integration is carried out, and one or more of conversion of the inertia measurement data into a gravity system, bias removal, gravity removal and normalization are carried out in the data cache. And inputting the inertia measurement data subjected to the second preprocessing into a positioning model to obtain pose change information between the first historical image frame and the target image frame. Then, after the pre-integration and the image preprocessing are performed, whether the positioning system is initialized is judged, and if so, abnormal point detection is performed, wherein the abnormal point detection is the step of removing the data meeting the second preset condition. And then, constructing a total energy relation based on the image information, the pre-integration information and the pose change information, optimizing and updating the corresponding pose of each image frame, and otherwise, initializing the system. And optimizing the parameters of the positioning system after the optimization updating of the pose. Of course, in other embodiments, the total energy relationship in this positioning process may also be constructed based on historical total energy relationships.

The method comprises the steps of obtaining inertial measurement data between a first historical image frame and a target image frame, and positioning according to the inertial measurement data to obtain pose change information between the two images, wherein if the first historical image frame is the origin of a world coordinate system or the pose of the first historical image frame is known, the pose information of the target image frame can be obtained according to the pose change information. In addition, after the pose change information is obtained, the pose of the target image frame is determined by combining the reference factors, and therefore more accurate pose can be obtained. The process reduces the influence of the vision on positioning, thereby reducing the situation of low positioning accuracy caused by factors such as external environment and the like.

Furthermore, the motion prior (pose change information), IMU information and visual information corresponding to the positioning model are tightly coupled, so that a high-precision positioning effect can be obtained in a normal visual environment, and more robust tracking can be obtained due to the robustness of the motion prior in an extremely challenging visual environment.

Further, the technical scheme provided by the embodiment of the disclosure can be coupled with other positioning algorithms or sensors to perform positioning navigation.

The positioning method provided by the embodiment of the disclosure can be applied to scenes such as augmented reality, virtual reality, robots, automatic driving, games, movies, education, electronic commerce, tourism, smart medical treatment, indoor decoration equipment, smart home, smart manufacturing, maintenance and assembly and the like.

The main body of the positioning method may be a positioning apparatus, for example, the positioning method may be performed by a terminal device or a server or other processing device, where the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like. In some possible implementations, the location method may be implemented by a processor calling computer readable instructions stored in a memory.

Referring to fig. 4, fig. 4 is a schematic structural diagram of an embodiment of the positioning device of the present application. The positioning apparatus 40 includes a data acquisition module 41, a positioning processing module 42, and a pose determination module 43. A data obtaining module 41, configured to obtain a plurality of pieces of inertia measurement data measured during a shooting period from a shooting time of the first history image frame to a shooting time of the target image frame; the positioning processing module 42 is configured to perform positioning processing based on the plurality of inertial measurement data to obtain pose change information between the first historical image frame and the target image frame; a pose determination module 43, configured to determine a pose of the target image frame based on the pose change information and at least one reference factor, where the at least one reference factor includes a number of inertial measurement data.

In some disclosed embodiments, the at least one reference factor further comprises image information about the target image frame and the first historical image frame.

According to the scheme, the pose of the target image frame is obtained by combining the image information besides the pose change information and the inertial measurement data, so that the positioning accuracy can be improved.

In some disclosed embodiments, the pose determination module 43 determines the pose of the target image frame based on the pose change information and at least one reference factor, including: determining a first pose of the target image frame based on the pose change information; constructing a total energy relation corresponding to the positioning by using the pose change information and at least one reference factor, wherein the total energy relation corresponding to the positioning is used for determining the pose deviation of the target image frame to be optimized; and optimizing the first pose by using the total energy relation corresponding to the positioning to obtain the pose of the target image frame.

According to the scheme, the total energy relation is constructed by utilizing the pose change information and at least one reference factor, and the pose deviation is determined through the total energy relation, so that the first pose of the target image frame is optimized.

In some disclosed embodiments, the pose determining module 43 constructs a total energy relationship corresponding to the current positioning by using the pose change information and at least one reference factor, including: respectively determining a measurement energy relation corresponding to each reference factor by using each reference factor, and determining a motion prior energy relation by using pose change information; and constructing a total energy relation corresponding to the positioning based on the motion prior energy relation and the measurement energy relation corresponding to each reference factor.

According to the scheme, the corresponding measurement energy relation is established for each reference factor, and the pose deviation determined by the acquired total energy relation is more accurate by combining the motion prior energy relation.

In some disclosed embodiments, the pose determination module 43 determines the measurement energy relationship corresponding to each reference factor by using each reference factor, respectively, including: acquiring pre-integral information of a plurality of inertial measurement data; and constructing an inertia measurement energy relation by using pre-integration information of a plurality of inertia measurement data.

According to the scheme, the inertia measurement energy relation constructed by the pre-integral information is combined with the motion prior energy relation, so that the determined pose deviation is more accurate compared with a single motion prior energy relation.

In some disclosed embodiments, the pose change information includes at least one pose change amount, the at least one pose change amount including a position change amount and/or a pose change amount; the pose determination module 43 determines a motion prior energy relationship using pose change information, including: determining a motion prior energy relationship by utilizing a plurality of parameters corresponding to at least one positioning, wherein the at least one positioning comprises the positioning, and the plurality of parameters corresponding to each positioning comprise pose variation obtained in the positioning process, position difference between the initial measurement time and the final measurement time of a plurality of inertial measurement data adopted in the positioning process, and initial pose information corresponding to the initial measurement time.

According to the scheme, the motion prior energy relationship is determined by acquiring the historical positioning pose variation and the position difference of a plurality of inertial measurement data during the measurement period, so that the constructed motion prior energy relationship is more accurate, and the optimized pose is more accurate.

In some disclosed embodiments, the pose change information further includes certainty characterizing information of the pose change amount; the pose determination module 43 determines a motion prior energy relationship using a plurality of parameters corresponding to at least one of the positions, including: obtaining the weight of corresponding positioning based on the certainty factor representation information obtained in each positioning process; determining a motion prior energy relation by using the weight of at least one positioning and a plurality of parameters; and/or, the at least one positioning is a plurality of times of positioning, and the motion prior energy relation is determined by utilizing a plurality of parameters corresponding to the at least one positioning, and the method comprises the following steps: removing the positioning meeting the removal condition from the multiple positioning; the removing condition is that the pose variation corresponding to positioning and a preset processing result between the determination degree representation information of the pose variation meet a first preset requirement; and determining the prior energy relationship of the motion by utilizing a plurality of parameters corresponding to the eliminated residual times of positioning.

According to the scheme, the corresponding weight is obtained through the certainty degree characterization information, so that the constructed motion prior energy relationship is more accurate. In addition, the parameters are screened to eliminate abnormal parameters, so that the acquired motion prior energy relationship is more accurate.

In some disclosed embodiments, the starting attitude information includes yaw angle information; and/or positioning the corresponding first historical image frame and the target image frame each time, wherein the image frame which is shot earliest is a starting image frame, and the image frame which is shot latest is an ending image frame; before determining the motion prior energy relationship using a number of parameters corresponding to at least one of the positions, the pose determination module 43 method is further configured to: and determining initial attitude information, a position of an initial measurement moment and a position of an end measurement moment corresponding to the target positioning based on pre-integral information of a plurality of inertial measurement data corresponding to the target positioning, the position of a starting image frame and the position of an end image frame.

According to the scheme, the motion prior energy is constructed by using the yaw angle information, so that the constructed motion prior energy relationship is more accurate. In addition, the positions of the initial measurement time and the ending measurement time of the inertial measurement data are determined through the pre-integral information and the poses of the related image frames, the correction of the positions of the initial measurement time and the ending measurement time is realized, and the accuracy of the motion prior energy relation is further improved.

In some disclosed embodiments, the reference factors further include image information about the target image frame and the first historical image frame, and determining a measured energy relationship corresponding to the reference factors using each of the reference factors, respectively, includes determining a visual measured energy relationship corresponding to the image information using the image information. Before determining the vision measurement energy relationship corresponding to the image information by using the image information, the pose determination module 43 is further configured to: matching the feature points of a plurality of reference image frames to obtain a feature point matching result, wherein the plurality of reference images comprise a first historical image frame and a target image frame in at least one positioning process, and the at least one positioning process comprises the current positioning; determining a vision measurement energy relation corresponding to the image information by using the image information, wherein the method comprises the following steps: determining at least one pair of matched image frames from a plurality of reference image frames based on the feature point matching result, wherein each pair of matched image frames has a matched feature point pair; and determining the visual measurement energy relation by using the pose of each pair of matched image frames and the positions of the matched feature point pairs in the matched image frames.

According to the scheme, the vision measurement energy relation is established through the two-dimensional point information, and the vision measurement energy relation is not established through the three-dimensional point information, so that the condition that the vision measurement energy relation is inaccurate due to the precision problem of the three-dimensional point is reduced, and the obtained vision measurement energy relation is more accurate.

In some disclosed embodiments, before constructing the total energy relationship corresponding to the current location based on the motion prior energy relationship and the measurement energy relationship corresponding to each reference factor, the pose determination module 43 is further configured to: determining an optimized prior energy relation based on a total energy relation corresponding to historical positioning; based on the motion prior energy relationship and the measurement energy relationship corresponding to each reference factor, constructing a total energy relationship corresponding to the current positioning, comprising: and constructing a total energy relation corresponding to the positioning based on the motion prior energy relation, the optimized prior energy relation and the measured energy relation corresponding to each reference factor.

According to the scheme, the total energy relation is constructed by combining the optimization prior energy relation, so that the determined pose deviation is more accurate.

In some disclosed embodiments, the total energy relationship corresponding to the historical positioning is the total energy relationship corresponding to the last positioning; and/or the pose deviation corresponding to the target image frame is determined by at least the pose of the target image frame, the poses corresponding to the first number of image frames before the target image frame and the inertia information corresponding to the target image frame; the pose determination module 43 determines an optimized prior energy relationship based on the total energy relationship corresponding to the historical positioning, including: updating to obtain a new pose deviation corresponding to a second historical image frame by using the pose of the second historical image frame, the poses corresponding to a second number of image frames before the second historical image frame, the pose of the target image frame and the inertia information corresponding to the second historical image frame, wherein the second historical image frame is the target image frame in the historical positioning, and the second number is smaller than the first number; and replacing the pose deviation in the total energy relation corresponding to the historical positioning with a new pose deviation to obtain an optimized prior energy relation.

According to the scheme, the position of the earliest image frame of the first number of image frames before the second historical image frame is replaced by the position of the target image frame to update the position deviation corresponding to the second historical image frame, so that the determined optimized prior energy relation is associated with the position of the target image frame, and the position deviation of the target image frame determined by the energy relation is more accurate.

In some disclosed embodiments, the total energy relationship represents a relationship between pose deviation and total energy; the pose determining module 43 optimizes the first pose by using the total energy relationship corresponding to the current positioning to obtain the pose of the target image frame, which includes: determining the pose deviation enabling the total energy to meet a second preset requirement by utilizing the total energy relation corresponding to the positioning; optimizing a first pose of the target image frame based on the determined pose deviation; and/or the pose change information comprises at least one pose change amount; determining a first pose of the target image frame based on the pose change information, comprising: and determining the pose of the target image frame by using the pose variation quantity corresponding to the target image frame.

According to the scheme, the total energy relation meets the pose deviation of the second preset requirement, and the first pose of the target image frame is optimized based on the pose deviation, so that the pose of the final target image frame is more accurate. In addition, the pose of the target image frame can be determined through the pose variation, and the whole process is convenient and fast.

In some disclosed embodiments, the positioning process on the inertial measurement data is performed by a positioning model; and/or the positioning processing module 42 performs positioning processing based on a plurality of inertial measurement data to obtain pose change information between the first historical image frame and the target image frame, including: determining final motion state information of the object to be positioned obtained in the current positioning processing by using the inertial measurement data and the reference motion state information, wherein the reference motion state information is the final motion state information corresponding to the last positioning processing; and obtaining pose change information between the first historical image frame and the target image frame based on the final motion state information obtained by the positioning processing.

According to the scheme, the final motion state information obtained by the last positioning processing is combined, so that the final motion state information obtained by the current positioning processing is more accurate.

In some disclosed embodiments, the positioning method is performed by the positioning system, and before determining the pose of the target image frame based on the pose change information and the at least one reference factor, the pose determination module 43 is further configured to: judging whether parameters of a positioning system are initialized, wherein the parameters comprise at least one of a gravity direction and an inertia offset; in response to the parameters having been initialized, performing pose determination for the target image frame based on the pose change information and the at least one reference factor; and in response to the non-initialization of the parameters, selecting an initialization mode matched with the state corresponding to the target image frame, initializing the parameters of the positioning system, and determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the states comprise a motion state and a static state.

According to the scheme, the initialization mode matched with the state corresponding to the target image frame is selected under the condition that the positioning system is not initialized, and the parameters of the positioning system are initialized, so that the initialized parameters are more accurate.

In some disclosed embodiments, the positioning method is performed by the positioning system, and after determining the pose of the target image frame based on the pose change information and the at least one reference factor, the pose determination module 43 is further configured to: and optimizing parameters of the positioning system based on the pose of the target image frame, wherein the parameters comprise at least one of a gravity direction and an inertia offset.

According to the scheme, the parameters of the positioning system are optimized, so that the next positioning precision is higher.

In some disclosed embodiments, the pose of the target image frame represents the pose of the object to be positioned at the shooting time of the target image frame, the target image frame and the first historical image frame are shot by a shooting device fixed relative to the object to be positioned, and the inertial measurement data is measured by an inertial measurement device fixed relative to the object to be positioned; and/or after acquiring a plurality of pieces of inertia measurement data measured during the shooting period from the shooting time of the first history image frame to the shooting time of the target image frame, the data acquisition module 41 is further configured to: pre-processing the inertial measurement data, wherein the pre-processed inertial measurement data is used for positioning processing, the pre-processing including one or more of converting the inertial measurement data into a gravitational system, removing bias, removing gravity, and normalizing.

According to the scheme, the target image frame is obtained by shooting through the shooting device fixed relative to the object to be positioned, and the inertial measurement data is obtained through the inertial measurement device fixed relative to the object to be positioned, so that any object can be positioned. In addition, the obtained pose change information is more accurate by preprocessing the inertia measurement data.

Referring to fig. 5, fig. 5 is a schematic structural diagram of an embodiment of an electronic device according to the present application. The electronic device 50 comprises a memory 51 and a processor 52, the processor 52 being configured to execute program instructions stored in the memory 51 to implement the steps in any of the above-described embodiments of the positioning method. In one particular implementation scenario, electronic device 50 may include, but is not limited to: medical equipment, a microcomputer, a desktop computer, a server, and the electronic equipment 50 may also include mobile equipment such as a notebook computer, a tablet computer, and the like, which is not limited herein.

In particular, the processor 52 is configured to control itself and the memory 51 to implement the steps in any of the above-described embodiments of the positioning method. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip having signal processing capabilities. The Processor 52 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 52 may be commonly implemented by an integrated circuit chip.

Referring to fig. 6, fig. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application. The computer readable storage medium 60 stores program instructions 61 executable by the processor, the program instructions 61 for implementing the steps in any of the positioning method embodiments described above.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like.

The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

The foregoing description of the various embodiments is intended to highlight various differences between the embodiments, and the same or similar parts may be referred to each other, and for brevity, will not be described again herein.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely one type of logical division, and an actual implementation may have another division, for example, a unit or a component may be combined or integrated with another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electrical, mechanical or other form.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. A method of positioning, comprising:

acquiring a plurality of inertia measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame;

positioning processing is carried out on the basis of the plurality of inertial measurement data, and pose change information between the first historical image frame and the target image frame is obtained;

determining a pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor comprises the number of inertial measurement data.

2. The method of claim 1, wherein the at least one reference factor further comprises image information regarding the target image frame and a first historical image frame.

3. The method of claim 1, wherein the determining the pose of the target image frame based on the pose change information and at least one reference factor comprises:

determining a first pose of the target image frame based on the pose change information; and the number of the first and second groups,

constructing a total energy relation corresponding to the positioning by using the pose change information and the at least one reference factor, wherein the total energy relation corresponding to the positioning is used for determining the pose deviation of the target image frame to be optimized;

and optimizing the first pose by using the total energy relation corresponding to the positioning to obtain the pose of the target image frame.

4. The method according to claim 3, wherein the constructing the total energy relationship corresponding to the current location by using the pose change information and the at least one reference factor comprises:

determining a measurement energy relationship corresponding to each of the reference factors by using each of the reference factors, respectively, and,

determining a motion prior energy relation by using the pose change information;

and constructing a total energy relation corresponding to the positioning based on the motion prior energy relation and the measurement energy relation corresponding to each reference factor.

5. The method of claim 4, wherein the determining the measured energy relationship corresponding to each of the reference factors by using each of the reference factors respectively comprises:

acquiring pre-integral information of the plurality of inertial measurement data;

and constructing an inertia measurement energy relation by using the pre-integration information of the plurality of inertia measurement data.

6. The method according to claim 4, characterized in that the pose change information comprises at least one pose change amount, the at least one pose change amount comprising a position change amount and/or a pose change amount; the determining a motion prior energy relationship by using the pose change information includes:

determining a motion prior energy relationship by utilizing a plurality of parameters corresponding to at least one positioning, wherein the at least one positioning comprises the positioning, and the plurality of parameters corresponding to each positioning comprise the pose variation obtained in the positioning process, the position difference between the initial measurement time and the final measurement time of the plurality of inertial measurement data adopted in the positioning process, and the initial attitude information corresponding to the initial measurement time.

7. The method according to claim 6, characterized in that the pose change information further includes certainty-degree characterizing information of the pose change amount; the determining the motion prior energy relationship by using a plurality of parameters corresponding to at least one positioning comprises:

obtaining the weight of corresponding secondary positioning based on the certainty degree representation information obtained in each positioning process;

determining a motion prior energy relationship by using the weight of the at least one positioning and the parameters;

and/or, the at least one positioning is a plurality of times of positioning, and the determining of the motion prior energy relationship by using a plurality of parameters corresponding to the at least one positioning comprises:

removing the positioning meeting the removal condition from the multiple positioning; the removing condition is that a preset processing result between pose variation corresponding to the positioning and certainty degree representation information of the pose variation meets a first preset requirement;

and determining the prior energy relationship of the motion by utilizing a plurality of parameters corresponding to the eliminated residual times of positioning.

8. The method of claim 6, wherein the starting pose information comprises yaw angle information;

and/or positioning the corresponding first historical image frame and the target image frame each time, wherein the image frame which is shot earliest is a starting image frame, and the image frame which is shot latest is an ending image frame; before determining the motion prior energy relationship using a number of parameters corresponding to the at least one location, the method further comprises:

and taking each positioning in the at least one positioning as a target positioning, and determining the initial attitude information, the position of the initial measurement moment and the position of the end measurement moment corresponding to the target positioning based on pre-integration information of a plurality of pieces of inertial measurement data corresponding to the target positioning, the position of a starting image frame and the position of an end image frame.

9. The method of claim 4, wherein the reference factors further include image information about the target image frame and a first historical image frame, and wherein the determining, using each of the reference factors, a measured energy relationship corresponding to the reference factor, respectively, comprises:

determining a visual measurement energy relation corresponding to the image information by using the image information;

before the determining, by using the image information, a vision measurement energy relationship corresponding to the image information, the method further includes:

matching the feature points of a plurality of reference image frames to obtain a feature point matching result, wherein the plurality of reference images comprise a first historical image frame and a target image frame in at least one positioning process, and the at least one positioning process comprises the current positioning;

the determining, by using the image information, a vision measurement energy relationship corresponding to the image information includes:

determining at least one pair of matched image frames from the plurality of reference image frames based on the feature point matching result, wherein a matched feature point pair exists in each pair of matched image frames;

and determining the visual measurement energy relation by using the pose of each pair of the matched image frames and the positions of the matched feature point pairs in the matched image frames.

10. The method according to claim 4, wherein before the constructing the total energy relationship corresponding to the current location based on the motion prior energy relationship and the measured energy relationship corresponding to each of the reference factors, the method further comprises:

determining an optimized prior energy relation based on a total energy relation corresponding to historical positioning;

the method for constructing the total energy relationship corresponding to the positioning based on the motion prior energy relationship and the measured energy relationship corresponding to each reference factor comprises the following steps:

and constructing a total energy relation corresponding to the positioning based on the motion prior energy relation, the optimized prior energy relation and the measurement energy relation corresponding to each reference factor.

11. The method of claim 10, wherein the total energy relationship corresponding to the historical position is the total energy relationship corresponding to the last position;

and/or the pose deviation corresponding to the target image frame is determined at least by the pose of the target image frame, the poses corresponding to a first number of image frames before the target image frame and the inertia information corresponding to the target image frame; determining an optimized prior energy relationship based on a total energy relationship corresponding to historical positioning, comprising:

updating to obtain a new pose deviation corresponding to a second historical image frame by using the pose of the second historical image frame, the poses corresponding to a second number of image frames before the second historical image frame, the pose of the target image frame and the inertial information corresponding to the second historical image frame, wherein the second historical image frame is the target image frame in the historical positioning, and the second number is smaller than the first number;

and replacing the pose deviation in the total energy relation corresponding to the historical positioning with the new pose deviation to obtain the optimized prior energy relation.

12. The method according to claim 3, wherein the total energy relationship represents a relationship between the pose deviation and a total energy; the optimizing the first pose by using the total energy relationship corresponding to the current positioning to obtain the pose of the target image frame includes:

determining the pose deviation which enables the total energy to meet a second preset requirement by utilizing the total energy relation corresponding to the positioning;

optimizing the first pose based on the determined pose deviation;

and/or the pose change information comprises at least one pose change amount; the determining a first pose of the target image frame based on the pose change information includes:

and determining a first pose of the target image frame by using the pose variation corresponding to the target image frame.

13. The method of claim 1, wherein said performing a location determination process based on said number of inertial measurement data is performed by a location determination model;

and/or the positioning processing is carried out based on the plurality of inertial measurement data to obtain pose change information between the first historical image frame and the target image frame, and the method comprises the following steps:

determining final motion state information obtained by the positioning processing at this time by using the inertia measurement data and reference motion state information, wherein the reference motion state information is the final motion state information obtained by historical positioning processing;

and obtaining pose change information between the first historical image frame and the target image frame based on the final motion state information obtained by the positioning processing.

14. The method of claim 1, wherein the localization method is performed by a localization system, and prior to the determining the pose of the target image frame based on the pose change information and at least one reference factor, the method further comprises:

judging whether parameters of the positioning system are initialized, wherein the parameters comprise at least one of a gravity direction and an inertia offset;

in response to the parameter having been initialized, performing the determining the pose of the target image frame based on the pose change information and at least one reference factor;

and in response to the non-initialization of the parameters, selecting an initialization mode matched with the state corresponding to the target image frame, initializing the parameters of the positioning system, and determining the pose of the target image frame based on the pose change information and at least one reference factor, wherein the states comprise a motion state and a static state.

15. The method of claim 1, wherein the localization method is performed by a localization system, and after the determining the pose of the target image frame based on the pose change information and at least one reference factor, the method further comprises:

optimizing parameters of the positioning system based on the pose of the target image frame, wherein the parameters include at least one of a direction of gravity, an amount of inertial bias.

16. The method according to claim 1, characterized in that the pose of the target image frame represents the pose of the object to be positioned at the moment of shooting of the target image frame, the target image frame and the first historical image frame being shot by a shooting device fixed relative to the object to be positioned, the inertial measurement data being measured by an inertial measurement device fixed relative to the object to be positioned;

and/or after acquiring a plurality of pieces of inertia measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame, the method further comprises the following steps:

pre-processing the inertial measurement data, wherein the pre-processed inertial measurement data is used to perform the positioning processing, the pre-processing including one or more of converting the inertial measurement data to a gravitational system, removing bias, removing gravity, and normalizing.

17. A positioning device, comprising:

the data acquisition module is used for acquiring a plurality of inertia measurement data measured from a first historical image frame to the shooting period of the target image frame;

the positioning processing module is used for performing positioning processing based on the plurality of inertial measurement data to obtain pose change information between the first historical image frame and the target image frame;

a pose determination module for determining a pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor comprises the number of inertial measurement data.

18. An electronic device comprising a memory and a processor for executing program instructions stored in the memory to implement the method of any of claims 1 to 16.

19. A computer readable storage medium having stored thereon program instructions, which when executed by a processor implement the method of any of claims 1 to 16.