WO2023050634A1 - 定位方法及装置、设备、存储介质及计算机程序产品 - Google Patents

定位方法及装置、设备、存储介质及计算机程序产品 Download PDF

Info

Publication number
WO2023050634A1
WO2023050634A1 PCT/CN2021/143905 CN2021143905W WO2023050634A1 WO 2023050634 A1 WO2023050634 A1 WO 2023050634A1 CN 2021143905 W CN2021143905 W CN 2021143905W WO 2023050634 A1 WO2023050634 A1 WO 2023050634A1
Authority
WO
WIPO (PCT)
Prior art keywords
pose
positioning
image frame
energy relationship
information
Prior art date
Application number
PCT/CN2021/143905
Other languages
English (en)
French (fr)
Inventor
陈丹鹏
王楠
Original Assignee
深圳市慧鲤科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市慧鲤科技有限公司 filed Critical 深圳市慧鲤科技有限公司
Publication of WO2023050634A1 publication Critical patent/WO2023050634A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the embodiment of the present disclosure is based on the Chinese patent application with the application number 202111154872.X, the application date is September 29, 2021, and the application name is "Positioning method and device, equipment, storage medium", and requires the priority of the Chinese patent application Right, the entire content of this Chinese patent application is hereby incorporated into this disclosure by way of introduction.
  • the present disclosure relates to the field of positioning, and in particular to a positioning method and device, equipment, storage medium and computer program product.
  • the positioning methods mainly include visual inertial positioning.
  • the way of visual inertial positioning mainly constructs a three-dimensional map through the image information between the images captured by the device, and then determines the position of the device.
  • this positioning method is more dependent on the external environment.
  • dynamics drastic changes in lighting, weak textures, distant views, and occlusions
  • visual positioning cannot work well. In this case, less information can be extracted from the image captured by the device, so positioning cannot be performed well.
  • Embodiments of the present disclosure at least provide a positioning method and device, equipment, storage medium, and computer program product.
  • An embodiment of the present disclosure provides a positioning method, including: acquiring several inertial measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame; performing positioning processing based on the several inertial measurement data, and obtaining Pose change information between the first historical image frame and the target image frame; determine the pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor includes several inertial measurement data.
  • An embodiment of the present disclosure provides a positioning device, including: a data acquisition module configured to acquire a number of inertial measurement data measured during the shooting period from the first historical image frame to the target image frame; a positioning processing module configured to Perform positioning processing on the data to obtain pose change information between the first historical image frame and the target image frame; the pose determination module is configured to determine the pose of the target image frame based on the pose change information and at least one reference factor, wherein , at least one reference factor includes several inertial measurement data.
  • An embodiment of the present disclosure provides an electronic device, including a memory and a processor, and the processor is configured to execute program instructions stored in the memory, so as to implement the above positioning method.
  • An embodiment of the present disclosure provides a computer-readable storage medium, on which program instructions are stored, and the above positioning method is implemented when the program instructions are executed by a processor.
  • An embodiment of the present disclosure also provides a computer program product, including a computer-readable storage medium storing program codes, and when instructions included in the program codes are executed by a processor of a computer device, the steps of the positioning method described above are implemented.
  • the pose change information between the two frames of images is obtained, and then the pose can be obtained according to the The change information obtains the pose information of the target image frame.
  • the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, thus reducing the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.
  • FIG. 1 is a schematic flowchart of a positioning method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic flowchart of step S13 in a positioning method provided by an embodiment of the present disclosure
  • Fig. 3 is a schematic flowchart of another positioning method provided by an embodiment of the present disclosure.
  • Fig. 4 is a schematic structural diagram of a positioning device provided by an embodiment of the present disclosure.
  • FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • Fig. 6 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present disclosure.
  • FIG. 1 is a schematic flowchart of a positioning method provided by an embodiment of the present disclosure. Described method is carried out by electronic equipment, comprises the steps:
  • Step S11 Obtain several inertial measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame.
  • the target image frame and the first historical image frame can be obtained by the object to be positioned, or can be obtained by shooting a camera component relatively fixed to the position of the target to be positioned, that is, the pose of the camera component can represent the position of the target to be positioned The pose of the object.
  • the inertial measurement data can also be obtained by the object to be positioned, or by an inertial sensor (Inertial Measurement Unit, IMU) whose position is relatively fixed to the object to be positioned, that is, the pose of the inertial sensor can represent the The pose of the object to be positioned.
  • IMU Inertial Measurement Unit
  • the object to be positioned can be a device or any living animal.
  • the object to be positioned may be a vehicle, a robot, a person, a kitten, a puppy, and the like.
  • the aforementioned camera component and inertial sensor may be components in the device, or components other than the device.
  • Inertial measurement data refers to data measured by inertial sensors.
  • the number of inertial sensors may be multiple.
  • Inertial sensors can be accelerometers, gyroscopes, etc.
  • the position between the object to be positioned and the inertial sensor is relatively fixed. That is, the inertial measurement data measured by the inertial sensor within the target time period may represent the inertial measurement data of the object to be positioned within the target time period.
  • the inertial measurement data includes the inertial measurement data between the shooting times of each adjacent image frame.
  • Step S12 Perform positioning processing based on several inertial measurement data to obtain pose change information between the first historical image frame and the target image frame.
  • the pose change information may include the relative displacement between the first historical image frame and the target image frame, and may also include the relative rotation between the two.
  • the pose change information includes relative displacement and relative rotation as an example.
  • Step S13 Determine the pose of the target image frame based on the pose change information and at least one reference factor, wherein the at least one reference factor includes several inertial measurement data.
  • the pose information of the first historical image frame and the pose change information between the first historical image frame and the target image frame may be used to determine the first pose corresponding to the target image frame.
  • the pose of the target image frame can be considered as the pose of the object to be positioned when the target image frame is captured.
  • At least one reference factor is used to optimize the first pose corresponding to the target image frame to obtain the optimized pose corresponding to the target image frame.
  • At least one reference factor can also be used to optimize the first pose, so as to determine the pose of the intermediate frame.
  • the intermediate frame here refers to one frame between the first historical image frame and the target image frame.
  • the pose change information between the two frames of images is obtained, and then the pose can be obtained according to the The change information obtains the pose information of the target image frame.
  • the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, which reduces the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.
  • the general visual-inertial positioning method needs to calculate the pre-integration, so it is necessary to accurately calibrate the parameters such as the offset of the inertial sensor, and the technical solution provided by the embodiment of the present disclosure only needs to perform positioning processing on the inertial measurement data, without Pre-integration, therefore, there is no need to accurately calibrate parameters such as the offset of the inertial sensor, which simplifies the positioning process.
  • positioning processing of pieces of inertial measurement data is performed by a positioning model.
  • the manner of obtaining the pose change information between the first historical image frame and the target image frame may be:
  • the final motion state information obtained by this positioning process is determined by using the inertial measurement data and the reference motion state information. Then, based on the final motion state information obtained in this positioning process, the pose change information between the first historical image frame and the target image frame is obtained.
  • the pose change information can be regarded as at least one of position change and pose change of the object to be positioned between the shooting time of the first historical image frame and the shooting time of the target image frame.
  • a change in position between two points in time may be a relative displacement between the two.
  • a pose change may be a change in the orientation of the object to be located.
  • the reference motion state information is the final motion state information obtained through the historical positioning process.
  • this positioning process is the fourth positioning process performed on the object to be positioned
  • the historical positioning process can be the third positioning process performed on the same object to be positioned, or a combination of the second and third times, It can also be a combination of all previous positioning processes this time.
  • the embodiment of the present disclosure selects the final motion state information obtained in the last positioning process as the reference motion state information in the current positioning process.
  • the final motion state information obtained in this positioning process can be used to derive the motion of the object to be positioned between the shooting time point of the first historical image frame and the shooting time point of the target image frame.
  • the reference motion state information can be used to determine the motion between the shooting time point corresponding to the first historical image frame and the shooting time point of the corresponding target image frame in the historical positioning process.
  • the final motion state information corresponding to the current positioning processing and the historical positioning processing may include motion parameters of the object to be positioned at various moments in the corresponding time period.
  • the motion parameters may include the acceleration and angular velocity of the object to be positioned at each moment, or the speed and direction of motion of the object to be positioned at each moment.
  • the final motion state information can be a local window used to represent the acceleration and angular velocity obtained based on inertial measurement data, as the acceleration and angular velocity of human motion, but because of various factors, which may contain some noise.
  • determining the final motion state information obtained in this positioning process includes the following steps: based on the inertial measurement data, obtaining the initial distance between the first historical image frame and the target image frame The motion state information is then fused with the reference motion state information and the initial motion state information to obtain the final motion state information.
  • the initial motion state information may include motion parameters of the object to be positioned at various moments between the shooting time point of the first historical image frame and the shooting time point of the target image frame.
  • the initial motion state information may include the acceleration and angular velocity of the object to be positioned at each moment, or the speed and direction of motion of the object to be positioned at each moment.
  • the initial motion state information may be in the form of a hidden layer state matrix used to describe the motion of the object to be positioned within the target time period.
  • the initial motion state information can be used to derive the motion of the object to be positioned within the target time period.
  • the obtained initial motion state contains noise, so if only the initial motion state information is used to deduce the motion of the object to be positioned within the target time period, there may be certain errors.
  • the form of the final motion state information and the reference motion state information may also be a hidden layer state matrix used to describe the motion of the object to be positioned within a corresponding time period.
  • the embodiment of the present disclosure considers that the motion of the object to be positioned is continuous and regular. Therefore, the initial motion state information used to represent the motion of the object to be positioned in the corresponding time period corresponds to the historical positioning process used to represent the object to be positioned. In order to estimate the more accurate final motion state information of the object to be positioned in the corresponding time period of this positioning process, it is possible to estimate the object to be positioned at the corresponding time according to the final motion state information. movement within the segment.
  • the positioning model includes a first sub-network and a second sub-network and a third sub-network.
  • the first subnetwork may be a residual network, such as a ResNet18 network.
  • the second sub-network may be a long short-term memory network (Long Short-Term Memory, LSTM).
  • the first sub-network can be used to perform the step of obtaining initial motion state information between the first historical image frame and the target image frame based on the inertial measurement data.
  • the second sub-network is used to perform the step of fusing the reference motion state information with the initial motion state information to obtain the final motion state information
  • the third sub-network is used to execute the final motion state information obtained based on this positioning process to obtain the first motion state information.
  • the positioning method further includes the following steps:
  • Preprocessing of inertial measurement data is used for positioning processing.
  • Preprocessing includes one or more of transforming the inertial measurement data into a gravitational frame, removing bias, removing gravity, and normalizing.
  • the preprocessing further includes performing fixed frame rate interpolation on the inertial measurement data, and caching the interpolated data. By preprocessing the inertial measurement data, the obtained pose change information is more accurate.
  • the positioning method is performed by a positioning system. Before step S13 is executed, the following step is further included: judging whether the parameters of the positioning system have been initialized. Wherein, the parameters include at least one of gravity direction and inertial offset. In response to the parameter being initialized, step S13 is executed. If the parameters are not initialized, select an initialization mode that matches the state corresponding to the target image frame, initialize the parameters of the positioning system, and then perform the above step S13. Among them, the state includes motion state and static state. In some embodiments, before selecting an initialization mode matching the state corresponding to the target image frame, first determine the state corresponding to the target image frame.
  • the manner of determining the state corresponding to the target image frame may include at least one of the following: the average displacement of the two-dimensional features tracked in several consecutive frames of images on the image plane is lower than the first threshold; the standard deviation of the inertial measurement data is low If it is less than the second threshold, the state corresponding to the target image frame is considered to be a static state; otherwise, the state corresponding to the target image frame is considered to be a moving state.
  • the initialization method corresponding to the static state is static initialization
  • the initialization method corresponding to the motion state is motion initialization.
  • the static initialization method includes setting the initial translation to 0, and the initial local gravity is the average value of the accelerometer measurement values between the latest two frames of images.
  • the initial rotation is aligned with the local initial gravity.
  • the initial gyroscope bias is the average of the gyroscope measurements between the last two images.
  • the initial acceleration bias is set to 0.
  • the last two frames refer to the target image frame and the previous frame of the target image frame.
  • the initialization corresponding to the motion state is to obtain the pose without size only through visual positioning, and then align the pre-integration information of the inertial measurement data with the result of visual positioning to restore the size, velocity, gravity and inertial offset.
  • the parameters of the positioning system are initialized, so that the parameters obtained by initialization are more accurate.
  • FIG. 2 is a schematic flowchart of step S13 in a positioning method provided by an embodiment of the present disclosure. As shown in Figure 2, above-mentioned step S13 comprises the following steps:
  • Step S131 Determine the first pose of the target image frame based on the pose change information.
  • the first pose of the target image frame represents the pose of the object to be positioned at the shooting moment of the target image frame.
  • the target image and the first historical image frame are captured by a photographing device that is relatively fixed to the object to be positioned, and the inertial measurement data is measured by an inertial measurement device that is relatively fixed to the object to be positioned.
  • Positioning of any object can be realized by obtaining the target image frame captured by the photographing device relatively fixed to the object to be positioned and the inertial measurement data obtained by the inertial measurement device relatively fixed to the object to be positioned.
  • the pose change information includes at least one pose change amount.
  • at least one kind of pose variation includes a position variation and an attitude variation.
  • the manner of acquiring the first pose of the target image frame may be to determine the first pose of the target image frame by using the amount of change in pose corresponding to the target image frame.
  • the pose of the first historical image frame is known, and the first pose of the target image frame can be obtained based on the amount of change in pose between the first historical image frame and the target image frame. By determining the position change amount and the attitude change amount, the determined first pose is more accurate.
  • Step S132 Using the pose change information and at least one reference factor, construct the total energy relationship corresponding to this positioning.
  • the total energy relationship corresponding to this positioning is used to determine the pose deviation to be optimized of the target image frame.
  • each reference factor is used to determine the measured energy relationship corresponding to the reference factor. And, using the pose change information to determine the motion prior energy relationship. Finally, based on the motion prior energy relationship and the measured energy relationship corresponding to each reference factor, the total energy relationship corresponding to this positioning is constructed.
  • pre-integration information for several inertial measurement data is obtained.
  • the inertial measurement energy relation is then constructed using the pre-integrated information of several inertial measurement data.
  • the way of constructing the energy relationship of the inertial measurement can refer to generally known technologies.
  • H k is the Jacobian matrix corresponding to the last positioning process, is the Jacobian matrix of this positioning about the pose, is the Jacobian matrix of this positioning with respect to the inertial offset, is the pose deviation corresponding to the last positioning, is the pose deviation corresponding to this positioning, It is the deviation corresponding to the inertial offset corresponding to this positioning.
  • ⁇ u is the covariance matrix corresponding to the energy relationship of the inertial measurement
  • r uk+1 is the residual error of the inertial sensor measurement.
  • the way to obtain the prior energy relationship of motion can be:
  • At least one positioning is multiple positioning, multiple times is two or more.
  • at least one positioning includes this positioning, and several parameters corresponding to each positioning include the pose change amount obtained during the positioning process, the position difference between the initial measurement time and the end measurement time corresponding to the positioning, and the corresponding initial measurement time. initial pose information.
  • the position difference between the start measurement time and the end measurement time refers to the difference between the position of the object to be positioned at the start measurement time and the position of the object to be positioned at the end of the measurement time.
  • the initial attitude information at the initial measurement moment refers to the orientation of the object to be positioned at the initial measurement moment.
  • the motion prior energy relationship is determined, so that the constructed motion prior energy relationship is more accurate, and the optimized pose is more accurate.
  • the initial pose information of the initial image frame includes yaw angle information.
  • the inertial measurement data is not presented in the form of yaw angle, and the inertial measurement data is converted into a quaternion form.
  • the yaw angle converted from the XYZ coordinate axis is 90°, the yaw angle information is acquired again by adjusting the coordinate system to the YXZ coordinate axis.
  • the pose change information further includes certainty characterizing information of the pose change amount.
  • the degree of certainty characterization information can be used to represent the degree of certainty, and can also be used to represent the degree of uncertainty.
  • the weight of the corresponding positioning is obtained.
  • the positioning weight is determined based on a preset multiple of certainty representation information corresponding to the positioning, where the preset multiple is a natural number.
  • the weight may be a covariance matrix obtained by transforming certainty characterization information based on a preset multiple.
  • the preset multiple is generally less than or equal to 1, for example, the preset multiple may be 0.1, etc.
  • the preset multiple The multiplier can also be greater than 1.
  • the preset multiple is generally greater than or equal to 1, for example, the preset multiple can be 10, etc. Of course, this is only an example. In other embodiments, the preset multiple can also be less than 1.
  • a motion prior energy relationship is determined.
  • the motion prior energy relationship of each positioning may be determined first, and then based on the weight of each positioning, the motion prior energy relationship of each positioning may be combined to obtain the final motion prior energy relationship.
  • the corresponding weight is obtained by characterizing the information with the degree of certainty, so that the constructed motion prior energy relationship is more accurate.
  • the way to obtain the motion prior energy relationship c n may be:
  • the start measurement time of several inertial measurement data corresponding to each positioning is time i
  • the end measurement time is time j
  • p i and p j represent the position of the object to be positioned corresponding to time i and j time respectively
  • d ij represents the amount of pose change between the i-th moment and the j-th moment output by the localization model.
  • T stands for transpose.
  • ⁇ ij represents the covariance matrix (weight) corresponding to the positioning model, and the covariance matrix is obtained from the certainty characterization information.
  • the degree of certainty characterization information is a three-dimensional vector, and three elements in the three-dimensional vector are logarithms of diagonal elements of the covariance matrix.
  • a preset multiple of the degree of certainty characterization information may be used as the logarithm of the diagonal elements of the covariance matrix.
  • the certainty characteristic information is used to represent the uncertainty, and the certainty characteristic information may be the original certainty characteristic information, or the certainty characteristic information enlarged by a preset multiple.
  • the pose change information includes at least one pose change amount.
  • the at least one pose change amount includes at least one of a position change amount and an attitude change amount.
  • Embodiments of the present disclosure may also provide the following method to associate the motion prior energy with the optimal state, so that the optimal value of the optimal state can be obtained when the energy is minimum.
  • the following steps are also included: taking each positioning in at least one positioning as the target positioning, based on the pre-integration information of several inertial measurement data corresponding to the target positioning, and the bit position of the initial image frame The pose and the pose of the end image frame are used to determine the initial pose information corresponding to the target positioning, the position at the start measurement time, and the position at the end measurement time.
  • the pose of the start image frame and the pose of the end image frame in the target positioning can both be the first pose before optimization, or the optimized pose. posture.
  • the pose of the start image frame and the pose of the end image frame are the first pose before optimization. That is, if the target positioning is the last positioning, the initial attitude information, the position of the initial measurement moment and the position of the end measurement moment in the previous positioning can be obtained from the pre-integration information of several inertial measurement data in the previous positioning process, starting The pose of the start image frame and the pose of the end image frame are determined.
  • the pose of the object to be positioned at the initial measurement moment is the same as the pose when the initial image frame is taken, and the pose at the end of the measurement moment is the same as the pose when the end image frame is taken. same.
  • the way to determine the position of the start pose information, the start measurement time and the end measurement time corresponding to the target positioning can be as follows :
  • m represents the start image frame
  • n represents the end image frame
  • i represents the start measurement time
  • j represents the end measurement time
  • p Ii represents the position of the initial measurement moment
  • w represents the world coordinate system, for example W p Ii represents the position of the initial measurement moment in the world coordinate system, and the same applies to others.
  • p Ij represents the position at the end of the measurement moment
  • R Ii represents the initial attitude information.
  • p Im represents the position of the initial image frame
  • v Im represents the velocity of the initial image frame
  • ⁇ t m represents the time interval between the shooting time of the initial image frame and the initial measurement moment.
  • R Im represents the attitude information corresponding to the initial image frame, Indicates the position obtained by pre-integrating the inertial measurement data between the initial image frame and the initial measurement moment without considering the initial velocity and gravity.
  • p Ij represents the position at the end of the measurement moment, Represents the attitude obtained by pre-integrating the inertial measurement data between the initial image frame and the initial measurement moment.
  • H represents the corresponding Jacobian matrix
  • r represents the corresponding measurement residual
  • ⁇ n is the covariance matrix corresponding to the final motion prior energy relationship.
  • the covariance matrix can be obtained from the certainty characterization information output by the positioning model and the pre-integration information. See above for the meaning of the subscripts.
  • ⁇ ij is a covariance matrix determined based on a preset multiple of the certainty characterization information output by the positioning model. and are the pose parts of the covariance matrix corresponding to the pre-integration at time j and i, respectively.
  • w represents the world coordinate system
  • T represents the transpose. See above for the meaning of the subscripts.
  • the pre-integration information is used in the calculation of the motion prior energy relationship, it can be considered that the final motion prior energy relationship contains the inertial measurement energy relationship, so the total energy relationship can be constructed without using the inertial measurement energy relationship .
  • the pre-integration information and the pose of the relevant image frame determine the position of the start measurement time and the end measurement time of several inertial measurement data, realize the correction of the position of the start measurement time and end measurement time, and then improve the motion prior energy accuracy of the relationship.
  • the positioning that satisfies the removal condition is eliminated from the multiple positionings.
  • the removal condition is that the preset processing result between the pose change amount corresponding to the positioning and the certainty characteristic information of the pose change amount satisfies the first preset condition.
  • the motion prior energy relationship is determined by using several parameters corresponding to the remaining times of positioning after elimination.
  • the abnormal location is removed by acquiring the Mahalanobis distance corresponding to the pose variation. where the removal condition can be greater than the threshold.
  • d represents the amount of pose change initially estimated by the filter, Indicates the amount of pose change output by the localization model.
  • H P denote the measured Jacobians and corresponding state covariances in the motion prior energy relation.
  • the covariance matrix corresponding to the motion prior energy relationship can also be enlarged by several times to reduce the inaccurate measurement data, for example, by ten times. By screening the parameters to eliminate abnormal parameters, the acquired motion prior energy relationship is more accurate.
  • the total energy relation constructed based on the inertial measurement energy relation and the motion prior energy relation can be:
  • H u is the Jacobian matrix in the inertial measurement energy relation
  • r u is the measurement residual in the inertial measurement energy relation
  • H n is the Jacobian matrix in the motion prior energy relation
  • r n is the motion prior energy relation
  • Measure residuals ⁇ n is the covariance matrix corresponding to the motion prior energy relationship
  • ⁇ z is the covariance matrix corresponding to the inertial measurement energy relationship. It can be considered that the covariance can be the weight in the corresponding relationship.
  • the total energy relationship may not include relevant parameters of the inertial measurement energy relationship.
  • the at least one reference factor further includes image information about the target image frame and the first historical image frame.
  • image information of each image frame between the target image frame and the first historical image frame may also be included.
  • the pose of the target image frame is obtained by combining the image information, so the positioning accuracy can be improved.
  • the manner of determining the measurement energy relationship corresponding to the reference factor includes: using image information to determine the visual measurement energy relationship corresponding to the image information.
  • the following steps are further included: matching feature points of several reference image frames to obtain feature point matching results.
  • the matching feature points may use the sparse optical flow method to track the feature points of the previous frame image to the feature points of the current image frame to obtain a matching result of the feature points.
  • epipolar geometric constraints are used to remove erroneous matching results.
  • the several reference images include the first historical image frame and the target image frame in at least one positioning process, and at least one positioning includes the current positioning.
  • the feature point matching result includes a set of feature points. If a feature point exists in two reference image frames at the same time, the feature point will be added to the feature point set. Of course, in other embodiments, only when a feature point exists in three or more reference image frames at the same time, it is added to the feature point set. Wherein, the feature point is a two-dimensional feature point.
  • each pair of matching image frames is determined from the reference image frames.
  • each pair of matching image frames has matching feature point pairs.
  • the visual measurement energy relationship is determined.
  • the matching image frame is not the target image frame in the historical positioning, that is, the matching image frame is the image frame between the first historical image frame and the target image frame in this positioning or the target image frame, then the bit of the matching image frame
  • the pose is the first pose. If the matching image frame is the target image frame in the historical positioning, the pose of the matching image frame can be the first pose before optimization or the pose after optimization.
  • the visual measurement energy relationship is constructed through two-dimensional point information instead of the three-dimensional point information, which reduces the inaccurate visual measurement energy relationship caused by the accuracy of the three-dimensional point, thus making the obtained visual measurement energy relationship is more accurate.
  • the way to obtain the visual energy relationship is:
  • F is the set of feature points that can be tracked by each reference image frame. If a feature point is observed by two or more reference image frames, the feature point can be added to the set of feature points.
  • C is the camera state set that can track the two-dimensional feature point set F. is the two-dimensional position of the kth feature point on the jth frame reference image frame.
  • K is the projection matrix of the camera component, and R i represents the rotation matrix of the camera component corresponding to the i-th reference image frame.
  • T stands for transpose.
  • ⁇ Z is the corresponding covariance matrix.
  • the i frame proposed in this formula is the reference image frame captured earlier, and the j frame is the reference image frame captured later.
  • the following step may also be included: determining and optimizing the prior energy relationship based on the total energy relationship corresponding to the historical location.
  • This optimized prior energy relation can be used to construct the total energy relation.
  • the total energy relationship is constructed by combining the optimized prior energy relationship, so that the deviation of the determined pose is more accurate.
  • the way to obtain the optimized prior energy relationship may be: using the pose of the second historical image frame, the pose corresponding to the second number of image frames before the second historical image frame, and the pose of the target image frame And the inertial information corresponding to the second historical image frame is updated to obtain a new pose deviation corresponding to the second historical image frame.
  • the second historical image frame is the target image frame in the historical positioning.
  • the pose deviation corresponding to the target image frame is at least determined by the pose of the target image frame, the pose corresponding to the first number of image frames before the target image frame, and the inertial information corresponding to the target image frame.
  • the second quantity is smaller than the first quantity.
  • the first number of image frames may be the corresponding target image frames in the previous positioning processes.
  • the pose of the first number of image frames may be a pose optimized through a historical positioning process, or may be a first pose before optimization.
  • the first historical image frame used in this positioning is not the same frame as the first historical image frame used in the previous positioning.
  • the first historical image frame can be the first frame of the shooting, and the first frame is used as the origin of the world coordinate system, and the target image frame is the third frame of the shooting, because after the positioning is performed for the first time, the first frame
  • the first historical image frame used for the second positioning execution can be the 2nd frame or the 3rd frame, if the first historical image frame used for the second positioning execution is In the second frame, the target image frame is the fourth frame.
  • the target image frame may be the fifth frame.
  • the second historical image frame is the third frame.
  • the pose of the second historical image frame used may be the pose after the last positioning optimization, and other disclosed embodiments may also be the pose before optimization.
  • the pose deviation in the total energy relationship corresponding to the historical positioning is replaced with a new pose deviation to obtain an optimized prior energy relationship.
  • the inertial information corresponding to the target image frame may be an inertial offset.
  • the inertial bias may include acceleration bias, angular velocity bias, and the like.
  • obtaining an optimized prior energy relation The way is:
  • H k+1 is the Jacobian matrix corresponding to the k+1 times of positioning. is the pose deviation corresponding to the target image frame, r k+1 is the corresponding measurement residual.
  • the total energy relationship corresponding to this positioning is constructed.
  • the total energy relationship corresponding to this positioning is constructed based on the motion prior energy relationship and the inertial measurement energy relationship.
  • the current positioning correspondence is constructed based on the motion prior energy relationship and the visual measurement energy relationship. The total energy relationship of the positioning, or construct the total energy relationship corresponding to this positioning based on the motion prior energy relationship, visual measurement energy relationship and visual measurement energy relationship, or based on the motion prior energy relationship, visual measurement energy relationship and optimization prior The energy relationship and the inertial measurement energy relationship construct the total energy relationship corresponding to this positioning.
  • Step S133 Using the total energy relationship corresponding to this positioning, optimize the first pose to obtain the pose of the target image frame.
  • the total energy relationship represents the relationship between the pose deviation and the total energy.
  • using the total energy relationship corresponding to this positioning to optimize the first pose to obtain the pose of the target image frame may be: using the total energy relationship corresponding to this positioning to determine that the total energy satisfies The pose deviation of the second preset condition.
  • the second preset condition may be that the total energy is minimum.
  • the first pose of the target image frame is then optimized based on the determined pose bias.
  • the obtained pose deviation is summed with the first pose of the target image frame to obtain the optimized pose of the target image frame.
  • the way to minimize the total energy relationship is:
  • H k+1 is the Jacobian matrix in the optimized prior energy relationship
  • r k+1 is the measurement residual in the optimized prior energy relationship
  • H z is the Jacobian matrix in the visual measurement energy relation
  • r z is the measurement residual in the visual measurement energy relation
  • H n is the Jacobian matrix in the motion prior energy relation
  • r n is the motion prior energy relation
  • Measure residuals ⁇ n is the covariance matrix corresponding to the motion prior energy relationship
  • ⁇ z is the covariance matrix corresponding to the visual measurement energy relationship.
  • the final Sum with the first pose to get the optimized pose can replace the original offset in the positioning system, so that the subsequent pose calculation is more accurate.
  • parameters such as the bias of the inertial sensor and the direction of gravity need to be optimized, the above Then it can also represent the deviation corresponding to all parameters to be optimized, that is, in this case, Including the total deviation of the deviation corresponding to the parameters such as pose deviation and each offset amount.
  • the parameters of the positioning system may also be optimized based on the pose of the target image frame.
  • the parameter deviation that needs to be optimized in this positioning process is obtained, and then the obtained deviation is added to the corresponding parameter to obtain the optimized parameter.
  • the optimized parameters include at least one of gravity direction and inertial offset.
  • the optimized parameters can be used for positioning. By optimizing the parameters of the positioning system, the accuracy of the next positioning is higher.
  • FIG. 3 is a schematic flowchart of another positioning method provided by an embodiment of the present disclosure. As shown in Figure 3, the positioning method provided by the embodiment of the present disclosure includes the following steps:
  • Step S21 The camera component acquires several image frames.
  • image frames refer to image frames between the first historical image frame and the target image frame.
  • Step S22 Perform feature extraction and tracking.
  • feature extraction and tracking are performed on several image frames to obtain image information corresponding to the several image frames.
  • Step S23 The inertial sensor acquires several inertial measurement data.
  • the several inertial measurement data refer to the inertial measurement data during the shooting period between the shooting time of the first historical image frame and the shooting time of the target image frame.
  • Step S24 Perform data caching.
  • one or more of converting inertial measurement data to a gravitational frame, removing bias, removing gravity, and normalizing is performed in the data cache.
  • Step S25 Input the positioning model.
  • the positioning model performs positioning processing based on several inertial measurement data to obtain the pose change information of this positioning processing.
  • the manner in which the positioning model performs positioning processing on several inertial measurement data is as described above.
  • Step S26 Perform pre-integration.
  • Step S27 Determine whether the positioning system is initialized.
  • the determination method is as above.
  • step S28 If the judging result is that the positioning system has not been initialized, execute step S28, otherwise execute step S29.
  • Step S28 Perform static initialization/motion initialization.
  • Step S29 Perform abnormal point detection.
  • the outlier detection is the step of removing the data satisfying the first preset condition above.
  • Step S30 Optimizing and updating.
  • the total energy relationship is constructed based on the image information, pre-integration information, and pose change information, and the pose corresponding to each image frame is optimized and updated to obtain the final pose of the target image frame.
  • Step S31 Optimizing parameters.
  • the parameters of the positioning system that can be optimized may include acceleration bias Ba, angular velocity bias Bg, and other parameters R of the positioning system that can be optimized.
  • step S21 and step S23 may be executed synchronously.
  • step S22, step S24 and step S26 may also be executed synchronously.
  • the input of the positioning system may include the output of the camera assembly and the inertial sensor, that is, all image frames between the first historical image and the target image frame and several inertial measurement data.
  • Each image frame and inertial measurement data are preprocessed separately.
  • the way to preprocess the image frame can be feature extraction and tracking.
  • the preprocessing of inertial measurement data can also be divided into two steps, one is to perform pre-integration, and the other is to perform one of the steps of converting inertial measurement data to the gravity system, removing bias, removing gravity, and normalizing in the data cache. one or more species.
  • the total energy relationship in this positioning process can also be constructed according to the historical total energy relationship.
  • the pose change information between the two frames of images can be obtained. If the first historical image frame is the world If the origin of the coordinate system or its pose is known, the pose information of the target image frame can be obtained according to the pose change information. In addition, after the pose change information is obtained, the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose. The above process reduces the impact of vision on positioning, thereby reducing the situation of low positioning accuracy due to external environment and other factors.
  • the motion prior (pose change information) corresponding to the positioning model, IMU information, and visual information are tightly coupled.
  • high-precision positioning effects can be achieved.
  • more robust tracking can be achieved.
  • the technical solutions provided by the embodiments of the present disclosure can be coupled with other positioning algorithms or sensors to perform positioning and navigation.
  • the positioning method provided by the embodiments of the present disclosure can be applied to scenarios such as augmented reality, virtual reality, robots, automatic driving, games, film and television, education, e-commerce, tourism, smart medical care, interior decoration equipment, smart home, smart manufacturing, maintenance and assembly, etc. middle.
  • the execution subject of the positioning method may be a positioning device, for example, the positioning method may be executed by a terminal device or a server or other processing equipment, wherein the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal , cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc.
  • the positioning method may be implemented by a processor calling computer-readable instructions stored in a memory.
  • FIG. 4 is a schematic structural diagram of a positioning device provided by an embodiment of the present disclosure.
  • the positioning device 40 includes a data acquisition module 41 , a positioning processing module 42 and a pose determination module 43 .
  • the data acquisition module 41 is configured to acquire some inertial measurement data measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame;
  • the positioning processing module 42 is configured to perform positioning processing based on the several inertial measurement data,
  • the pose change information between the first historical image frame and the target image frame is obtained;
  • the pose determination module 43 is configured to determine the pose of the target image frame based on the pose change information and at least one reference factor, wherein at least one reference Factors include several inertial measurements.
  • the pose change information between the two frames of images is obtained, and then the pose can be obtained according to the The change information obtains the pose information of the target image frame.
  • the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, thus reducing the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.
  • the at least one reference factor further includes image information about the target image frame and the first historical image frame.
  • the pose of the target image frame is obtained by combining the image information, so the positioning accuracy can be improved.
  • the pose determination module 43 when the pose determination module 43 determines the pose of the target image frame based on the pose change information and at least one reference factor, it is configured to: determine the first position of the target image frame based on the pose change information. Pose; and, using the pose change information and at least one reference factor to construct a total energy relationship corresponding to this positioning, wherein the total energy relationship corresponding to this positioning is used to determine the pose deviation of the target image frame to be optimized; Using the total energy relationship corresponding to this positioning, the first pose is optimized to obtain the pose of the target image frame.
  • the first pose of the target image frame is optimized by using the pose change information and at least one reference factor to construct a total energy relationship, and determining the pose deviation through the total energy relationship.
  • the pose determination module 43 is configured to use each reference factor to determine the corresponding The measured energy relationship, and use the pose change information to determine the motion prior energy relationship; based on the motion prior energy relationship and the measurement energy relationship corresponding to each reference factor, the total energy relationship corresponding to this positioning is constructed.
  • the pose determination module 43 uses each reference factor to determine the measured energy relationship corresponding to the reference factor, it is configured to: acquire pre-integration information of several inertial measurement data; Pre-integrate information to construct inertial measurement energy relations.
  • the determined pose deviation is more accurate than a single motion prior energy relationship.
  • the pose change information includes at least one pose change amount, and the at least one pose change amount includes at least one of a position change amount and an attitude change amount; the pose determination module 43 uses the pose change amount
  • the cooperation is as follows: determine the prior energy relationship of motion by using several parameters corresponding to at least one positioning, wherein at least one positioning includes this positioning, and the several parameters corresponding to each positioning include the positioning process The amount of pose change obtained in the positioning process, the position difference between the start measurement time and the end measurement time of several inertial measurement data used in the positioning process, and the initial attitude information corresponding to the start measurement time.
  • the above-mentioned solution determines the motion prior energy relationship by obtaining the position change of historical positioning and the position difference during the measurement of several inertial measurement data, so that the constructed motion prior energy relationship is more accurate, and the optimized pose is more accurate .
  • the pose change information also includes the degree of certainty characterization information of the pose change; the pose determination module 43 is configured to: Based on the certainty characterization information obtained in each positioning process, the weight of the corresponding positioning is obtained; the weight of at least one positioning and several parameters are used to determine the motion prior energy relationship.
  • At least one positioning is multiple positionings
  • the pose determination module 43 is configured to: from the multiple positionings, eliminate the satisfying The positioning of the removal condition; wherein, the removal condition is that the preset processing result between the corresponding pose change amount and the certainty characterization information of the pose change amount satisfies the first preset condition; Several parameters of , determine the motion prior energy relation.
  • the corresponding weights are obtained by characterizing the information with the degree of certainty, so that the constructed motion prior energy relationship is more accurate.
  • abnormal parameters are eliminated by screening the parameters, so that the acquired motion prior energy relationship is more accurate.
  • the initial attitude information includes yaw angle information.
  • the pose determination module 43 is also configured to: take each positioning in at least one positioning as the target positioning, based on the pre-integration information, initial The pose of the image frame and the pose of the end image frame determine the initial pose information corresponding to the target positioning, the position at the start measurement time, and the position at the end measurement time.
  • the constructed motion prior energy relationship is more accurate.
  • the positions of the start measurement time and the end measurement time of several inertial measurement data are determined, and the position correction of the start measurement time and the end measurement time is realized, thereby improving the motion first. Check the accuracy of the energy relationship.
  • the reference factors also include image information about the target image frame and the first historical image frame, and the pose determination module 43 uses each reference factor to determine the measured energy relationship corresponding to the reference factor, and configures To: use the image information to determine the visual measurement energy relationship corresponding to the image information.
  • the pose determination module 43 is also used to: match the feature points of several reference image frames to obtain the feature point matching results, wherein the several reference images include at least one The first historical image frame and the target image frame in the positioning process, and at least one positioning includes this positioning; using the image information to determine the visual measurement energy relationship corresponding to the image information, including: based on the feature point matching results, from several reference image frames Determine at least one pair of matching image frames, wherein each pair of matching image frames has a matching feature point pair; use the pose of each pair of matching image frames and the position of the matching feature point pair in the matching image frame to determine the visual measurement energy relationship.
  • the above solution constructs the visual measurement energy relationship through two-dimensional point information instead of the three-position point information, which reduces the inaccurate visual measurement energy relationship caused by the accuracy of the three-dimensional point, so that the acquisition can be obtained
  • the visual measurement of the energy relationship is more accurate.
  • the pose determination module 43 is further configured to: based on the total energy relationship corresponding to the historical positioning Energy relationship, determine and optimize the prior energy relationship; based on the motion prior energy relationship and the measured energy relationship corresponding to each reference factor, construct the total energy relationship corresponding to this positioning, including: based on the motion prior energy relationship, optimize the prior energy relationship relationship and the measured energy relationship corresponding to each reference factor to construct the total energy relationship corresponding to this positioning.
  • the total energy relationship is constructed by combining the optimization of the prior energy relationship, so that the determined pose deviation is more accurate.
  • the total energy relationship corresponding to the historical positioning is the total energy relationship corresponding to the previous positioning.
  • the pose deviation corresponding to the target image frame is at least determined by the pose of the target image frame, the pose corresponding to the first number of image frames before the target image frame, and the inertial information corresponding to the target image frame; pose determination
  • module 43 is configured to: use the pose of the second historical image frame, the position corresponding to the second number of image frames before the second historical image frame
  • the pose, the pose of the target image frame, and the inertia information corresponding to the second historical image frame are updated to obtain the new pose deviation corresponding to the second historical image frame, wherein the second historical image frame is the target image frame in the historical positioning , the second number is smaller than the first number; the pose deviation in the total energy relationship corresponding to the historical positioning is replaced with a new pose deviation to obtain an optimized prior energy relationship.
  • the pose deviation corresponding to the second historical image frame is updated, so that the determined optimized prior
  • the experimental energy relationship is associated with the pose of the target image frame, and then the pose deviation of the target image frame determined by using the energy relationship is more accurate.
  • the total energy relationship represents the relationship between the pose deviation and the total energy; the pose determination module 43 optimizes the first pose by using the total energy relationship corresponding to this positioning, and obtains the target image frame
  • the configuration is: using the total energy relationship corresponding to this positioning to determine the pose deviation that makes the total energy meet the second preset condition; based on the determined pose deviation, the first pose of the target image frame is determined. optimization.
  • the pose change information includes at least one pose change amount; the pose determination module 43 is configured to: use the target image to determine the first pose of the target image frame based on the pose change information The pose change amount corresponding to the frame determines the pose of the target image frame.
  • the pose of the final target image frame is more accurate.
  • the pose of the target image frame can be determined through the amount of pose change, and the whole process is convenient and fast.
  • positioning processing of the pieces of inertial measurement data is performed by a positioning model.
  • the positioning processing module 42 is configured to: use the inertial measurement data and the reference The motion state information determines the final motion state information of the object to be positioned in this positioning process, wherein the reference motion state information is the final motion state information corresponding to the previous positioning process; based on the final motion state information obtained in this positioning process, The pose change information between the first historical image frame and the target image frame is obtained.
  • the final motion state information obtained by this positioning processing is more accurate.
  • the positioning method is performed by a positioning system.
  • the pose determination module 43 is further configured to: determine whether the parameters of the positioning system have been Initialization, wherein the parameters include at least one of the gravity direction and the inertial offset; in response to the parameters being initialized, the pose of the target image frame is determined based on the pose change information and at least one reference factor; in response to the parameters not being initialized Initialize, then select the initialization method corresponding to the state corresponding to the target image frame, initialize the parameters of the positioning system, and then perform based on the pose change information and at least one reference factor to determine the pose of the target image frame, wherein the state includes motion state and static state.
  • the parameters of the positioning system are initialized by selecting an initialization mode that matches the state corresponding to the target image frame when the positioning system has not been initialized, so that the parameters obtained by initialization are more accurate.
  • the positioning method is performed by a positioning system.
  • the pose determination module 43 is further configured to: based on the pose of the target image frame , to optimize the parameters of the positioning system, wherein the parameters include at least one of the direction of gravity and the amount of inertial bias.
  • the pose of the target image frame represents the pose of the object to be positioned at the shooting moment of the target image frame
  • the target image frame and the first historical image frame are captured by a shooting device relatively fixed to the object to be positioned
  • the inertial measurement data is obtained by measuring the inertial measurement device relatively fixed to the object to be positioned.
  • the data acquisition module 41 is further configured to: preprocess the inertial measurement data, wherein, the preprocessed inertial measurement data is used for positioning processing, and the preprocessing includes one or more of converting the inertial measurement data into a gravity system, removing bias, removing gravity, and normalizing.
  • the target image frame is captured by the photographing device relatively fixed to the object to be positioned and the inertial measurement data is obtained by the inertial measurement device relatively fixed to the object to be positioned, so any object can be positioned.
  • the obtained pose change information is more accurate.
  • the pose change information between the two frames of images is obtained, and then the pose can be obtained according to the The change information obtains the pose information of the target image frame.
  • the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, thus reducing the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.
  • FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device 50 includes a memory 51 and a processor 52, and the processor 52 is configured to execute program instructions stored in the memory 51, so as to implement the steps in any of the positioning method embodiments described above.
  • the electronic device 50 may include but not limited to: medical equipment, microcomputers, desktop computers, and servers.
  • the electronic device 50 may also include mobile devices such as notebook computers and tablet computers, which are not limited here.
  • the processor 52 is used to control itself and the memory 51 to implement the steps in any one of the positioning method embodiments above.
  • the processor 52 may also be called a central processing unit (Central Processing Unit, CPU).
  • the processor 52 may be an integrated circuit chip and has signal processing capabilities.
  • the processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the processor 52 may be jointly realized by an integrated circuit chip.
  • the pose change information between the two frames of images is obtained, and then the pose can be obtained according to the The change information obtains the pose information of the target image frame.
  • the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, thus reducing the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.
  • FIG. 6 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present disclosure.
  • the computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps in any one of the positioning method embodiments described above.
  • An embodiment of the present disclosure also provides a computer program product, including a computer-readable storage medium storing program code.
  • program code including a computer-readable storage medium storing program code.
  • the computer program product can be realized by hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.
  • the pose change information between the two frames of images is obtained, and then the pose can be obtained according to the The change information obtains the pose information of the target image frame.
  • the pose of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, thus reducing the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.
  • This disclosure relates to the field of augmented reality.
  • acquiring the image information of the target object in the real environment and then using various visual correlation algorithms to detect or identify the relevant features, states and attributes of the target object, and thus obtain the image information that matches the specific application.
  • AR effect combining virtual and reality.
  • the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places, etc.
  • Vision-related algorithms can involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc.
  • Application scenarios can not only involve interactive scenarios such as tours, navigation, explanations, reconstructions, virtual effect superimposition and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display.
  • the relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network.
  • the above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.
  • the functions or modules included in the apparatus provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments, and the implementation can refer to the descriptions of the above method embodiments.
  • the disclosed methods and devices may be implemented in other ways.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage
  • the medium includes several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) or a processor (Processor) to execute all or part of the steps of the methods in various embodiments of the present disclosure.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .
  • the embodiment of the present disclosure discloses a positioning method, device, storage medium, and computer program product.
  • the positioning method includes: acquiring a number of inertias measured during the shooting period from the shooting time of the first historical image frame to the shooting time of the target image frame Measurement data; perform positioning processing based on several inertial measurement data, and obtain pose change information between the first historical image frame and the target image frame; determine the pose of the target image frame based on the pose change information and at least one reference factor, wherein , at least one reference factor includes several inertial measurement data.
  • the pose change information between the two frames of images is obtained, and then can be obtained according to the pose change information
  • the pose information of the target image frame is determined in combination with reference factors, thereby obtaining a more accurate pose.
  • the above process uses the inertial measurement data to calculate the pose change and then realize the positioning, thus reducing the adverse impact of the vision on the positioning, thereby reducing the low positioning accuracy caused by factors such as the external environment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

一种定位方法及装置（40）、设备（50）、存储介质（60），定位方法包括：获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据（S11）；基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息（S12）；基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，至少一个参考因子包括若干惯性测量数据（S13）。方案能够提高定位准确度。

Description

定位方法及装置、设备、存储介质及计算机程序产品
相关申请的交叉引用
本公开实施例基于申请号为202111154872.X、申请日为2021年09月29日、申请名称为“定位方法及装置、设备、存储介质”的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本公开。
技术领域
本公开涉及定位领域，尤其涉及一种定位方法及装置、设备、存储介质及计算机程序产品。
背景技术
目前，定位方式主要有视觉惯性定位。其中，视觉惯性定位的方式主要通过设备拍摄到的图像之间的图像信息，构建三维地图，然后确定设备的位置。其中，该定位方式比较依赖外部环境，在动态、光照剧烈变化、弱纹理、远景和遮挡等情况下，视觉定位无法很好正常工作。在这种情况下，设备拍摄得到的图像中提取得到的信息较少，则无法很好地进行定位。
发明内容
本公开实施例至少提供一种定位方法及装置、设备、存储介质及计算机程序产品。
本公开实施例提供了一种定位方法，包括：获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据；基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息；基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，至少一个参考因子包括若干惯性测量数据。
本公开实施例提供了一种定位装置，包括：数据获取模块，配置为获取第一历史图像帧至目标图像帧的拍摄期间测量得到的若干惯性测量数据；定位处理模块，配置为基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息；位姿确定模块，配置为基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，至少一个参考因子包括若干惯性测量数据。
本公开实施例提供了一种电子设备，包括存储器和处理器，处理器用于执行存储器中存储的程序指令，以实现上述定位方法。
本公开实施例提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述定位方法。
本公开实施例还提供一种计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令被计算机设备的处理器运行时，实现上述定位方法的步骤。
上述方案，通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，故减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开实施例。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开实施例的技术方案。
图1是本公开实施例提供的一种定位方法的流程示意图；
图2是本公开实施例提供的一种定位方法中步骤S13的流程示意图；
图3是本公开实施例提供的另一种定位方法的流程示意图；
图4是本公开实施例提供的一种定位装置的结构示意图；
图5是本公开实施例提供的一种电子设备的结构示意图；
图6是本公开实施例提供的一种计算机可读存储介质的结构示意图。
具体实施方式
下面结合说明书附图，对本公开实施例的方案进行详细说明。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、接口、技术之类的具体细节，以便透彻理解本公开实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
请参阅图1，图1是本公开实施例提供的一种定位方法的流程示意图。所述方法由电子设备执行，包括如下步骤：
步骤S11：获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据。
其中，目标图像帧和第一历史图像帧可以是由待定位对象获取，也可以是由与待定位对象的位置相对固定的摄像组件拍摄得到，即摄像组件的位姿情况可以表示所述待定位对象的位姿情况。当然，惯性测量数据同样可以是由待定位对象获取，也可以是由与待定位对象的位置相对固定的惯性传感器(Inertial Measurement Unit，IMU)获取得到，即惯性传感器的位姿情况可以表示所述待定位对象的位姿情况。
待定位对象可以是设备，也可以是任意具有生命的动物体。例如，待定位对象可以是车辆、机器人、人、小猫、小狗等。可以理解的是，当待定位对象为设备时，上述摄像组件和惯性传感器可以为该设备中的组件，或者为该设备以外的组件。
惯性测量数据指的是惯性传感器测量得到的数据。其中，惯性传感器的数量可以是多个。惯性传感器可以是加速度计，也可以是陀螺仪等。待定位对象与惯性传感器之间的位置相对固定。即，惯性传感器在目标时间段内测量得到的惯性测量数据可以代表待定位对象在目标时间段内的惯性测量数据。其中，若目标图像帧和第一历史图像帧之间包括多帧图像时，惯性测量数据则包括各相邻图像帧的拍摄时间之间的惯性测量数据。
步骤S12：基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息。
其中，位姿变化信息可以包括第一历史图像帧与目标图像帧之间的相对位移，也可以包括二者之间的相对旋转。
本公开实施例中，位姿变化信息以包括相对位移和相对旋转为例。
步骤S13：基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，至少一个参考因子包括若干惯性测量数据。
在一些实施例中，可以使用第一历史图像帧的位姿信息以及第一历史图像帧与目标图像帧之间的位姿变化信息，确定目标图像帧对应的第一位姿。其中，可以认为目标图像帧的位姿为拍摄目标图像帧时待定位对象的位姿。
然后，使用至少一个参考因子，对目标图像帧对应的第一位姿进行优化，得到优化后的目标图像帧对应的位姿。
当然，若目标图像帧与第一历史图像帧之间包括若干图像帧时，同样可通过至少一个参考因子，对第一位姿进行优化，从而确定中间帧的位姿。这里的中间帧指的是第一历史图像帧与目标图像帧之间的其中一帧。
上述方案，通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
另，一般的视觉惯性定位方法中需要计算预积分，因此需要对惯性传感器的偏置量等参数进行精确的标定，而本公开实施例提供的技术方案只需要对惯性测量数据进行定位处理，无需预积分，因此，无需对惯性传感器的偏置量等参数进行精确的标定，简化了定位流程。
一些公开实施例中，对若干惯性测量数据进行定位处理是由定位模型执行的。其中，获取第一历史图像帧与目标图像帧之间的位姿变化信息的方式可以是：
利用惯性测量数据以及参考运动状态信息，确定本次定位处理得到的最终运动状态信息。然后，基于本次定位处理得到的最终运动状态信息，得到第一历史图像帧与目标图像帧之间的位姿变化信息。该位姿变化信息可以认为是待定位对象在第一历史图像帧的拍摄时间至目标图像帧的拍摄时间之间的位置变化和姿态变化中的至少之一。两个时间点之间的位置变化可以是二者之间的相对位移。姿态变化可以是待定位对象朝向的变化。
其中，参考运动状态信息为历史定位处理得到的最终运动状态信息。例如，本次定位处理为对待定位对象执行的第四次定位处理，则历史定位处理可以是对同一待定位对象执行的第三次定位处理，还可以是第二次和第三次的结合，也可以是本次以前所有次定位处理的结合。其中，本公开实施例选择将上一次定位处理中获取得到的最终运动状态信息，作为本次定位处理过程中的参考运动状态信息。本次定位处理得到的最终运动状态信息能够用于推导待定位对象在第一历史图像帧的拍摄时间点至目标图像帧的拍摄时间点之间的运动。同理，参考运动状态信息能够用于确定历史定位处理中对应第一历史图像帧的拍摄时间点至对应的目标图像帧的拍摄时间点之间的运动。本次定位处理和历史定位处理对应的最终运动状态信息可以包括待定位对象在对应时间段内各个时刻的运动参数。例如，运动参数可以包括待定位对象在各个时刻的加速度和角速度，或待定位对象在各个时刻的运动速度和运动方向等。一些应用场景中，在待定位对应为人类的情况下，最终运动状态信息可以是用于表示基于惯性测量数据得到的加速度和角速度的一个局部窗口，作为人类运动的加速度和角速度，但因为各种因素的作用，其可能带有一定的噪声。
其中，对若干惯性测量数据进行定位处理是由定位模型执行的。在一些实施例中，利用惯性测量数据以及参考运动状态信息，确定本次定位处理得到的最终运动状态信息包括以下步骤：基于惯性测量数据，得到第一历史图像帧至目标图像帧之间的初始运动状态信息，再对参考运动状态信息与初始运动状态信息进行融合，得 到该最终运动状态信息。初始运动状态信息可以包括待定位对象在第一历史图像帧的拍摄时间点至目标图像帧的拍摄时间点之间各个时刻的运动参数。例如，初始运动状态信息可以包括待定位对象在各个时刻的加速度和角速度，或待定位对象在各个时刻的运动速度和运动方向等。在一些实施例中，初始运动状态信息的形式可以是用于描述待定位对象在目标时间段内的运动的隐层状态矩阵。其中，初始运动状态信息可以用于推导待定位对象在目标时间段内的运动。但因为传感器的偏置等因素的影响，导致获取得到的初始运动状态含有噪声，所以若仅使用初始运动状态信息推导待定位对象在目标时间段内的运动，或许会存在一定的误差。
在一些实施例中，最终运动状态信息和参考运动状态信息的形式也可以是用于描述待定位对象在对应时间段内运动的隐层状态矩阵。本公开实施例认为待定位对象的运动是连续且有规律的，因此，通过用于表示待定位对象在对应时间段内的运动的初始运动状态信息与用于表示待定位对象在历史定位处理对应的时间段内运动的参考运动状态信息进行融合，以估计待定位对象在本次定位处理对应时间段内更准确的最终运动状态信息，从而可以根据该最终运动状态信息估计待定位对象在对应时间段内的运动。
在一些实施例中，定位模型包括第一子网络和第二子网络以及第三子网络。其中，第一子网络可以是残差网络，如ResNet18网络。第二子网络可以是长短期记忆网络(Long Short-Term Memory，LSTM)。其中，第一子网络可以用于执行基于惯性测量数据，得到第一历史图像帧至目标图像帧之间的初始运动状态信息的步骤。第二子网络用于执行对参考运动状态信息与初始运动状态信息进行融合，得到该最终运动状态信息的步骤，第三子网络用于执行基于本次定位处理得到的最终运动状态信息，得到第一历史图像帧与目标图像帧之间的位姿变化信息的步骤。通过结合上一次定位处理对应的最终运动状态信息，使得此次定位处理得到的最终运动状态信息更为准确。
本公开实施例中，获取到若干惯性测量数据之后，定位方法还包括以下步骤：
对惯性测量数据进行预处理。其中，经预处理的惯性测量数据用于进行定位处理。预处理包括将惯性测量数据转换到重力系下、去除偏置、去除重力和归一化中的一种或多种。在一些实施例中，预处理还包括对惯性测量数据进行固定帧率插值，并将插值后的数据进行缓存。通过对惯性测量数据进行预处理，使得得到的位姿变化信息更为准确。
一些公开实施例中，定位方法由定位系统执行。在执行步骤S13之前，还包括以下步骤：判断定位系统的参数是否已初始化。其中，参数包括重力方向、惯性偏置量中的至少一者。响应于参数已初始化，则执行步骤S13。响应于参数未初始化，则选择与目标图像帧对应的状态匹配的初始化方式，对定位系统的参数进行初始化，然后再执行上述步骤S13。其中，状态包括运动状态和静止状态。在一些实施例中，选择与目标图像帧对应的状态匹配的初始化方式以前，先确定目标图像帧对应的状态。其中，确定目标图像帧对应的状态的方式可以包括满足以下至少之一：连续若干帧图像中跟踪到的二维特征在图像平面上的平均位移低于第一阈值；惯性测量数据的标准差低于第二阈值，则认为目标图像帧对应的状态为静止状态，否则，认为目标图像帧对应的状态为运动状态。
静止状态对应的初始化方式为静止初始化，而运动状态对应的初始化方式为运动初始化。其中，静止初始化方式包括将初始平移设置为0，初始局部重力为最近两帧图像之间的加速度计测量值的平均值。初始旋转与局部初始重力对齐。初始陀 螺仪偏置是最近两帧图像之间陀螺仪测量值的平均值。初始加速度偏置设置为0。最近两帧指的是目标图像帧以及目标图像帧的前一帧。
运动状态对应的初始化为仅通过视觉定位获取没有尺寸的姿势，然后将惯性测量数据的预积分信息与视觉定位的结果进行对齐以恢复尺寸、速度、重力和惯性偏置量。
通过在定位系统未经初始化的情况下，选择与目标图像帧对应的状态匹配的初始化方式，对定位系统的参数进行初始化，使得初始化得到的参数更准确。
一些公开实施例中，请同时参见图2，图2是本公开实施例提供的一种定位方法中步骤S13的流程示意图。如图2所示，上述步骤S13包括以下步骤：
步骤S131：基于位姿变化信息，确定目标图像帧的第一位姿。
其中，目标图像帧的第一位姿表示待定位对象在目标图像帧的拍摄时刻的位姿。本公开实施例中，目标图像和第一历史图像帧是由待定位对象相对固定的拍摄装置拍摄得到，惯性测量数据是由与待定位对象相对固定的惯性测量装置测量得到的。通过与待定位对象相对固定的拍摄装置拍摄得到目标图像帧以及与待定位对象相对固定的惯性测量装置得到惯性测量数据，可以实现对任意的对象进行定位。
在一些实施例中，位姿变化信息包括至少一种位姿变化量。其中，至少一种位姿变化量包括位置变化量以及姿态变化量。获取目标图像帧的第一位姿的方式可以是利用目标图像帧对应的位姿变化量，确定目标图像帧的第一位姿。在一些实施例中，第一历史图像帧的位姿已知，基于第一历史图像帧与目标图像帧之间的位姿变化量，即可得到目标图像帧的第一位姿。通过确定位置变化量以及姿态变化量，使得确定的第一位姿更准确。
步骤S132：利用位姿变化信息以及至少一个参考因子，构建本次定位对应的总能量关系。
其中，本次定位对应的总能量关系用于确定目标图像帧的需优化的位姿偏差。
在一些实施例中，分别利用每个参考因子，确定参考因子对应的测量能量关系。以及，利用位姿变化信息确定运动先验能量关系。最后，基于运动先验能量关系以及每个参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
一些公开实施例中，获取若干惯性测量数据的预积分信息。然后利用若干惯性测量数据的预积分信息，构建惯性测量能量关系。
其中，基于惯性测量数据的预积分信息，构建定惯性测量能量关系的方式可参见一般公知的技术。
此处简单列举获取惯性测量能量关系c u的方式：
Figure PCTCN2021143905-appb-000001
其中，H k是上一次定位过程对应的雅可比矩阵，
Figure PCTCN2021143905-appb-000002
是本次定位关于位姿的雅可比矩阵，
Figure PCTCN2021143905-appb-000003
是本次定位关于惯性偏置量的雅可比矩阵，
Figure PCTCN2021143905-appb-000004
是上一次定位对应的位姿偏差，
Figure PCTCN2021143905-appb-000005
是本次定位对应的位姿偏差，
Figure PCTCN2021143905-appb-000006
为本次定位对应的惯性偏置量对应的偏差。∑ u是惯性测量能量关系对应的协方差矩阵，r uk+1是惯性传感器测量残差。其中，获取上述各参数的方式可参见一般公知的技术。
其中，获取运动先验能量关系的方式可以是：
利用至少一次定位对应的若干参数，确定运动先验能量关系。至少一次定位为多次定位，多次为两次及以上。其中，至少一次定位包括本次定位，每次定位对应 的若干参数包括定位过程中得到的位姿变化量、定位对应的起始测量时刻与结束测量时刻之间的位置差以及起始测量时刻对应的起始姿态信息。其中，起始测量时刻与结束测量时刻之间的位置差指的是待定位对象在起始测量时刻的位置与待定位对象在结束测量时刻之间的位置之差。起始测量时刻的起始姿态信息指的是待定位对象在起始测量时刻的朝向。
通过获取历史定位的位姿变化量以及若干惯性测量数据的测量期间的位置差，确定运动先验能量关系，使得构建的运动先验能量关系更准确，进而得到优化的位姿更准确。
在一些实施例中，起始图像帧的起始姿态信息包括偏航角信息。一般惯性测量数据并非以偏航角的形式呈现，通过将惯性测量数据转换为四元数形式。其中，若以XYZ坐标轴的方式转换得到的偏航角为90°，则调整坐标系为YXZ坐标轴的方式重新获取偏航角信息。通过使用偏航角信息构建运动先验能量，使得构建得到的运动先验能量关系更准确。
在一些实施例中，位姿变化信息还包括位姿变化量的确定度表征信息。其中，确定度表征信息可以用于表示确定度，也可用于表示不确定度。基于每次定位过程中得到的确定度表征信息，得到对应次定位的权重。其中，定位的权重是基于定位对应的确定度表征信息的预设倍数确定得到，其中，预设倍数为自然数。在一些实施例中，该权重可以为基于预设倍数的确定度表征信息变换得到的协方差矩阵。在一些实施例中，在确定度表征信息用于表示确定度时，预设倍数一般小于或等于1，例如预设倍数可以是0.1等，当然，这仅为举例，在其他实施例中，预设倍数还可以大于1。在确定度表征信息用于表示不确定度时，预设倍数一般大于或等于1，例如预设倍数可以是10等，当然，这仅为举例，在其他实施例中，预设倍数还可以小于1。
利用至少一次定位的权重以及若干参数，确定运动先验能量关系。在一些实施例中，可以先确定各次定位的运动先验能量关系，再基于各次定位的权重，将各次定位的运动先验能量关系进行结合，得到最终的运动先验能量关系。通过确定度表征信息，得到对应的权重，使得构建的运动先验能量关系更准确。
在一些实施例中，获取运动先验能量关系c n的方式可以是：
Figure PCTCN2021143905-appb-000007
其中，每次定位对应的若干惯性测量数据的起始测量时刻为i时刻、结束测量时刻为j时刻，p i和p j分别表示第i时刻和第j时刻对应的待定位对象的位置、d ij表示定位模型输出的第i时刻和第j时刻之间的位姿变化量。
Figure PCTCN2021143905-appb-000008
表示第i时刻对应的目标对象的偏航角矩阵。T表示转置。∑ ij表示定位模型对应的协方差矩阵(权重)，该协方差矩阵由确定度表征信息获取得到。在一些实施例中，确定度表征信息为三维向量，三维向量中的三个元素为该协方差矩阵的对角元素的对数。在一些实施例中，可以将确定度表征信息的预设倍数作为该协方差矩阵的对角元素的对数。其中，此处确定度表征信息用于表示不确定度，该确定度表征信息可以是原始的确定度表征信息，也可以是扩大预设倍数后的确定度表征信息。
在一些实施例中，每次定位对应的第一历史图像帧与目标图像帧中，拍摄最早的图像帧为起始图像帧，拍摄最晚的图像帧为结束图像帧。本公开实施例中，认为第一历史图像帧即为起始图像帧，目标图像帧即为结束图像帧。以及，位姿变化信 息包括至少一种位姿变化量。至少一种位姿变化量包括位置变化量和姿态变化量中的至少之一。
为减少时间戳误差导致定位精度下降的问题出现，获取更为准确的运动先验能量关系。本公开实施例还可提供以下方式，让运动先验能量关联上优化状态，使最小能量时，能取得优化状态的最优值。实施时，在获取运动先验能量关系之前，还包括以下步骤：将至少一次定位中的每次定位作为目标定位，基于目标定位对应的若干惯性测量数据的预积分信息、起始图像帧的位姿和结束图像帧的位姿，确定目标定位对应的起始姿态信息、起始测量时刻的位置和结束测量时刻的位置。在一些实施例中，若目标定位为历史定位，则目标定位中起始图像帧的位姿和结束图像帧的位姿均可以是优化前的第一位姿，也可以是经优化后的位姿。而目标定位为本次定位时，起始图像帧的位姿和结束图像帧的位姿为优化前的第一位姿。即，若目标定位为上一次定位，则上一次定位中起始姿态信息、起始测量时刻的位置和结束测量时刻的位置可以由上一次定位过程中的若干惯性测量数据的预积分信息、起始图像帧的位姿和结束图像帧的位姿确定。当然，在另一些公开实施例中，可以认为待定位对象在起始测量时刻的位姿与拍摄起始图像帧时的位姿相同，结束测量时刻的位姿与拍摄结束图像帧时的位姿相同。
在一些实施例中，通过预积分信息以及起始图像帧的位姿、结束图像帧的位姿，确定目标定位对应的起始姿态信息、起始测量时刻和结束测量时刻的位置的方式可以是：
Figure PCTCN2021143905-appb-000009
Figure PCTCN2021143905-appb-000010
Figure PCTCN2021143905-appb-000011
其中，m表示起始图像帧，n表示结束图像帧。i表示起始测量时刻，j表示结束测量时刻。p Ii表示起始测量时刻的位置，w表示世界坐标系，例如 Wp Ii表示起始测量时刻在世界坐标系下的位置，其他同理。p Ij表示结束测量时刻的位置，R Ii表示起始姿态信息。p Im表示起始图像帧的位置，v Im表示起始图像帧的速度，Δt m表示起始图像帧的拍摄时间与起始测量时刻之间的时间间隔。R Im表示起始图像帧对应的姿态信息，
Figure PCTCN2021143905-appb-000012
表示不考虑初速度和重力情况下，使用起始图像帧和起始测量时刻之间的惯性测量数据进行预积分得到的位置。p Ij表示结束测量时刻的位置，
Figure PCTCN2021143905-appb-000013
表示起始图像帧和起始测量时刻之间的惯性测量数据进行预积分得到的姿态，上述三个公式中其他相同符号的不同上下标，可参见上述解析。通过时间戳的方式从若干图像帧中选择拍摄时间与i时刻最接近图像帧作为起始图像帧，与j时刻最接近的图像帧作为结束图像帧。
由此，得到的最终的运动先验能量关系
Figure PCTCN2021143905-appb-000014
Figure PCTCN2021143905-appb-000015
其中，
Figure PCTCN2021143905-appb-000016
表示第k+1次定位中的位姿偏差。H表示对应的雅可比矩阵，r表示对应的测量残差。∑ n为最终的运动先验能量关系对应的协方差矩阵。该协方差矩阵可以由定位模型输出的确定度表征信息以及预积分信息获取得到。下标的含义参见上述。
获取的方式 ∑n可以是：
Figure PCTCN2021143905-appb-000017
其中，∑ ij是基于定位模型输出的确定度表征信息的预设倍数确定的协方差矩阵。
Figure PCTCN2021143905-appb-000018
和
Figure PCTCN2021143905-appb-000019
分别是j、i时刻预积分对应的协方差矩阵的位姿部分。w表示世界坐标系，T表示转置。下标的含义参见上述。
其中，在计算运动先验能量关系中若使用了预积分信息，则可以认为最终得到的运动先验能量关系包含了惯性测量能量关系，因此，可以不必再使用惯性测量能量关系，构建总能量关系。通过预积分信息以及相关图像帧的位姿，确定若干惯性测量数据的起始测量时刻和结束测量时刻的位置，实现对起始测量时刻和结束测量时刻的位置的校正，进而提高运动先验能量关系的准确性。
一些公开实施例中，从多次定位中，剔除满足去除条件的定位。其中，去除条件为定位对应的位姿变化量以及位姿变化量的确定度表征信息之间的预设处理结果满足第一预设条件。然后利用经剔除后的剩余次定位对应的若干参数，确定运动先验能量关系。在一些实施例中，通过获取位姿变化量对应的马氏距离来去除异常定位。其中，去除条件可以是
Figure PCTCN2021143905-appb-000020
大于阈值。d表示滤波器初步估计的位姿变化量，
Figure PCTCN2021143905-appb-000021
表示定位模型输出的位姿变化量。其中，H，P表示运动先验能量关系中的测量雅可比矩阵和相应的状态协方差。
Figure PCTCN2021143905-appb-000022
表示定位系统的状态协方差矩阵，其中，该矩阵由运动先验能量关系以及视觉测量能量关系对应的协方差矩阵融合得到。当然，除了将其剔除，还可采用将运动先验能量关系对应的协方差矩阵放大若干倍用于减轻测量数据不准确的情况，例如放大十倍。通过对参数进行筛选，以剔除异常参数，进而使得获取到的运动先验能量关系更为准确。
其中，基于惯性测量能量关系和运动先验能量关系构建的总能量关系可以是：
Figure PCTCN2021143905-appb-000023
其中，这里的
Figure PCTCN2021143905-appb-000024
为k+1次定位确定的位姿偏差。H u为惯性测量能量关系中的雅可比矩阵，r u为惯性测量能量关系中的测量残差，H n为运动先验能量关系中的雅可比矩阵，r n为运动先验能量关系中的测量残差。∑ n为运动先验能量关系对应的协方差矩阵，∑ z为惯性测量能量关系对应的协方差矩阵，可以认为协方差可以是对应关系中的权重。
在一些实施例中，若运动先验能量关系的获取结合了预积分信息，则该总能量关系中可以不包括惯性测量能量关系的相关参数。
一些公开实施例中，至少一个参考因子还包括关于目标图像帧和第一历史图像帧的图像信息。在一些实施例中，目标图像帧和第一历史图像帧之间包括多帧图像时，则还可包括目标图像帧和第一历史图像帧之间各图像帧的图像信息。除基于位姿变化信息、惯性测量数据以外，还结合图像信息得到目标图像帧的位姿，故能够提高定位精度。
在一些实施例中，分别利用每个参考因子，确定参考因子对应的测量能量关系的方式，包括：利用图像信息，确定图像信息对应的视觉测量能量关系。在一些实施例中，利用图像信息，确定图像信息对应的视觉测量能量关系之前，还包括以下步骤：对若干参考图像帧的特征点进行匹配，得到特征点匹配结果。在一些实施例中，匹配特征点可以利用稀疏光流法将上一帧图像特征点跟踪到当前图像帧特征点，得到特征点的匹配结果。在一些实施例中，利用极线几何约束去除错误的匹配结果。其中，若干参考图像包括至少一次定位过程中的第一历史图像帧和目标图像帧，且至少一次定位包括本次定位。其中，特征点匹配结果包括特征点集合。若一特征点同时存在与两帧参考图像帧中，则该特征点会加入特征点集合。当然，在其他实施例中，只有在特征点同时存在于三帧及以上的参考图像帧中，才将其加入特征点集合。其中，该特征点为二维特征点。
基于特征点匹配结果，从若干参考图像帧中确定至少一对匹配图像帧。其中，每对匹配图像帧存在匹配特征点对。利用每对匹配图像帧的位姿以及匹配特征点对在匹配图像帧中的位置，确定视觉测量能量关系。其中，若匹配图像帧不为历史定位中的目标图像帧，即匹配图像帧为本次定位中第一历史图像帧与目标图像帧之间的图像帧或目标图像帧，则匹配图像帧的位姿为第一位姿，若匹配图像帧为历史定位中的目标图像帧，则匹配图像帧的位姿可以是优化前的第一位姿或优化后的位姿。通过二维点信息构建视觉测量能量关系，而非通过三维点信息构建视觉测量能量关系，减少了因为三维点的精度问题造成视觉测量能量关系不准确的情况出现，从而使得获取得到的视觉测量能量关系更准确。
在一些实施例中，获取视觉能量关系的方式为：
Figure PCTCN2021143905-appb-000025
其中，F为各参考图像帧能跟踪上的特征点集合，其中，若一个特征点被两帧及以上的参考图像帧观测到，该特征点即可加入特征点集合。C为能跟踪到二维特征点集合F的相机状态集合。
Figure PCTCN2021143905-appb-000026
为第k个特征点在第j帧参考图像帧上的二维位置。K是拍摄组件的投影矩阵，R i表示第i帧参考图像帧对应的摄像组件的旋转矩阵。T表示转置。∑ Z为对应的协方差矩阵。本式中提出的i帧为在先拍摄的参考图像帧，j帧为在后拍摄的参考图像帧。
在一些实施例中，在获取总能量关系之前，还可包括以下步骤：基于历史定位对应的总能量关系，确定优化先验能量关系。该优化先验能量关系可用于构建总能量关系。通过结合优化先验能量关系构建总能量关系，使得确定的位姿的偏差更准确。
在一些实施例中，获取优化先验能量关系的方式可以是：利用第二历史图像帧的位姿、第二历史图像帧之前的第二数量图像帧对应的位姿、目标图像帧的位姿以及第二历史图像帧对应的惯性信息，更新得到第二历史图像帧对应的新的位姿偏差。其中，第二历史图像帧为历史定位中的目标图像帧。其中，目标图像帧对应的位姿偏差至少由目标图像帧的位姿以及目标图像帧之前的第一数量图像帧对应的位姿以 及目标图像帧对应的惯性信息确定。其中，第二数量小于第一数量。其中，第一数量图像帧可以是前若干次定位过程中对应的目标图像帧。其中，第一数量图像帧的位姿可以是经历史定位过程优化后的位姿，还可以是优化前的第一位姿。
在一些实施例中，本次定位所使用的第一历史图像帧与上一次定位所使用的第一历史图像帧并非同一帧。例如，若第一次执行定位，第一历史图像帧可以是拍摄的首帧，该首帧作为世界坐标系的原点，目标图像帧为拍摄的第3帧，因第一次执行定位后，第2帧和第3帧的位姿已知，第二次执行定位使用的第一历史图像帧可以是第2帧也可以是第3帧，若第二次执行定位使用的第一历史图像帧为第2帧，则目标图像帧为第4帧，同理若第二次执行定位所使用的第一历史图像帧为第3帧，则目标图像帧可以是第5帧。对于第二次执行定位而言，第二历史图像帧为第3帧。当然，在此次定位过程中，所使用的第二历史图像帧的位姿可以是经上一次定位优化后的位姿，另一些公开实施例还可以是经优化前的位姿。
然后将历史定位对应的总能量关系中的位姿偏差替换为新的位姿偏差，得到优化先验能量关系。其中，目标图像帧对应的惯性信息可以是惯性偏置。其中，惯性偏置可以包括加速度偏置以及角速度偏置等。通过利用目标图像帧的位姿替换第二历史图像帧之前第一数量图像帧的最早图像帧的位姿，以更新第二历史图像帧对应的位姿偏差，进而使得确定得到的优化先验能量关系关联于目标图像帧的位姿，进而使得利用能量关系确定的目标图像帧的位姿偏差更准确。
在一些实施例中，获取优化先验能量关系
Figure PCTCN2021143905-appb-000027
的方式为：
Figure PCTCN2021143905-appb-000028
其中，H k+1为k+1次定位对应的雅可比矩阵。
Figure PCTCN2021143905-appb-000029
为目标图像帧对应的位姿偏差，r k+1为对应的测量残差。
然后，基于运动先验能量关系、视觉测量能量关系以及优化先验能量关系、惯性测量能量关系中的一者或多者，构建本次定位对应的总能量关系。
一些公开实施例中，基于运动先验能量关系以及惯性测量能量关系构建本次定位对应的总能量关系，另一些公开实施例中，基于运动先验能量关系以及视觉测量能量关系构建本次定位对应的总能量关系，或者基于运动先验能量关系、视觉测量能量关系以及视觉测量能量关系构建本次定位对应的总能量关系，还可以是基于运动先验能量关系、视觉测量能量关系以及优化先验能量关系、惯性测量能量关系构建本次定位对应的总能量关系。
步骤S133：利用本次定位对应的总能量关系，对第一位姿进行优化，得到目标图像帧的位姿。
其中，总能量关系表示位姿偏差与总能量之间的关系。在一些实施例中，利用本次定位对应的总能量关系，对第一位姿进行优化，得到目标图像帧的位姿的方式可以是：利用本次定位对应的总能量关系，确定总能量满足第二预设条件的位姿偏差。其中，第二预设条件可以是总能量最小。然后基于确定的位姿偏差对目标图像帧的第一位姿进行优化。在一些实施例中，将获取到的位姿偏差与目标图像帧的第一位姿进行求和，得到目标图像帧优化后的位姿。
从而，最小化总能量关系，更新各个图像帧对应的状态以及惯性传感器的偏置。
在一些实施例中，最小化总能量关系的方式为：
Figure PCTCN2021143905-appb-000030
其中，这里的
Figure PCTCN2021143905-appb-000031
为k+1次定位确定的位姿偏差。H k+1为优化先验能量关系中的雅可比矩阵，r k+1为优化先验能量关系中的测量残差。H z为视觉测量能量关系中的雅可比矩阵，r z为视觉测量能量关系中的测量残差，H n为运动先验能量关系中的雅可比矩阵，r n为运动先验能量关系中的测量残差。∑ n为运动先验能量关系对应的协方差矩阵，∑ z为视觉测量能量关系对应的协方差矩阵。
通过上述公式，能够求解得到最终的
Figure PCTCN2021143905-appb-000032
与第一位姿进行求和，得到优化后的位姿。在一些实施例中，求取得到的惯性传感器的偏置可以取代定位系统中原始的偏置，以便后续的位姿计算更准确。当然，若需要优化的还包括惯性传感器的偏置、重力方向等参数时，上述
Figure PCTCN2021143905-appb-000033
则还可表示包括所有待优化的参数对应的偏差，也就是在这种情况下，
Figure PCTCN2021143905-appb-000034
包括位姿偏差以及各偏置量等参数对应的偏差的总偏差。
另一些公开实施例中，在确定目标图像帧的位姿后，还可基于目标图像帧的位姿，对定位系统的参数进行优化。在一些实施例中，通过调整包含位姿偏差和参数的偏差的总偏差，得到本次定位过程中需要优化的参数偏差，再将获取到的偏差与对应的参数进行相加，得到优化后的参数。最后使用该优化后的参数替换定位系统中的参数。其中，参数包括重力方向、惯性偏置量中的至少一者。下一次定位过程中，可以使用优化后的参数进行定位。通过对定位系统的参数进行优化，使得下一次定位的精度更高。
为更好地理解本公开实施例提供的技术方案，请参考下例。请同时参见图3，图3是本公开实施例提供的另一种定位方法的流程示意图。如图3所示，本公开实施例提供的定位方法包括如下步骤：
步骤S21：摄像组件获取若干图像帧。
其中，若干图像帧指的是第一历史图像帧至目标图像帧之间的图像帧。
步骤S22：进行特征提取以及跟踪。
在一些实施例中，对若干图像帧进行特征提取以及跟踪，得到若干图像帧对应的图像信息。
步骤S23：惯性传感器获取若干惯性测量数据。
其中，若干惯性测量数据指的是第一历史图像帧的拍摄时间至目标图像帧的拍摄时间之间的拍摄期间的惯性测量数据。
惯性测量数据的获取方式如上述。
步骤S24：进行数据缓存。
在一些实施例中，在数据缓存中执行将惯性测量数据转换到重力系下、去除偏置、去除重力和归一化中的一种或多种。
步骤S25：输入定位模型。
在一些实施例中，将若干惯性测量数据输入定位模型。定位模型基于若干惯性测量数据进行定位处理得到本次定位处理的位姿变化信息。其中，定位模型对若干惯性测量数据进行定位处理的方式如上述。
步骤S26：进行预积分。
这里，对惯性测量数据进行预积分的方式可参见一般的公知技术。
步骤S27：判断定位系统是否初始化。
这里，所述判断方式如上述。
若判断结果为定位系统未进行初始化，则执行步骤S28，否则执行步骤S29。
步骤S28：进行静止初始化/运动初始化。
这里，进行静止初始化/运动初始化的方式如上述。
步骤S29：进行异常点检测。
其中，异常点检测就是上文中去除满足第一预设条件的数据的步骤。
步骤S30：进行优化和更新。
在一些实施例中，基于图像信息、预积分信息以及位姿变化信息构建总能量关系并对各图像帧对应的位姿进行优化和更新，得到最终的目标图像帧的位姿。
步骤S31：优化参数。
在一些实施例中，可以进行优化的定位系统参数可以包括加速度偏置Ba、角速度偏置Bg以及定位系统其他可以优化的参数R。
其中，上述步骤之间的关系并非严格按照上述顺序执行，例如，步骤S21和步骤S23可同步执行。步骤S22和步骤S24以及步骤S26也可同步执行等。
在一些实施例中，定位系统的输入可以包括摄像组件和惯性传感器的输出，即第一历史图像至目标图像帧之间的所有图像帧以及若干惯性测量数据。分别对各图像帧和惯性测量数据进行预处理。对图像帧进行预处理的方式可以是特征提取和跟踪。对惯性测量数据进行预处理也可分为两步，一是进行预积分，二是在数据缓存中执行将惯性测量数据转换到重力系下、去除偏置、去除重力和归一化中的一种或多种。将经过第二种预处理后的惯性测量数据输入定位模型得到第一历史图像帧和目标图像帧之间的位姿变化信息。然后，在进行预积分和图像预处理之后，判断定位系统是否进行了初始化，若是，则进行异常点检测，其中，异常点检测就是上文中去除满足第一预设条件的数据的步骤。然后基于图像信息、预积分信息以及位姿变化信息构建总能量关系并对各图像帧对应的位姿进行优化和更新，否则进行系统初始化。在对位姿进行优化更新之后，对定位系统的参数进行优化。当然，在其他实施例中，还可根据历史的总能量关系构建此次定位过程中的总能量关系。
通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位，即可获取这两帧图像之间的位姿变化信息，若第一历史图像帧为世界坐标系的原点或其位姿已知，则可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程减少了视觉方面对定位的影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
在一些实施例中，将定位模型对应的运动先验(位姿变化信息)、IMU信息和视觉信息进行紧耦合，在正常视觉环境下，能取得高精度定位效果，在极端挑战视觉环境下，由于运动先验的鲁棒性，能取得较为鲁棒的跟踪。
在一些实施例中，本公开实施例提供的技术方案可以与其他定位算法或传感器进行耦合，进行定位导航。
本公开实施例提供的定位方法可应用于增强现实、虚拟现实、机器人、自动驾驶、游戏、影视、教育、电子商务、旅游、智慧医疗、室内装修设备、智慧家居、智能制造以及维修装配等场景中。
其中，定位方法的执行主体可以是定位装置，例如，定位方法可以由终端设备或服务器或其它处理设备执行，其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些可能的实现方式中，该定位方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
请参阅图4，图4是本公开实施例提供的一种定位装置的结构示意图。定位装置40包括数据获取模块41、定位处理模块42以及位姿确定模块43。数据获取模块41，配置为获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据；定位处理模块42，配置为基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息；位姿确定模块43，配置为基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，至少一个参考因子包括若干惯性测量数据。
上述方案，通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，故减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
一些公开实施例中，至少一个参考因子还包括关于目标图像帧和第一历史图像帧的图像信息。
上述方案，除基于位姿变化信息、惯性测量数据以外，还结合图像信息得到目标图像帧的位姿，故能够提高定位精度。
一些公开实施例中，位姿确定模块43在基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿的情况下，配置为：基于位姿变化信息，确定目标图像帧的第一位姿；以及，利用位姿变化信息以及至少一个参考因子，构建本次定位对应的总能量关系，其中，本次定位对应的总能量关系用于确定目标图像帧的需优化的位姿偏差；利用本次定位对应的总能量关系，对第一位姿进行优化，得到目标图像帧的位姿。
上述方案，通过利用位姿变化信息以及至少一个参考因子构建总能量关系，通过总能量关系确定位姿偏差，从而对目标图像帧的第一位姿进行优化。
一些公开实施例中，位姿确定模块43在利用位姿变化信息以及至少一个参考因子，构建本次定位对应的总能量关系的情况下，配置为：分别利用每个参考因子，确定参考因子对应的测量能量关系，以及，利用位姿变化信息确定运动先验能量关系；基于运动先验能量关系以及每个参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
上述方案，对每一参考因子都构建对应的测量能量关系，并结合运动先验能量关系，使得获取到的总能量关系确定的位姿偏差更准确。
一些公开实施例中，位姿确定模块43在分别利用每个参考因子，确定参考因子对应的测量能量关系的情况下，配置为：获取若干惯性测量数据的预积分信息；利用若干惯性测量数据的预积分信息，构建惯性测量能量关系。
上述方案，通过将预积分信息构建的惯性测量能量关系与运动先验能量关系进行结合，相比于单一的运动先验能量关系而言，确定的位姿偏差更准确。
一些公开实施例中，位姿变化信息包括至少一种位姿变化量，至少一种位姿变化量包括位置变化量和姿态变化量中的至少之一；位姿确定模块43在利用位姿变化信息确定运动先验能量关系的情况下，配合为：利用至少一次定位对应的若干参数，确定运动先验能量关系，其中，至少一次定位包括本次定位，每次定位对应的若干参数包括定位过程中得到的位姿变化量、定位过程中采用的若干惯性测量数据的起始测量时刻和结束测量时刻的位置差、以及起始测量时刻对应的起始姿态信息。
上述方案，通过获取历史定位的位姿变化量以及若干惯性测量数据的测量期间的位置差，确定运动先验能量关系，使得构建的运动先验能量关系更准确，进而得到优化的位姿更准确。
一些公开实施例中，位姿变化信息还包括位姿变化量的确定度表征信息；位姿确定模块43在利用至少一次定位对应的若干参数，确定运动先验能量关系的情况下，配置为：基于每次定位过程中得到的确定度表征信息，得到对应次定位的权重；利用至少一次定位的权重以及若干参数，确定运动先验能量关系。
一些公开实施例中，至少一次定位为多次定位，位姿确定模块43在利用至少一次定位对应的若干参数，确定运动先验能量关系的情况下，配置为：从多次定位中，剔除满足去除条件的定位；其中，去除条件为定位对应的位姿变化量以及位姿变化量的确定度表征信息之间的预设处理结果满足第一预设条件；利用经剔除后的剩余次定位对应的若干参数，确定运动先验能量关系。
上述方案，通过确定度表征信息，得到对应的权重，使得构建的运动先验能量关系更准确。另外，通过对参数进行筛选，以剔除异常参数，进而使得获取到的运动先验能量关系更为准确。
一些公开实施例中，起始姿态信息包括偏航角信息。
一些公开实施例中，每次定位对应的第一历史图像帧与目标图像帧中，拍摄最早的图像帧为起始图像帧，拍摄最晚的图像帧为结束图像帧；在利用至少一次定位对应的若干参数，确定运动先验能量关系之前，位姿确定模块43还配置为：将至少一次定位中的每次定位作为目标定位，基于目标定位对应的若干惯性测量数据的预积分信息、起始图像帧的位姿和结束图像帧的位姿，确定目标定位对应的起始姿态信息、起始测量时刻的位置和结束测量时刻的位置。
上述方案，通过使用偏航角信息构建运动先验能量，使得构建得到的运动先验能量关系更准确。另外，通过预积分信息以及相关图像帧的位姿，确定若干惯性测量数据的起始测量时刻和结束测量时刻的位置，实现对起始测量时刻和结束测量时刻的位置的校正，进而提高运动先验能量关系的准确性。
一些公开实施例中，参考因子还包括关于目标图像帧和第一历史图像帧的图像信息，位姿确定模块43在分别利用每个参考因子，确定参考因子对应的测量能量关系的情况下，配置为：利用图像信息，确定图像信息对应的视觉测量能量关系。在利用图像信息，确定图像信息对应的视觉测量能量关系之前，位姿确定模块43还用于：对若干参考图像帧的特征点进行匹配，得到特征点匹配结果，其中，若干参考图像包括至少一次定位过程中的第一历史图像帧和目标图像帧，且至少一次定位包括本次定位；利用图像信息，确定图像信息对应的视觉测量能量关系，包括：基于特征点匹配结果，从若干参考图像帧中确定至少一对匹配图像帧，其中，每对匹配图像帧存在匹配特征点对；利用每对匹配图像帧的位姿以及匹配特征点对在匹配图像帧中的位置，确定视觉测量能量关系。
上述方案，通过二维点信息构建视觉测量能量关系，而非通过三位点信息构建视觉测量能量关系，减少了因为三维点的精度问题造成视觉测量能量关系不准确的情况出现，从而使得获取得到的视觉测量能量关系更准确。
一些公开实施例中，在基于运动先验能量关系以及每个参考因子对应的测量能量关系，构建本次定位对应的总能量关系之前，位姿确定模块43还配置为：基于历史定位对应的总能量关系，确定优化先验能量关系；基于运动先验能量关系以及每个参考因子对应的测量能量关系，构建本次定位对应的总能量关系，包括：基于运动先验能量关系、优化先验能量关系以及每个参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
上述方案，通过结合优化先验能量关系构建总能量关系，使得确定的位姿偏差更准确。
一些公开实施例中，历史定位对应的总能量关系为上一次定位对应的总能量关系。
一些公开实施例中，目标图像帧对应的位姿偏差至少由目标图像帧的位姿以及目标图像帧之前的第一数量图像帧对应的位姿以及目标图像帧对应的惯性信息确定；位姿确定模块43在基于历史定位对应的总能量关系，确定优化先验能量关系的情况下，配置为：利用第二历史图像帧的位姿、第二历史图像帧之前的第二数量图像帧对应的位姿、目标图像帧的位姿、以及第二历史图像帧对应的惯性信息，更新得到第二历史图像帧对应的新的位姿偏差，其中，第二历史图像帧为历史定位中的目标图像帧，第二数量小于第一数量；将历史定位对应的总能量关系中的位姿偏差替换为新的位姿偏差，得到优化先验能量关系。
上述方案，通过利用目标图像帧的位姿替换第二历史图像帧之前第一数量图像帧的最早图像帧的位置，以更新第二历史图像帧对应的位姿偏差，进而使得确定得到的优化先验能量关系关联于目标图像帧的位姿，进而使得利用能量关系确定的目标图像帧的位姿偏差更准确。
一些公开实施例中，总能量关系表示位姿偏差与总能量之间的关系；位姿确定模块43在利用本次定位对应的总能量关系，对第一位姿进行优化，得到目标图像帧的位姿的情况下，配置为：利用本次定位对应的总能量关系，确定使总能量满足第二预设条件的位姿偏差；基于确定的位姿偏差对目标图像帧的第一位姿进行优化。
一些公开实施例中，位姿变化信息包括至少一种位姿变化量；位姿确定模块43在基于位姿变化信息，确定目标图像帧的第一位姿的情况下，配置为：利用目标图像帧对应的位姿变化量，确定目标图像帧的位姿。
上述方案，通过使得总能量关系满足第二预设条件的位姿偏差，并基于该位姿偏差对目标图像帧的第一位姿进行优化，使得最终目标图像帧的位姿更准确。另外，通过位姿变化量，即可确定目标图像帧的位姿，整个过程方便快捷。
一些公开实施例中，对若干惯性测量数据进行定位处理是由定位模型执行。
一些公开实施例中，定位处理模块42在基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息的情况下，配置为：利用惯性测量数据以及参考运动状态信息，确定待定位对象在本次定位处理得到的最终运动状态信息，其中，参考运动状态信息为上一次定位处理对应的最终运动状态信息；基于本次定位处理得到的最终运动状态信息，得到第一历史图像帧与目标图像帧之间的位姿变化信息。
上述方案，通过结合上一次定位处理得到的最终运动状态信息，使得此次定位处理得到的最终运动状态信息更为准确。
一些公开实施例中，定位方法由定位系统执行，在基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿之前，位姿确定模块43还配置为：判断定位系统的参数是否已初始化，其中，参数包括重力方向、惯性偏置量中的至少一者；响应于参数已初始化，则执行基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿；响应于参数未初始化，则选择与目标图像帧对应的状态匹配的初始化方式，对定位系统的参数进行初始化，再执行基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，状态包括运动状态和静止状态。
上述方案，通过在定位系统未经初始化的情况下，选择与目标图像帧对应的状态匹配的初始化方式，对定位系统的参数进行初始化，使得初始化得到的参数更准确。
一些公开实施例中，定位方法由定位系统执行，在基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿之后，位姿确定模块43还配置为：基于目标图像帧的位姿，对定位系统的参数进行优化，其中，参数包括重力方向、惯性偏置量中的至少一者。
上述方案，通过对定位系统的参数进行优化，使得下一次定位的精度更高。
一些公开实施例中，目标图像帧的位姿表示待定位对象在目标图像帧的拍摄时刻的位姿，目标图像帧和第一历史图像帧是由与待定位对象相对固定的拍摄装置拍摄得到，惯性测量数据是由与待定位对象相对固定的惯性测量装置测量得到。
一些公开实施例中，获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据之后，数据获取模块41还配置为：对惯性测量数据进行预处理，其中，经预处理的惯性测量数据用于进行定位处理，预处理包括将惯性测量数据转换到重力系下、去除偏置、去除重力和归一化中的一种或多种。
上述方案，通过由与待定位对象相对固定的拍摄装置拍摄得到目标图像帧以及与待定位对象相对固定的惯性测量装置得到惯性测量数据，因此，可以对任意的对象进行定位。另，通过对惯性测量数据进行预处理，使得得到的位姿变化信息更为准确。
上述方案，通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，故减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
请参阅图5，图5是本公开实施例提供的一种电子设备的结构示意图。电子设备50包括存储器51和处理器52，处理器52用于执行存储器51中存储的程序指令，以实现上述任一定位方法实施例中的步骤。在一个实施场景中，电子设备50可以包括但不限于：医疗设备、微型计算机、台式电脑、服务器，此外，电子设备50还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。
在一些实施例中，处理器52用于控制其自身以及存储器51以实现上述任一定位方法实施例中的步骤。处理器52还可以称为中央处理单元(Central Processing Unit，CPU)。处理器52可以是一种集成电路芯片，具有信号的处理能力。处理器52还可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶 体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器52可以由集成电路芯片共同实现。
上述方案，通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，故减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
请参阅图6，图6是本公开实施例提供的一种计算机可读存储介质的结构示意图。计算机可读存储介质60存储有能够被处理器运行的程序指令61，程序指令61用于实现上述任一定位方法实施例中的步骤。
本公开实施例还提供一种计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令被计算机设备的处理器运行时，实现上述方法的步骤，可参见上述定位方法实施例。
该计算机程序产品可以通过硬件、软件或其结合的方式实现。在一个可选实施例中，所述计算机程序产品体现为计算机存储介质，在另一个可选实施例中，计算机程序产品体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
上述方案，通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，故减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。
本公开涉及增强现实领域，通过获取现实环境中的目标对象的图像信息，进而借助各类视觉相关算法实现对目标对象的相关特征、状态及属性进行检测或识别处理，从而得到与具体应用匹配的虚拟与现实相结合的AR效果。在一些实施例中，目标对象可涉及与人体相关的脸部、肢体、手势、动作等，或者与物体相关的标识物、标志物，或者与场馆或场所相关的沙盘、展示区域或展示物品等。视觉相关算法可涉及视觉定位、SLAM、三维重建、图像注册、背景分割、对象的关键点提取及跟踪、对象的位姿或深度检测等。应用场景不仅可以涉及跟真实场景或物品相关的导览、导航、讲解、重建、虚拟效果叠加展示等交互场景，还可以涉及与人相关的特效处理，比如妆容美化、肢体美化、特效展示、虚拟模型展示等交互场景。
可通过卷积神经网络，实现对目标对象的相关特征、状态及属性进行检测或识别处理。上述卷积神经网络是基于深度学习框架进行模型训练而得到的网络模型。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其实现可以参照上文方法实施例的描述。
上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考。
在本公开所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不 执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(Processor)执行本公开各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
工业实用性
本公开实施例公开了一种定位方法及装置、设备、存储介质及计算机程序产品，定位方法包括：获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据；基于若干惯性测量数据进行定位处理，得到第一历史图像帧与目标图像帧之间的位姿变化信息；基于位姿变化信息以及至少一个参考因子，确定目标图像帧的位姿，其中，至少一个参考因子包括若干惯性测量数据。通过获取第一历史图像帧至目标图像帧之间的惯性测量数据，并依据该惯性测量数据进行定位处理，获取这两帧图像之间的位姿变化信息，进而可以根据该位姿变化信息得到目标图像帧的位姿信息。另外，在获取位姿变化信息之后，再结合参考因子，确定目标图像帧的位姿，由此，可以获取得到更为准确的位姿。上述过程利用惯性测量数据进行位姿变化的计算进而实现定位，故减少了视觉方面对定位的不良影响，从而减少了因为外部环境等因素造成定位准确度较低的情况。

Claims (35)

  1. 一种定位方法，包括：
    获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据；
    基于所述若干惯性测量数据进行定位处理，得到所述第一历史图像帧与所述目标图像帧之间的位姿变化信息；
    基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿，其中，所述至少一个参考因子包括所述若干惯性测量数据。
  2. 根据权利要求1所述的方法，其中，所述至少一个参考因子还包括关于所述目标图像帧和第一历史图像帧的图像信息。
  3. 根据权利要求1所述的方法，其中，所述基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿，包括：
    基于所述位姿变化信息，确定所述目标图像帧的第一位姿；以及，
    利用所述位姿变化信息以及所述至少一个参考因子，构建本次定位对应的总能量关系，其中，本次定位对应的总能量关系用于确定所述目标图像帧的需优化的位姿偏差；
    利用本次定位对应的的总能量关系，对所述第一位姿进行优化，得到所述目标图像帧的位姿。
  4. 根据权利要求3所述的方法，其中，所述利用所述位姿变化信息以及所述至少一个参考因子，构建本次定位对应的总能量关系，包括：
    分别利用每个所述参考因子，确定所述参考因子对应的测量能量关系，以及，
    利用所述位姿变化信息确定运动先验能量关系；
    基于所述运动先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
  5. 根据权利要求4所述的方法，其中，所述分别利用每个所述参考因子，确定所述参考因子对应的测量能量关系，包括：
    获取所述若干惯性测量数据的预积分信息；
    利用所述若干惯性测量数据的预积分信息，构建惯性测量能量关系。
  6. 根据权利要求4所述的方法，其中，所述位姿变化信息包括至少一种位姿变化量，所述至少一种位姿变化量包括位置变化量和姿态变化量中的至少之一；所述利用所述位姿变化信息确定运动先验能量关系，包括：
    利用至少一次定位对应的若干参数，确定运动先验能量关系，其中，所述至少一次定位包括本次定位，每次所述定位对应的若干参数包括所述定位过程中得到的所述位姿变化量、所述定位过程中采用的所述若干惯性测量数据的起始测量时刻和结束测量时刻的位置差、以及所述起始测量时刻对应的起始姿态信息。
  7. 根据权利要求6所述的方法，其中，所述位姿变化信息还包括所述位姿变化量的确定度表征信息；所述利用至少一次定位对应的若干参数，确定运动先验能量关系，包括：
    基于每次定位过程中得到的所述确定度表征信息，得到对应次定位的权重；
    利用所述至少一次定位的权重以及所述若干参数，确定运动先验能量关系。
  8. 根据权利要求6或7所述的方法，其中，所述至少一次定位为多次定位，所述利用至少一次定位对应的若干参数，确定运动先验能量关系，包括：
    从所述多次定位中，剔除满足去除条件的定位；其中，所述去除条件为所述定位对应的位姿变化量以及所述位姿变化量的确定度表征信息之间的预设处理结果满足第一预设条件；
    利用经剔除后的剩余次定位对应的若干参数，确定运动先验能量关系。
  9. 根据权利要求6所述的方法，其中，所述起始姿态信息包括偏航角信息。
  10. 根据权利要求6或9所述的方法，其中，每次定位对应的所述第一历史图像帧与所述目标图像帧中，拍摄最早的图像帧为起始图像帧，拍摄最晚的图像帧为结束图像帧；在所述利用至少一次定位对应的若干参数，确定运动先验能量关系之前，所述方法还包括：
    将所述至少一次定位中的每次定位作为目标定位，基于所述目标定位对应的若干惯性测量数据的预积分信息、起始图像帧的位姿和所述结束图像帧的位姿，确定所述目标定位对应的所述起始姿态信息、所述起始测量时刻的位置和所述结束测量时刻的位置。
  11. 根据权利要求4所述的方法，其中，所述参考因子还包括关于所述目标图像帧和第一历史图像帧的图像信息，所述分别利用每个所述参考因子，确定所述参考因子对应的测量能量关系，包括：
    利用所述图像信息，确定所述图像信息对应的视觉测量能量关系；
    在所述利用所述图像信息，确定所述图像信息对应的视觉测量能量关系之前，所述方法还包括：
    对若干参考图像帧的特征点进行匹配，得到特征点匹配结果，其中，所述若干参考图像包括至少一次定位过程中的第一历史图像帧和目标图像帧，且所述至少一次定位包括本次定位；
    所述利用所述图像信息，确定所述图像信息对应的视觉测量能量关系，包括：
    基于所述特征点匹配结果，从所述若干参考图像帧中确定至少一对匹配图像帧，其中，每对所述匹配图像帧存在匹配特征点对；
    利用每对所述匹配图像帧的位姿以及所述匹配特征点对在所述匹配图像帧中的位置，确定所述视觉测量能量关系。
  12. 根据权利要求4所述的方法，其中，在所述基于所述运动先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系之前，所述方法还包括：
    基于历史定位对应的总能量关系，确定优化先验能量关系；
    所述基于所述运动先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系，包括：
    基于所述运动先验能量关系、优化先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
  13. 根据权利要求12所述的方法，其中，所述历史定位对应的总能量关系为上一次定位对应的总能量关系。
  14. 根据权利要求12或13所述的方法，其中，所述目标图像帧对应的位姿偏差至少由所述目标图像帧的位姿以及所述目标图像帧之前的第一数量图像帧对应的位姿以及所述目标图像帧对应的惯性信息确定；所述基于历史定位对应的总能量关系，确定优化先验能量关系，包括：
    利用第二历史图像帧的位姿、所述第二历史图像帧之前的第二数量图像帧对应的位姿、所述目标图像帧的位姿、以及所述第二历史图像帧对应的惯性信息，更新 得到所述第二历史图像帧对应的新的位姿偏差，其中，所述第二历史图像帧为所述历史定位中的目标图像帧，所述第二数量小于所述第一数量；
    将所述历史定位对应的总能量关系中的所述位姿偏差替换为所述新的位姿偏差，得到所述优化先验能量关系。
  15. 根据权利要求3所述的方法，其中，所述总能量关系表示所述位姿偏差与总能量之间的关系；所述利用本次定位对应的总能量关系，对所述第一位姿进行优化，得到所述目标图像帧的位姿，包括：
    利用本次定位对应的总能量关系，确定使所述总能量满足第二预设条件的所述位姿偏差；
    基于确定的所述位姿偏差对所述第一位姿进行优化。
  16. 根据权利要求3或15所述的方法，其中，所述位姿变化信息包括至少一种位姿变化量；所述基于所述位姿变化信息，确定所述目标图像帧的第一位姿，包括：
    利用所述目标图像帧对应的位姿变化量，确定所述目标图像帧的第一位姿。
  17. 根据权利要求1所述的方法，其中，所述基于所述若干惯性测量数据进行定位处理是由定位模型执行。
  18. 根据权利要求1或17所述的方法，其中，所述基于所述若干惯性测量数据进行定位处理，得到所述第一历史图像帧与所述目标图像帧之间的位姿变化信息，包括：
    利用所述惯性测量数据以及参考运动状态信息，确定本次所述定位处理得到的最终运动状态信息，其中，所述参考运动状态信息为历史所述定位处理得到的最终运动状态信息；
    基于本次所述定位处理得到的最终运动状态信息，得到所述第一历史图像帧与所述目标图像帧之间的位姿变化信息。
  19. 根据权利要求1所述的方法，其中，所述定位方法由定位系统执行，在所述基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿之前，所述方法还包括：
    判断所述定位系统的参数是否已初始化，其中，所述参数包括重力方向、惯性偏置量中的至少一者；
    响应于所述参数已初始化，则执行所述基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿；
    响应于所述参数未初始化，则选择与所述目标图像帧对应的状态匹配的初始化方式，对所述定位系统的参数进行初始化，再执行所述基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿，其中，所述状态包括运动状态和静止状态。
  20. 根据权利要求1所述的方法，其中，所述定位方法由定位系统执行，在所述基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿之后，所述方法还包括：
    基于所述目标图像帧的位姿，对所述定位系统的参数进行优化，其中，所述参数包括重力方向、惯性偏置量中的至少一者。
  21. 根据权利要求1所述的方法，其中，所述目标图像帧的位姿表示待定位对象在所述目标图像帧的拍摄时刻的位姿，所述目标图像帧和第一历史图像帧是由与待定位对象相对固定的拍摄装置拍摄得到，所述惯性测量数据是由与所述待定位对象相对固定的惯性测量装置测量得到。
  22. 根据权利要求1或21所述的方法，其中，所述获取第一历史图像帧的拍摄时间至目标图像帧的拍摄时间的拍摄期间测量得到的若干惯性测量数据之后，所述方法还包括：
    对所述惯性测量数据进行预处理，其中，经所述预处理的所述惯性测量数据用于进行所述定位处理，所述预处理包括将所述惯性测量数据转换到重力系下、去除偏置、去除重力和归一化中的一种或多种。
  23. 一种定位装置，包括：
    数据获取模块，配置为获取第一历史图像帧至目标图像帧的拍摄期间测量得到的若干惯性测量数据；
    定位处理模块，配置为基于所述若干惯性测量数据进行定位处理，得到所述第一历史图像帧与所述目标图像帧之间的位姿变化信息；
    位姿确定模块，配置为基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿，其中，所述至少一个参考因子包括所述若干惯性测量数据。
  24. 根据权利要求23所述的装置，其中，所述位姿确定模块在基于所述位姿变化信息以及至少一个参考因子，确定所述目标图像帧的位姿的情况下，配置为：
    基于所述位姿变化信息，确定所述目标图像帧的第一位姿；以及，
    利用所述位姿变化信息以及所述至少一个参考因子，构建本次定位对应的总能量关系，其中，本次定位对应的总能量关系用于确定所述目标图像帧的需优化的位姿偏差；
    利用本次定位对应的的总能量关系，对所述第一位姿进行优化，得到所述目标图像帧的位姿。
  25. 根据权利要24所述的装置，其中，所述位姿确定模块在利用所述位姿变化信息以及所述至少一个参考因子，构建本次定位对应的总能量关系的情况下，配置为：
    分别利用每个所述参考因子，确定所述参考因子对应的测量能量关系，以及，
    利用所述位姿变化信息确定运动先验能量关系；
    基于所述运动先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
  26. 根据权利要25所述的装置，其中，所述位姿确定模块在分别利用每个所述参考因子，确定所述参考因子对应的测量能量关系的情况下，配置为：
    获取所述若干惯性测量数据的预积分信息；
    利用所述若干惯性测量数据的预积分信息，构建惯性测量能量关系。
  27. 根据权利要求25所述的装置，其中，所述位姿变化信息包括至少一种位姿变化量，所述至少一种位姿变化量包括位置变化量和姿态变化量中的至少之一；所述位姿确定模块在利用所述位姿变化信息确定运动先验能量关系的情况下，包括：
    利用至少一次定位对应的若干参数，确定运动先验能量关系，其中，所述至少一次定位包括本次定位，每次所述定位对应的若干参数包括所述定位过程中得到的所述位姿变化量、所述定位过程中采用的所述若干惯性测量数据的起始测量时刻和结束测量时刻的位置差、以及所述起始测量时刻对应的起始姿态信息。
  28. 根据权利要求27所述的装置，其中，所述位姿变化信息还包括所述位姿变化量的确定度表征信息；所述位姿确定模块在利用至少一次定位对应的若干参数，确定运动先验能量关系的情况下，配置为：
    基于每次定位过程中得到的所述确定度表征信息，得到对应次定位的权重；
    利用所述至少一次定位的权重以及所述若干参数，确定运动先验能量关系。
  29. 根据权利要求27所述的装置，其中，每次定位对应的所述第一历史图像帧与所述目标图像帧中，拍摄最早的图像帧为起始图像帧，拍摄最晚的图像帧为结束图像帧；所述位姿确定模块在所述利用至少一次定位对应的若干参数，确定运动先验能量关系之前，配置为：
    将所述至少一次定位中的每次定位作为目标定位，基于所述目标定位对应的若干惯性测量数据的预积分信息、起始图像帧的位姿和所述结束图像帧的位姿，确定所述目标定位对应的所述起始姿态信息、所述起始测量时刻的位置和所述结束测量时刻的位置。
  30. 根据权利要求25所述的装置，其中，所述参考因子还包括关于所述目标图像帧和第一历史图像帧的图像信息，所述位姿确定模块在分别利用每个所述参考因子，确定所述参考因子对应的测量能量关系的情况下，配置为：
    利用所述图像信息，确定所述图像信息对应的视觉测量能量关系；
    在所述利用所述图像信息，确定所述图像信息对应的视觉测量能量关系之前，所述装置还包括：
    对若干参考图像帧的特征点进行匹配，得到特征点匹配结果，其中，所述若干参考图像包括至少一次定位过程中的第一历史图像帧和目标图像帧，且所述至少一次定位包括本次定位；
    所述利用所述图像信息，确定所述图像信息对应的视觉测量能量关系，包括：
    基于所述特征点匹配结果，从所述若干参考图像帧中确定至少一对匹配图像帧，其中，每对所述匹配图像帧存在匹配特征点对；
    利用每对所述匹配图像帧的位姿以及所述匹配特征点对在所述匹配图像帧中的位置，确定所述视觉测量能量关系。
  31. 根据权利要求25所述的装置，其中，所述位姿确定模块在所述基于所述运动先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系之前，还配置为：
    基于历史定位对应的总能量关系，确定优化先验能量关系；
    所述基于所述运动先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系，包括：
    基于所述运动先验能量关系、优化先验能量关系以及每个所述参考因子对应的测量能量关系，构建本次定位对应的总能量关系。
  32. 根据权利要求24所述的装置，其中，所述总能量关系表示所述位姿偏差与总能量之间的关系；所述利用本次定位对应的总能量关系，对所述第一位姿进行优化，得到所述目标图像帧的位姿，包括：
    利用本次定位对应的总能量关系，确定使所述总能量满足第二预设条件的所述位姿偏差；
    基于确定的所述位姿偏差对所述第一位姿进行优化。
  33. 一种电子设备，包括存储器和处理器，所述处理器用于执行所述存储器中存储的程序指令，以实现权利要求1至22任一项所述的方法。
  34. 一种计算机可读存储介质，其上存储有程序指令，所述程序指令被处理器执行时实现权利要求1至22任一项所述的方法。
  35. 一种计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令被计算机设备的处理器运行时，实现权利要求1至22中任一项所述的方法。
PCT/CN2021/143905 2021-09-29 2021-12-31 定位方法及装置、设备、存储介质及计算机程序产品 WO2023050634A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111154872.XA CN113899364B (zh) 2021-09-29 2021-09-29 定位方法及装置、设备、存储介质
CN202111154872.X 2021-09-29

Publications (1)

Publication Number Publication Date
WO2023050634A1 true WO2023050634A1 (zh) 2023-04-06

Family

ID=79189637

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/143905 WO2023050634A1 (zh) 2021-09-29 2021-12-31 定位方法及装置、设备、存储介质及计算机程序产品

Country Status (3)

Country Link
CN (1) CN113899364B (zh)
TW (1) TW202314593A (zh)
WO (1) WO2023050634A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116718153A (zh) * 2023-08-07 2023-09-08 成都云智北斗科技有限公司 一种基于gnss和ins的形变监测方法及系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114543797B (zh) * 2022-02-18 2024-06-07 北京市商汤科技开发有限公司 位姿预测方法和装置、设备、介质
CN117889853B (zh) * 2024-03-15 2024-06-04 歌尔股份有限公司 Slam定位方法、终端设备及可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017172778A1 (en) * 2016-03-28 2017-10-05 Sri International Collaborative navigation and mapping
CN107747941A (zh) * 2017-09-29 2018-03-02 歌尔股份有限公司 一种双目视觉定位方法、装置及系统
CN111507132A (zh) * 2019-01-31 2020-08-07 杭州海康机器人技术有限公司 一种定位方法、装置及设备
CN112884840A (zh) * 2021-03-29 2021-06-01 湖南国科微电子股份有限公司 一种视觉定位方法、装置、设备及存储介质
CN113847907A (zh) * 2021-09-29 2021-12-28 深圳市慧鲤科技有限公司 定位方法及装置、设备、存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108492316A (zh) * 2018-02-13 2018-09-04 视辰信息科技(上海)有限公司 一种终端的定位方法和装置
CN112902953B (zh) * 2021-01-26 2022-10-04 中国科学院国家空间科学中心 一种基于slam技术的自主位姿测量方法
CN112819860B (zh) * 2021-02-18 2023-12-22 Oppo广东移动通信有限公司 视觉惯性系统初始化方法及装置、介质和电子设备
CN113012224B (zh) * 2021-03-12 2022-06-03 浙江商汤科技开发有限公司 定位初始化方法和相关装置、设备、存储介质
CN113052897A (zh) * 2021-03-25 2021-06-29 浙江商汤科技开发有限公司 定位初始化方法和相关装置、设备、存储介质
CN113256718B (zh) * 2021-05-27 2023-04-07 浙江商汤科技开发有限公司 定位方法和装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017172778A1 (en) * 2016-03-28 2017-10-05 Sri International Collaborative navigation and mapping
CN107747941A (zh) * 2017-09-29 2018-03-02 歌尔股份有限公司 一种双目视觉定位方法、装置及系统
CN111507132A (zh) * 2019-01-31 2020-08-07 杭州海康机器人技术有限公司 一种定位方法、装置及设备
CN112884840A (zh) * 2021-03-29 2021-06-01 湖南国科微电子股份有限公司 一种视觉定位方法、装置、设备及存储介质
CN113847907A (zh) * 2021-09-29 2021-12-28 深圳市慧鲤科技有限公司 定位方法及装置、设备、存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116718153A (zh) * 2023-08-07 2023-09-08 成都云智北斗科技有限公司 一种基于gnss和ins的形变监测方法及系统
CN116718153B (zh) * 2023-08-07 2023-10-27 成都云智北斗科技有限公司 一种基于gnss和ins的形变监测方法及系统

Also Published As

Publication number Publication date
CN113899364A (zh) 2022-01-07
CN113899364B (zh) 2022-12-27
TW202314593A (zh) 2023-04-01

Similar Documents

Publication Publication Date Title
CN111156984B (zh) 一种面向动态场景的单目视觉惯性slam方法
CN109307508B (zh) 一种基于多关键帧的全景惯导slam方法
WO2023050634A1 (zh) 定位方法及装置、设备、存储介质及计算机程序产品
CN111882607B (zh) 一种适用于增强现实应用的视觉惯导融合位姿估计方法
JP5987823B2 (ja) 画像センサおよび運動または位置センサから生じたデータを融合するための方法およびシステム
CN112219087A (zh) 位姿预测方法、地图构建方法、可移动平台及存储介质
CN110717927A (zh) 基于深度学习和视惯融合的室内机器人运动估计方法
JP7131994B2 (ja) 自己位置推定装置、自己位置推定方法、自己位置推定プログラム、学習装置、学習方法及び学習プログラム
JP2015532077A (ja) 少なくとも１つの画像を撮影する撮影装置に関連する装置の位置及び方向の決定方法
US10636190B2 (en) Methods and systems for exploiting per-pixel motion conflicts to extract primary and secondary motions in augmented reality systems
US20160210761A1 (en) 3d reconstruction
WO2022188334A1 (zh) 定位初始化方法和装置、设备、存储介质、程序产品
WO2023087681A1 (zh) 定位初始化方法、装置、计算机可读存储介质和计算机程序产品
CN109544632B (zh) 一种基于层次主题模型的语义slam对象关联方法
CN112284381B (zh) 视觉惯性实时初始化对准方法及系统
CN112907633A (zh) 动态特征点识别方法及其应用
CN112731503A (zh) 一种基于前端紧耦合的位姿估计方法及系统
TWI812053B (zh) 定位方法、電子設備及電腦可讀儲存媒體
CN113847907A (zh) 定位方法及装置、设备、存储介质
Zhang et al. Robust orientation estimate via inertial guided visual sample consensus
CN116576866B (zh) 导航方法和设备
Andrés López et al. VCIT-Visually Corrected inertial Tracking
López et al. VCIT-Visually Corrected inertial Tracking.
CN117111038A (zh) 标定方法、系统、电子设备和存储介质
CN116698023A (zh) 状态区分方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21959192

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE