WO2022193508A1

WO2022193508A1 - Method and apparatus for posture optimization, electronic device, computer-readable storage medium, computer program, and program product

Info

Publication number: WO2022193508A1
Application number: PCT/CN2021/106997
Authority: WO
Inventors: 章国锋; 鲍虎军; 叶智超; 刘浩敏
Original assignee: 浙江商汤科技开发有限公司
Priority date: 2021-03-16
Filing date: 2021-07-19
Publication date: 2022-09-22
Also published as: CN113034594A

Abstract

The present disclosure relates to a method and apparatus for posture optimization, an electronic device, a computer-readable storage medium, a computer program, and a program product. The method for posture optimization comprises: determining a current predicted posture of a mobile device on the basis of first images captured by the mobile device, the first images comprising a current frame of image captured by the mobile device when located at the current position and an initial frame of image captured by the mobile device when located at an initial position before the current position; determining movement parameter information corresponding to a target movement process that the mobile device experienced when moving to the current position; acquiring multiple pieces of estimated posture information of the mobile device, the multiple pieces of estimated posture information comprising estimated posture information corresponding respectively to the initial frame of image and to the current frame of image and estimated posture information corresponding respectively to multiple frames of images between the initial frame of image and the current frame of image; and optimizing the current predicted posture on the basis of the movement parameter information and of the multiple pieces of estimated posture information to produce a current optimized posture.

Description

Pose optimization method, apparatus, electronic device, computer-readable storage medium, computer program and program product

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is based on the Chinese patent application with the application number of 202110279737.1, the application date of March 16, 2021, and the application title of "Pose Optimization Method, Device, Electronic Device and Storage Medium", and claims the priority of the Chinese patent application. , the entire content of the Chinese patent application is incorporated herein by reference.

technical field

The present disclosure relates to the field of computer vision technology, and in particular, to a pose optimization method, apparatus, electronic device, computer-readable storage medium, computer program, and program product.

Background technique

At present, the positioning of unmanned equipment through vision is an important part of applications in the fields of robots, unmanned aerial vehicles, automatic navigation and augmented reality. has been widely used. However, the image data collected by the monocular camera cannot provide the real three-dimensional size information of these objects, so the scale of the monocular visual odometry cannot be determined, which leads to certain errors in the pose calculation.

SUMMARY OF THE INVENTION

This embodiment provides at least one pose optimization method, apparatus, electronic device, computer-readable storage medium, computer program, and program product.

In a first aspect, an embodiment of the present disclosure provides a pose optimization method, including:

Determine the current predicted pose of the mobile device based on the first image captured by the mobile device; the first image includes the current frame image captured when the mobile device is at the current position and before the mobile device is at the current position The starting frame image taken when the starting position is ;

Determine the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position; the target movement process is located between the starting position and the current position;

Acquire multiple pieces of estimated pose information of the mobile device, where the pieces of estimated pose information include estimated pose information corresponding to the starting frame image, estimated pose information corresponding to the current frame image, and the starting frame image. The estimated pose information corresponding to the multi-frame images between the initial frame image and the current frame image respectively;

The current predicted pose is optimized based on the movement parameter information and the plurality of estimated pose information to obtain a current optimized pose.

In the embodiment of the present disclosure, the scale information of the visual odometer is recovered by acquiring the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position, and the current predicted pose is optimized based on the scale information. , which can improve the calculation accuracy of pose. In addition, this method uses the acquired movement parameter information as a scale constraint, and then obtains an optimized pose information. The computational complexity is low, which reduces the performance requirements of the device (for example, it is applied to some low-end mobile phones), so The adaptability of the method is improved.

According to the first aspect, in a possible implementation manner, the movement parameter information includes a movement distance;

The determining of the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position includes:

Determine, based on the pedestrian dead reckoning algorithm, the moving distance corresponding to the moving process of the target experienced by the mobile device when it moves to the current position;

The current predicted pose is optimized based on the movement parameter information and the plurality of estimated pose information to obtain the current optimized pose, including:

The current predicted pose is optimized based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose.

In the embodiment of the present disclosure, the moving distance corresponding to the moving process of the target is determined based on the pedestrian dead reckoning algorithm, which can simplify the process of determining the moving distance, improve the efficiency of determining the moving distance, and further improve the efficiency of the pose optimization method. Operational efficiency.

According to the first aspect, in a possible implementation manner, the current predicted pose is optimized based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose, including: :

Determine the movement start time and movement end time corresponding to the target movement process;

According to the image capturing time, the initial moving frame image corresponding to the moving start time and the moving ending frame image corresponding to the moving ending time are determined, and the moving initial frame image corresponding to the moving initial frame image and the moving ending frame image are respectively determined. Estimated pose and estimated pose at the end of the move;

Determining the moving distance as the distance between the initial estimated pose of the movement and the estimated pose at the end of the movement, and combining the multiple estimated pose information, optimize the current predicted pose to obtain the current Optimize the pose.

In the embodiment of the present disclosure, according to the image capture time and the movement start time and movement end time corresponding to the target movement process, the movement initial estimated pose and the movement end estimated position corresponding to the movement initial frame image and the movement end frame image are determined The movement distance is determined as the distance between the initial estimated pose of the movement and the estimated pose of the end of the movement, so that the distance constraint is obtained between the initial estimated pose of the movement and the estimated pose of the end of the movement, and then It can realize the optimization of the current predicted pose and improve the accuracy of visual positioning.

According to the first aspect, in a possible implementation manner, the moving distance corresponding to the moving process of the target experienced by the mobile device when moving to the current position is determined based on a pedestrian dead reckoning algorithm, include:

Obtain the number of steps taken by the user of the mobile device during the movement of the target;

The moving distance is determined based on the number of walking steps and the step length of the user.

In the embodiment of the present disclosure, since the moving distance is determined by using the user's walking steps and step length, only an Inertial Measurement Unit (Inertial Measurement Unit) with lower precision can be used to achieve this, and thus the positioning accuracy can be improved at the same time. The requirement for the accuracy of the IMU is reduced, so that the method can perform indoor positioning in scenarios with limited computing power (for example, on the applet of a low-end mobile phone).

According to the first aspect, in a possible implementation manner, the movement parameter information further includes a movement speed;

The determining of the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position further includes:

determining the movement speed corresponding to the target movement process experienced by the mobile device when it moves to the current position;

The optimizing the current predicted pose based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose includes:

The current predicted pose is optimized based on the moving distance, the moving speed, and the plurality of estimated pose information to obtain the current optimized pose.

In the embodiment of the present disclosure, after the moving distance is determined, the moving speed is also determined, so that the corresponding estimated pose information of the multi-frame images corresponding to the distance that the user walks one step is also constrained by the distance, thereby further improving the optimization of the current predicted pose precision.

According to the first aspect, in a possible implementation manner, the determining of the moving speed corresponding to the target moving process experienced by the mobile device when moving to the current position includes:

Obtain the historical motion speed of the user of the mobile device;

The movement speed is predicted based on a movement model and the historical movement speed.

In the embodiment of the present disclosure, the moving speed of the target process is predicted by the motion model and the historical motion speed, which can improve the accuracy of parameter acquisition compared to the user's walking speed measured directly based on the IMU.

According to the first aspect, in a possible implementation manner, after obtaining the current optimized pose, the method further includes:

judging whether the error of the current optimized pose is less than a preset threshold;

When the error of the current optimized pose exceeds the preset threshold, acquiring the current predicted speed and current orientation of the mobile device;

Based on the current optimized pose, the current predicted velocity, and the current orientation, real-time pose information of the mobile device is determined.

In the embodiment of the present disclosure, when the error of the current optimized pose exceeds the preset threshold, that is, the visual tracking result is relatively poor, the current predicted speed and the current orientation can be directly used to continue tracking for a period of time until the visual tracking Track recovery. In this way, even in the scene where the visual odometry is difficult to locate, the navigation and positioning results can still be obtained through the pedestrian dead position information, which improves the applicability of the method.

Based on the current optimized pose, the destination input by the user, and the stored map, a navigation route is planned, and the navigation route is displayed on the map.

In this disclosed embodiment, the current route planning is performed by using the optimized current predicted pose, which can improve the accuracy of the planned route.

According to the first aspect, in a possible implementation manner, the navigation route is planned based on the current optimized pose, the destination input by the user, and the stored map, and the navigation route is displayed on the map, include:

determining the location information of the destination in the map in response to the destination input request of the user entering and leaving;

Based on the current optimized pose and the location information of the destination, determining at least one connection line between the current optimized pose and the destination;

The at least one connection line is displayed in the map.

In the embodiment of the present disclosure, in the route planning process, the destination information input by the user is first determined in the map, and then at least one connection route is planned according to the current optimized pose and the location information of the destination, and is The at least one connection line is displayed on the map, so that multiple choices can be provided for the user, so that the user can select one of the multiple lines for navigation according to the actual situation, thereby improving the applicability of the navigation line.

In a second aspect, an embodiment of the present disclosure provides a pose optimization device, including:

The pose estimation part is configured to determine the current predicted pose of the mobile device based on the first image captured by the mobile device; the first image includes the current frame image captured when the mobile device is in the current position and the movement The starting frame image taken when the device is at the starting position before the current position;

A parameter determination part, configured to determine the movement parameter information corresponding to the target movement process experienced by the mobile device when moving to the current position; the target movement process is located between the starting position and the current position ;

a pose acquisition part, configured to obtain a plurality of estimated pose information of the mobile device, the plurality of estimated pose information including the estimated pose information corresponding to the initial frame image, the estimated pose information corresponding to the current frame image pose information and estimated pose information respectively corresponding to the multi-frame images between the initial frame image and the current frame image;

The pose optimization part is configured to optimize the current predicted pose based on the movement parameter information and the plurality of estimated pose information to obtain the current optimized pose.

According to the second aspect, in a possible implementation manner, the movement parameter information includes a movement distance;

The parameter determination part is configured as:

The pose optimization part is configured as:

According to the second aspect, in a possible implementation manner, the pose optimization part is configured as:

According to the second aspect, in a possible implementation manner, the parameter determining part is configured to:

According to the second aspect, in a possible implementation manner, the movement parameter information further includes a movement speed;

The parameter determination part is also configured as:

The pose optimization part is configured as:

Obtain the historical motion speed of the user of the mobile device;

According to the second aspect, in a possible implementation manner, the pose optimization part is further configured to:

According to the second aspect, in a possible implementation manner, the pose optimization device further includes a route planning part, and the route planning part is configured to:

According to the second aspect, in a possible implementation manner, the route planning part is configured as:

The at least one connection line is displayed in the map.

In a third aspect, embodiments of the present disclosure provide an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processing The processor and the memory communicate through a bus, and the machine-readable instructions are executed by the processor to execute the pose optimization method according to the first aspect.

In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the pose optimization method according to the first aspect .

In a fifth aspect, an embodiment of the present disclosure provides a computer program, including computer-readable code, where the computer-readable code runs in an electronic device and is executed by a processor in the electronic device, and executes the following steps: The pose optimization method described in the first aspect.

In a sixth aspect, an embodiment of the present disclosure provides a computer program product that, when executed on a computer, causes the computer to execute the pose optimization method described in the first aspect.

In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description serve to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

FIG. 1 shows a flowchart of a pose optimization method provided by an embodiment of the present disclosure;

FIG. 2 shows a flowchart of a method for optimizing a current predicted pose based on a moving distance provided by an embodiment of the present disclosure;

FIG. 3 shows a flowchart of a method for obtaining a moving distance provided by an embodiment of the present disclosure;

FIG. 4 shows a flowchart of a method for obtaining a moving speed provided by an embodiment of the present disclosure;

FIG. 5 shows a flowchart of another pose optimization method provided by an embodiment of the present disclosure;

FIG. 6 shows a schematic structural diagram of a pose optimization apparatus provided by an embodiment of the present disclosure;

FIG. 7 shows a schematic structural diagram of another pose optimization apparatus provided by an embodiment of the present disclosure;

FIG. 8 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. The components of the disclosed embodiments generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The term "and/or" in this paper only describes an association relationship, which means that there can be three kinds of relationships, for example, A and/or B, which can mean: the existence of A alone, the existence of A and B at the same time, the existence of B alone. a situation. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

At present, the positioning of unmanned equipment through vision is an important part of applications in the fields of robots, unmanned aerial vehicles, automatic navigation, and augmented reality. The vision method only relies on the image information collected by the camera, based on the camera model and visual geometry Learning models, etc. to calculate the six-degree-of-freedom motion of the machine itself. This self-localization orientation method based on image information is similar to traditional odometer dead reckoning, so it is called visual odometry.

After research, it is found that monocular cameras are widely used in visual positioning due to their low cost, light weight and easy installation. However, the image data collected by the monocular camera cannot provide the real three-dimensional size information of these objects, so the scale of the monocular visual odometry cannot be determined, which leads to certain errors in the pose calculation.

In this regard, the following solutions are proposed in the related art:

One type of solution is to combine additional accurate sensors to obtain more information and restore scales, such as visual odometry combined with high-precision GPS and IMU; however, this method requires the device to be equipped with additional high-precision sensing devices and to the computing power of the device. Accuracy is required.

Another type of solution is to build in advance through the navigation scene, and restore the scale of the visual odometry by continuously positioning in the map. However, this method has high requirements on the frequency of pre-mapping and positioning.

In view of this, how to improve the positioning accuracy of the monocular visual odometry is a technical problem to be solved by the present disclosure.

Based on the above research, the present disclosure provides a pose optimization method, which determines the current predicted pose of the mobile device based on the first image captured by the mobile device; the first image includes the current frame image captured when the mobile device The initial frame image taken when the device is in the initial position before the current position; determine the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position; the target movement process is located at the initial position and the current position; obtain multiple estimated pose information of the mobile device, the multiple estimated pose information includes the estimated pose information corresponding to the starting frame image and the current frame image respectively, as well as the starting frame image and the current frame image The estimated pose information corresponding to the multi-frame images between them respectively; the current predicted pose is optimized based on the movement parameter information and the multiple estimated pose information to obtain the current optimized pose.

That is, in the method in the embodiment of the present disclosure, during the process of visual positioning, the scale information of the visual odometer is recovered by acquiring the movement parameter information corresponding to the movement process of the target experienced by the mobile device when it moves to the current position. , and optimize the current predicted pose based on the scale information, which can improve the accuracy of pose calculation.

In order to facilitate the understanding of this embodiment, a pose optimization method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the pose optimization method provided by the embodiment of the present disclosure is generally an electronic device with a certain computing capability. The electronic device includes, for example, a terminal device or a server or other processing device, and the terminal device may be a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like. In some possible implementations, the pose optimization method may be implemented by the processor calling computer-readable instructions stored in the memory.

Referring to FIG. 1, which is a flowchart of a pose optimization method provided by an embodiment of the present disclosure, the pose optimization method includes the following S101-S104:

S101: Determine the current predicted pose of the mobile device based on a first image captured by the mobile device; the first image includes a current frame image captured when the mobile device is at the current position and a start image captured when the mobile device is at a starting position before the current position Start frame image.

In the process of visual localization, it usually includes the process of visual tracking, estimating pose and optimizing. Among them, the correlation between the feature points between the images can be obtained by visual tracking, and then the rough pose of the camera is estimated according to the correlation between the feature points. For example, for two images, some representative points should be selected first, called feature points; after that, only the motion of the camera is estimated for these feature points, and the spatial position of the feature points and the information of other non-feature points in the image are estimated at the same time. , is discarded. For example, common feature point extraction methods, including corner points, color blocks, etc. in the image.

A mobile device refers to a device that can be mounted on a carrier (such as a human body) and can move with the carrier. Exemplarily, the mobile device may specifically include a mobile phone, a tablet computer, augmented reality (Augmented Reality, AR) glasses, and the like. That is, the mobile device may be a mobile device or a handheld device among the aforementioned electronic devices having a certain computing capability. The mobile device can have a built-in image acquisition component or an external image acquisition component. After the mobile device enters the working state, the image acquisition component can be used to capture images of the real scene to obtain several images. Wherein, the first image is a part of the several images, that is, the first image includes the current frame image taken when the mobile device is at the current position and the starting position before the mobile device is at the current position The starting frame image taken at the time.

Among them, the starting position can be set according to the complexity of the specific real scene environment. For example, if the current real scene is relatively empty and simple, the position far from the current position before the current position can be used as the starting position; If the real scene environment is more complex, the position that is closer to the current position before the current position can be used as the starting position, which is not limited here.

It can be understood that since the image acquisition component in the embodiment of the present disclosure is a monocular camera, there is a problem of uncertain scale, that is, only the relative position of the current position relative to the starting position can be obtained by using the feature point method based on the first image, Therefore, the obtained current predicted pose is only a relative pose, which is not accurate, and the current predicted pose needs to be optimized.

S102: Determine the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position; the target movement process is located between the initial position and the current position.

Exemplarily, the target moving process may be a process with the starting position as the starting point and the current position as the ending point, or may be any process between the starting position and the current position, here Not limited.

In some embodiments, the movement parameter information may only include the movement distance; in other embodiments, the movement parameter information includes the movement distance and the movement speed, which will be described later for details.

S103: Acquire multiple estimated pose information of the mobile device, where the multiple estimated pose information includes estimated pose information corresponding to the initial frame image, estimated pose information corresponding to the current frame image, and the difference between the initial frame image and the current frame image The estimated pose information corresponding to the multi-frame images in between.

It can be understood that since a plurality of images are captured during the movement of the mobile device, according to the feature point method, each image has estimated pose information of the camera (mobile device) corresponding to the image. Specifically, after matching the feature points, two one-to-one corresponding pixel point sets can be obtained, and then the motion of the camera is estimated according to the two sets of matched point sets.

S104, optimize the current predicted pose based on the movement parameter information and the plurality of estimated pose information to obtain the current optimized pose.

Exemplarily, the current predicted pose may be optimized based on the light velocity adjustment method (Bundle Adjustment). Among them, the beam method adjustment method uses the camera attitude and the three-dimensional coordinates of the measurement point as unknown parameters, and uses the coordinates of the feature points detected on the image for forward intersection as the observation data to adjust to obtain the optimal camera parameters and World point coordinates.

In the embodiment of the present disclosure, during the process of visual positioning, the scale information of the visual odometer is recovered by acquiring the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position, and based on the scale information The information optimizes the current predicted pose, which in turn can improve the accuracy of pose calculation. In addition, this method uses the acquired movement parameter information as a constraint, and then obtains an optimized pose information, the computational complexity is low, and the performance requirements of the device are reduced (for example, it is applied to some low-end mobile phones), thereby improving the adaptability of the method.

In some embodiments, the movement parameter information includes the movement distance. Therefore, for the above S102, when determining the movement parameter information corresponding to the target movement process experienced by the mobile device when moving to the current position, the movement parameter information may include: Based on the pedestrian dead reckoning algorithm, determine the moving distance corresponding to the target moving process experienced by the mobile device when moving to the current position; for the above S104, in the based on the moving parameter information and The multiple estimated pose information optimizes the current predicted pose, and when the current optimized pose is obtained, the method may include: performing an optimization on the current predicted pose based on the moving distance and the multiple estimated pose information. Perform optimization to obtain the current optimized pose.

Among them, dead reckoning is a typical process of calculating the user's current position based on a previously determined position, estimated speed and elapsed time period, which can be determined based on sensor data generated by inertial sensors.

In this embodiment, the scale information of the visual odometry is recovered by obtaining the moving distance corresponding to the moving process of the target experienced by the mobile device when it moves to the current position, thereby optimizing the pose and improving the accuracy of visual positioning.

In addition, determining the moving distance corresponding to the target moving process based on the pedestrian dead reckoning algorithm can simplify the process of determining the moving distance, improve the efficiency of determining the moving distance, and further improve the computing efficiency of the pose optimization method.

Referring to Fig. 2, the flow chart of the method for optimizing the current predicted pose based on the moving distance includes the following S1041-1043:

S1041: Determine the movement start time and the movement end time corresponding to the target movement process.

S1042, according to the image capturing time, determine the initial movement frame image corresponding to the movement start time and the movement end frame image corresponding to the movement end time, and respectively determine the movement initial estimated pose corresponding to the movement initial frame image and the movement end frame image and Estimate the pose at the end of the move.

S1043 , determining the moving distance as the distance between the initial estimated pose of the movement and the estimated pose of the moving end, and combining a plurality of estimated pose information to optimize the current predicted pose to obtain the current optimized pose.

In the embodiment of the present disclosure, according to the image capture time and the movement start time and movement end time corresponding to the target movement process, the movement initial estimated pose and the movement end estimated position corresponding to the movement initial frame image and the movement end frame image are determined The movement distance is determined as the distance between the initial estimated pose of the movement and the estimated pose of the end of the movement, so that the distance constraint is obtained between the initial estimated pose of the movement and the estimated pose of the end of the movement, and then The optimization of the current predicted pose can be achieved.

For the above S102, when determining the moving distance corresponding to the target movement process experienced by the mobile device when moving to the current position based on the pedestrian dead reckoning algorithm, referring to FIG. 3, the following S1021 may be included. ~1022:.

S1021: Acquire the number of steps taken by the user of the mobile device during the movement of the target.

S1022, determining the moving distance based on the number of walking steps and the step length of the user.

For example, the number of steps taken by the user of the mobile device during the movement of the target may be acquired based on the IMU on the mobile device. For example, onboard inertial sensors such as accelerometers and magnetometers enable mobile computing devices to count a user's steps and calculate the distance the user has moved in one step, ie, step length, through pedestrian dead reckoning.

In the embodiment of the present disclosure, since the pedestrian dead reckoning algorithm is used to determine the moving distance of the target moving process, only an IMU with lower precision can be used to realize the realization, thereby improving the positioning accuracy and reducing the requirement for the accuracy of the IMU, so that the This method can perform indoor navigation in scenarios with limited computing power (for example, on small programs on low-end mobile phones).

Since the pedestrian dead reckoning algorithm can obtain the distance information only after the pedestrian completes one step, there is no distance information when the pedestrian is in the process of walking one step, so the few frames of images within one step of the pedestrian are constrained by the lack of distance information, so , in order to further improve the accuracy of visual positioning, in other embodiments, the movement parameter further includes a movement speed.

In this embodiment, for the above S102, the determining of the movement parameter information corresponding to the target movement process experienced by the mobile device when the mobile device moves to the current position further includes: determining that the mobile device is moving to the current location. position, the moving speed corresponding to the experienced target movement process; for the above S104, the current predicted pose is optimized based on the moving distance and the plurality of estimated pose information, and the obtained When optimizing the current pose, it may include: optimizing the current predicted pose based on the moving distance, the moving speed, and the plurality of estimated pose information to obtain the current optimized pose.

Regarding the above S102, when determining the moving speed corresponding to the target moving process experienced by the mobile device when moving to the current position, referring to FIG. 4, the following S102a-102b may be included:

S102a, acquiring the historical motion speed of the user of the mobile device.

S102b, predicting the movement speed based on the movement model and the historical movement speed.

For example, the historical motion speed of the user of the mobile device may be acquired based on the IMU. The motion model may be a uniform motion model, a uniform acceleration motion model, or a more complex motion model, which is not limited herein.

In this way, according to the historical movement speed and the movement model, the movement speed of the user during the movement of the target can be predicted, and the predicted movement speed is added to the optimization algorithm of the visual odometry to calculate the optimal pose information of the current position. That is, according to the predicted moving speed combined with the user's walking time, the scale information corresponding to any two frames of images during the movement of the target can be obtained, thereby further improving the accuracy of visual positioning.

It should be noted that, in this embodiment, the historical motion speed refers to the average motion speed of the user for a period of time before the target moving process. In addition, if the user's walking speed is directly measured based on the IMU, the error will be relatively large. In this embodiment, the moving speed of the target process is predicted by using the motion model and the historical motion speed, which can improve the accuracy of parameter acquisition.

Referring to FIG. 5 , which is a flowchart of another pose optimization method provided by an embodiment of the present disclosure, different from the pose optimization method in FIG. 1 , the pose optimization method further includes the following S105 after step S104 ~S108:

S105, determine whether the error of the current optimized pose is less than a preset threshold; if yes, go to step S108; if not, go to step S106.

It can be understood that in some environments (strong reflection, weak texture, dynamic physics), the visual odometry is difficult to track, which will lead to a large error in the current optimized pose, so it is necessary to judge the current optimized pose. For example, it may be determined based on the reprojection error whether the error of the optimized pose exceeds a preset threshold, and if the error of the current optimized pose information exceeds the preset threshold, step S106 is performed.

S106: Obtain the current predicted speed and the current orientation of the mobile device.

Among them, the prediction method of the current predicted speed is the same as the prediction method of the aforementioned movement speed. It should be noted that the historical movement speed at this time is the average speed of the user's movement for a period of time before the current moment, that is, although the prediction method The same, but because the historical movement speed of the user in different time periods is different, the obtained predicted speed is also different. In addition, the current heading can be obtained through IMU measurement.

S107: Determine real-time pose information of the mobile device based on the current optimized pose, the current predicted speed, and the current orientation.

When the error of the current optimized pose exceeds the preset threshold, that is, in the case of poor visual tracking, the current predicted speed and the current orientation can be directly used based on the current optimized pose to continue tracking for a period of time until The error of the current optimized pose is less than the preset threshold.

In this way, in the scene where the visual odometer is difficult to locate, a rough navigation and positioning result can still be obtained through the pedestrian dead position information, which improves the applicability of the method.

S108 , based on the current optimized pose (or real-time pose information), the destination input by the user, and the stored map, plan a navigation route, and display the navigation route on the map.

In this embodiment of the present disclosure, in response to a user's destination input request, the location information of the destination may be determined in the map; and then the current optimized pose and location information of the destination may be determined based on the current At least one connecting line between the pose and the destination is optimized; finally, the at least one connecting line is displayed in the map.

In the embodiment of the present disclosure, route planning can be performed based on the current optimized pose, the destination input by the user, and the map, so that even in an indoor environment, a more accurate navigation function can be provided for the user, and the user experience can be improved. In addition, in the route planning process, the destination information input by the user is first determined in the map, and then at least one connecting route is planned according to the current optimized pose and the location information of the destination, and all the routes are displayed on the map. The at least one connection line is described, so that more choices can be provided for the user, so that the user can select one of the multiple lines for navigation according to the actual situation, thereby improving the applicability of the navigation line.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

Based on the same technical concept, the embodiments of the present disclosure also provide a pose optimization device corresponding to the pose optimization method. For the implementation of the apparatus, reference may be made to the implementation of the method, and the repetition will not be repeated.

Referring to FIG. 6 , which is a schematic diagram of a pose optimization apparatus 500 according to an embodiment of the present disclosure, the pose optimization apparatus 500 includes:

The pose estimation part 501 is configured to determine the current predicted pose of the mobile device based on a first image captured by the mobile device; the first image includes the current frame image captured when the mobile device is in the current position and the The initial frame image taken when the mobile device is at the initial position before the current position;

The parameter determination part 502 is configured to determine the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position; the target movement process is located between the starting position and the current position. between;

The pose obtaining part 503 is configured to obtain a plurality of estimated pose information of the mobile device, the plurality of estimated pose information includes the estimated pose information corresponding to the starting frame image, the estimated pose information corresponding to the current frame image Estimated pose information and estimated pose information respectively corresponding to multiple frames between the initial frame image and the current frame image;

The pose optimization part 504 is configured to optimize the current predicted pose based on the movement parameter information and the plurality of estimated pose information to obtain a current optimized pose.

In some embodiments, the movement parameter information includes movement distance;

The parameter determination part 502 is configured as:

The pose optimization part 504 is configured as:

In some embodiments, the pose optimization section 504 is configured to:

In some embodiments, the parameter determination section 502 is configured to:

In some embodiments, the movement parameter information further includes movement speed;

The parameter determination part 502 is further configured to:

The pose optimization part 504 is configured as:

In some embodiments, the parameter determination section 502 is configured to:

Obtain the historical motion speed of the user of the mobile device;

In some embodiments, the pose optimization part 504 is further configured to:

In some embodiments, as shown in FIG. 7 , the pose optimization apparatus 500 further includes a route planning part 505, and the route planning part 505 is configured as:

In some embodiments, the route planning section 505 is configured to:

The at least one connection line is displayed in the map.

For the description of the processing flow of each part in the apparatus and the interaction flow between the various parts, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

Based on the same technical concept, an embodiment of the present disclosure also provides an electronic device. Referring to FIG. 8 , a schematic structural diagram of an electronic device 700 provided by an embodiment of the present disclosure includes a processor 701 , a memory 702 , and a bus 703 . Among them, the memory 702 is used to store the execution instructions, including the memory 7021 and the external memory 7022; the memory 7021 here is also called the internal memory, which is used to temporarily store the operation data in the processor 701 and the data exchanged with the external memory 7022 such as the hard disk, The processor 701 exchanges data with the external memory 7022 through the memory 7021 .

In this embodiment of the present application, the memory 702 is specifically used to store the application program code for executing the solution of the present application, and the execution is controlled by the processor 701 . That is, when the electronic device 700 is running, the processor 701 communicates with the memory 702 through the bus 703, so that the processor 701 executes the application code stored in the memory 702, thereby executing the method described in any of the foregoing embodiments.

The memory 702 may be, but not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read only memory (Programmable Read-Only Memory, PROM), or Erasable Programmable Read-Only Memory (EPROM), Electrical Erasable Programmable Read-Only Memory (EEPROM), etc.

The processor 701 may be an integrated circuit chip with signal processing capability. The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC) , Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 700 . In other embodiments of the present application, the electronic device 700 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the pose optimization method in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

Embodiments of the present disclosure also provide a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the steps configured to implement the above-mentioned pose optimization method .

Embodiments of the present disclosure further provide a computer program product, where the computer program product carries program codes, and the instructions included in the program codes can be used to execute the steps of the pose optimization method in the foregoing method embodiments. For details, please refer to the foregoing method implementation. For example, it will not be repeated here.

Wherein, the above-mentioned computer program product can be specifically implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that contribute to the prior art or the parts of the technical solutions. The computer software products are stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure rather than limit them. The protection scope of the present disclosure is not limited thereto, although referring to the foregoing The embodiments describe the present disclosure in detail. Those of ordinary skill in the art should understand that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure. Changes can be easily thought of, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered in the present disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Industrial Applicability

In this embodiment of the present disclosure, the current predicted pose of the mobile device is determined based on a first image captured by the mobile device; the first image includes a current frame image captured when the mobile device is at the current position and a start before the mobile device is at the current position The initial frame image captured when the mobile device moves to the current position; determine the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position; obtain multiple estimated pose information of the mobile device, and the multiple estimated pose information includes the starting frame. The estimated pose information corresponding to the initial frame image and the current frame image respectively, and the estimated pose information corresponding to the multi-frame images between the initial frame image and the current frame image respectively; based on the movement parameter information and multiple estimated pose information, the current The predicted pose is optimized to obtain the current optimized pose, which further improves the accuracy of visual positioning.

Claims

A pose optimization method, including:

Determine the current predicted pose of the mobile device based on the first image captured by the mobile device; the first image includes the current frame image captured when the mobile device is at the current position and before the mobile device is at the current position The starting frame image taken when the starting position is ;

Determine the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position; the target movement process is located between the starting position and the current position;

Acquire multiple pieces of estimated pose information of the mobile device, where the pieces of estimated pose information include estimated pose information corresponding to the starting frame image, estimated pose information corresponding to the current frame image, and the starting frame image. The estimated pose information corresponding to the multi-frame images between the initial frame image and the current frame image respectively;

The current predicted pose is optimized based on the movement parameter information and the plurality of estimated pose information to obtain a current optimized pose.
The method of claim 1, wherein the movement parameter information includes a movement distance;

The determining of the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position includes:

Determine, based on the pedestrian dead reckoning algorithm, the moving distance corresponding to the moving process of the target experienced by the mobile device when it moves to the current position;

The current predicted pose is optimized based on the movement parameter information and the plurality of estimated pose information to obtain the current optimized pose, including:

The current predicted pose is optimized based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose.
The method according to claim 2, wherein the optimizing the current predicted pose based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose comprises:

Determine the movement start time and movement end time corresponding to the target movement process;

According to the image capturing time, the initial moving frame image corresponding to the moving start time and the moving ending frame image corresponding to the moving ending time are determined, and the moving initial frame image corresponding to the moving initial frame image and the moving ending frame image are respectively determined. Estimate the pose and the estimated pose at the end of the move;

Determining the moving distance as the distance between the initial estimated pose of the movement and the estimated pose at the end of the movement, and combining the multiple estimated pose information, optimize the current predicted pose to obtain the current Optimize the pose.
The method according to claim 2 or 3, wherein the determining, based on the pedestrian dead reckoning algorithm, determines the moving distance corresponding to the moving process of the target experienced by the mobile device when moving to the current position, comprising: :

Obtain the number of steps taken by the user of the mobile device during the movement of the target;

The moving distance is determined based on the number of walking steps and the step length of the user.
The method according to any one of claims 2-4, wherein the movement parameter information further comprises a movement speed;

The determining of the movement parameter information corresponding to the target movement process experienced by the mobile device when it moves to the current position further includes:

determining the movement speed corresponding to the target movement process experienced by the mobile device when it moves to the current position;

The optimizing the current predicted pose based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose includes:

The current predicted pose is optimized based on the moving distance, the moving speed, and the plurality of estimated pose information to obtain the current optimized pose.
The method according to claim 5, wherein the determining the moving speed corresponding to the target moving process experienced by the mobile device when moving to the current position comprises:

Obtain the historical motion speed of the user of the mobile device;

The movement speed is predicted based on a movement model and the historical movement speed.
The method according to any one of claims 1-6, wherein after obtaining the current optimized pose, the method further comprises:

judging whether the error of the current optimized pose is less than a preset threshold;

When the error of the current optimized pose exceeds the preset threshold, acquiring the current predicted speed and current orientation of the mobile device;

Based on the current optimized pose, the current predicted velocity, and the current orientation, real-time pose information of the mobile device is determined.
The method according to any one of claims 1-7, wherein after obtaining the current optimized pose, the method further comprises:

Based on the current optimized pose, the destination input by the user, and the stored map, a navigation route is planned, and the navigation route is displayed on the map.
The method according to claim 8, wherein the planning of a navigation route based on the current optimized pose, a destination input by the user and a stored map, and displaying the navigation route on the map, comprises:

determining the location information of the destination in the map in response to the destination input request of the user entering and leaving;

Based on the current optimized pose and the location information of the destination, determining at least one connection line between the current optimized pose and the destination;

The at least one connection line is displayed in the map.
A pose optimization device, comprising:

The pose estimation part is configured to determine the current predicted pose of the mobile device based on the first image captured by the mobile device; the first image includes the current frame image captured when the mobile device is at the current position and the movement The starting frame image taken when the device is at the starting position before the current position;

A parameter determination part, configured to determine movement parameter information corresponding to a target movement process experienced by the mobile device when moving to the current position; the target movement process is located between the starting position and the current position ;

a pose acquisition part, configured to obtain a plurality of estimated pose information of the mobile device, the plurality of estimated pose information including the estimated pose information corresponding to the starting frame image, the estimated pose information corresponding to the current frame image pose information and estimated pose information respectively corresponding to the multi-frame images between the initial frame image and the current frame image;

The pose optimization part is configured to optimize the current predicted pose based on the movement parameter information and the plurality of estimated pose information to obtain the current optimized pose.
The pose optimization apparatus according to claim 10, wherein the movement parameter information includes a movement distance;

The parameter determination part is configured to determine the moving distance corresponding to the target moving process experienced by the mobile device when moving to the current position based on a pedestrian dead reckoning algorithm;

The pose optimization part is configured to optimize the current predicted pose based on the moving distance and the plurality of estimated pose information to obtain the current optimized pose.
The pose optimization device according to claim 11, wherein,

The pose optimization part is configured to determine the movement start time and movement end time corresponding to the target movement process; and according to the image capture time, determine the movement initial frame image corresponding to the movement start time and the movement end time The movement end frame image corresponding to the time, and respectively determine the movement initial estimation pose and the movement end estimation pose corresponding to the movement initial frame image and the movement end frame image; and determine the movement distance as the movement initial estimation The distance between the pose and the estimated pose at the end of the movement, and combined with the plurality of estimated pose information, the current predicted pose is optimized to obtain the current optimized pose.
The pose optimization device according to claim 11 or 12, wherein,

The parameter determination part is configured to acquire the number of steps taken by the user of the mobile device during the movement of the target; and determine the moving distance based on the number of steps taken and the step length of the user.
The pose optimization device according to any one of claims 11 to 13, wherein the movement parameter information further includes a movement speed;

The parameter determination part is configured to determine the movement speed corresponding to the target movement process experienced by the mobile device when it moves to the current position;

The pose optimization part is configured to optimize the current predicted pose based on the moving distance, the moving speed, and the plurality of estimated pose information to obtain the current optimized pose.
The pose optimization device according to claim 14, wherein,

The parameter determination part is configured to acquire the historical movement speed of the user of the mobile device; and predict the movement speed based on the movement model and the historical movement speed.
The pose optimization device according to any one of claims 10 to 15, wherein,

The pose optimization part is configured to judge whether the error of the current optimized pose is less than a preset threshold; and in the case that the error of the current optimized pose exceeds the preset threshold, obtain the information of the mobile device. a current predicted speed and a current orientation; and determining real-time pose information of the mobile device based on the current optimized pose, the current predicted speed, and the current orientation.
The pose optimization device according to any one of claims 10 to 16, wherein the pose optimization device further comprises:

The route planning part is configured to: plan a navigation route based on the current optimized pose, the destination input by the user and the stored map, and display the navigation route on the map.
The pose optimization device according to claim 17, wherein,

The route planning part is configured to determine the location information of the destination in the map in response to a destination input request of the user entering and leaving; and based on the current optimized pose and the location information of the destination , determining at least one connecting line between the current optimized pose and the destination; and displaying the at least one connecting line in the map.
An electronic device, comprising: a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus , when the machine-readable instructions are executed by the processor, the pose optimization method according to any one of claims 1-9 is executed.
A computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the pose optimization method according to any one of claims 1-9 is executed.
A computer program, comprising computer-readable codes, in the case that the computer-readable codes are run in an electronic device and executed by a processor in the electronic device, to implement the method described in any one of claims 1 to 9 Pose optimization method.
A computer program product, when running on a computer, causes the computer to execute the pose optimization method of any one of claims 1 to 9.