TWI795885B - Visual positioning method, device and computer-readable storage medium - Google Patents

Abstract

The present disclosure discloses a visual positioning method, a device and a computer-readable storage medium, wherein the visual positioning method includes: obtaining gravity information of a camera; using the gravity information to obtain the camera pose parameters of the current image taken by the camera in the preset motion state; based on the camera pose parameters of the current image, the camera pose parameters of the image to be processed after the current image are obtained. The above solution reduces the use cost of visual positioning technology and expands the scope of use of visual positioning technology.

Description

Visual positioning method, device and computer-readable storage medium

The present invention relates to the technical field of computer vision, in particular to a visual positioning method, device and computer-readable storage medium.

With the development of electronic information technology, visual positioning technologies such as SLAM (Simultaneous Localization And Mapping, real-time positioning and map construction) have been gradually applied to automatic driving, indoor navigation, AR (Augmented Reality, augmented reality), VR (Virtual Reality , virtual reality) and other fields.

Visual positioning technologies such as SLAM complete tasks such as autonomous positioning and navigation of mobile devices by obtaining the camera pose of the mobile device, which is essentially a complex mathematical problem. At present, visual positioning technologies such as SLAM rely on sensors in hardware, and usually require sensors such as cameras, accelerometers, gravimeters, and IMUs (Inertial Measurement Units). However, in practical applications, generally only mid-to-high-end mobile devices are fully equipped with the above-mentioned sensors. Low-end mobile devices are generally equipped with fewer sensors and generally do not have an IMU, resulting in the existing visual positioning The use cost of technology is high, and the scope of use is narrow. In view of this, how to reduce the cost of using visual positioning technology and expand the scope of use of visual positioning technology has become an urgent problem to be solved.

The invention provides a visual positioning method, equipment and computer-readable storage medium.

The first aspect of the present invention provides a visual positioning method, including: obtaining the gravity information of the camera; using the gravity information to obtain the camera pose parameters of the current image captured by the camera in a preset motion state; the camera position based on the current image Pose parameters, to obtain the camera pose parameters of the image to be processed after the current image.

Therefore, by obtaining the gravity information of the camera, the gravity information is used to obtain the camera pose parameters of the current image captured by the camera in a preset motion state, and based on the camera pose parameters of the current image, the The camera pose parameters of the image to be processed can then perform visual positioning only relying on the camera and gravity information, so the cost of using the visual positioning technology can be reduced and the scope of use of the visual positioning technology can be expanded.

Among them, the gravity information includes the gravity direction information, based on the camera pose parameters of the current image, before obtaining the camera pose parameters of the image to be processed after the current image, it also includes: obtaining the feature directions of the feature points in the current image information; use the feature direction information and gravity direction information of the feature points to obtain the depth information of the feature points in the current image; based on the camera pose parameters of the current image, obtain the camera pose parameters of the image to be processed after the current image The number includes: based on the depth information of the feature points in the current image and the camera pose parameters of the current image, obtain the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed.

Therefore, by obtaining the feature direction information of the feature points in the current image, and using the feature direction information of the feature points and the gravity direction information contained in the gravity information to obtain the depth information of the feature points in the current image, it is possible to Image to initialize the depth information of the feature points in the current image and the camera pose parameters of the current image, and based on the depth information of the feature points in the current image and the camera pose parameters of the current image, the image after the current image can be obtained The depth information of the feature points in the image to be processed and the camera pose parameters of the image to be processed do not need to scan multiple frames of images for initialization, which can improve the response speed of visual positioning.

Among them, the feature direction information includes the direction vector of the feature point, the gravity direction information includes the gravity vector, and the depth information includes the depth value of the feature point; use the feature direction information and gravity direction information of the feature point to obtain the depth information of the feature point in the current image Including: performing the first preset operation on the direction vector and the gravity vector of the feature point to obtain the angle between the direction vector and the gravity vector of the feature point; performing the second preset operation on the preset height and angle of the camera to obtain the feature point the depth value.

Therefore, the feature direction information is set to include the direction vector of the feature point, the gravity direction information is set to include the gravity vector, and the depth information is set to include the depth value of the feature point, so that the first preset operation is performed on the direction vector and the gravity vector of the feature point , to obtain the angle between the direction vector of the feature point and the gravity vector, so as to perform the second preset operation on the preset height and angle of the camera, Obtaining the depth value of the feature point can help reduce the computational complexity of obtaining the depth value of the feature point.

Wherein, the first preset operation includes inner product operation, and/or, the second preset operation includes dividing the preset height by the cosine value of the included angle.

Therefore, setting the first preset operation to include the inner product operation can help reduce the complexity of obtaining the angle between the direction vector and the gravity vector, and setting the second preset operation to include dividing the preset height by the cosine of the included angle value, which can help reduce the complexity of obtaining the depth value.

Wherein, based on the depth information of the feature points in the current image and the camera pose parameters of the current image, obtaining the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed include: Use the preset pose tracking method to track the depth information of the feature points in the current image and the camera pose parameters of the current image to obtain the depth information of the feature points in the next frame of the current image and the next frame The camera pose parameters of the image; take the next frame of the image as the current image, and re-execute the depth information of the feature points in the current image and the camera pose parameters of the current image by using the preset pose tracking method Processed steps and subsequent steps.

Therefore, the depth information of the feature points in the current image and the camera pose parameters of the current image are tracked by using the preset pose tracking method, and the depth information and the next frame of the feature point in the current image are obtained. The camera pose parameters of one frame of image, so that the next frame of image is used as the current image, and the depth information of the feature points in the current image and the camera pose of the current image are re-executed using the preset pose tracking method. Parameters for tracking processing steps and subsequent steps, so that the camera pose parameters can be calculated frame by frame, which is beneficial To reduce the cumulative error of camera pose parameters.

Among them, the depth information of the feature points in the current image and the camera pose parameters of the current image are tracked by using the preset pose tracking method, and the depth information and the next frame of the feature point in the current image are obtained. The camera pose parameters of a frame of image, including: using the depth information of the feature points in the current image to determine the projection point of the feature point in the next frame of image, based on the primitive value of the feature point in the local area of the current image and the difference between the primitive values of the projection point in the local area in the next frame image to obtain the pose transformation parameters between the current image and the next frame image, using the pose transformation parameters and the camera position of the current image pose parameters to obtain the camera pose parameters of the next frame of image, use the converged 3D points to optimize the camera pose parameters of the next frame of image, obtain the probability distribution of the depth information of the feature points, and use the probability distribution to obtain The depth information of the feature points in the next image frame.

Therefore, by using the depth information of the feature points in the current image to determine the projection points of the feature points in the next frame of the image, based on the primitive values and projection points of the feature points in the local area in the current image The difference between the primitive values in the local area in the middle is used to obtain the pose transformation parameters between the current image and the next frame image, and the next frame is obtained by using the pose transformation parameters and the camera pose parameters of the current image The camera pose parameters of the image, using the converged 3D points, optimize the camera pose parameters of the next frame of image, so that the camera pose parameters can be further optimized, which is conducive to improving the accuracy of the camera pose parameters; By obtaining the probability distribution of the depth information of the feature points, and using the probability distribution, the depth information of the feature points in the next frame of image can be obtained, so that based on the distribution probability of the depth information, during the shooting process Optimize the depth information; here, the camera pose parameters and depth information are optimized separately, which reduces the amount of optimization calculation.

Wherein, the camera pose parameters include rotation parameters and displacement parameters; based on the camera pose parameters of the current image, after obtaining the camera pose parameters of the image to be processed after the current image, the method further includes: responding to the image to be processed The camera pose parameters do not meet the preset steady state conditions, and it is determined that the displacement parameters of the image to be processed cannot be obtained; and, using the primitive value of the previous frame image of the image to be processed and the camera position of the previous frame image Attitude parameters to obtain the rotation parameters of the image to be processed.

Therefore, the camera pose parameters are set to include rotation parameters and displacement parameters, and after acquiring the camera pose parameters of the image to be processed after the current image, responding that the camera pose parameters of the image to be processed do not satisfy the preset stability State conditions, determine that the displacement parameter of the image to be processed cannot be obtained, so that the rotation parameter of the image to be processed can be obtained by using the primitive value of the previous frame image of the image to be processed and the camera pose parameter of the previous frame image , and then when the camera pose parameters are inaccurate, the image primitives can be directly used to estimate the rotation parameters, which can help reduce the probability of problems such as virtual objects sticking to the screen in virtual reality caused by the inability to update the rotation parameters.

Wherein, using the primitive value of the previous frame image of the image to be processed and the camera pose parameter of the previous frame image, obtaining the rotation parameter of the image to be processed includes: using the image to be processed and the previous frame image The pose transformation parameters between perform projection transformation on at least some of the primitive points in the previous frame image, and obtain at least some of the primitive points in the projection points of the image to be processed; use at least some of the primitive points in the previous frame image Primitive values in the image and projections corresponding to at least some of the primitive points The difference between the primitive values of the shadow point in the image to be processed is used to construct an objective function about the pose transformation parameters; the pose transformation parameters obtained by solving the objective function are used to transform the camera pose parameters of the previous frame image, Get the rotation parameters of the image to be processed.

Therefore, use the pose transformation parameters between the image to be processed and the previous frame image to perform projection transformation on at least part of the primitive points in the previous frame image, and obtain the projection of at least part of the primitive points on the image to be processed points, and using the difference between the primitive values of at least some of the primitive points in the previous frame image and the primitive values of the projection points corresponding to at least some of the primitive points in the image to be processed, to construct the parameters of the pose transformation The objective function, so that the camera pose parameters of the previous frame image are transformed using the pose transformation parameters obtained by solving the objective function, and the rotation parameters of the image to be processed are obtained, so the rotation parameters can be obtained based on at least some primitive points , which can help to reduce the amount of calculation for calculating the rotation parameters.

Wherein, using the pose transformation parameters between the image to be processed and the previous frame image to perform projection transformation on at least part of the primitive points in the previous frame image, to obtain the projection of at least part of the primitive points on the image to be processed Before the point, the method also includes: downsampling the previous frame image to obtain a thumbnail image of the previous frame image; using the pose transformation parameters between the image to be processed and the previous frame image to be processed Performing projection transformation on at least some of the primitive points in the image, and obtaining the projection points of at least some of the primitive points on the image to be processed includes: using the pose transformation parameters between the image to be processed and the previous frame image to shorten the The primitive points in the image are subjected to projection transformation to obtain the projection points of the primitive points in the thumbnail image on the image to be processed.

Therefore, by down-sampling the previous frame image, the thumbnail image of the previous frame image is obtained, and the thumbnail image is transformed using the pose transformation parameters between the previous frame images of the image to be processed Projective transformation is performed on the primitive points in the thumbnail image to obtain the projection points of the primitive points in the thumbnail image on the image to be processed, so as to construct and solve the subsequent objective function, which can help reduce the amount of calculation for calculating the rotation parameters.

Wherein, after obtaining the rotation parameter of the image to be processed by using the primitive value of the previous frame image of the image to be processed and the camera pose parameter of the previous frame image, the method further includes: detecting the current acceleration information of the camera, And determine whether the acceleration information is in the preset motion state; if yes, re-execute the step of obtaining the gravity information of the camera and the subsequent steps; if not, re-execute the step of detecting the current acceleration information of the camera and the subsequent steps.

Therefore, after obtaining the rotation parameters of the image to be processed, further detect the current acceleration information of the camera, and judge whether the acceleration information is in the preset motion state, so that in the case of the preset motion state, re-execute the acquisition of the camera's gravity information The step and the subsequent steps, and re-execute the step of detecting the current acceleration information of the camera and the subsequent steps without being in the preset motion state, which can help improve the robustness of the visual positioning.

Among them, the gravity information includes the gravity direction information, and the camera pose parameters include rotation parameters and displacement parameters. Using the gravity information, obtaining the camera pose parameters of the current image captured by the camera in the preset motion state includes: using the gravity direction information to obtain The rotation angles of the camera relative to the x coordinate axis, y coordinate axis and z coordinate axis of the world coordinate system respectively; among them, the camera rotates according to the rotation angle The direction of gravity after the degree rotation is the same as the opposite direction of the z coordinate axis; use the rotation angle to obtain the rotation parameter, and set the displacement parameter to a preset value.

Therefore, by using the gravity direction information, the rotation angles of the camera relative to the x coordinate axis, y coordinate axis and z coordinate axis of the world coordinate system are obtained respectively, and the gravity direction after the camera is rotated according to the rotation angle is the same as the opposite direction of the z coordinate axis, thus using the rotation angle , to obtain the rotation parameters, and set the displacement parameters to preset values, the rotation parameters can be obtained through gravity alignment, and then the camera pose parameters can be initialized, which is beneficial to reduce the calculation amount of camera pose parameter initialization.

Wherein, the origin of the world coordinate system is the position where the camera captures the current image, and the default value is 0.

Therefore, setting the origin of the world coordinate system to the position where the camera captures the current image, and setting the preset value to 0 can help reduce the complexity of initializing the displacement parameters.

Wherein, the preset motion state is a static state or a uniform motion state; and/or, the gravity information is obtained by using the acceleration information of the camera in the preset state.

Therefore, setting the preset motion state to a static state or a uniform motion state can help improve the accuracy of initializing the camera pose parameters of the current image; and using the acceleration information of the camera in the preset state to obtain gravity information can only Using the accelerometer to obtain the gravity information can further reduce the cost of using the visual positioning technology and expand the application range of the visual positioning technology.

The second aspect of the present invention provides a visual positioning device, including: The gravity information acquisition part, the first pose acquisition part and the second pose acquisition part, the gravity information acquisition part is configured to obtain the gravity information of the camera; the first pose acquisition part is configured to use the gravity information to obtain the camera in the preset motion state The camera pose parameters of the current image captured below; the second pose acquisition part is configured to acquire the camera pose parameters of the image to be processed after the current image based on the camera pose parameters of the current image.

The third aspect of the present invention provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory, so as to realize the visual positioning method in the first aspect above.

A fourth aspect of the present invention provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the visual positioning method in the above-mentioned first aspect is implemented.

The fifth aspect of the present invention provides a computer program, including computer-readable codes, when the computer-readable codes run in an electronic device and are executed by a processor in the electronic device, the above-mentioned first aspect can be implemented. visual positioning method.

In the above solution, by obtaining the gravity information of the camera, the gravity information is used to obtain the camera pose parameters of the current image captured by the camera in the preset motion state, and based on the camera pose parameters of the current image, after obtaining the current image The camera pose parameters of the image to be processed, and then can only rely on the camera and gravity information for visual positioning, so the cost of using visual positioning technology can be reduced, and the scope of use of visual positioning technology can be expanded.

60: Visual positioning device

61: Gravity information acquisition part

62: The first pose acquisition part

63: The second pose acquisition part

70: Electronic equipment

71: Memory

72: Processor

80: computer readable storage medium

801: Program instruction

S11~S13, S131~S133, S41~S47, S451~S453: steps

h : the preset height of the camera

:特徵點的重力向量

: Gravity vector of feature point

α: The angle between the direction vector of the feature point and the gravity vector

:特徵點的方向向量

: The direction vector of the feature point

Fig. 1 is a schematic flow diagram of an embodiment of the visual positioning method of the present invention; Fig. 2 is a schematic diagram of an embodiment of obtaining depth information; Fig. 3 is a schematic flow diagram of an embodiment of step S13 in Fig. 1; Fig. 4 is a visual tracking method of the present invention A schematic flow diagram of another embodiment; FIG. 5 is a schematic flow diagram of an embodiment of step S45 in FIG. 4; FIG. 6 is a schematic diagram of the frame of an embodiment of the visual positioning device of the present invention; FIG. Schematic diagram; FIG. 8 is a schematic diagram of the framework of an embodiment of the computer-readable storage medium of the present invention.

The solutions of the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

In the following description, for purposes of illustration rather than limitation, details such as specific system structures, interfaces, and techniques are set forth in order to provide a thorough understanding of the present invention.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship. In addition, "many" in this article means two or more than two.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of an embodiment of the visual positioning method of the present invention. The visual positioning method may include the following steps.

Step S11: Obtain the gravity information of the camera.

In the embodiment of the present invention, the execution subject of the visual positioning method may be a visual positioning device. For example, the visual positioning method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (User Equipment, UE), mobile devices, user terminals, terminals, cellular phones, wireless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. In some possible implementation manners, the visual positioning method may be implemented by calling a computer-readable instruction stored in a memory by a processor.

In the embodiment of the present invention, the camera may be integrated into a mobile device; the mobile device may include but not limited to: a mobile phone, a tablet computer, a robot, and the like. Here, the steps in the embodiments of the present invention and the following disclosed embodiments may be executed by a mobile device, and a visual positioning device is provided in the mobile device. In addition, the mobile device may also be integrated with other sensors, such as an accelerometer, a gravity meter, an IMU, etc., which may be set according to actual application scenarios, and are not limited here. For example, due to cost constraints, low-end mobile devices can only integrate cameras and accelerometers; or, they can also integrate cameras and gravimeters. In addition, mid-to-high-end mobile devices can also integrate cameras, accelerometers, IMUs, etc. Here No limit.

In an implementation scenario, the gravity information can be obtained by using the acceleration information of the camera in a preset motion state, so that the gravity information can be obtained only by the accelerometer without an IMU. Wherein, the preset motion state is a static state or a uniform motion state. For example, the difference between the detected camera acceleration and the acceleration of gravity in the preset motion state is within the preset range. For example, the acceleration of gravity is 9.8m/s ² , and the preset range is 0~1m/s ² . When the acceleration of the camera is 10m/s ² , it can be considered that the camera is in a preset motion state, and the preset range can be set according to actual application needs, which is not limited here. In addition, when the mobile device is integrated with a gravimeter, it is also possible to obtain gravity information directly through the gravimeter without an IMU.

。 In an implementation scenario, the mobile device can also judge whether it is in a static state or a state of constant motion according to the detected camera acceleration. state of motion. Among them, when an accelerometer is configured, the root of the sum of the squares of the acceleration components (such as a _x , a _y , a _z ) of the accelerometer on the three axes can be calculated as the camera acceleration a _camera , namely

.

In another implementation scenario, when it is detected that the camera is not in the preset motion state, the mobile device may perform detection again until it is detected that the camera is in the preset motion state. Here, the frequency of detection can be consistent with the frequency of camera shooting. For example, if the camera takes 25 images per second, it can detect whether it is in the preset motion state 25 times per second. For this, it can be set according to actual application needs , is not limited here.

+

+

)，並將向量和作為重力向量，或者，還可以將與向量和同方向的單位向量作為重力向量，對於重力向量，可以根據實際應用需要進行設置，在此不做限定。 In an implementation scenario, the gravity information of the camera may include gravity direction information; wherein, the gravity direction information may include a gravity vector. In an implementation scenario, when an accelerometer is configured, the vector sum of the acceleration components of the accelerometer on three axes (ie

+

+

), and the vector sum is used as a gravity vector, or a unit vector in the same direction as the vector sum can also be used as a gravity vector. For the gravity vector, it can be set according to actual application needs, and is not limited here.

Step S12: Obtain the camera pose parameters of the current image captured by the camera in a preset motion state by using the gravity information.

In an implementation scenario, the camera pose parameters can include displacement parameters and rotation parameters, so that the gravity direction information can be used to obtain the rotation angles of the camera relative to the x coordinate axis, y coordinate axis, and z coordinate axis of the world coordinate system, and the camera rotates according to the rotation angle The direction of gravity after rotation is the same as the opposite direction of the z coordinate axis, and then the rotation parameter can be obtained by using the rotation angle, and the displacement parameter can be set to a preset value, so the camera pose parameters of the current image can be simplified and initialized through gravity alignment, reducing Calculations.

，則相機相對於世界座標系x座標軸的旋轉參數R _x ，相對於世界座標系y座標軸的旋轉參數R _y ，相對於世界座標系z座標軸的旋轉參數R _z 可以分別表示為：

In an implementation scenario, the rotation angle of the camera relative to the x coordinate axis can be expressed as θ, the rotation angle of the camera relative to the y coordinate axis can be expressed as φ, and the rotation angle of the camera relative to the z coordinate axis can be expressed as

, then the rotation parameter R _x of the camera relative to the x coordinate axis of the world coordinate system, the rotation parameter R _y relative to the y coordinate axis of the world coordinate system, and the rotation parameter R _z relative to the z coordinate axis of the world coordinate system can be expressed as:

The rotation parameter R can be obtained from the rotation parameter R _x relative to the x coordinate axis of the world coordinate system, the rotation parameter R _y relative to the y coordinate axis of the world coordinate system, and the rotation parameter R _z relative to the z coordinate axis of the world coordinate system; here, The product of the above rotation parameter R _x , rotation parameter R _y and rotation parameter R _z is used as the rotation parameter R , that is, the rotation parameter R can be expressed as: R=R _x R _y R _z  …(2)

In another implementation scenario, the mobile device can use the origin of the world coordinate system as the position where the camera takes the current image, that is, the displacement of the camera relative to the x-coordinate axis, y-coordinate axis, and z-coordinate axis is all 0, so the preset Set the value to 0, that is, the displacement parameter can be set to 0.

Step S13: Based on the camera pose parameters of the current image, acquire the camera pose parameters of the image to be processed after the current image.

In an implementation scenario, the mobile device can scan and acquire multiple frames of images, and use triangulation to process the matching feature points in the current image and the adjacent images of the current image to obtain the depth of the matching feature points information; among them, the depth information can include the depth value of the feature point, so that the three-dimensional coordinates of the feature point in the world coordinate system can be obtained by using the calculated depth value; and then the next frame image of the current image and the current image The pose transformation parameters between images, reproject the three-dimensional coordinates of the feature points to the next frame of image, and obtain the projection point in the next frame of image; thus use the primitive value of the projection point in the next frame of image and the corresponding The difference between the feature points in the primitive values of the current image constructs an objective function about the pose transformation parameters; by minimizing the objective function, the pose transformation parameters can be obtained, and using the pose transformation parameters and the current image The camera pose parameters of the image can be used to obtain the camera pose parameters of the next frame of image; and so on, the mobile device can obtain frame by frame Get the camera pose parameters of the image to be processed after the current image. In an implementation scenario, the triangulation method refers to observing the same 3D point at different positions, and knowing the 2D projection points of the 3D points observed at different positions, using the triangular relationship to restore the depth information of the 3D point , which will not be repeated here.

In another implementation scenario, in order to reduce the additional response time caused by scanning multiple frames of images and improve the response speed of visual positioning, the mobile device can also obtain the feature direction information and gravity direction information of the feature points in the current image. The depth information of the feature points in the current image, so that based on the depth information of the feature points in the current image and the camera pose parameters of the current image, the depth information and the depth information of the feature points in the image to be processed after the current image are obtained The camera pose parameters of the image can then initialize the depth information with only the current image, so it can avoid scanning multiple frames of images, which is beneficial to improve the response speed of visual positioning.

In an implementation scenario, the feature direction information can include the direction vector of the feature point, and the gravity direction information can include the gravity vector; wherein, the direction vector and the gravity vector can be unit vectors, and the depth information can include the depth value of the feature point; the feature point can include The primitive points that describe the image features, for example, can include the contour edge primitive points in the image, the primitive points with sudden changes in the primitive value, etc.; here, for the feature points, you can set them according to actual needs, and do not do it here limited. For example, FAST (Features from Accelerated Segment Test, features of accelerated segment test), BRIEF (Binary Robust Independent Elementary Features, binary robust independent feature descriptor), SIFT (Scale Invariant Feature Transform (Scale Invariant Feature Transform), ORB and other detection methods to obtain feature points and direction vectors of feature points. For this, feature point detection methods can be selected according to actual application needs, and are not limited here.

和重力向量

進行第一預設運算，得到特徵點的方向向量

和重力向量

之間的夾角，第一預設運算可以包括內積運算，即方向向量

和重力向量

之間的夾角α可以表示為：

In another implementation scenario, please refer to FIG. 2. FIG. 2 is a schematic diagram of an embodiment of obtaining depth information. The mobile device can calculate the direction vector of the feature point

and the gravity vector

Perform the first preset operation to obtain the direction vector of the feature point

and the gravity vector

The angle between , the first preset operation can include the inner product operation, that is, the direction vector

and the gravity vector

The angle α between can be expressed as:

After obtaining the included angle α, a second preset operation can be performed on the preset height h of the camera and the included angle α to obtain the depth value z of the feature point. The second preset operation includes dividing the preset height h by the cosine value of the included angle, The preset height h can be set according to the actual application situation. Taking the AR application as an example, it can be set according to the size of the virtual object. For example, if the virtual object is a pet such as a cat or a dog of a normal size, the preset height can be set to 0.5 m ~ 1 m, other applications can be set according to the actual situation, no more examples here. Among them, the depth value z can be expressed as:

In an implementation scenario, the steps in the embodiments of the present invention and the following disclosed embodiments can be integrated into indoor navigation, automatic driving, AR, VR and other applications and webpages running on mobile devices. The actual application needs to be set, which is not limited here.

In the above solution, the mobile device obtains the gravity information of the camera, thereby using the gravity information to obtain the camera pose parameters of the current image captured by the camera in a preset motion state, and based on the camera pose parameters of the current image, obtains the current image The camera pose parameters of the image to be processed after the imaging can only rely on the camera and gravity information for visual positioning, so the cost of using visual positioning technology can be reduced and the scope of use of visual positioning technology can be expanded.

Please refer to FIG. 3 . FIG. 3 is a schematic flowchart of an embodiment of step S13 in FIG. 1 . Figure 3 is based on the depth information of the feature points in the current image and the camera pose parameters of the current image to obtain the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed A schematic flow chart of an embodiment. Wherein, S13 may include the following steps.

Step S131: Use the preset pose tracking method to track the depth information of the feature points in the current image and the camera pose parameters of the current image to obtain the depth information and the depth information of the feature points in the next frame of the current image. The camera pose parameters of the next frame image.

The preset pose tracking method can be set according to actual application needs. In the embodiment of the present invention, the preset pose tracking method may include steps such as sparse image alignment, feature point alignment, and pose optimization, so as to obtain the camera pose parameters of the next frame of image through the above steps. In addition, the preset The pose tracking method may also include a map point optimization step, so as to obtain the depth information of the feature points in the next frame image through the map point optimization and other steps.

In an implementation scenario, when performing sparse image alignment, the mobile device may first use the depth information of the feature points in the current image to determine the projection points of the feature points in the next frame of image , may include: back-projecting the first coordinate information ( u , du ) including the feature point coordinate u and the feature point depth value du to the 3D space by using the 2D-to-3D back-projection function π ^-1 , to obtain the 3D feature point Coordinate π ^-1 ( u , du ), and use the pose transformation parameter T between the current image k -1 and the next frame image k and the projection function π from 3D to 2D to transform the 3D of the feature points of the current image The coordinate π ^-1 ( u , du ) is projected to the next frame of image k , and the projected point π( T ‧π ^-1 ( u , du )) of the feature point in the next frame of image k is obtained; thus the projected point is obtained in the following The primitive value W _k (π( T ‧π ^-1 ( u , du ))) of the local area in a frame of image k and the primitive value W of the local area of the feature point corresponding to the current image k -1 There is a difference between _{k -1} ( u ); based on this difference, the pose transformation parameters between the current image k -1 and the next frame image k can be obtained. Among them, the local area can be a rectangular area (such as 3 * 3 area, 4 * 4 area, 8 * 8 area, etc.) centered on the feature point (or projection point), as shown in the following formula: r ( T , u )= W _k (π( T ‧π ^-1 ( u , du )))- W _{k -1} ( u )...(5)

It should be noted that there are generally multiple feature points, so the above differences can be calculated for multiple feature points and summed to further construct the objective function, as shown in the following formula:

表示目標函數，其中，ρ表示魯棒函數，用於降低雜訊影響，∥‧∥表示範數運算，

表示以位姿變換參數T為優化對象最小化目標函數，T _k,k-1表示求解目標函數所得到的位姿變換參數。 In the above formula (6),

Represents the objective function, where ρ represents a robust function for reducing the impact of noise, ∥‧∥ represents a norm operation,

Indicates that the objective function is minimized with the pose transformation parameter T as the optimization object, and T _{k,k -1} represents the pose transformation parameters obtained by solving the objective function.

After calculating the pose transformation parameter T _{k,k -1} , the mobile device can use the pose transformation parameter T _{k,k -1} and the camera pose parameter T _{k -1} of the current image k -1 to obtain the next The camera pose parameter T _k of the frame image. Here, the pose transformation parameter T _{k,k -1} can be multiplied by the camera pose parameter T _k -1 of the current image k -1 to obtain the camera pose parameter T _k of the next frame image.

In addition, in order to reduce the computational complexity of sparse image alignment, the mobile device can also down-sample the current image k -1 and the next frame image k to obtain the current image k -1 and the next frame image k Pyramid image, and take a layer of images or multi-layer images whose resolution is the preset resolution in the pyramid image to perform the above sparse image alignment processing, so that the computational complexity can be reduced.

；從 而獲取投影點

在下一幀圖像中局部區域圖元值W _k (

)，進而可以利用局部區域圖元值W _r (u _i )和局部區域圖元值W _k (

)，構建關於投影點

的目標函數，參閱下式：

In an implementation scenario, the above operation of sparse image alignment will inevitably bring cumulative errors, resulting in low accuracy of the camera pose parameter T _k of the acquired next frame image. In order to improve the accuracy, the mobile device can use the converged 3D points (for example, 3D points in the 3D model) to optimize the camera pose parameter T _k of the next frame of image. Here, the converged three-dimensional points can be used for matching and alignment to obtain the projection points, and then the projection points can be used to optimize the camera pose parameters T _k of the next frame image obtained by the above sparse image alignment. The step of feature point alignment can include : Select the 3D point that can be projected to the next frame of image k from the converged 3D points as the target 3D point, and select the earliest captured image among the images that the target 3D point can project from the captured images image, as a reference image, and obtain the primitive value W _r ( u _i ) of the local area of the target 3D point in the reference image, and use the camera pose parameter T _k of the next frame image roughly estimated above to convert the target 3D point Project to the next frame image to get the projection point of the target 3D point in the next frame image

; to get the projected point

In the next frame image, the primitive value W _k of the local area (

), and then the local area primitive value W _r ( u _i ) and the local area primitive value W _k (

), constructing about the projected point

The objective function of , see the following formula:

表示目標函數，其中，∥‧∥表示範數運算，A _i 表示仿射變換矩陣，用於補償不同視角帶來的圖像扭曲，

表示以投影點

的位置為優化對象最小化目標函數。 In the above formula (7),

Represents the objective function, where ∥‧∥ represents the norm operation, A _i represents the affine transformation matrix, which is used to compensate the image distortion caused by different viewing angles,

expressed as a projected point

The location of is the optimization object to minimize the objective function.

之後，移動設備可以基於上述特徵點對齊所得到的投影點

可以對上述稀疏圖像對齊所得到的下一幀圖像的相機位姿參數T _k 進行優化，最終優化得到下一幀圖像的相機位姿參數T _w,k 。位姿優化的步驟可以包括：利用下一幀圖像的相機位姿參數T _w,k 和三維到二維的投影函數π將目標三維點 ^w p _i 重投影至下一幀圖像k中，得到投影點

，並利用投影點

和特徵點對齊步驟中優化得到的下一幀圖像的投影點

之間的位置差異，構建關於相機位姿參數T _w,k 目標函數，參閱下式：

get the projected point

Afterwards, the mobile device can align the resulting projected points based on the above feature points

The camera pose parameter T _k of the next frame of image obtained by the above sparse image alignment can be optimized, and finally the camera pose parameter T _w,k of the next frame of image is optimized. The step of pose optimization may include: using the camera pose parameter T _w,k of the next frame of image and the projection function π from 3D to 2D to re-project the target 3D point ^w p _i into the next frame of image k , get projected point

, and using the projected point

The projection point of the next frame image optimized in the feature point alignment step

The position difference between , to construct the objective function about the camera pose parameters T _w,k , refer to the following formula:

表示目標函數，其中，ρ表示魯棒函數，用於降低雜訊影響，∥‧∥表示範數運算，

表示以T _w,k 為優化對象最小化目標函數。 In the above formula (8),

Represents the objective function, where ρ represents a robust function for reducing the impact of noise, ∥‧∥ represents a norm operation,

Indicates to minimize the objective function with T _w,k as the optimization object.

By solving the objective function shown in formula (8), the camera pose parameter T _w,k of the next frame image can be finally obtained.

In an implementation scenario, the essence of map point optimization is to optimize the inverse depth (ie, the reciprocal of the depth value) of the corresponding position on the reference image where the 3D point is observed for the first time. Here, the probability distribution of the depth information of the feature point can be obtained, and the interior point probability γ and the inverse depth value z of the feature point approximately conform to the Beta Gaussian Mixture Model Distribution (Beta Gaussian Mixture Model Distribution), refer to the following formula:

表示逆深度高斯分佈的均值和方差。在得到概率分佈之後，移動設備可以利用獲取到的概率分佈，得到下一幀圖像中特徵點的深度資訊。例如，當逆深度高斯分佈的方差

小於一預設深度範圍(如，1/200)，則可以認為深度值收斂，取此時逆深度高斯分佈的均值μ _k 的倒數作為特徵點的深度值，從而可以在拍攝過程中，對特徵點的深度值不斷進行優化。 Among them, the above formula (9) represents the probability distribution after the kth observation of a certain feature point p , a _k , b _k represent the parameters of the Beta distribution, μ _k ,

Represents the mean and variance of the inverse depth Gaussian distribution. After obtaining the probability distribution, the mobile device can use the obtained probability distribution to obtain the depth information of the feature points in the next frame of image. For example, when the variance of the inverse depth Gaussian distribution

is less than a preset depth range (eg, 1/200), it can be considered that the depth value converges, and the reciprocal of the mean value μ _k of the inverse depth Gaussian distribution at this time is taken as the depth value of the feature point, so that during the shooting process, the feature The depth value of the point is continuously optimized.

Step S132: Use the next frame of image as the current image.

After obtaining the camera pose parameters of the next frame of image and the depth information of feature points, the mobile device can use the next frame of image as the current image, and re-execute the above step S131 and subsequent steps, so that the image can be calculated frame by frame. The camera pose parameters of the image and the depth information of the feature points in the image.

Step S133: Re-execute step S131 and subsequent steps.

Different from the foregoing embodiments, the mobile device uses the preset pose to track the The method tracks the depth information of the feature points in the current image and the camera pose parameters of the current image, and obtains the depth information of the feature points in the next frame of the current image and the camera pose of the next frame of the image. Parameters; so that the next frame image is used as the current image, and the steps of using the preset pose tracking method to track the depth information of the feature points in the current image, the camera pose parameters of the current image, and the follow-up Steps, and then the camera pose parameters can be calculated frame by frame, which is beneficial to reduce the cumulative error of the camera pose parameters.

Please refer to FIG. 4 . FIG. 4 is a schematic flowchart of another embodiment of a visual tracking method according to the present invention. The visual tracking method may include the following steps.

Step S41: Obtain the gravity information of the camera.

Please refer to the relevant steps in the foregoing embodiments.

Step S42: Obtain the camera pose parameters of the current image captured by the camera in a preset motion state by using the gravity information.

Please refer to the relevant steps in the foregoing embodiments.

Step S43: Based on the camera pose parameters of the current image, acquire the camera pose parameters of the image to be processed after the current image.

Please refer to the relevant steps in the foregoing embodiments.

Step S44: Determine whether the camera pose parameters of the image to be processed meet the preset stable state condition, if not, execute step S45, and if yes, execute step S46.

The preset steady-state conditions may include at least one of the following: the camera pose parameters have no abnormal values, and the difference between the camera pose parameters of the image to be processed and the camera pose parameters of the previous frame of the image to be processed is within a preset range Inside. In one implementation scenario, outliers may include displacement parameters greater than a displacement threshold, Rotation parameters greater than the rotation threshold; wherein, the displacement threshold, the rotation threshold and the preset range can be set according to actual application needs, and are not limited here.

Step S45: Determine that the displacement parameter of the image to be processed cannot be obtained, and use the primitive value of the image in the previous frame of the image to be processed and the camera pose parameter of the image in the previous frame to obtain the rotation parameter of the image to be processed .

In the actual application process, factors such as fast movement and drastic changes in lighting conditions will lead to inaccurate camera pose parameters, resulting in inaccurate visual positioning. In order to improve the robustness of visual positioning, when judging the camera pose of an image When the parameters do not meet the preset stable state conditions, the mobile device may determine that the camera pose parameters obtained through the above steps are inaccurate, especially the displacement parameters. Therefore, in order to reduce the problem of sticking virtual objects on the screen caused by the failure to update the rotation parameters, the mobile device can use the primitive values of the previous frame image of the image to be processed and the camera pose parameters of the previous frame image to obtain the Handles the image's rotation parameters, thus keeping the rotation parameters updated.

In an implementation scenario, please refer to FIG. 5 , which is a schematic flowchart of an embodiment of step S45 in FIG. 4 . Wherein, S45 may include the following steps.

Step S451: Use the pose transformation parameters between the image to be processed and the previous frame of image to perform projective transformation on at least some of the primitive points in the previous frame of image, to obtain at least some of the primitive points in the image to be processed projected point.

For the convenience of description, the image to be processed can be denoted as k , the previous frame image can be denoted as k -1 , the pose transformation parameters can be denoted as T _{k,k -1} , and at least some primitives in the previous frame image The two-dimensional coordinates of a point are expressed as u , and the depth value of at least some primitive points is expressed as du , the back projection function from 2D to 3D can be expressed as π ^-1 , and the projection function from 3D to 2D can be expressed as π, then the projection The point can be expressed as π( T _{k,k -1} ‧π ^-1 ( u , du )). Here, reference can be made to the relevant steps in the foregoing embodiments, which will not be repeated here.

In an implementation scenario, in order to reduce the computational complexity, the mobile device may also downsample the previous frame image to obtain a thumbnail image of the previous frame image (eg, 40*30 or smaller image) , so that the primitive points in the thumbnail image are projectively transformed using the pose transformation parameters between the image to be processed and the previous frame image, and the primitive points in the thumbnail image are in the position of the image to be processed projected point. In another implementation scenario, in order to reduce the computational complexity, the mobile device can also project the primitive points in the thumbnail image onto the unit sphere, that is, the depth values of the primitive points in the thumbnail image can be uniformly set is 1. In addition, the depth value can also be uniformly set to other values according to actual application needs, which is not limited here.

Step S452: Using the difference between the primitive values of at least some of the primitive points in the previous frame image and the primitive values of the projected points corresponding to at least some of the primitive points in the image to be processed, construct parameters for pose transformation the objective function of .

In an implementation scenario, the mobile device may use at least some of the primitive points in the local area primitive value W _{k -1} ( u ) in the previous frame image, and at least some of the primitive points corresponding to the projection point π( T _{k, k -1} ‧π ^-1 ( u , du )) in the image to be processed local area primitive value W _k (π( T _{k,k -1} ‧π ^-1 ( u , du ))) difference, construct Objective function with respect to pose transformation parameters.

In another implementation scenario, when down-sampling the previous frame image, the mobile device can use the primitive values of the primitive points in the thumbnail image and the projection points corresponding to these primitive values in the image to be processed The difference between the primitive values in the image is used to construct the objective function with respect to the pose transformation parameters.

Wherein, for the objective function, reference may be made to the relevant steps in the foregoing embodiments, which will not be repeated here.

Step S453: Use the pose transformation parameters obtained by solving the objective function to transform the camera pose parameters of the previous frame image to obtain the rotation parameters of the image to be processed.

The above objective function is optimized and solved. During the optimization and solution process, the mobile device can only optimize the rotation parameters, so that the pose transformation parameters obtained by the solution are used to transform the camera pose parameters of the previous frame image, and the image to be processed is obtained. The camera pose parameters of the image, and the rotation parameters in the camera pose parameters are extracted as the rotation parameters of the image to be processed.

In an implementation scenario, in order to improve the robustness of visual positioning, the mobile device can continue to detect the current acceleration information of the camera after obtaining the rotation parameters of the image to be processed, and determine whether the acceleration information is in the preset motion state, and obtain the acceleration The information and the steps of judging whether the acceleration information is in the preset motion state can refer to the related steps in the foregoing disclosed embodiments, and will not be repeated here. If it is in the preset motion state, it can be considered that the camera is in a static state or a uniform motion state at this time, and the steps of obtaining the gravity information of the camera and subsequent steps can be re-executed. If it is not in the preset motion state, it can be considered that the camera If you are still in the state of vigorous exercise, you can re-execute the steps of detecting the current acceleration information of the camera and next steps. By repeatedly detecting the current acceleration information of the camera in the case of inaccurate visual positioning, and judging whether the acceleration information is in the preset motion state, and re-executing the steps of obtaining the gravity information of the camera and in the preset motion state As a subsequent step, the robustness of visual localization can be improved.

Step S46: Take the image to be processed as the current image.

After obtaining the rotation parameters of the image to be processed, the mobile device can use the image to be processed as the current image, and re-execute the above-mentioned camera pose parameters based on the current image to obtain the camera of the image to be processed after the current image The step of pose parameters and subsequent steps to keep updating the rotation parameters in the case of severe motion or drastic changes in lighting conditions.

Step S47: Re-execute step S43 and subsequent steps.

Different from the previous embodiments, the camera pose parameters are set to include rotation parameters and displacement parameters, and after the mobile device acquires the camera pose parameters of the image to be processed after the current image, the response to the camera pose of the image to be processed The parameters do not meet the preset steady state conditions, and it is determined that the displacement parameters of the image to be processed cannot be obtained, so that the image to be processed can be obtained by using the primitive value of the previous frame image of the image to be processed and the camera pose parameters of the previous frame image. Process the rotation parameters of the image, and then directly use the image primitives to estimate the rotation parameters when the camera pose parameters are inaccurate, which can help avoid problems such as virtual object sticking in virtual reality caused by the inability to update the rotation parameters. screen etc.

Please refer to FIG. 6 . FIG. 6 is a schematic frame diagram of an embodiment of a visual positioning device 60 of the present invention. The visual positioning device 60 includes a gravity information acquisition part 61, a first pose acquisition part 62 and a second pose acquisition part 63, The gravity information acquisition part 61 is used to obtain the gravity information of the camera; the first pose acquisition part 62 is configured to use the gravity information to obtain the camera pose parameters of the current image captured by the camera in a preset motion state; the second pose acquisition Part 63 is configured to acquire camera pose parameters of images to be processed after the current image based on the camera pose parameters of the current image.

In the above solution, by obtaining the gravity information of the camera, the gravity information is used to obtain the camera pose parameters of the current image captured by the camera in the preset motion state, and based on the camera pose parameters of the current image, after obtaining the current image The camera pose parameters of the image to be processed, and then can only rely on the camera and gravity information for visual positioning, so the cost of using visual positioning technology can be reduced, and the scope of use of visual positioning technology can be expanded.

In some disclosed embodiments, the gravity information includes gravity direction information, and the visual positioning device 60 further includes a feature direction acquisition part configured to acquire feature direction information of feature points in the current image, and the visual positioning device 60 also includes a depth information acquisition part , configured to use the feature direction information and gravity direction information of the feature points to obtain the depth information of the feature points in the current image, and the second pose acquisition part 63 is also configured to be based on the depth information of the feature points in the current image and the current image The camera pose parameters, obtain the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed.

Different from the foregoing embodiments, by obtaining the feature direction information of the feature points in the current image, and using the feature direction information of the feature points and the gravity direction information included in the gravity information, the depth information of the feature points in the current image can be obtained, so it can Initialize the feature points in the current image only based on the current image Depth information and the camera pose parameters of the current image, and based on the depth information of the feature points in the current image and the camera pose parameters of the current image, the depth information of the feature points in the image to be processed after the current image can be obtained And the camera pose parameters of the image to be processed, without scanning multiple frames of images for initialization, which can improve the response speed of visual positioning.

In some disclosed embodiments, the feature direction information includes the direction vector of the feature point, the gravity direction information includes the gravity vector, the depth information includes the depth value of the feature point, and the depth information acquisition part includes a first operator part configured to Perform the first preset operation on the direction vector and the gravity vector to obtain the angle between the direction vector and the gravity vector of the feature point. The second operator part of the depth information acquisition part is configured to perform a second preset on the preset height and angle of the camera. Set the operation to get the depth value of the feature point.

Different from the above-mentioned embodiments, the feature direction information is set to include the direction vector of the feature point, the gravity direction information is set to include the gravity vector, and the depth information is set to include the depth value of the feature point, so that the direction vector and the gravity vector of the feature point are set for the first time. The first preset operation obtains the angle between the direction vector and the gravity vector of the feature point, so as to perform the second preset operation on the preset height and angle of the camera to obtain the depth value of the feature point, so it can help reduce the cost of obtaining the feature point Computational complexity of depth values.

In some disclosed embodiments, the first preset operation includes an inner product operation, and/or, the second preset operation includes dividing the preset height by the cosine of the included angle.

Different from the foregoing embodiments, setting the first preset operation to include the inner product operation can help reduce the complexity of obtaining the angle between the direction vector and the gravity vector, and setting the second preset operation to include dividing the preset height by by The cosine value of the included angle can help reduce the complexity of obtaining the depth value.

In some disclosed embodiments, the second pose acquisition part 63 includes a pose tracking subsection configured to use a preset pose tracking method to carry out the depth information of the feature points in the current image and the camera pose parameters of the current image Tracking processing, to obtain the depth information of the feature points in the next frame image of the current image and the camera pose parameters of the next frame image, the second pose acquisition part 63 includes a repeat execution subsection, configured to convert the next frame The image is used as the current image, and the steps of tracking the depth information of the feature points in the current image and the camera pose parameters of the current image by using the preset pose tracking method and subsequent steps are re-executed.

Different from the above-mentioned embodiments, the depth information of the feature points in the current image and the camera pose parameters of the current image are tracked by using the preset pose tracking method, and the feature points in the next frame of the current image are obtained. Depth information and the camera pose parameters of the next frame image, so that the next frame image is used as the current image, and the depth information of the feature points in the current image, the current image are re-executed using the preset pose tracking method The steps of tracking and subsequent steps of the camera pose parameters, and then the camera pose parameters can be calculated frame by frame, which is conducive to reducing the cumulative error of the camera pose parameters.

In some disclosed embodiments, the pose tracking subpart includes: a feature point projection part configured to use the depth information of the feature points in the current image to determine the projection points of the feature points in the next frame image; pose transformation parameter calculation The part is configured to obtain the difference between the current image and the next frame image based on the difference between the primitive value of the feature point in the local area in the current image and the primitive value of the projected point in the local area in the next frame image. The pose transformation parameters of ; the camera position The pose parameter calculation part is configured to use the pose transformation parameters and the camera pose parameters of the current image to obtain the camera pose parameters of the next frame image; the camera pose parameter optimization part is configured to use the converged 3D points, Optimize the camera pose parameters of the next frame image; the depth information acquisition part is configured to obtain the probability distribution of the depth information of the feature point, and use the probability distribution to obtain the depth information of the feature point in the next frame image.

Different from the previous embodiments, by using the depth information of the feature points in the current image, the projection points of the feature points in the next frame of image are determined, so that based on the primitive values and projection points of the feature points in the local area of the current image, the The difference between the primitive values of the local area in one frame of image is used to obtain the pose transformation parameters between the current image and the next frame of image, and using the pose transformation parameters and the camera pose parameters of the current image, Get the camera pose parameters of the next frame of image, use the converged 3D points to optimize the camera pose parameters of the next frame of image, so that the camera pose parameters can be further optimized, which is conducive to improving the camera pose parameters Accuracy; and by obtaining the probability distribution of the depth information of the feature points, and using the probability distribution, the depth information of the feature points in the next frame of image can be obtained, so that the depth information can be calculated based on the distribution probability of the depth information during the shooting process. optimize.

In some disclosed embodiments, the camera pose parameters include rotation parameters and displacement parameters, and the visual positioning device 60 further includes a camera pose detection part configured to respond to the camera pose parameters of the image to be processed not satisfying the preset steady state condition , it is determined that the displacement parameter of the image to be processed cannot be obtained, and the visual positioning device 60 also includes a rotation parameter updating part configured to use the primitive value of the previous frame image of the image to be processed and the camera pose of the previous frame image parameter, get the rotation parameter of the image to be processed.

Different from the foregoing embodiments, the camera pose parameters are set to include rotation parameters and displacement parameters, and after acquiring the camera pose parameters of the image to be processed after the current image, in response to the fact that the camera pose parameters of the image to be processed are not Satisfy the preset steady state conditions, determine that the displacement parameters of the image to be processed cannot be obtained, so that the image to be processed can be obtained by using the primitive value of the previous frame image of the image to be processed and the camera pose parameters of the previous frame image The rotation parameters of the image can be directly used to estimate the rotation parameters when the camera pose parameters are inaccurate. probability of such problems occurring.

In some disclosed embodiments, the rotation parameter update part includes a projective transformation subsection configured to use the pose transformation parameters between the image to be processed and the previous frame image to update at least some primitive points in the previous frame image Perform projection transformation to obtain the projection points of at least some of the primitive points in the image to be processed, and the rotation parameter update part includes a function construction subsection configured to use at least some of the primitive points in the previous frame image and the sum of The difference between the primitive values of the projection points corresponding to at least some of the primitive points in the image to be processed is used to construct an objective function about the pose transformation parameters, and the rotation parameter update part includes a parameter acquisition subsection, which is configured to use the obtained by solving the objective function The pose transformation parameters transform the camera pose parameters of the previous frame image to obtain the rotation parameters of the image to be processed.

Different from the foregoing embodiments, at least some of the primitive points in the previous frame image are adjusted using the pose transformation parameters between the image to be processed and the previous frame image Perform projection transformation to obtain the projection points of at least some of the primitive points in the image to be processed, and use the primitive values of at least some of the primitive points in the previous frame image and the projection points corresponding to at least some of the primitive points in the image to be processed The difference between the primitive values in the image is used to construct an objective function about the pose transformation parameters, so that the pose transformation parameters obtained by solving the objective function are used to transform the camera pose parameters of the previous frame image, and the image to be processed is obtained. The rotation parameter of the image, so the rotation parameter can be obtained based on at least some of the primitive points, which can help reduce the calculation amount of the calculation of the rotation parameter.

In some disclosed embodiments, the rotation parameter update part includes a downsampling subsection configured to perform downsampling processing on the previous frame image to obtain a thumbnail image of the previous frame image, and the projection transformation subsection is also configured to use The pose transformation parameters between the image to be processed and the previous frame image perform projection transformation on the primitive points in the thumbnail image to obtain the projection points of the primitive points in the thumbnail image on the image to be processed.

Different from the foregoing embodiments, the thumbnail image of the previous frame image is obtained by down-sampling the previous frame image, so that the pose transformation parameters between the previous frame images of the image to be processed are used to The primitive points in the thumbnail image are projected and transformed to obtain the projection points of the primitive points in the thumbnail image on the image to be processed, so as to construct and solve the subsequent objective function, which can help reduce the cost of calculating the rotation parameters Calculations.

In some disclosed embodiments, the visual positioning device 60 also includes an acceleration detection part configured to detect the current acceleration information of the camera and determine whether the acceleration information is in a preset motion state, the gravity information acquisition part 61, the first pose acquisition part 62 It is also matched with the second pose acquisition part 63 It is configured to re-execute the step of obtaining the gravity information of the camera and the subsequent steps if the judgment result is Yes, and the acceleration detection part is also configured to re-execute the steps of detecting the current acceleration information of the camera and the subsequent steps if the judgment result is No. step.

Different from the above-mentioned embodiments, after obtaining the rotation parameters of the image to be processed, the current acceleration information of the camera is further detected, and it is judged whether the acceleration information is in the preset motion state, so that in the case of the preset motion state, re-execute the acquisition The step of detecting the gravity information of the camera and the subsequent steps, and re-executing the step of detecting the current acceleration information of the camera and the subsequent steps when it is not in the preset motion state, can help improve the robustness of visual positioning.

In some disclosed embodiments, the gravity information includes gravity direction information, the camera pose parameters include rotation parameters and displacement parameters, and the first pose acquisition part 62 includes a rotation angle acquisition subsection, configured to use the gravity direction information to obtain the camera relative to each other. The rotation angle of the x coordinate axis, y coordinate axis and z coordinate axis of the world coordinate system; and the direction of gravity after the camera is rotated according to the rotation angle is the same as the opposite direction of the z coordinate axis. The first pose acquisition part 62 parameter initialization subsection is configured to use the rotation Angle, get the rotation parameter, and set the displacement parameter to the preset value.

In the embodiments of the present invention and other embodiments, a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a unit, and it may also be a module or non-modular.

Different from the previous embodiments, by using the gravity direction information, the camera is obtained relative to the world coordinate system x-coordinate axis, y-coordinate axis and z-coordinate The rotation angle of the axis, and the direction of gravity after the camera rotates according to the rotation angle is the same as the opposite direction of the z coordinate axis, so that the rotation parameter can be obtained by using the rotation angle, and the displacement parameter can be set to a preset value, and the rotation parameter can be obtained through gravity alignment. In turn, the camera pose parameters are initialized, which is beneficial to reduce the calculation amount of camera pose parameter initialization.

In some disclosed embodiments, the origin of the world coordinate system is the position where the camera captures the current image, and the default value is 0.

Different from the foregoing embodiments, the origin of the world coordinate system is set to the position where the camera captures the current image, and the preset value is set to 0, which can help reduce the complexity of initializing the displacement parameters.

In some disclosed embodiments, the preset motion state is a static state or a uniform motion state; and/or, the gravity information is obtained by using the acceleration information of the camera in the preset state.

Different from the foregoing embodiments, setting the preset motion state to a static state or a uniform motion state can help improve the accuracy of initializing the camera pose parameters of the current image; and use the acceleration information of the camera in the preset state to obtain gravity Information can only use the accelerometer to obtain gravity information, which can help to further reduce the cost of using the visual positioning technology and expand the scope of use of the visual positioning technology.

Please refer to FIG. 7 . FIG. 7 is a schematic frame diagram of an embodiment of an electronic device 70 of the present invention. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other. The processor 72 is used to execute the program instructions stored in the memory 71 to implement the steps of any one of the above embodiments of the visual positioning method. In an implementation scenario, the electronic device 70 may include but not limited to: a mobile phone, Mobile devices such as tablet computers and robots are not limited here.

In the embodiment of the present invention, the processor 72 is used to control itself and the memory 71 to implement the steps of any of the above embodiments of the visual positioning method. The processor 72 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 72 may be an integrated circuit chip with signal processing capability. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. In addition, the processor 72 may be jointly realized by an integrated circuit chip.

The above solution can reduce the use cost of the visual positioning technology and expand the application range of the visual positioning technology.

Please refer to FIG. 8 . FIG. 8 is a frame diagram of an embodiment of a computer-readable storage medium 80 of the present invention. The computer-readable storage medium 80 stores program instructions 801 that can be executed by the processor, and the program instructions 801 are used to implement the steps of any one of the above embodiments of the visual positioning method.

The above solution can reduce the use cost of the visual positioning technology and expand the application range of the visual positioning technology.

In the several embodiments provided by the present invention, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, with The device implementation described above is only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated into other A system, or some feature, can be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium. Include a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) or A processor (processor) executes all or part of the steps of the methods in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc., which can store program codes. medium.

Industrial Applicability

In the embodiment of the present invention, by obtaining the gravity information of the camera, the gravity information is used to obtain the camera pose parameters of the current image captured by the camera in a preset motion state, and based on the camera pose parameters of the current image, the current The camera pose parameters of the image to be processed after the image can then only rely on the camera and gravity information for visual positioning, so the cost of using visual positioning technology can be reduced and the scope of use of visual positioning technology can be expanded.

S11~S13: Steps

Claims (13)

A visual positioning method applicable to a visual positioning device, the method comprising: acquiring gravity information of a camera; using the gravity information to acquire camera pose parameters of a current image captured by the camera in a preset motion state; based on The camera pose parameters of the current image, acquire the camera pose parameters of the image to be processed after the current image; wherein, the camera pose parameters include rotation parameters and displacement parameters; in response to the to-be-processed The camera pose parameters of the image do not meet the preset steady state conditions, and it is determined that the displacement parameters of the image to be processed cannot be obtained; and, using the primitive value of the previous frame of the image to be processed and the The camera pose parameters of the last frame of image to obtain the rotation parameters of the image to be processed; The camera pose parameters of the image, obtaining the rotation parameters of the image to be processed includes: using the pose transformation parameters between the image to be processed and the previous frame image to At least some of the primitive points are subjected to projection transformation to obtain the projection points of the at least some of the primitive points in the image to be processed; using the primitive values of the at least some of the primitive points in the previous frame image and the difference between the primitive values of the projected points corresponding to the at least part of the primitive points in the image to be processed, constructing the object of the pose transformation parameters A scalar function; using the pose transformation parameters obtained by solving the objective function to transform the camera pose parameters of the previous frame image to obtain the rotation parameters of the image to be processed. According to the method described in claim 1, wherein the gravity information includes gravity direction information, and the camera pose parameters of the image to be processed after the current image are obtained based on the camera pose parameters of the current image Before parameters, the method further includes: acquiring feature direction information of feature points in the current image; using feature direction information of feature points and the gravity direction information to obtain the feature in the current image The depth information of the point; the camera pose parameters based on the current image, obtaining the camera pose parameters of the image to be processed after the current image includes: based on the feature points in the current image The depth information and the camera pose parameters of the current image are obtained by acquiring the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed. According to the method described in claim 2, wherein, the feature direction information includes the direction vector of the feature point, the gravity direction information includes a gravity vector, and the depth information includes the depth value of the feature point; The feature direction information and the gravity direction information of the feature point, obtaining the depth information of the feature point in the current image includes: performing a first prediction on the direction vector and the gravity vector of the feature point Set the operation to obtain the direction vector and the gravity direction of the feature point An included angle between the quantities; performing a second preset operation on the preset height of the camera and the included angle to obtain the depth value of the feature point. The method according to claim 3, wherein the first preset operation includes an inner product operation; and/or, the second preset operation includes dividing the preset height by a cosine value of the included angle. According to the method described in claim 2, wherein, based on the depth information of the feature points in the current image and the camera pose parameters of the current image, the image to be processed after the current image is acquired The depth information of the feature points in the image and the camera pose parameters of the image to be processed include: using a preset pose tracking method to track the depth information of the feature points in the current image, the current image The camera pose parameters of the current image are tracked to obtain the depth information of the feature points in the next frame image of the current image and the camera pose parameters of the next frame image; the next frame image image as the current image, and re-execute the step of tracking the depth information of the feature points in the current image and the camera pose parameters of the current image using the preset pose tracking method and next steps. According to the method described in claim 5, wherein, the depth information of the feature points in the current image and the camera pose parameters of the current image are tracked by using the preset pose tracking method to obtain The depth information of the feature points in the next frame image of the current image and the The camera pose parameters of the next frame image include: using the depth information of the feature point in the current image to determine the projection point of the feature point in the next frame image; based on the feature The difference between the primitive value of the point in the local area in the current image and the primitive value of the projected point in the local area in the next frame image, to obtain the current image and the next The pose transformation parameters between frame images; using the pose transformation parameters and the camera pose parameters of the current image to obtain the camera pose parameters of the next frame image; using the converged three-dimensional points , optimizing the camera pose parameters of the next frame of image; obtaining the probability distribution of the depth information of the feature point, and using the probability distribution to obtain the depth information of the feature point in the next frame of image. According to the method described in claim 1, wherein the use of the pose transformation parameters between the image to be processed and the image of the previous frame is used to modify at least some primitive points in the image of the previous frame Performing projection transformation to obtain that at least some of the primitive points are before the projection point of the image to be processed, the method further includes: performing downsampling processing on the previous frame image to obtain the previous frame image The thumbnail image of the image; the use of the pose transformation parameters between the image to be processed and the image of the previous frame to perform projective transformation on at least some primitive points in the image to be processed, to obtain The projection points of at least some of the primitive points on the image to be processed include: Using the pose transformation parameters between the image to be processed and the previous frame image to perform projective transformation on the primitive points in the thumbnail image to obtain the primitive points in the thumbnail image At the projection point of the image to be processed. According to the method described in claim 1, wherein the image to be processed is obtained by using the primitive value of the previous frame image of the image to be processed and the camera pose parameter of the previous frame image After the rotation parameters of the image, the method further includes: detecting the current acceleration information of the camera, and judging whether the acceleration information is in the preset motion state; if so, re-executing the step of acquiring the gravity information of the camera and subsequent steps; if not, re-execute the step of detecting the current acceleration information of the camera and subsequent steps. According to the method described in claim 1, wherein the gravity information includes gravity direction information, the camera pose parameters include rotation parameters and displacement parameters, and the camera is in a preset motion state obtained by using the gravity information The camera pose parameters of the current image captured below include: using the gravity direction information to obtain the rotation angles of the camera relative to the x coordinate axis, y coordinate axis and z coordinate axis of the world coordinate system respectively; The direction of gravity after the rotation angle is rotated is the same as the opposite direction of the z coordinate axis; the rotation parameter is obtained by using the rotation angle, and the displacement parameter is set as a preset value. The method according to claim 9, wherein the world The origin of the coordinate system is the position where the camera captures the current image, and the preset value is 0. The method according to any one of claims 1 to 10, wherein the preset motion state is a static state or a uniform motion state; and/or, the gravity information is obtained by using the camera in the preset motion state Get the acceleration information below. An electronic device, comprising a memory and a processor coupled to each other, the processor is used to execute program instructions stored in the memory, so as to realize the visual positioning method described in any one of claims 1 to 11. A computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the visual positioning method described in any one of claims 1 to 11 is implemented.

Images