JP7280385B2 - Visual positioning method and related apparatus, equipment and computer readable storage medium - Google Patents

Description

(Cross reference to related applications)
This application is filed based on and claims priority from a Chinese patent application with application number 202011148780.6 filed with the Chinese Patent Office on October 23, 2020. , the entire content of the Chinese patent application is incorporated herein by reference.

TECHNICAL FIELD The present invention relates to the field of computer vision technology, and more particularly to visual positioning methods and related apparatus, apparatus, and computer readable storage media.

With the development of electronic information technology, visual positioning technology such as Simultaneous Localization And Mapping (SLAM) is widely used in autonomous driving, indoor navigation, augmented reality (AR), virtual reality (VR). : Virtual Reality).

Visual positioning techniques such as SLAM complete tasks such as mobile device autonomous positioning, navigation, etc. by acquiring the mobile device's camera pose, which is essentially a complex mathematical problem. Currently, visual positioning techniques such as SLAM rely on sensors in hardware, typically requiring sensors such as cameras, accelerometers, gravimeters, and inertial measurement units (IMUs). However, in practical applications, usually only mid-end/high-end mobile devices are equipped with the above sensors. Low-end mobile devices typically have relatively few sensors and no IMU, which makes the existing visual positioning technology relatively expensive to use and relatively narrow in scope. In view of this, reducing the cost of using visual positioning technology and expanding the scope of use of visual positioning technology are urgent issues.

The present invention provides a visual positioning method and related apparatus, apparatus and computer readable storage medium.

A first aspect of the present invention provides a visual positioning method, the method comprising obtaining gravity information of a camera and using the gravity information to determine the current image taken by the camera in a preset motion state. and obtaining camera pose parameters of an image to be processed after the current image based on the camera pose parameters of the current image.

Therefore, by obtaining the gravity information of the camera, the gravity information is used to obtain the camera pose parameters of the current image taken by the camera in the preset motion state, and based on the camera pose parameters of the current image , the camera pose parameters of the image to be processed after the current image can be obtained, and furthermore, visual positioning can be performed only by relying on the camera and gravity information, thus reducing the cost of using visual positioning technology and extend the range of use of visual positioning techniques.

Here, the gravity information includes gravity direction information, and before obtaining the camera pose parameters of the image to be processed after the current image based on the camera pose parameters of the current image, the method includes: further comprising obtaining feature direction information for the feature points in the image, and using the feature direction information and gravity direction information for the feature points to obtain depth information for the feature points in the current image; Obtaining the camera pose parameters of the image to be processed after the current image based on the camera pose parameters of the current image is based on the depth information of the feature points in the current image and the camera pose parameters of the current image. Based on this, obtaining depth information of feature points in the image to be processed after the current image and camera pose parameters of the image to be processed.

Therefore, obtain the feature direction information of the feature points in the current image, and use the feature direction information of the feature points and the gravity direction information included in the gravity information to obtain the depth information of the feature points in the current image can initialize the depth information of the feature points in the current image and the camera pose parameters of the current image based on the current image, and obtain the depth information of the feature points in the current image and the camera pose parameters of the current image Based on the camera pose parameters, the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed can be obtained, and the multi-frame image for the initial initiation operation. can improve the response speed of visual positioning without the need to scan the

Here, the feature direction information includes the direction vector of the feature point, the gravity direction information includes the gravity vector, the depth information includes the depth value of the feature point, and the feature direction information and the gravity direction information of the feature point are used. to obtain the depth information of the feature point in the current image by performing a first preset operation on the feature point's direction vector and the gravity vector to obtain the feature point's direction vector and the gravity vector obtaining an included angle; and performing a second preset operation on the preset height and included angle of the camera to obtain the depth value of the feature point.

Therefore, the feature direction information is set to include the direction vector of the feature point, the gravity direction information is set to include the gravity vector, the depth information is set to include the depth value of the feature point, and Perform a first preset operation on the feature point's direction vector and gravity vector to obtain the included angle between the feature point's direction vector and the gravity vector, thereby for the preset height and included angle of the camera A second preset operation may be performed to obtain the depth values of the feature points to help reduce the computational complexity of obtaining the feature point depth values.

Here, the first preset operation includes an inner product operation and/or the second preset operation includes dividing the preset height by the cosine value of the included angle.

Therefore, the first preset operation can be set to include an inner product operation to help reduce the complexity of obtaining the included angle between the direction vector and the gravity vector, and the second preset operation can be set to include a preset can be set to include a cosine value dividing the height of by the included angle to help reduce the complexity of obtaining depth values.

Here, based on the depth information of the feature points in the current image and the camera pose parameters of the current image, the depth information of the feature points in the image to be processed after the current image and the camera of the image to be processed Obtaining the pose parameters uses the preset pose tracking method to perform the tracking process on the depth information of the feature points in the current image, the camera pose parameters of the current image, and the Obtaining the depth information of the feature points in the next frame image and the camera pose parameters of the next frame image and using the next frame image as the current image and using the preset pose tracking method , the depth information of the feature points in the current image, the step of performing the tracking process on the camera pose parameters of the current image, and re-performing the subsequent steps.

Therefore, using the preset pose tracking method, the depth information of the feature points in the current image, the camera pose parameters of the current image, and the tracking process are performed in the next frame of the current image. feature point depth information and the camera pose parameters of the next frame image, thereby using the next frame image as the current image and using the preset pose tracking method, feature point depth information, the step of performing the tracking process on the camera pose parameters of the current image and the subsequent steps can be re-executed, and the camera pose parameters can be calculated for each frame, and the camera pose parameters It helps to reduce the cumulative error of

Here, using the preset pose tracking method, the depth information of the feature points in the current image, the camera pose parameters of the current image, the tracking process is performed to obtain the image of the next frame of the current image. Obtaining the depth information of the feature points in the image of the next frame and the camera pose parameters of the image of the next frame uses the depth information of the feature points in the current image to determine the projection points of the feature points in the image of the next frame. and based on the difference between the pixel values of the feature points in the local region in the current image and the pixel values of the projection points in the local region in the image of the next frame, the current image and the next obtaining the pose transformation parameters between the images of the frame; using the pose transformation parameters and the camera pose parameters of the current image to obtain the camera pose parameters of the image of the next frame; Using the original point to optimize the camera pose parameters of the image of the next frame, obtaining the probability distribution of the depth information of the feature points, and using the probability distribution to determine the features in the image of the next frame obtaining depth information for the points.

Therefore, by using the depth information of the feature points in the current image to determine the projection points of the feature points in the image of the next frame, the pixel values of the feature points in the local region in the current image and the Get the pose transformation parameters between the current image and the image of the next frame, based on the difference between the pixel values of the projection points in the local region in the image of the frame, and obtain the pose transformation parameters and the current image get the camera pose parameters of the next frame image using the camera pose parameters of , and use the converged 3D points to optimize the camera pose parameters of the next frame image to It can be further optimized, which helps improve the accuracy of the camera pose parameters, obtains the probability distribution of the depth information of the feature points, and uses the probability distribution to identify the feature points in the image of the next frame. By obtaining the depth information, the depth information can be optimized in the shooting process based on the distribution probability of the depth information, where the camera pose parameters and the depth information are optimized respectively to reduce the computational complexity of optimization reduce

Here, the camera pose parameters include rotation parameters and displacement parameters, and after obtaining the camera pose parameters of the image to be processed after the current image based on the camera pose parameters of the current image, the method includes processing Determining a displacement parameter at which the image to be processed cannot be obtained in response to the camera pose parameters of the image to be processed failing to satisfy a preset steady state condition, and the previous frame of the image to be processed. using the pixel values of the image of the previous frame and the camera pose parameters of the image of the previous frame to obtain the rotation parameters of the image to be processed.

Therefore, after setting the camera pose parameters to include rotation and displacement parameters and obtaining the camera pose parameters of the image to be processed after the current image, the camera pose parameters of the image to be processed are set to the preset determining the displacement parameters for which the image to be processed cannot be obtained, in response to failure to satisfy the steady state condition of to obtain the rotation parameters of the image to be processed, and if the camera pose parameters are inaccurate, the image pixels are directly used to estimate the rotation parameters, and the rotation parameters It can help to reduce the probability of problems such as virtual objects sticking to the screen in virtual reality caused by the inability to update.

Here, using the pixel values of the image of the previous frame of the image to be processed and the camera pose parameters of the image of the previous frame to obtain the rotation parameters of the image to be processed is pixel points in the image to be processed by performing a projection transformation on at least some of the pixel points in the image of the previous frame using the pose transformation parameters between the image and the image of the previous frame pixel values of at least some of the pixel points in the image of the previous frame; and pixels of the projection points corresponding to at least some of the pixel points in the image to be processed. constructing a target function for the pose transform parameters using the difference between the values of and performing a transformation process on the image to obtain rotation parameters of the image to be processed.

Therefore, using the pose transformation parameters between the image to be processed and the image of the previous frame, a projective transformation is performed on at least some of the pixel points in the image of the previous frame to be processed. obtaining projection points of at least some of the pixel points in the image to be processed, corresponding to pixel values of at least some of the pixel points in the image of the previous frame and at least some of the pixel points in the image to be processed. Using the difference between the pixel values of the projection points to construct a target function for the pose transformation parameters, thereby using the pose transformation parameters obtained by solving the target function, the image of the previous frame. Calculation for calculating the rotation parameter, which can be performed on the camera pose parameter to obtain the rotation parameter for the image to be processed, to obtain the rotation parameter based at least in part on the pixel points; It can help you cut down on volume.

where the pose transformation parameters between the image to be processed and the image of the previous frame are used to perform a projective transformation on at least some of the pixel points in the image of the previous frame to Before obtaining projection points of at least some of the pixel points in the image to be projected, the method performs a downsampling operation on the previous frame image to obtain a thumbnail image of the previous frame image. performing a projective transformation on at least some of the pixel points in the image to be processed using the pose transformation parameters between the image to be processed and the image of the previous frame , obtaining the projection points of at least some of the pixel points in the image to be processed uses pose transformation parameters between the image to be processed and the image of the previous frame to obtain the pixel points in the thumbnail image Performing a projection transformation on the points to obtain projection points of the pixel points in the thumbnail image in the image to be processed.

Therefore, by performing a downsampling process on the image of the previous frame to obtain a thumbnail image of the image of the previous frame, the image to be processed uses the pose transformation parameters between the images of the previous frame. to perform a projective transformation on the pixel points in the thumbnail image to obtain the projection points of the pixel points in the thumbnail image in the image to be processed for subsequent target function construction and solution. This can help reduce the computational complexity of calculating the rotation parameters.

Here, after obtaining the rotation parameters of the image to be processed using the pixel values of the image of the previous frame of the image to be processed and the camera pose parameters of the image of the previous frame, the method uses the camera's detecting the current acceleration information and determining whether the acceleration information is in a preset motion state; if yes, re-performing the step of obtaining gravity information of the camera and subsequent steps; detecting current acceleration information of the camera and re-performing subsequent steps.

Therefore, after obtaining the rotation parameters of the image to be processed, further detect the current acceleration information of the camera, determine whether the acceleration information is in the preset motion state, and determine whether the acceleration information is in the preset motion state. If so, re-execute the step of obtaining the gravity information of the camera and subsequent steps, and re-execute the step of detecting the current acceleration information of the camera and subsequent steps if it is not in the preset motion state, and additionally perform the visual Positioning robustness can be improved.

Here, the gravity information includes gravity direction information, the camera pose parameters include rotation parameters and displacement parameters, and the gravity information is used to calculate the camera pose parameters of the current image taken by the camera in the preset motion state. is using the gravitational direction information to obtain the rotation angles of the camera respectively corresponding to the acquisition world coordinate system x-coordinate axis, y-coordinate axis and z-coordinate axis, wherein the camera is according to the rotation angle The direction of gravity after rotation is the same as the opposite direction of the z-coordinate axis; and using the rotation angle to obtain the rotation parameter and set the displacement parameter to a preset numerical value.

Therefore, the direction of gravity information is used to obtain the rotation angles of the camera corresponding to the world coordinate system x-, y-, and z-coordinate axes, and the direction of gravity after the camera rotates according to the rotation angle is the z-coordinate axis of It is the same as the reverse direction, using the rotation angle to get the rotation parameter, setting the displacement parameter to a preset number to get the rotation parameter by gravity alignment, and also initializing the camera pose parameter , which helps reduce the amount of computation for initializing the camera pose parameters.

Here, the origin of the world coordinate system is the position when the camera takes the current image, and the preset value is 0.

Therefore, the origin of the world coordinate system can be set to the position when the camera takes the current image, and the preset numerical value can be set to 0 to help reduce the complexity of initializing the displacement parameters.

Here, the preset motion state is a static state or a uniform velocity motion state, and/or the gravity information is obtained using the acceleration information of the camera in the preset state.

Therefore, the preset motion state can be set to a static state or a uniform velocity motion state to help improve the accuracy of initializing the camera pose parameters of the current image, and the acceleration of the camera in the preset state Information can be used to obtain gravity information to obtain gravity information using only the accelerometer, thereby further reducing the cost of using visual positioning technology and increasing the scope of use of visual positioning technology can be useful in expanding the

A second aspect of the present invention provides a visual positioning device, the device comprising a gravity information acquisition unit, a first pose acquisition unit and a second pose acquisition unit, the gravity information acquisition unit obtaining gravity information of a camera. a first pose obtaining unit configured to obtain camera pose parameters of a current image taken by the camera in a preset motion state using the gravity information; and a second pose obtaining The unit is configured to obtain camera pose parameters of an image to be processed after the current image based on the camera pose parameters of the current image.

A third aspect of the present invention provides an electronic device. The electronic device comprises a memory and a processor coupled to each other, the processor executing program instructions stored in the memory to implement the visual positioning method in the first aspect above.

A fourth aspect of the present invention provides a computer-readable storage medium, said computer-readable storage medium storing program instructions, said program instructions, when executed by a processor, instructing said processor to perform the steps of the above first aspect. Have a visual positioning method implemented.

A fifth aspect of the present invention provides a computer program product, said computer program product comprising computer readable code, said computer readable code, when executed in an electronic device, instructing a processor in said electronic device to perform the above-described process. The visual positioning method in the first aspect is performed.

The above technical solution is to obtain the gravity information of the camera, use the gravity information to obtain the camera pose parameters of the current image taken by the camera in the preset motion state, and obtain the camera pose parameters of the current image Based on the parameters, the camera pose parameters of the image to be processed after the current image can be obtained, and furthermore, visual positioning can be performed only by relying on the camera and gravity information, so the use of visual positioning technology The cost of use can be reduced and the range of use of visual positioning technology can be expanded.

1 is an exemplary flow chart of one embodiment of the visual positioning method of the present invention; FIG. 4 is a schematic diagram of an example of obtaining depth information; FIG. 2 is an exemplary flow chart of one embodiment of step S13 in FIG. 1; FIG. 4 is an exemplary flow chart of another embodiment of the visual tracking method of the present invention; FIG. 5 is an exemplary flow chart of one implementation of step S45 in FIG. 4; FIG. 1 is a schematic diagram of the framework of one embodiment of the visual positioning device of the present invention; FIG. 1 is a schematic diagram of the framework of an embodiment of an electronic device of the present invention; FIG. 1 is a schematic diagram of the framework of an embodiment of a computer-readable storage medium of the present invention; FIG.

The technical solutions of the embodiments of the present invention are described in detail below with reference to the drawings.

In the following description, for purposes of illustration rather than limitation, details of specific system architectures, interfaces, techniques, etc. are suggested in order to provide a thorough understanding of the present invention.

The terms "system" and "network" are always used interchangeably herein. The term "and/or" herein is simply an association relationship describing the objects that are associated, indicating that there can be three relationships, e.g., A and/or B if A exists independently , where A and B exist simultaneously, and where B exists independently. Furthermore, the symbol "/" in this specification generally indicates that the associated objects before and after are in an "or" relationship. In addition, "plurality" in this specification represents two or two or more.

Referring to FIG. 1, FIG. 1 is an exemplary flow chart of one embodiment of the visual positioning method of the present invention. A visual positioning method may include the following steps.

In step S11, gravity information of the camera is acquired.

In the embodiments of the present invention, the execution subject of the visual positioning method can be a visual positioning device, for example, the visual positioning method can be executed by a terminal device or a server or other processing device, wherein: Terminal Equipment includes User Equipment (UE), Mobile Equipment, User Terminal, Terminal, Mobile Phone, Cordless Phone, Personal Digital Assistant (PDA), Handheld Device, Computing Device, Vehicle Device, Wearable Device. and so on. In some possible embodiments, the visual positioning method can be implemented by a processor via a method of calling computer readable instructions stored in memory.

In embodiments of the present invention, the camera can be integrated into a mobile device, which can include, but is not limited to, mobile phones, tablets, robots, and the like. Here, the steps in the embodiments of the present invention and the following disclosed embodiments can be performed by a mobile device, in which a visual positioning device is arranged. In addition, mobile devices can integrate other sensors, such as accelerometers, gravimeters, IMUs, etc., which can be installed according to actual application scenarios, not limited here. For example, due to cost constraints, low-end mobile devices may integrate only a camera and accelerometer, or may integrate a camera and a gravimeter, while mid-/high-end mobile devices may integrate a camera, accelerometer, controller, IMU, etc., and is not limited here.

In one implementation scenario, the gravity information may have been obtained using the acceleration information of the camera in a preset motion state, thereby relying solely on the accelerometer to obtain the gravity information without the need for an IMU. be able to. Here, the preset motion state is a static state or a uniform speed motion state. For example, the difference between the camera acceleration detected in the preset motion state and the gravitational acceleration is within a preset range, for example, the gravitational acceleration is 9.8 m/ ^s2 and the preset range is 0-1 m /s ² , and the detected camera acceleration is 10m/s ² , consider that the camera is in the preset motion state, the preset range can be set according to the actual application needs , not limited here. Furthermore, even if the gravimeter is integrated into the mobile device, gravity information can be obtained directly through the gravimeter without the need for an IMU.

，

，

）二乗和の根を計算して、カメラ加速度

として使用することができ、即ち、

である。 In one implementation scenario, the mobile device can further determine whether it is in a static state or a uniform-velocity motion state according to the detected camera acceleration, for example, the detected camera acceleration is the gravitational acceleration can be considered to be in a state of static or uniform velocity motion if it is close to . Now, if the accelerometer is placed, the acceleration components of the accelerometer in three axes (e.g.,

，

，

) to calculate the root of the sum of squares to give the camera acceleration

can be used as, i.e.

is.

In another implementation scenario, if the camera is detected not to be in a preset motion state, the mobile device can redetect until the camera is detected to be in a preset motion state. Here, the detected frequency can correspond to the frequency captured by the camera, for example, if the camera captures 25 images per second, correspondingly whether it is in a preset motion state or not. It can be detected 25 times per second, while it can be set according to the actual application needs and is not limited here.

）を計算し、ベクトル合計を重力ベクトルとして使用することができ、または、ベクトルと同じ方向の単位ベクトルを重力ベクトルとして使用することもでき、重力ベクトルに対して、実際の適用ニーズに従って設定することができ、ここでは限定しない。 In one implementation scenario, the camera's gravity information may include gravity direction information, where the gravity direction information may include a gravity vector. In one implementation scenario, when the accelerometer is positioned, the vector sum of the acceleration components of the accelerometer in three axes (i.e.

), the vector sum can be used as the gravity vector, or the unit vector in the same direction as the vector can be used as the gravity vector, and the gravity vector can be set according to the actual application needs , not limited here.

In step S12, the gravity information is used to obtain the camera pose parameters of the current image taken by the camera in the preset motion state.

In one implementation scenario, the camera pose parameters may include a displacement parameter and a rotation parameter, whereby the gravity direction information is used to determine the rotation angles of the camera corresponding to the world coordinate system x-, y-, and z-coordinate axes, respectively. and the direction of gravity after the camera rotates according to the rotation angle is the same as the opposite direction of the z-coordinate axis, and the rotation angle can be used to obtain the rotation parameters, The displacement parameters can be set to preset numerical values to simplify initializing the camera pose parameters of the current image via gravity alignment and reduce the amount of computation.

に表すことができ、ｙ座標軸に対応するカメラの回転角度は、

に表すことができ、ｚ座標軸に対応するカメラの回転角度は、

に表すことができ、世界座標系ｘ座標軸に対応するカメラの回転パラメータ

、世界座標系ｙ座標軸に対応する回転パラメータ

、世界座標系ｚ座標軸に対応する回転パラメータ

は、それぞれ以下のように表すことができる。

……（１） In one implementation scenario, the rotation angle of the camera corresponding to the x coordinate axis is

and the rotation angle of the camera corresponding to the y coordinate axis is

and the rotation angle of the camera corresponding to the z coordinate axis is

and the rotation parameter of the camera corresponding to the world coordinate system x coordinate axis

, the rotation parameter corresponding to the world coordinate system y coordinate axis

, the rotation parameter corresponding to the world coordinate system z coordinate axis

can be expressed as follows.

……(1)

、世界座標系ｙ座標軸に対応する回転パラメータ

、世界座標系ｚ座標軸に対応する回転パラメータ

によって得られることができ、ここで、上記の回転パラメータ

、回転パラメータ

および回転パラメータ

の乗積を、回転パラメータＲとして使用することができ、即ち、回転パラメータＲは、以下の通りに表すことができる。

……（２） The rotation parameter R is the rotation parameter corresponding to the world coordinate system x coordinate axis.

, the rotation parameter corresponding to the world coordinate system y coordinate axis

, the rotation parameter corresponding to the world coordinate system z coordinate axis

where the above rotation parameters

, rotation parameter

and rotation parameters

can be used as the rotation parameter R, ie the rotation parameter R can be expressed as:

……(2)

In another implementation scenario, the mobile device can use the origin of the world coordinate system as the position when the camera takes the current image, i.e., the camera are all zero, so the preset number can be set to zero, ie the displacement parameter can be set to zero.

In step S13, the camera pose parameters of the image to be processed after the current image are obtained according to the camera pose parameters of the current image.

In one implementation scenario, the mobile device scans to obtain a multi-frame image, uses a triangulation scheme to process feature points that match each other in the current image and neighboring images of the current image, Depth information of matching feature points can be obtained. wherein the depth information may include depth values of the feature points, so that the calculated depth values can be used to obtain three-dimensional coordinates of the feature points in the world coordinate system; , using the pose transformation parameters between the image of the next frame of the current image and the current image, reprojecting the 3D coordinates of the feature points to the image of the next frame to obtain A projection point can be obtained, thereby using the difference between the pixel value of the projection point in the image of the next frame and the pixel value of the corresponding feature point in the current image to determine the pose transformation parameters A target function can be constructed. The pose transformation parameters can be obtained by minimizing the target function, and the pose transformation parameters and the camera pose parameters of the current image can be used to obtain the camera pose parameters of the next frame image. By analogy, the mobile device can obtain the camera pose parameters of the image to be processed after the current image on a frame-by-frame basis. In one implementation scenario, the triangulation scheme observes the same 3D point at different locations, knows the 2D projection points of the 3D points observed at different locations, and uses trigonometry to determine the 3D point It refers to restoring the depth information of , and will not be described in detail here.

In another implementation scenario, in order to reduce the additional response time of scanning multi-frame images and improve the response speed of visual positioning, the mobile device calculates the feature directions of feature points in the current image. Using the information and the gravity direction information, the depth information of the feature points in the current image can be obtained, so that based on the depth information of the feature points in the current image and the camera pose parameters of the current image to obtain the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed, and furthermore, the depth information can be initialized only in the current image, thus , avoiding scanning multi-frame images, can help improve the response speed of visual positioning.

In one implementation scenario, the feature direction information may include the feature point direction vector, the gravity direction information may include the gravity vector, where the direction vector and the gravity vector may be unit vectors, and the depth information may include the feature point The feature points may include pixel points that can describe image features, such as contour edge pixel points in an image, pixel points with abrupt changes in pixel values, etc., where for feature points, actual can be set according to your needs and is not limited here. For example, Features from Accelerated Segment Test (FAST), Binary Robust Independent Elementary Features (BRIEF), Scale Invariant Feature Transform (SIFT), OR Detection of B, etc. Through the method, the feature points and the direction vectors of the feature points can be obtained, for which the feature point detection method can be selected according to the actual application needs, which is not limited here.

と重力ベクトル

との夾角

は、以下に通りに表すことができる。

……（３） In another implementation scenario, please refer to FIG. 2 , which is a schematic diagram of an embodiment of obtaining depth information, wherein the mobile device sets a first preset for the direction vector and the gravity vector of the feature point. An operation can be performed to obtain the included angle between the direction vector of the feature point and the gravity vector, and the first preset operation can include an inner product operation, namely, the direction vector

and the gravity vector

included angle with

can be expressed as

……(3)

を取得した後、カメラのプリセットの高さｈおよび夾角

に対して第２プリセットの演算を実行して、特徴点の深度値ｚを取得することができ、第２プリセットの演算は、プリセットの高さを夾角で除算するコサイン値を含み、プリセットの高さｈは、実際の適用状況に従って設定することができる。ＡＲ適用を例として、仮想物体のサイズに従って設定することができ、例えば、仮想物体は、一般的なサイズの猫や犬などのペットである場合、プリセットの高さを０．５米～１米に設定することができ、他の適用状況は、実際の状況に従って設定することができ、ここではいちいち列挙をしない。ここで、深度値ｚは、以下の通りに表すことができる。

……（４） included angle

After obtaining the camera preset height h and included angle

to obtain the depth value z of the feature point, the second preset operation including a cosine value dividing the preset height by the included angle, the preset height h can be set according to the actual application situation. Taking AR application as an example, it can be set according to the size of the virtual object. For example, if the virtual object is a pet such as a cat or dog of general size, the preset height is 0.5 to 1 meter. , and other applicable situations can be set according to the actual situation, not listed here. Here, the depth value z can be expressed as follows.

……(4)

In one implementation scenario, the embodiments of the present invention and the steps in the following disclosed embodiments can be integrated into application programs, websites, such as indoor navigation, autonomous driving, AR, VR, etc. operated by mobile devices. can be configured according to actual application needs, and is not limited here.

In the above technical solution, the mobile device obtains the gravity information of the camera, uses the gravity information to obtain the camera pose parameters of the current image taken by the camera in the preset motion state, and obtains the current image Based on the camera pose parameters of the image, the camera pose parameters of the image to be processed after the current image can be obtained, and furthermore, visual positioning can be performed by relying only on the camera and gravity information, so the visual The cost of using visual positioning technology can be reduced and the range of use of visual positioning technology can be expanded.

Referring to FIG. 3, FIG. 3 is an exemplary flow chart of one embodiment of step S13 in FIG. FIG. 3 shows the depth information of the feature points in the image to be processed after the current image and the depth information of the image to be processed based on the depth information of the feature points in the current image and the camera pose parameters of the current image. FIG. 4 is an exemplary flowchart of one embodiment of obtaining camera pose parameters; FIG. Here, S13 may include the following steps.

In step S131, using the preset pose tracking method, the depth information of the feature points in the current image and the camera pose parameters of the current image are tracked to obtain the next frame of the current image. Obtain the depth information of the feature points in the image and the camera pose parameters of the image of the next frame.

The preset pose tracking schemes can be set according to actual application needs. In embodiments of the present invention, the preset pose tracking scheme may include steps such as sparse image alignment, feature point alignment, pose optimization, etc., thereby processing through the above steps to obtain the next frame's image In addition, the preset pose tracking scheme may include a map point optimization step, thereby processing through steps such as map point optimization to obtain the camera pose parameters in the image of the next frame Get the depth information of feature points.

を使用して、特徴点座標ｕおよび特徴点深度値ｄｕを含む第１座標情報

を三次元空間に逆投影して、特徴点の三次元座標

を取得し、現在の画像ｋ－１と次のフレームの画像ｋとの間のポーズ変換パラメータＴおよび三次元から二次元への投影関数

を使用して、現在の画像特徴点の三次元座標

を次のフレームの画像ｋに投影して、次のフレームの画像における特徴点の投影点

を取得することを含み得、それにより、次のフレームの画像ｋ内のローカル領域における投影点の画素値

と、現在の画像ｋ－１の特徴点ローカル領域に対応する画素値

との間に差異があることを取得し、さらに、当該差異に基づいて、現在の画像ｋ－１と次のフレームの画像ｋとの間のポーズ変換パラメータを取得することができる。ここで、ローカル領域は、特徴点（または投影点）を中心とする一長方形領域（例えば、３＊３領域、４＊４領域、８＊８領域など）であり得、以下の式に示された通りである。

……（５） In one implementation scenario, when performing sparse image alignment, the mobile device first uses depth information of the feature points in the current image to determine projection points of the feature points in the next frame image. can be determined, said step being a two-dimensional to three-dimensional backprojection function

to obtain first coordinate information including feature point coordinates u and feature point depth values du using

to the 3D space, the 3D coordinates of the feature points

, and the pose transformation parameter T between the current image k−1 and the next frame image k and the 3D to 2D projection function

to find the 3D coordinates of the current image feature point using

is projected onto the image k of the next frame, and the projected point of the feature point in the image of the next frame is

by obtaining the pixel value of the projection point in the local region in the image k of the next frame

and the pixel value corresponding to the feature point local region of the current image k−1

, and based on the difference, the pose transformation parameters between the current image k−1 and the next frame image k can be obtained. Here, the local region can be one rectangular region (eg, 3*3 region, 4*4 region, 8*8 region, etc.) centered on the feature point (or projection point), and is shown in the following formula: That's right.

……(5)

……（６） Since the number of feature points is usually multiple, the above differences can be calculated and summed for multiple feature points, and the target function can be further constructed, as shown in the equation below. Please note.

……(6)

は、ターゲット関数を表し、ここで、

は、ロバスト関数を表し、ノイズ影響を低減するために使用され、

は、ノルム演算を表し、

は、ポーズ変換パラメータＴを最適化対象としてターゲット関数を最小化することを表し、

は、ターゲット関数を解けて得られたポーズ変換パラメータを表す。 In the above formula (6),

represents the target function, where

represents a robust function and is used to reduce noise effects,

represents the norm operation,

represents minimization of the target function with the pose transformation parameter T as the optimization target,

represents the pose transformation parameters obtained by solving the target function.

を取得した後、モバイル機器は、ポーズ変換パラメータ

および現在の画像ｋ－１のカメラポーズパラメータ

を使用して、次のフレームの画像のカメラポーズパラメータ

を取得することができる。ここで、ポーズ変換パラメータ

を現在の画像ｋ－１のカメラポーズパラメータ

と乗算して、次のフレームの画像のカメラポーズパラメータ

を取得することができる。 Calculate pose transformation parameters

, the mobile device uses the pose transformation parameters

and the camera pose parameters of the current image k−1

Use the camera pose parameters for the next frame's image

can be obtained. where the pose transformation parameters

be the camera pose parameters of the current image k−1

to obtain the camera pose parameters for the next frame's image

can be obtained.

Furthermore, in order to reduce the computational complexity of the sparse image alignment, the mobile device performs a downsampling process on the current image k−1 and the image k of the next frame to obtain the current image k−1 and Obtaining a pyramid image of the image k of the next frame, taking a single layer image or a multi-layer image whose resolution is the preset resolution from the pyramid image and performing the above sparse image alignment process, thereby: It can reduce computational complexity.

の精度が比較的に低くする。精度を向上させるために、モバイル機器は、収束した三次元点（例えば、三次元モデル内の三次元点）を使用して、次のフレームの画像のカメラポーズパラメータ

を最適化することができる。ここで、収束した三次元点をマッチングしアライメントすることにより投影点を取得することができ、投影点を利用して、上記のスパース画像アライメントによって得られた次のフレームの画像のカメラポーズパラメータ

に対して特徴点アライメントの最適化を実行するステップは、収束した三次元点から次のフレームの画像ｋに投影できる三次元点を選択して、ターゲット三次元点として使用し、撮影された画像から、ターゲット三次元点が投影できる画像のうち、最も早く撮影された画像を選択して、参照画像として使用し、参照画像内のローカル領域におけるターゲット三次元点の画素値

を取得し、上記の概算された次のフレームの画像のカメラポーズパラメータ

を使用して、ターゲット三次元点を次のフレームの画像に投影して、次のフレームの画像におけるターゲット三次元点の投影点

を取得し、それにより、次のフレームの画像内のローカル領域における投影点

の画素値

を取得し、さらに、ローカル領域画素値

およびローカル領域画素値

を使用して、投影点

に関するターゲット関数を構築することができ、以下の式を参照されたい。

……（７） In one implementation scenario, the sparse image alignment operation described above inevitably introduces an accumulated error, thereby causing the camera pose parameter

The accuracy of is relatively low. To improve accuracy, the mobile device uses converged 3D points (e.g., 3D points in a 3D model) to determine the camera pose parameters for the next frame's image.

can be optimized. Now, the projection points can be obtained by matching and aligning the converged 3D points, and the projection points are used to obtain the camera pose parameters of the next frame's image obtained by the sparse image alignment above.

The step of performing feature point alignment optimization on the captured image is by selecting a 3D point that can be projected from the converged 3D point onto the image k of the next frame to be used as the target 3D point and the captured image , among the images that the target 3D point can be projected onto, the earliest captured image is selected and used as the reference image, and the pixel value of the target 3D point in the local region within the reference image is

and the camera pose parameters of the next frame image estimated above

is used to project the target 3D point onto the next frame's image to find the projected point of the target 3D point in the next frame's image

so that the projected point in the local region in the image of the next frame

pixel value of

and the local region pixel value

and local region pixel values

using the projection point

You can construct a target function for , see the formula below.

……(7)

は、ターゲット関数を表し、ここで、

は、ノルム演算を表し、

は、アフィン変換行列を表し、異なる視角による画像の歪みを補正するために使用され、

は、投影点

の位置を最適化対象としてターゲット関数を最小化することを表す。 In the above formula (7),

represents the target function, where

represents the norm operation,

represents the affine transformation matrix and is used to correct image distortion due to different viewing angles,

is the projection point

It means minimizing the target function with the position of as the optimization target.

を取得した後、モバイル機器は、上記の特徴点アライメントによって得られた投影点

に基づいて、上記のスパース画像アライメントによって得られた次のフレームの画像のカメラポーズパラメータ

を最適化し、最終的に、最適化して次のフレームの画像のカメラポーズパラメータ

を取得することができる。ポーズ最適化のステップは、次のフレームの画像のカメラポーズパラメータ

および三次元から二次元への投影関数

を使用して、ターゲット三次元点

を次のフレームの画像ｋに再投影して、投影点

を取得し、投影点

と、特徴点アライメントステップで最適化して得られた次のフレームの画像の投影点

との位置差異を使用して、カメラポーズパラメータ

に関するターゲット関数を構築することができ、以下の式を参照されたい。

……（８） projection point

After obtaining , the mobile device uses the projected points

camera pose parameters of the next frame image obtained by the above sparse image alignment based on

and finally, optimize the camera pose parameters for the next frame image

can be obtained. The pose optimization step is to use the camera pose parameters of the next frame image

and the projection function from 3D to 2D

using the target 3D point

to the image k of the next frame, and the projection point

and get the projection point

and the projection point of the next frame image optimized in the feature point alignment step

Using the position difference between the camera pose parameters

You can construct a target function for , see the formula below.

……(8)

は、ターゲット関数を表し、ここで、

は、ロバスト関数を表し、ノイズ影響を低減するために使用され、

は、ノルム演算を表し、

は、ポーズ変換パラメータ

を最適化対象としてターゲット関数を最小化することを表す。 In the above formula (8),

represents the target function, where

represents a robust function and is used to reduce noise effects,

represents the norm operation,

is the pose transform parameter

is the optimization target and the target function is minimized.

を取得することができる。 By solving the target function shown in equation (8), we finally obtain the camera pose parameters of the next frame image

can be obtained.

および逆深度値ｚは、ベータとガウスの混合モデル分布（Ｂｅｔａ Ｇａｕｓｓｉａｎ Ｍｉｘｔｕｒｅ Ｍｏｄｅｌ Ｄｉｓｔｒｉｂｕｔｉｏｎ）とほぼ一致し、以下の式を参照されたい。

……（９） In one implementation scenario, the essence of map point optimization is the optimization for the inverse depth (ie, the inverse of the depth value) of the corresponding location on the reference image where the 3D point was originally observed. Here, the probability distribution of the depth information of the feature points can be obtained, and the internal point probability of the feature points

and the inverse depth value z approximately match the Beta Gaussian Mixture Model Distribution, see the equation below.

……(9)

を第ｋ回観察した後の確率分布を表し、

，

は、ベータ分布のパラメータを表し、

，

は、逆深度ガウス分布の平均値と分散を表す。確率分布を取得した後、モバイル機器は、取得された確率分布を使用して、次のフレームの画像内の特徴点の深度情報を取得することができる。例えば、逆深度ガウス分布の分散

が、一プリセットの深度範囲（例えば、１／２００）より小さい場合、深度値収束に見なされることができ、この場合の逆深度ガウス分布の平均値

の逆数を取って特徴点の深度値として使用して、撮影プロセスで、特徴点の深度値を最適化し続けることができる。 Here, in the above equation (9), a feature point

represents the probability distribution after the k-th observation of

，

is the parameter of the beta distribution, and

，

represents the mean and variance of the inverse depth Gaussian distribution. After obtaining the probability distribution, the mobile device can use the obtained probability distribution to obtain the depth information of the feature points in the image of the next frame. For example, the variance of the inverse depth Gaussian distribution

is less than one preset depth range (e.g., 1/200), it can be considered depth value convergence, where the mean value of the inverse depth Gaussian distribution

can be taken and used as the feature point depth values to continue optimizing the feature point depth values in the imaging process.

In step S132, the image of the next frame is used as the current image.

After obtaining the camera pose parameters and the depth information of the feature points of the next frame image, the mobile device may use the next frame image as the current image and re-perform the above step S131 and subsequent steps. , whereby the camera pose parameters of the image and the depth information of the feature points in the image can be calculated for each frame.

In step S133, step S131 and subsequent steps are re-executed.

Distinguished from the above embodiments, the mobile device uses a preset pose tracking scheme to perform tracking processing on depth information of feature points in the current image, camera pose parameters of the current image, Get the depth information of the feature points in the next frame image of the current image and the camera pose parameters of the next frame image, thereby using the next frame image as the current image and preset pose tracking method is used to re-execute the step of performing tracking processing on the depth information of the feature points in the current image, the camera pose parameters of the current image and the subsequent steps, and furthermore, the camera pose parameters are re-performed for each frame. , which helps reduce the cumulative error of the camera pose parameters.

Referring to FIG. 4, FIG. 4 is an exemplary flowchart of another embodiment of the visual tracking method of the present invention, which may include the following steps.

In step S41, the gravitational information of the camera is obtained.

Please refer to the relevant steps in the above example.

In step S42, the gravity information is used to obtain the camera pose parameters of the current image taken by the camera in the preset motion state.

Please refer to the relevant steps in the above example.

In step S43, the camera pose parameters of the image to be processed after the current image are obtained according to the camera pose parameters of the current image.

Please refer to the relevant steps in the above example.

In step S44, it is determined whether the camera pose parameter of the image to be processed satisfies a preset steady state condition, if not, step S45 is performed; if yes, step S46 is performed.

The preset steady-state conditions are that there are no outliers in the camera pose parameters, and that the difference between the camera pose parameters of the image to be processed and the camera pose parameters of the previous frame of the image to be processed is within the preset range. at least one of In one implementation scenario, the outliers may include displacement parameters greater than the displacement threshold, rotation parameters greater than the rotation threshold, where the displacement threshold, rotation threshold and preset ranges can be set according to actual application needs. , not limited here.

In step S45, determine the displacement parameters that fail to obtain the image to be processed, and use the pixel values of the previous frame image of the image to be processed and the camera pose parameters of the previous frame image to be processed. Get the rotation parameter of the image to power.

In the actual application process, factors such as rapid motion, drastic changes in lighting conditions, etc. will all make the camera pose parameters inaccurate, resulting in inaccurate visual positioning, and to improve the robustness of visual positioning. , if it is determined that the camera pose parameters of the processed image do not satisfy the preset steady state conditions, the mobile device may determine the inaccuracy of the camera pose parameters obtained by the above steps, particularly the displacement parameters. can. Therefore, in order to reduce problems such as the virtual object sticking to the screen due to failure to update the rotation parameter, the mobile device sets the pixel value of the image of the previous frame and the camera pose of the image of the previous frame to the image to be processed. The parameter can be used to obtain the rotation parameters of the image to be processed, thereby keeping the rotation parameters updated.

In one implementation scenario, refer to FIG. 5, which is an exemplary flowchart of one implementation of step S45 in FIG. Here, S45 may include the following steps.

performing a projective transformation on at least some of the pixel points in the previous frame image using the pose transformation parameters between the image to be processed and the previous frame image, in step S451; Obtain projection points of at least some of the pixel points in the image to be processed.

に表し、前のフレームの画像内の画素点の少なくとも一部の二次元座標をｕに表し、画素点の少なくとも一部の深度値をｄｕに表し、二次元から三次元への逆投影関数を

に表すことができ、三次元から二次元への投影関数を

に表すことができ、投影点は、

に表すことができ、ここで、上記の実施例における関連ステップを参照でき、ここでは詳細に説明しない。 For convenience of explanation, let the image to be processed be k, the image of the previous frame be k−1, and the pose transformation parameters be

, the two-dimensional coordinates of at least some of the pixel points in the image of the previous frame in u, the depth values of at least some of the pixel points in du, and the two-dimensional to three-dimensional backprojection function as

and the projection function from 3D to 2D is

and the projection point is

where reference can be made to the relevant steps in the above embodiments, which will not be described in detail herein.

In one implementation scenario, to reduce computational complexity, the mobile device further performs downsampling on the previous frame image to provide a thumbnail image of the previous frame image (e.g., 40*30 or smaller image), thereby performing a projective transformation on the pixel points in the thumbnail image using the pose transformation parameters between the image to be processed and the image of the previous frame to obtain the projection point of the pixel point in the thumbnail image in the image to be processed. In another implementation scenario, to reduce computational complexity, the mobile device can also project the pixel points in the thumbnail image onto the unit sphere, i.e., the depth values of the pixel points in the thumbnail image are It can be uniformly set to 1, and further according to actual application needs, the depth value can be uniformly set to other numerical values, and is not limited here.

In step S452, using the difference between the pixel values of at least some of the pixel points in the image of the previous frame and the pixel values of the projected points corresponding to at least some of the pixel points in the image to be processed. to construct a target function for the pose transformation parameters.

と、処理されるべき画像内のローカル領域における、画素点の少なくとも一部に対応する投影点

の画素値

との間の差異を使用して、ポーズ変換パラメータに関するターゲット関数を構築することができる。 In one implementation scenario, the mobile device determines the pixel values of at least some of the pixel points in the local region within the image of the previous frame.

and projection points corresponding to at least some of the pixel points in the local region in the image to be processed

pixel value of

can be used to construct a target function for the pose transformation parameters.

In another implementation scenario, when performing downsampling on the image of the previous frame, the mobile device corresponds the pixel values of the pixel points in the thumbnail image with those pixel values in the image to be processed. The difference between the pixel values of the projection points can be used to construct a target function for the pose transformation parameters.

Here, the target function can refer to the relevant steps in the above embodiments and will not be described in detail here.

In step S453, using the pose transformation parameters obtained by solving the target function, perform transformation processing on the camera pose parameters of the image of the previous frame to obtain the rotation parameters of the image to be processed. .

In the process of optimizing and solving the above target function, the mobile device only optimizes the rotation parameters and uses the pose transformation parameters obtained by solving to the camera pose parameters of the image of the previous frame. A transformation process may be performed on it to obtain the camera pose parameters of the image to be processed, and the rotation parameter within the camera pose parameters may be extracted and used as the rotation parameter of the image to be processed.

In one implementation scenario, in order to improve the robustness of visual positioning, the mobile device continues to detect the current acceleration information of the camera after obtaining the rotation parameters of the image to be processed, and the acceleration information is preset and the steps of obtaining acceleration information and determining whether the acceleration information is in a preset exercise state are the relevant steps of the above disclosed embodiments. and will not be described in detail here. When in a preset state of motion, the camera in this case can be considered to be in a static state or a state of uniform velocity motion, and then re-execute the step of obtaining the gravity information of the camera and subsequent steps. If the camera is not in a preset motion state, then the camera can be considered to be in a state of significant motion, and the step of detecting the camera's current acceleration information and subsequent steps can then be repeated. can be executed. If the visual positioning is inaccurate, repeat detecting the current acceleration information of the camera, determine whether the acceleration information is in the preset motion state, and if it is in the preset motion state, the camera's gravity Re-executing the step of obtaining information and subsequent steps can improve the robustness of visual positioning.

At step S46, the image to be processed is used as the current image.

After obtaining the rotation parameters of the image to be processed, the mobile device may use the image to be processed in order to still be able to continue updating the rotation parameters in the event of significant motion or significant changes in lighting conditions. as the current image, and based on the camera pose parameters of the current image above, the step of obtaining the camera pose parameters of the image to be processed after the current image and subsequent steps can be reperformed .

In step S47, step S43 and subsequent steps are re-executed.

Distinct from the above example, the camera pose parameters are set to include rotation parameters and displacement parameters, and after the mobile device obtains the camera pose parameters of the image to be processed after the current image, the Determining displacement parameters at which the image to be processed cannot be obtained in response to the camera pose parameters of the image to be processed failing to satisfy a preset steady state condition, thereby causing the previous frame of the image to be processed to be Use the pixel values of the image and the camera pose parameters of the previous frame image to obtain the rotation parameters of the image to be processed, and if the camera pose parameters are inaccurate, use the image pixels directly. By estimating the rotation parameter, it is possible to avoid the problem that the virtual object in the virtual reality sticks to the screen due to the inability to update the rotation parameter.

Referring to FIG. 6, FIG. 6 is a schematic diagram of the framework of one embodiment of the visual positioning device 60 of the present invention. The visual positioning device 60 comprises a gravity information acquisition unit 61, a first pose acquisition unit 62 and a second pose acquisition unit 63, the gravity information acquisition unit 61 being configured to acquire the gravity information of the camera, and the first A pose obtainer 62 is configured to obtain camera pose parameters of a current image taken by the camera in a preset motion state using the gravity information, and a second pose obtainer 63 is configured to obtain the camera pose parameters of the current image. It is configured to obtain camera pose parameters of an image to be processed after the current image based on the camera pose parameters.

The above technical solution is to obtain the gravity information of the camera, use the gravity information to obtain the camera pose parameters of the current image taken by the camera in the preset motion state, and obtain the camera pose parameters of the current image Based on the parameters, the camera pose parameters of the image to be processed after the current image can be obtained, and furthermore, visual positioning can be performed only by relying on the camera and gravity information, so the use of visual positioning technology The cost of use can be reduced and the range of use of visual positioning technology can be expanded.

In some disclosed embodiments, the gravity information includes gravity direction information, and the visual positioning device 60 is further configured to obtain feature direction information for feature points in the current image. The visual positioning device 60, comprising an obtaining unit, is further configured to obtain depth information of the feature points in the current image using the feature direction information and the gravity direction information of the feature points. , wherein the second pose obtaining unit 63 further extracts feature points in the image to be processed after the current image based on the depth information of the feature points in the current image and the camera pose parameters of the current image. and camera pose parameters of the image to be processed.

Different from the above embodiment, the feature direction information of the feature point in the current image is obtained, and the feature direction information of the feature point and the gravity direction information included in the gravity information are used to obtain the feature point in the current image. Based on the current image, the depth information of the feature points in the current image and the camera pose parameters of the current image can be initialized by obtaining the depth information of the feature points in the current image Based on the information and the camera pose parameters of the current image, the depth information of the feature points in the image to be processed after the current image and the camera pose parameters of the image to be processed can be obtained, and the initial initiation operation is performed. Therefore, the response speed of visual positioning can be improved without the need to scan multi-frame images.

In some disclosed embodiments, the feature direction information includes a feature point direction vector, the gravity direction information includes a gravity vector, the depth information includes a depth value of the feature point, and the depth information obtaining unit comprises: , a first computing sub-unit configured to perform a first preset operation on the direction vector and the gravity vector of the feature point to obtain an included angle between the direction vector and the gravity vector of the feature point; The information obtaining unit comprises a second calculation sub-unit configured to perform a second preset calculation on the preset height and included angle of the camera to obtain the depth value of the feature point.

Different from the above embodiment, the feature direction information is set to include the direction vector of the feature point, the gravity direction information is set to include the gravity vector, and the depth information is set to include the depth value of the feature point. and perform a first preset operation on the direction vector and the gravity vector of the feature point to obtain the included angle between the direction vector and the gravity vector of the feature point, thereby obtaining the preset height of the camera A second preset operation may be performed on height and included angle to obtain feature point depth values to help reduce the computational complexity of obtaining feature point depth values.

In some disclosed embodiments, the first preset operation includes a dot product operation and/or the second preset operation includes dividing the preset height by the cosine value of the included angle.

Distinct from the above example, the first preset operation can be set to include an inner product operation to help reduce the complexity of obtaining the included angle between the direction vector and the gravity vector; The preset arithmetic can be set to include a cosine value that divides the preset height by the included angle to help reduce the complexity of obtaining depth values.

In some disclosed embodiments, the second pose obtainer 63 uses a preset pose tracking scheme to track depth information of feature points in the current image, relative to camera pose parameters of the current image. a second pose acquisition comprising a pose tracking sub-unit configured to perform processing to obtain depth information of feature points in the image of the next frame of the current image and camera pose parameters of the image of the next frame; A unit 63 uses the image of the next frame as the current image, and uses the preset pose tracking method to track the depth information of the feature points in the current image and the camera pose parameters of the current image. and a re-executing sub-unit configured to re-execute the steps following.

Distinguished from the above embodiment, the preset pose tracking method is used to perform tracking processing on the depth information of the feature points in the current image, the camera pose parameters of the current image, and the Get the depth information of the feature points in the next frame image and the camera pose parameters of the next frame image, thereby using the next frame image as the current image and using the preset pose tracking method and re-performing the step of performing a tracking process on the depth information of the feature points in the current image, the camera pose parameters of the current image and subsequent steps, and further calculating the camera pose parameters for each frame. can help reduce the cumulative error of the camera pose parameters.

In some disclosed embodiments, the pose tracking subunit is configured to use the depth information of the feature points in the current image to determine the projection points of the feature points in the image of the next frame. A feature point projection unit calculates a current image and a next image based on the difference between the pixel values of the feature points in the local region in the current image and the pixel values of the projection points in the local region in the image of the next frame. a pose transformation parameter calculation unit configured to obtain pose transformation parameters between the image of the next frame and a camera pose of the image of the next frame using the pose transformation parameters and the camera pose parameters of the current image; A camera pose parameter calculator configured to obtain the parameters and a camera pose parameter optimizer configured to use the converged 3D points to optimize the camera pose parameters for the image of the next frame. and a depth information obtaining unit configured to obtain a probability distribution of depth information of the feature points and use the probability distribution to obtain depth information of the feature points in the image of the next frame.

Distinguishing from the above example, by using the depth information of the feature points in the current image to determine the projection points of the feature points in the image of the next frame, the feature points in the local region in the current image are and the pixel values of the projection point in the local region in the image of the next frame, get the pose transformation parameters between the current image and the image of the next frame, and pose Use the transformation parameters and the camera pose parameters of the current image to get the camera pose parameters of the next frame image, and use the converged 3D points to optimize the camera pose parameters of the next frame image. can further optimize the camera pose parameters, which helps improve the accuracy of the camera pose parameters, obtain the probability distribution of the depth information of the feature points, and use the probability distribution to determine the next frame By obtaining the depth information of the feature points in the image, the depth information can be optimized in the imaging process based on the distribution probability of the depth information.

In some disclosed embodiments, the camera pose parameters include rotation parameters and displacement parameters, and visual positioning device 60 further determines that the camera pose parameters of the image to be processed satisfy preset steady state conditions. The visual positioning device 60 further comprises a camera pose detector configured to determine a displacement parameter for which the image to be processed cannot be obtained in response to the fact that the previous frame of the image to be processed. a rotation parameter updater configured to obtain a rotation parameter of the image to be processed using the pixel values of the image of the previous frame and the camera pose parameters of the image of the previous frame.

Distinct from the above embodiment, the camera pose parameters are set to include rotation parameters and displacement parameters, and after obtaining the camera pose parameters of the image to be processed after the current image, Determining a displacement parameter for which the image to be processed cannot be obtained in response to the camera pose parameter not satisfying a preset steady state condition, thereby resulting in pixel values of an image of a frame preceding the image to be processed. and the camera pose parameters of the image of the previous frame to obtain the rotation parameters of the image to be processed, and if the camera pose parameters are inaccurate, the image pixels are directly used to obtain the rotation parameters , and can help reduce the probability of problems such as virtual objects sticking to the screen in virtual reality due to the inability to update the rotation parameters.

In some disclosed embodiments, the rotation parameter updater uses pose transformation parameters between the image to be processed and the previous frame image to at least a projection transformation sub-unit configured to perform a projection transformation on the portion to obtain projection points of at least some of the pixel points in the image to be processed, wherein the rotation parameter update unit is configured to perform a projection transformation on the portion of the for pose transformation parameters using differences between pixel values of at least some of the pixel points in the image of the frame and pixel values of projection points in the image to be processed corresponding to at least some of the pixel points; A function construction sub-unit configured to construct a target function, wherein the rotation parameter update unit uses the pose transformation parameters obtained by solving the target function to the camera pose parameters of the image of the previous frame. a parameter obtaining sub-unit configured to perform a transformation process on the image to obtain a rotation parameter of the image to be processed.

In contrast to the above embodiment, the pose transformation parameters between the image to be processed and the image of the previous frame are used to perform a projective transformation on at least some of the pixel points in the image of the previous frame. obtaining projection points of at least some of the pixel points in the image to be processed, pixel values of at least some of the pixel points in the image of the previous frame, and pixel values of the pixel points in the image to be processed; constructing a target function for the pose transformation parameters using the differences between the pixel values of the corresponding projection points at least in part, thereby using the pose transformation parameters obtained by solving the target function; performing a transform operation on the camera pose parameters of the image of the previous frame to obtain rotation parameters of the image to be processed, obtaining the rotation parameters based at least in part on the pixel points; It can help to avoid the computational complexity of calculating the rotation parameters.

In some disclosed embodiments, the rotation parameter updater is configured to perform a downsampling operation on the previous frame image to obtain a thumbnail image of the previous frame image. a sub-unit, the projection transformation sub-unit further performing projection transformation on the pixel points in the thumbnail image using the pose transformation parameters between the image to be processed and the image of the previous frame; to obtain projection points of pixel points in the thumbnail image in the image to be processed.

Distinct from the above embodiment, by performing a downsampling process on the previous frame image to obtain a thumbnail image of the previous frame image, the image between the previous frame image to be processed is performs a projective transformation on the pixel points in the thumbnail image using the pose transformation parameters of get points. This can help reduce the computational complexity of calculating the rotation parameters.

In some disclosed embodiments, the visual positioning device 60 is further configured to further detect current acceleration information of the camera and determine whether the acceleration information is in a preset motion state. Gravity information acquisition unit 61, first pose acquisition unit 62, and second pose acquisition unit 63 further perform a step of acquiring gravity information of the camera and subsequent steps if the determination is yes. The acceleration detection unit is further configured to re-execute the step of detecting the current acceleration information of the camera and subsequent steps if the determination result is no.

Distinct from the above embodiments, further detecting the current acceleration information of the camera after obtaining the rotation parameters of the image to be processed, determining whether the acceleration information is in the preset motion state, If it is in the preset motion state, re-execute the step of obtaining the gravity information of the camera and subsequent steps, and re-execute the step of detecting the current acceleration information of the camera and subsequent steps if it is not in the preset motion state. and further improve the robustness of visual positioning.

In some disclosed embodiments, the gravity information includes gravity direction information, the camera pose parameters include rotation parameters and displacement parameters, and the first pose obtainer 62 uses the gravity direction information to create a world Obtaining the rotation angles of the camera respectively corresponding to the coordinate system x-coordinate axis, y-coordinate axis and z-coordinate axis, the direction of gravity after the camera is rotated according to the rotation angle is configured to be the same as the opposite direction of the z-coordinate axis. With a rotation angle acquisition sub-unit, the first pose acquisition unit 62 comprises a parameter initialization sub-unit configured to use the rotation angle to acquire rotation parameters and set displacement parameters to preset numerical values.

In embodiments and other embodiments of the present invention, a "portion" may be a partial circuit, partial processor, partial program or software, etc., and of course may be a unit, modular or non-modular. may

Different from the above embodiment, the gravity direction information is used to obtain the rotation angles of the camera corresponding to the world coordinate system x-, y-, and z-coordinate axes, respectively, and the gravity after the camera rotates according to the rotation angle The direction is the same as the opposite direction of the z coordinate axis, the rotation angle is used to obtain the rotation parameter, the displacement parameter is set to a preset number to obtain the rotation parameter by gravity alignment, and the camera The pose parameters can be initialized, which helps reduce the amount of calculation for initializing the camera pose parameters.

In some disclosed embodiments, the origin of the world coordinate system is the position when the camera takes the current image, and the preset number is zero.

Distinct from the above example, the origin of the world coordinate system is set to the position when the camera takes the current image, and the preset numerical value is set to 0 to reduce the complexity of initializing the displacement parameters. can help.

In some disclosed embodiments, the preset motion state is a static state or a uniform velocity motion state, and/or the gravity information is obtained using the acceleration information of the camera in the preset state. It is a thing.

Different from the above embodiments, the preset motion state can be set to a static state or a uniform velocity motion state to help improve the accuracy of initializing the camera pose parameters of the current image. Gravity information can be obtained using the acceleration information of the camera in the state of , and only the accelerometer can be used to obtain the gravity information, thereby further reducing the cost of using visual positioning technology and reducing visual It can help expand the range of use of target positioning techniques.

Referring to FIG. 7, FIG. 7 is a schematic diagram of the framework of an embodiment of an electronic device 70 of the present invention. The electronic device 70 comprises a memory 71 and a processor 72 coupled together, the processor 72 executing program instructions stored in the memory 71 to implement the steps of any one of the visual positioning method embodiments described above. configured to In one implementation scenario, the electronic device 70 may include mobile devices such as, but not limited to, mobile phones, tablets, robots, etc., and is not limited herein.

In the disclosed embodiment, processor 72 is configured to control itself and memory 71 to implement the steps of any one of the visual positioning method embodiments described above. Processor 72 may also be referred to as a central processing unit (CPU). Processor 72 may be an integrated circuit chip having signal processing capabilities. Processor 72 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and so on. Further, processor 72 can be co-embodied by an integrated circuit chip.

The above technical solutions can reduce the cost of using visual positioning technology and expand the scope of using visual positioning technology.

Referring to FIG. 8, FIG. 8 is a schematic diagram of the framework of an embodiment of the computer-readable storage medium 80 of the present invention. Computer readable storage medium 80 stores program instructions 801 that can be executed by a processor, and program instructions 801 are used to implement the steps of any one of the visual positioning method embodiments described above.

The above technical solutions can reduce the cost of using visual positioning technology and expand the scope of using visual positioning technology.

It should be understood that in some embodiments according to the present invention, the disclosed method and apparatus can be implemented in other ways. For example, the embodiments of the apparatus described above are exemplary only, e.g., the division of modules or units is merely the division of logical functions, and the actual implementation may involve other division schemes. , for example, units or components may be integrated or integrated into another system, and some features may be ignored or not performed. Further, the mutual or direct couplings or communication connections shown or discussed may be realized using a number of interfaces, and the indirect couplings or communication connections between devices or units may be electrical or mechanical. It may be in a simple shape or in another shape.

Units described as separate parts may or may not be physically separated, and parts denoted as units may or may not be physical units. That is, it may be centrally located or distributed among network units. According to actual needs, some or all units therein can be selected to achieve the purpose of the technical solution of the present embodiment.

Furthermore, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be present. may be integrated into one unit. Said integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the said technical solution is embodied in the form of a software product. The computer software product may be stored in a single storage medium and implemented in a single computer device (which may be a personal computer, server, network device, etc.) or processor. It contains some instructions for performing all or part of the steps of the method. The above storage media include U disk, mobile hard disk, read-only memory (ROM: Read-Only Memory), random access memory (RAM: Random Access Memory), magnetic disk, optical disk, etc. media.

In an embodiment of the present invention, by obtaining the gravity information of the camera, the gravity information is used to obtain the camera pose parameters of the current image taken by the camera in the preset motion state, and the camera of the current image Based on the pose parameters, the camera pose parameters of the image to be processed after the current image can be obtained, and furthermore, the visual positioning can be performed only by relying on the camera and gravity information, so the visual positioning technology It can reduce the cost of using , and expand the scope of use of visual positioning technology.

Claims (15)

A visual positioning method comprising:
obtaining camera gravity information;
using the gravity information to obtain camera pose parameters for a current image taken by the camera in a preset motion state;
obtaining camera pose parameters for an image to be processed after the current image based on camera pose parameters for the current image ;
the camera pose parameters include rotation parameters and displacement parameters;
After obtaining camera pose parameters of an image to be processed after the current image based on the camera pose parameters of the current image, the visual positioning method comprises:
determining that a displacement parameter of the to-be-processed image cannot be obtained in response to a camera pose parameter of the to-be-processed image not meeting a preset steady state condition;
obtaining rotation parameters of the image to be processed using pixel values of a previous frame image of the image to be processed and camera pose parameters of the previous frame image. , visual positioning method. The gravity information includes gravity direction information, and before obtaining the camera pose parameters of an image to be processed after the current image based on the camera pose parameters of the current image, the visual positioning method comprises: ,
obtaining feature direction information for feature points in the current image;
obtaining depth information for the feature points in the current image using the feature direction information and the gravity direction information for the feature points;
Obtaining camera pose parameters of an image to be processed after the current image based on the camera pose parameters of the current image comprises:
Depth information of the feature points in an image to be processed after the current image and the depth information of the feature points in the image to be processed based on depth information of the feature points in the current image and camera pose parameters of the current image. including getting the camera pose parameters of the image,
A visual positioning method according to claim 1. the feature direction information includes a direction vector of the feature point, the gravity direction information includes a gravity vector, the depth information includes a depth value of the feature point;
obtaining depth information for the feature point in the current image using the feature direction information and the gravity direction information for the feature point;
performing a first preset operation on the direction vector and the gravity vector of the feature point to obtain an included angle between the direction vector and the gravity vector of the feature point;
performing a second preset operation on the camera preset height and the included angle to obtain a depth value for the feature point;
the first preset operation includes an inner product operation;
and/or, calculating the second preset comprises dividing the preset height by the cosine value of the included angle;
A visual positioning method according to claim 2. Depth information of the feature points in an image to be processed after the current image and the depth information of the feature points in the image to be processed based on depth information of the feature points in the current image and camera pose parameters of the current image. To get the camera pose parameters of an image,
Using a preset pose tracking scheme, performing a tracking process on the depth information of the feature points in the current image, the camera pose parameters of the current image, and the next frame of the current image. obtaining depth information for the feature points in an image and camera pose parameters for the next frame image;
using the image of the next frame as the current image, and using the preset pose tracking scheme, depth information of the feature points in the current image, relative to camera pose parameters of the current image; performing the tracking process and re-performing subsequent steps;
A visual positioning method according to claim 2. using the preset pose tracking scheme, performing a tracking process on depth information of the feature points in the current image, camera pose parameters of the current image, and performing a tracking process on the next frame of the current image; obtaining depth information of the feature points in the image of and camera pose parameters of the image of the next frame,
using depth information for the feature points in the current image to determine projection points for the feature points in the next frame image;
Based on the difference between the pixel values of the feature points in the local region in the current image and the pixel values of the projection points in the local region in the next frame image, the current image and the next frame obtaining pose transformation parameters to and from the image of the frame;
obtaining camera pose parameters for the next frame image using the pose transformation parameters and the current image camera pose parameters;
optimizing camera pose parameters for the next frame image using the converged 3D points;
obtaining a probability distribution of depth information for the feature points, and using the probability distribution to obtain depth information for the feature points in an image of a next frame;
A visual positioning method according to claim 4. obtaining rotation parameters of the image to be processed using pixel values of a previous frame image of the image to be processed and camera pose parameters of the previous frame image;
performing a projective transformation on at least some of the pixel points in the previous frame image using pose transformation parameters between the image to be processed and the previous frame image; obtaining projection points of at least some of said pixel points in an image to be processed;
using the difference between the pixel values of at least some of the pixel points in the previous frame image and the pixel values of the projected points corresponding to at least some of the pixel points in the image to be processed. to construct a target function for the pose transformation parameters;
Using the pose transformation parameters obtained by solving the target function, performing transformation processing on camera pose parameters of the previous frame image to obtain rotation parameters of the image to be processed. including
A visual positioning method according to claim 1 . performing a projective transformation on at least some of the pixel points in the previous frame image using pose transformation parameters between the image to be processed and the previous frame image; Before obtaining projection points of at least some of the pixel points in the image to be processed, the visual positioning method comprises:
further comprising performing a downsampling process on the previous frame image to obtain a thumbnail image of the previous frame image;
performing a projective transformation on at least some of the pixel points in the image to be processed using pose transformation parameters between the image to be processed and the image of the previous frame; obtaining projection points of at least some of said pixel points in an image to be processed,
performing a projective transformation on pixel points in the thumbnail image using pose transformation parameters between the image to be processed and the image of the previous frame to perform the projection transformation on the image to be processed; including obtaining a projected point of the pixel point in the thumbnail image;
A visual positioning method according to claim 6 . after obtaining a rotation parameter of the image to be processed using the pixel values of the image of the previous frame of the image to be processed and the camera pose parameters of the image of the previous frame, the visual positioning method teeth,
detecting current acceleration information of the camera and determining whether the acceleration information is in the preset motion state;
if in the preset motion state, re-performing the step of obtaining gravity information of the camera and subsequent steps;
if not in the preset motion state, detecting current acceleration information of the camera and re-performing subsequent steps;
A visual positioning method according to claim 1 . The gravity information includes gravity direction information, the camera pose parameters include rotation parameters and displacement parameters, and the gravity information is used to determine the camera pose of a current image taken by the camera in a preset motion state. Getting the parameters
using the gravity direction information to obtain rotation angles of the camera respectively corresponding to the world coordinate system x-coordinate axis, y-coordinate axis and z-coordinate axis, wherein the gravity direction after the camera is rotated according to the rotation angle is the same as the opposite direction of the z-coordinate axis;
using the rotation angle to obtain the rotation parameter and setting the displacement parameter to a preset number;
The origin of the world coordinate system is the position when the camera captures the current image, and the preset value is 0.
A visual positioning method according to claim 1. the preset motion state includes a static state or a uniform speed motion state;
and/or the gravity information is obtained using acceleration information of the camera in the preset motion state.
10. A visual positioning method according to any one of claims 1-9 . A visual positioning device comprising:
a gravity information acquisition unit configured to acquire gravity information of a camera;
a first pose obtaining unit configured to obtain camera pose parameters of a current image taken by the camera in a preset motion state using the gravity information;
a second pose obtaining unit configured to obtain camera pose parameters of an image to be processed after the current image based on the camera pose parameters of the current image , the camera pose parameters comprising: a second pose obtainer including rotation and displacement parameters ;
a camera pose detector configured to determine a displacement parameter at which the to-be-processed image cannot be obtained in response to the camera-pose parameter of the to-be-processed image failing to satisfy a preset steady state condition; ,
A rotation parameter configured to obtain a rotation parameter of the image to be processed using a pixel value of a previous frame image of the image to be processed and a camera pose parameter of the previous frame image. and an updater . The rotation parameter updating unit
performing a projective transformation on at least some of the pixel points in the previous frame image using pose transformation parameters between the image to be processed and the previous frame image; a projection transformation sub-unit configured to obtain projection points of at least some of the pixel points in the image to be processed;
using the difference between the pixel values of at least some of the pixel points in the previous frame image and the pixel values of the projected points corresponding to at least some of the pixel points in the image to be processed. a function construction sub-unit configured to construct a target function in terms of the pose transformation parameters as a
Using the pose transformation parameters obtained by solving the target function, performing transformation processing on camera pose parameters of the previous frame image to obtain rotation parameters of the image to be processed. a parameter acquisition sub-unit configured to
A visual positioning device according to claim 11. An electronic device comprising a memory and a processor coupled to each other, the processor executing program instructions stored in the memory to perform a visual positioning method according to any one of claims 1 to 10 . An electronic device that carries out A computer-readable storage medium storing a program for realizing the visual positioning method according to any one of claims 1 to 10 in a computer. A computer program for implementing a visual positioning method according to any one of claims 1 to 10 on a computer.

Images