CN115265529A

CN115265529A - Target object positioning method, system, device, electronic equipment and storage medium

Info

Publication number: CN115265529A
Application number: CN202210783952.XA
Authority: CN
Inventors: 庞业勇; 吴頔; 陈志远; 都大龙
Original assignee: Beijing Jianzhi Technology Co ltd
Current assignee: Beijing Jianzhi Technology Co ltd
Priority date: 2022-07-05
Filing date: 2022-07-05
Publication date: 2022-11-01

Abstract

The embodiment of the invention provides a method, a system, a device, electronic equipment and a storage medium for positioning a target object, wherein the method comprises the following steps: acquiring environmental data and motion data acquired by a target object; acquiring an original Hessian matrix and dimension information aiming at the original Hessian matrix, and marginalizing the original Hessian matrix according to the dimension information to obtain a target Hessian matrix; solving the target Hessian matrix to obtain a plurality of sub-matrices; performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is a matrix used for representing state information of the visual inertial fusion system, and the residual matrix is a matrix used for optimizing variable parameters; and calculating the environment data and the motion data according to the Jacobian matrix and the residual error matrix to obtain the positioning information of the target object.

Description

Target object positioning method, system, device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of machine learning technologies, and in particular, to a method for positioning a target object, a visual inertial fusion system, a device for positioning a target object, an electronic apparatus, and a computer-readable storage medium.

Background

Visual Inertial systems (VINS) are widely used in unmanned autonomous systems such as unmanned aerial vehicles and autonomous vehicles. Although most of modern unmanned autonomous systems use GNNS (Global Navigation Satellite System) and 4G/5G technologies as basic positioning technologies, the visual inertial System is still a very good auxiliary positioning technology for places with poor signal coverage, such as tunnels and remote mountainous areas. Although the visual inertial system products have been applied to some applications, the high-precision visual inertial system still has some technical challenges. How to fuse the data of a vision and Inertia Measurement Unit (IMU) and research and develop a vision inertia system with high precision and high robustness has important research and development significance.

For the visual inertial fusion system, the data processing may include two parts, namely pre-processing and post-processing, the pre-processing may be a method based on feature detection and tracking, and the post-processing may be an optimization method based on nonlinear solution. The post-processing enhances the robustness of the system, improves the positioning accuracy, and correspondingly, the system has larger computation amount, the pre-processing can improve the data computation speed, but the positioning accuracy is lower, and if a global optimization mode is adopted, the system has high accuracy but the computation efficiency is obviously reduced.

Disclosure of Invention

The embodiment of the invention provides a method, a system, a device, electronic equipment and a computer readable storage medium for positioning a target object, and aims to solve or partially solve the problems of low operation efficiency and low positioning accuracy in the positioning process of a visual inertial fusion system.

The embodiment of the invention discloses a method for positioning a target object, which comprises the following steps:

acquiring environmental data and motion data acquired by a target object;

acquiring an original Hessian matrix and dimension information aiming at the original Hessian matrix, and marginalizing the original Hessian matrix according to the dimension information to obtain a target Hessian matrix;

solving the target Hessian matrix to obtain a plurality of sub-matrices;

performing accelerated calculation according to the plurality of sub-matrixes to obtain a Jacobian matrix and a residual error matrix, wherein the Jacobian matrix is a matrix used for representing state information of the visual inertial fusion system, and the residual error matrix is a matrix used for optimizing variable parameters;

and calculating the environment data and the motion data according to the Jacobian matrix and the residual error matrix to obtain the positioning information of the target object.

Optionally, the performing accelerated computation according to the plurality of sub-matrices to obtain a jacobian matrix and a residual matrix includes:

adding the first sub-matrix and the transposed first sub-matrix to obtain a first target matrix;

performing matrix operation according to first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix;

multiplying the first inverse matrix with the second sub-matrix and the third sub-matrix respectively to obtain a second target matrix corresponding to the second sub-matrix and a third target matrix corresponding to the third sub-matrix;

performing matrix operation according to second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix;

performing matrix operation by adopting the third characteristic information of the diagonal matrix to obtain a Jacobian matrix;

and performing matrix operation by adopting the second inverse matrix and the third sub-matrix to obtain a residual error matrix.

Optionally, the performing matrix operation according to the first feature information of the first target matrix to obtain a first inverse matrix of the first target matrix includes:

acquiring a first eigenvalue and a first eigenvector of the first target matrix;

and multiplying a first matrix constructed by the first eigenvalue and a second matrix constructed by the first eigenvector to obtain a first inverse matrix corresponding to the first target matrix.

Optionally, the performing matrix operation according to the second feature information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix includes:

acquiring a second eigenvalue and a second eigenvector of the second target matrix;

multiplying a third matrix constructed by the second eigenvalue and a fourth matrix constructed by the second eigenvector to obtain a target vector and a target inverse vector corresponding to the second target matrix;

and carrying out square root operation on each scalar in the target vector to obtain a diagonal matrix corresponding to the target vector, and carrying out square root operation on each scalar in the target inverse vector to obtain a second inverse matrix corresponding to the diagonal matrix.

Optionally, the performing matrix operation by using the third feature information of the diagonal matrix to obtain a jacobian matrix includes:

acquiring a third eigenvector of the diagonal matrix;

and multiplying the diagonal matrix and the third eigenvector to obtain a Jacobian matrix.

Optionally, performing matrix operation by using the second inverse matrix and the third sub-matrix to obtain a residual matrix, including:

acquiring a fourth eigenvector of the second inverse matrix;

and multiplying the second inverse matrix, the fourth feature vector and the third sub-matrix to obtain a residual error matrix.

Optionally, the obtaining the target hessian matrix includes obtaining dimension information of the target hessian matrix, where the dimension information includes first dimension information that needs to be marginalized and second dimension information that needs to be retained, and performing marginalization processing on the original hessian matrix according to the dimension information, where the obtaining the target hessian matrix includes:

and performing marginalization processing on the original Hessian matrix by adopting the first dimension information and the second dimension information to construct a target Hessian matrix.

The embodiment of the invention also discloses a visual inertia fusion system which comprises a visual and inertia measurement unit, a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array and a tensor acceleration calculation control unit; the tensor data memory stores environment data and motion data acquired by the vision and inertia measurement unit; wherein the content of the first and second substances,

the tensor acceleration calculation control unit is used for responding to the received positioning operation instruction transmitted by the control instruction memory and transmitting the positioning operation instruction to the tensor arithmetic logic operation array;

the tensor arithmetic logic operation array is used for acquiring an original Hessian matrix from the tensor data storage according to the positioning operation instruction, acquiring dimension information aiming at the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimension information to acquire a target Hessian matrix; solving the target Hessian matrix to obtain a plurality of sub-matrices; performing accelerated calculation according to the plurality of sub-matrixes to obtain a Jacobian matrix and a residual error matrix; calculating the environment data and the motion data according to the Jacobian matrix and the residual error matrix to obtain the positioning information of the target object, wherein the Jacobian matrix is a matrix used for representing state information of a visual inertia fusion system, and the residual error matrix is a matrix used for optimizing variable parameters;

the tensor data memory is used for storing the target Hessian matrix, the Jacobian matrix, the residual error matrix and the positioning information.

Optionally, the tensor arithmetic logic operation array includes a computation instruction parsing unit, a number of arithmetic logic units, and an adder tree;

the calculation instruction analysis unit is used for analyzing the positioning operation instruction, acquiring the environment data, the motion data and an original Hessian matrix from the tensor data storage, and acquiring dimension information aiming at the original Hessian matrix;

the arithmetic logic unit is used for adding the first sub-matrix and the transposed first sub-matrix to obtain a first target matrix; performing matrix operation according to first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; multiplying the first inverse matrix with the second sub-matrix and the third sub-matrix respectively to obtain a second target matrix corresponding to the second sub-matrix and a third target matrix corresponding to the third sub-matrix; performing matrix operation according to second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; performing matrix operation by adopting the third characteristic information of the diagonal matrix to obtain a Jacobian matrix; performing matrix operation by adopting the second inverse matrix and the third sub-matrix to obtain a residual matrix;

the adder tree is used for carrying out operation acceleration when matrix multiplication is carried out.

Optionally, the arithmetic logic unit comprises a first arithmetic logic unit and a plurality of second arithmetic logic units; wherein the first arithmetic logic unit is used for executing at least one operation of addition, subtraction, multiplication, division and evolution, and the second arithmetic logic unit is used for executing at least one operation of addition, subtraction and multiplication.

The embodiment of the invention also discloses a device for positioning the target object, which comprises:

the data acquisition module is used for acquiring environmental data and motion data acquired by a target object;

the Hessian matrix acquisition module is used for acquiring an original Hessian matrix and dimension information aiming at the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimension information to acquire a target Hessian matrix;

the solving module is used for solving the target Hessian matrix to obtain a plurality of sub-matrices;

the acceleration calculation module is used for carrying out acceleration calculation according to the plurality of sub-matrixes to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is a matrix used for representing state information of the visual inertial fusion system, and the residual matrix is a matrix used for optimizing variable parameters;

and the positioning module is used for calculating the environment data and the motion data according to the Jacobian matrix and the residual error matrix to obtain the positioning information of the target object.

Optionally, the sub-matrices at least include a first sub-matrix, a second sub-matrix, and a third sub-matrix, and the accelerated computation module is specifically configured to:

Optionally, the acceleration calculation module is specifically configured to:

and performing square root operation on each scalar in the target vectors to obtain a diagonal matrix corresponding to the target vectors, and performing square root operation on each scalar in the target inverse vectors to obtain a second inverse matrix corresponding to the diagonal matrix.

Optionally, the acceleration calculation module is specifically configured to:

acquiring a third eigenvector of the diagonal matrix;

Optionally, the acceleration calculation module is specifically configured to:

acquiring a fourth eigenvector of the second inverse matrix;

Optionally, the dimension information includes first dimension information that needs marginalization and second dimension information that needs to be retained, and the hessian matrix acquisition module is specifically configured to:

The embodiment of the invention also discloses electronic equipment which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory finish mutual communication through the communication bus;

the memory is used for storing a computer program;

the processor is configured to implement the method according to the embodiment of the present invention when executing the program stored in the memory.

Also disclosed is a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform a method according to an embodiment of the invention.

The embodiment of the invention has the following advantages:

in the embodiment of the invention, when a target object is positioned, acquired environmental data and motion data can be acquired, then a hessian matrix for positioning operation and dimension information for the original hessian matrix can be acquired, marginalization processing is performed on the original hessian matrix according to the dimension information to acquire a target hessian matrix, then the target hessian matrix can be solved to acquire a plurality of sub-matrices, accelerated calculation is performed according to the plurality of sub-matrices to acquire a jacobian matrix and a residual matrix, the jacobian matrix is a matrix for representing state information of the visual inertial fusion system, the residual matrix is a matrix for optimizing variable parameters, and then the environmental data and the motion data are calculated according to the jacobian matrix and the residual matrix to acquire the positioning information of the target object, so that in the positioning process, especially when auxiliary positioning is performed through the visual inertial fusion system, the corresponding jacobian matrix and residual matrix are acquired by marginalized accelerated calculation on the hessian matrix in the system, the operation amount of the system can be effectively controlled under the condition that the positioning accuracy is ensured, and the operation efficiency is improved.

Drawings

Fig. 1 is a flowchart illustrating steps of a method for locating a target object according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a visual inertial fusion system provided in an embodiment of the invention;

fig. 3 is a schematic structural diagram of a tensor acceleration calculation control unit provided in an embodiment of the present invention;

FIG. 4 is a schematic diagram of an array of tensor arithmetic logic operations provided in an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a tensor data memory provided in an embodiment of the present invention;

FIG. 6 is a schematic illustration of a Hessian matrix provided in an embodiment of the invention;

fig. 7 is a block diagram of a visual inertial fusion system provided in an embodiment of the present invention;

fig. 8 is a block diagram of a target object locating apparatus provided in the embodiment of the present invention;

fig. 9 is a block diagram of an electronic device provided in an embodiment of the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention more comprehensible, the present invention is described in detail with reference to the accompanying drawings and the detailed description thereof.

As an example, for some positioning scenes, for example, a pilotless automobile runs in an underground parking lot, and a flying automobile moves in a remote mountain area, when no positioning signal exists or a 4G/5G signal is missing or poor in quality, a visual inertial fusion system is required to be relied on for positioning. The current implementation scheme for implementing the visual inertial fusion system mainly includes two modes, namely a microprocessor and an Application Specific Instruction Processor (ASIP), and the general microprocessor has low cost and a short development period, but the current visual inertial fusion system has a low data throughput rate for 1080P resolution, and cannot meet the performance requirements of high-level automatic driving Application scenes on the visual inertial fusion system.

For the visual inertial fusion system, the data processing may include two parts, namely pre-processing and post-processing, the pre-processing may be a method based on feature detection and tracking, and the post-processing may be an optimization method based on nonlinear solution. The post-processing enhances the robustness of the system, improves the positioning accuracy, correspondingly increases the calculation amount of the system, and the pre-processing can improve the data calculation speed, but the positioning accuracy is low, and if a global optimization mode is adopted, the calculation efficiency is obviously reduced although the accuracy of the system is high. Therefore, a sliding window method is introduced, and optimization operation is carried out on a specific number of frames each time, so that the precision and the efficiency are ensured. The new image frame and IMU data updates and replaces the old data during the window sliding process, and the Marginalization (Marginalization) operation can better utilize the information of the historical image frame and IMU. In the process, hessian matrix marginalization calculation is the part of the VINS system where the post-processing calculation accounts for the largest proportion, so that improving the calculation efficiency of hessian matrix marginalization is the key to improve the throughput capacity of the system.

In contrast, one of the core invention points in the invention is to construct a marginalized computation acceleration structure of the visual inertial fusion system, accelerate hessian matrix marginalized computation through the marginalized computation acceleration structure, and improve the computation efficiency of the portion with the largest processing computation proportion in positioning computation, thereby improving the throughput capacity of the visual inertial fusion system by improving the hessian matrix marginalized computation efficiency.

Referring to fig. 1, a flowchart illustrating steps of a method for positioning a target object provided in an embodiment of the present invention is shown, which may specifically include the following steps:

step 101, acquiring environmental data and motion data acquired by a target object;

optionally, for the target object, it may be an object that needs to be positioned, such as a vehicle, a flying automobile, an unmanned aerial vehicle, a ship, etc., and a corresponding sensor may be deployed on the target object for positioning. In the case of poor positioning signal or network signal, the target object cannot effectively position itself, thereby affecting the movement or parking of the target object. In this regard, a visual inertial fusion system may be deployed on the target object, and the auxiliary positioning may be performed based on the visual inertial fusion system when the target object has an abnormal positioning signal and/or network signal.

In particular implementations, the visual-inertial fusion system may respond to signal anomaly commands (including locating signal anomalies, network signal anomalies), followed by environmental data and motion data that may be collected by the visual and inertial measurement units. Alternatively, the visual inertial fusion system can be a fusion system of both binocular cameras and visual inertial system, and then a visual sensor and an inertial measurement sensor can be included in the system, for the visual sensor, it can be various cameras, which record sensor data in the form of images, generally speaking, the frequency is low (24 Hz to 60 Hz), we can estimate its motion through multi-view geometry, in the fusion, its advantage is that the drift is small in the case of nearly no motion, and the pose can be directly estimated; for the inertial measurement sensor, the acceleration and the angular velocity of the inertial measurement sensor can be measured through an accelerometer and a gyroscope, generally speaking, the frequency is very high (100 Hz-1 kHz), the displacement and the rotation of the self motion are obtained through the integration of the acceleration and the angular velocity, and therefore under the condition that a target object generates a positioning signal and/or a network signal is abnormal, the visual inertial fusion system can realize auxiliary positioning through processing the acquired environment data and the motion data.

102, acquiring an original Hessian matrix and dimension information aiming at the original Hessian matrix, and performing marginalization processing on the original Hessian matrix according to the dimension information to acquire a target Hessian matrix;

in the embodiment of the invention, before processing the environmental data and the motion data, the visual inertial fusion system can firstly carry out marginalized calculation acceleration on the hessian matrix, and then process the environmental data and the motion data according to an acceleration result, so as to realize the positioning of the target object.

Wherein, for the visual inertial fusion System, it may be a System based on a multi-core processor chip, and the multi-core processor chip System includes two parts of a Processor System (PS) and a Programmable Logic unit (PL). The visual inertial fusion system can specifically comprise a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array and a tensor calculation acceleration control unit. Optionally, the tensor calculation unit is divided into two organization forms, one is 64-way half precision (FP 16, floating Point 16 bits), the other is 32-way single precision (FP 32, floating Point 32 bits), and the precision of the FP32 can be higher than that of the FP 16.

In an example, referring to fig. 2, a schematic structural diagram of a visual inertial fusion system provided in an embodiment of the present invention is shown, and components related to tensor calculation acceleration in the visual inertial fusion system may include (1) a control instruction memory, (2) a tensor data memory, (3) a tensor arithmetic logic operation array, (4) a tensor calculation acceleration control unit, and (5) a data transmission unit. The control instruction memory (1) is used for storing operation control instruction data, and the data is generated offline through an assembly interpreter and is issued by a PS (packet switched); (2) the tensor data storage can be used for storing an original Hessian matrix, a Hessian matrix needing marginalization, a Jacobian matrix and residual items, and can comprise a plurality of URAM components, wherein the URAM components can be PL internal memory components; (3) the tensor arithmetic logic operation array can be used for completing various tensor calculation operations, including scalar division, square opening and the like, 1D tensor addition, subtraction, multiplication and the like, and the operation of the 2D tensor can be completed through the circulation operation of the 1D tensor, which can include a plurality of organization forms, such as FP16, FP32 and the like; (4) the tensor calculation acceleration control unit can be used for completing the control of storage and tensor calculation; (5) the data transmission unit may be configured to complete data interaction between a programmable logic unit PL side calculator and a processor system PS side DRAM (Dynamic Random Access Memory), and support a Load operation from the PS side data to the PL and a Fetch operation from the PL. The operation amount can be effectively controlled by a marginalized calculation mode; selectively deleting the next new frame or the oldest frame in the sliding window to ensure that the frames in the sliding window have enough parallax; when the oldest frame is deleted, the constraint (i.e., a priori information) of the old frame data to other frames is kept in the form of marginalization constraints.

Referring to fig. 3, a schematic structural diagram of a tensor acceleration calculation control unit provided in the embodiment of the present invention is shown, and the tensor acceleration calculation control unit may include (1) a control instruction unit (PC, program Counter), (2) a data transmission control interface unit, and (3) a tensor calculation control unit. The output of the control instruction unit is the input of the address port of the control instruction memory, the control instruction unit can point to different addresses of the instruction memory according to different operation types, and as mentioned above, the content of the control instruction memory is generated by the offline assembly interpreter, and all marginalized calculation operations can be covered; the instruction output by the control instruction memory directly drives two units, namely (2) a data transmission control interface unit and (3) a tensor calculation control unit; (2) the data transmission control unit can be used for controlling the data transmission module and completing an interface with the data transmission module; (3) the tensor calculation control unit can be used for acquiring corresponding data, an original Hessian matrix and the like from the tensor data storage, and performing Hessian matrix calculation and positioning operation through the tensor arithmetic logic operation array. In addition, the tensor acceleration calculation control unit can further comprise a selector which can be used for determining whether the output result of the control instruction analysis unit is transmitted to (2) the data transmission control interface unit or (3) the tensor calculation control unit according to the instruction type.

Referring to fig. 4, a schematic diagram of a structure of a tensor arithmetic logic operation array provided in the embodiment of the present invention is shown, and for the tensor arithmetic logic operation array, the tensor arithmetic logic operation array may include (1) a computation instruction parsing unit, several (2) - (3) arithmetic logic units, and (4) an adder tree, and the arithmetic logic units may include a homogeneous arithmetic logic unit and a heterogeneous arithmetic logic unit. The calculation instruction analysis unit (1) may be configured to obtain a calculation instruction from the tensor acceleration calculation control unit, select, according to the calculation instruction, corresponding data in the tensor data storage to transmit to the arithmetic logic unit (2) or the arithmetic logic unit (3) for calculation, and after the calculation is completed, select, according to the calculation instruction, a corresponding calculation result to write into the tensor data storage. For 64-path half-precision or 32-path single-precision organizational tensor arithmetic logic operation arrays, 64 or 32 arithmetic logic units are needed in total. In consideration of the actual algorithm and PL resources, the arithmetic logic units can be divided into 2 types, that is, (2) arithmetic logic unit 0 and (3) arithmetic logic unit N. Wherein (2) only one arithmetic logic unit 0 is deployed and can support addition, subtraction, multiplication, division and evolution operations. The rest of the arithmetic logic units are (3) arithmetic logic units N, and only support addition, subtraction and multiplication operations. Meanwhile, an addition tree (4) is arranged in the tensor arithmetic logic operation array and used for accelerating matrix multiplication. In addition, the tensor arithmetic logic operation array further comprises a selector which is used for determining which operation result is transmitted to which arithmetic logic unit according to the instruction type and determining which operation result of the arithmetic logic unit is transmitted to the tensor data storage according to the instruction type.

Referring to fig. 5, a schematic structural diagram of a tensor data memory provided in the embodiment of the present invention is shown, and for the tensor data memory, the tensor data memory may be composed of 16 (2) basic memory cells, and each (2) basic memory cell is composed of 2 (1) urams (UltraRAM). (1) The URAM may be a PL internal memory component, 64bitx4096 in size, with one read port and one write port, each address storing 4 half-precision floating point numbers (FP 16) or 2 single-precision floating point numbers (FP 32). It will be appreciated that, in view of the actual computational requirements, two (1) URAMs are used to form (2) basic memory cells, each (2) basic memory cell having one write port and two read ports. The write ports of the two (1) URAMs are connected together, so that the data stored in the two (1) URAMs are completely consistent; the read ports of the two (1) URAMs are directly used as the two read ports of the (2) basic storage units, so that 2 different numbers can be read from different addresses of the (2) basic storage units at the same time for calculation. Optionally, the tensor data memory is composed of 16 (2) basic memory units, and the effective memory data capacity is 1024bit x 4096. The tensor data memory controls the read-write address and the read-write enable of the tensor data memory by a tensor acceleration calculation control unit, provides operands for the tensor arithmetic logic array and stores the calculation result of the tensor arithmetic logic array.

By the aid of the acceleration calculation structure of the visual inertia fusion system, marginalized calculation acceleration can be effectively performed on the Hessian matrix of the visual inertia fusion system, so that the calculation amount is effectively controlled and the calculation efficiency is improved under the condition that the positioning accuracy of the system is guaranteed.

The sea-son matrix marginalization acceleration calculation can comprise three stages, namely, firstly, constructing a sea-son matrix which can be marginalized, secondly, solving the schur complement of the sea-son matrix, and thirdly, updating the jacobian matrix and the residual error matrix. The method comprises the steps of establishing a marginalizable Hessian matrix, obtaining first dimension information needing marginalization and second dimension information needing to be reserved, and then adopting the first dimension information and the second dimension information to conduct marginalization processing on an original Hessian matrix to establish a target Hessian matrix.

In an example, referring to fig. 6, a schematic diagram of a hessian matrix provided in the embodiment of the present invention is shown, an original hessian matrix may be loaded into a tensor data memory, and approximate second-order state information (i.e., the hessian matrix after marginalization) may be quickly obtained through a marginalization calculation method, specifically, the PS may load matrix data to the PL according to a dimension m that needs to be marginalized and a dimension n that needs to be reserved, so as to obtain a target hessian matrix.

103, solving the target hessian matrix to obtain a plurality of sub-matrices;

as shown in fig. 6, the objective hessian matrix may be solved to obtain Amm matrix, amr matrix, arm matrix, arr matrix and other different sub-matrices, so as to implement the acceleration calculation of hessian matrix marginalization according to the different sub-matrices.

104, performing accelerated calculation according to the plurality of sub-matrixes to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is a matrix used for representing state information of the visual inertial fusion system, and the residual matrix is a matrix used for optimizing variable parameters;

in a specific implementation, the jacobian matrix and the residual matrix can be obtained by performing marginalized accelerated calculation on the hessian matrix. Specifically, the submatrices at least include a first submatrix, a second submatrix, and a third submatrix, the first submatrix and the inverted first submatrix may be added to obtain a first target matrix, matrix operation may be performed according to first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix, the first inverse matrix may be multiplied by the second submatrix and the third submatrix, respectively, to obtain a second target matrix corresponding to the second submatrix and a third target matrix corresponding to the third submatrix, matrix operation may be performed according to second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix, matrix operation may be performed according to third characteristic information of the diagonal matrix to obtain a jacobian matrix, matrix operation may be performed by using the second inverse matrix and the third submatrix to obtain a residual error matrix, thereby edge-based accelerated calculation may be performed by using the hessian matrix to obtain a corresponding jacobian matrix and a residual error matrix, and may effectively control an amount of operation of a system under a condition that ensures positioning accuracy.

In an alternative embodiment, the marginalized acceleration calculation for the hessian matrix may specifically include:

and then, multiplying the first matrix constructed by the first characteristic value and the second matrix constructed by the first characteristic vector to obtain a first inverse matrix corresponding to the first target matrix. After the first inverse matrix is obtained, the first inverse matrix may be multiplied by the second sub-matrix and the third sub-matrix respectively to obtain a second target matrix corresponding to the second sub-matrix and a third target matrix corresponding to the third sub-matrix, then a second eigenvalue and a second eigenvector of the second target matrix may be obtained, the third matrix constructed by the second eigenvalue and a fourth matrix constructed by the second eigenvector may be multiplied to obtain a target vector and a target inverse vector corresponding to the second target matrix, then a square root operation may be performed on each scalar in the target vectors to obtain a diagonal matrix corresponding to the target vectors, and a square root operation may be performed on each scalar in the target inverse vectors to obtain a second inverse matrix corresponding to the diagonal matrix. After the diagonal matrix and the second inverse matrix corresponding to the diagonal matrix are obtained, a third eigenvector of the diagonal matrix can be obtained, the diagonal matrix and the third eigenvector are multiplied to obtain a jacobian matrix, a fourth eigenvector of the second inverse matrix can be obtained at the same time, and the second inverse matrix, the fourth eigenvector and the third sub-matrix are multiplied to obtain a residual error matrix.

In an example, for the first sub-matrix, the second sub-matrix, the third sub-matrix, the fourth sub-matrix, and so on, matrix elements of the target hessian matrix may respectively correspond to Amm matrix, an Amr matrix, an Arm matrix, and an Arr matrix in the target hessian matrix, in the process of performing marginalization on the hessian matrix, amm matrix and transposed Amm matrix in the target hessian matrix may be added to obtain target Amm matrix, then the first eigenvalue and the first eigenvector of the target Amm matrix may be calculated in PS, and the matrices respectively formed by the first eigenvalue and the second eigenvector may be multiplied to obtain Amm _ inv matrix. After obtaining the inverse Amm _ inv of the target Amm matrix, it may be multiplied by the Amr matrix and the Arm matrix in the target hessian matrix, respectively, to obtain the hessian matrix a and the hessian matrix B with updated dimension n × n. Then, the eigenvalue and the eigenvector of the hessian matrix a can be respectively calculated, then the matrix composed of the eigenvalue and the eigenvector is multiplied to obtain the S vector and the S _ inv inverse vector corresponding to the hessian matrix a, then the square root operation can be respectively carried out on each scalar quantity in the S vector and the S _ inv inverse vector point by point, and the diagonal matrix S _ sqrt composed of the eigenvalue array and the inverse matrix S _ sqrt _ inv corresponding to the diagonal matrix S _ sqrt are constructed. Then, a feature vector of the diagonal matrix S _ sqrt can be obtained, the diagonal matrix S _ sqrt and the corresponding feature vector are multiplied to obtain a Jacobian matrix which can be used for representing state information of the current fusion system, meanwhile, the inverse matrix S _ sqrt _ inv and the corresponding feature vector and the Hessian matrix B can be multiplied to obtain a residual matrix, marginalized accelerated calculation is carried out through the Hessian matrix to obtain the corresponding Jacobian matrix and the residual matrix, the operation amount of the system can be effectively controlled under the condition that the positioning accuracy is guaranteed, and the operation efficiency is improved.

After the Hessian matrix marginalization calculation acceleration is completed through the process, the collected environmental data and the collected motion data can be processed based on the Jacobian matrix and the residual error matrix obtained after the calculation acceleration so as to realize the positioning of the target object, and therefore under the condition that the target object is abnormal in positioning signals and/or network signals, the auxiliary positioning can be carried out through the visual inertial fusion system.

And 105, calculating the environment data and the motion data according to the Jacobian matrix and the residual error matrix to obtain the positioning information of the target object.

After the Hessian matrix marginalization calculation acceleration is completed, the target object can be calculated according to the obtained Jacobian matrix and the residual matrix and the environment data and the motion data acquired by the vision and inertia measurement unit, and the positioning information of the target object is obtained. Optionally, for the operation of the positioning process, which belongs to the related prior art, no further description is given here.

It should be noted that the embodiment of the present invention includes but is not limited to the above examples, and it is understood that, based on the teaching of the embodiment of the present invention, a person skilled in the art may also adopt other ways to adapt, and the present invention is not limited to this.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those of skill in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the embodiments of the invention.

Referring to fig. 7, a block diagram of a visual inertial fusion system provided in an embodiment of the present invention is shown, which may specifically include a visual and inertial measurement unit, a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array, and a tensor acceleration calculation control unit; the tensor data memory stores environment data and motion data acquired by the vision and inertia measurement unit; wherein the content of the first and second substances,

In an alternative embodiment, the tensor arithmetic logic operation array comprises a computation instruction parsing unit, a plurality of arithmetic logic units and an adder tree;

In an alternative embodiment, the arithmetic logic unit comprises a first arithmetic logic unit and a plurality of second arithmetic logic units; the first arithmetic logic unit is used for executing at least one operation of addition, subtraction, multiplication, division and evolution, and the second arithmetic logic unit is used for executing at least one operation of addition, subtraction and multiplication.

Referring to fig. 8, a block diagram of a structure of a target object positioning apparatus provided in the embodiment of the present invention is shown, which may specifically include the following modules:

a data obtaining module 801, configured to obtain environment data and motion data collected by a target object;

a hessian matrix acquisition module 802, configured to acquire an original hessian matrix and dimension information for the original hessian matrix, and perform marginalization processing on the original hessian matrix according to the dimension information to obtain a target hessian matrix;

a solving module 803, configured to solve the target hessian matrix to obtain a plurality of sub-matrices;

an accelerated calculation module 804, configured to perform accelerated calculation according to the plurality of sub-matrices to obtain a jacobian matrix and a residual matrix, where the jacobian matrix is a matrix used to represent state information of the visual inertial fusion system, and the residual matrix is a matrix used to optimize variable parameters;

a positioning module 805, configured to calculate the environmental data and the motion data according to the jacobian matrix and the residual matrix, so as to obtain positioning information of the target object.

In an optional embodiment, the sub-matrices at least include a first sub-matrix, a second sub-matrix, and a third sub-matrix, and the acceleration calculation module 804 is specifically configured to:

In an optional embodiment, the acceleration calculation module 804 is specifically configured to:

acquiring a third eigenvector of the diagonal matrix;

acquiring a fourth eigenvector of the second inverse matrix;

In an optional embodiment, the dimension information includes first dimension information that needs to be marginalized and second dimension information that needs to be retained, and the hessian matrix obtaining module 802 is specifically configured to:

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

In addition, an embodiment of the present invention further provides an electronic device, including: the processor, the memory, and the computer program stored in and executable on the memory are executed by the processor to implement the processes of the above-mentioned embodiment of the target object positioning method, and can achieve the same technical effects, and are not described herein again to avoid repetition.

The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when being executed by a processor, the computer program implements each process of the above-mentioned embodiment of the target object positioning method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

Fig. 9 is a schematic diagram of a hardware structure of an electronic device implementing various embodiments of the present invention.

The electronic device 900 includes, but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, a processor 910, and a power supply 911. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 9 does not constitute a limitation of the electronic device, and that the electronic device may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. In the embodiment of the present invention, the electronic device includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted terminal, a wearable device, a pedometer, and the like.

It should be understood that, in the embodiment of the present invention, the radio frequency unit 901 may be used for receiving and sending signals during information transceiving or a call, and specifically, after receiving downlink data from a base station, the downlink data is processed by the processor 910; in addition, the uplink data is transmitted to the base station. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 901 can also communicate with a network and other devices through a wireless communication system.

The electronic device provides wireless broadband internet access to the user via the network module 902, such as assisting the user in sending and receiving e-mails, browsing web pages, and accessing streaming media.

The audio output unit 903 may convert audio data received by the radio frequency unit 901 or the network module 902 or stored in the memory 909 into an audio signal and output as sound. Also, the audio output unit 903 may provide audio output related to a specific function performed by the electronic device 900 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 903 includes a speaker, a buzzer, a receiver, and the like.

The input unit 904 is used to receive audio or video signals. The input Unit 904 may include a Graphics Processing Unit (GPU) 9041 and a microphone 9042, and the Graphics processor 9041 processes image data of a still picture or video obtained by an image capturing device (such as a camera) in a video capture mode or an image capture mode. The processed image frames may be displayed on the display unit 906. The image frames processed by the graphic processor 9041 may be stored in the memory 909 (or other storage medium) or transmitted via the radio frequency unit 901 or the network module 902. The microphone 9042 can receive sounds and can process such sounds into audio data. The processed audio data may be converted into a format output transmittable to a mobile communication base station via the radio frequency unit 901 in case of the phone call mode.

The electronic device 900 also includes at least one sensor 905, such as light sensors, motion sensors, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 9061 according to the brightness of ambient light, and the proximity sensor may turn off the display panel 9061 and/or the backlight when the electronic device 900 is moved to the ear. As one type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), detect the magnitude and direction of gravity when stationary, and can be used to identify the posture of an electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), and vibration identification related functions (such as pedometer, tapping); the sensor 905 may further include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which are not further described herein.

The display unit 906 is used to display information input by the user or information provided to the user. The Display unit 906 may include a Display panel 9061, and the Display panel 9061 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The user input unit 907 may be used to receive input numerical or character information and generate signal inputs related to user settings and function control of the electronic device. Specifically, the user input unit 907 includes a touch panel 9071 and other input devices 9072. The touch panel 9071, also referred to as a touch screen, may collect touch operations by a user on or near the touch panel 9071 (e.g., operations by a user on or near the touch panel 9071 using a finger, a stylus, or any other suitable object or accessory). The touch panel 9071 may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 910, receives a command from the processor 910, and executes the command. In addition, the touch panel 9071 may be implemented by using various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The user input unit 907 may include other input devices 9072 in addition to the touch panel 9071. Specifically, the other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, and the like), a track ball, a mouse, and a joystick, which are not described herein again.

Further, the touch panel 9071 may be overlaid on the display panel 9061, and when the touch panel 9071 detects a touch operation on or near the touch panel 9071, the touch panel is transmitted to the processor 910 to determine the type of the touch event, and then the processor 910 provides a corresponding visual output on the display panel 9061 according to the type of the touch event. Although in fig. 9, the touch panel 9071 and the display panel 9061 are two independent components to implement the input and output functions of the electronic device, in some embodiments, the touch panel 9071 and the display panel 9061 may be integrated to implement the input and output functions of the electronic device, which is not limited herein.

The interface unit 908 is an interface for connecting an external device to the electronic apparatus 900. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 908 may be used to receive input from external devices (e.g., data information, power, etc.) and transmit the received input to one or more elements within the electronic device 900 or may be used to transmit data between the electronic device 900 and external devices.

The memory 909 may be used to store software programs as well as various data. The memory 909 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, etc. Further, the memory 909 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

The processor 910 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 909 and calling data stored in the memory 909, thereby performing overall monitoring of the electronic device. Processor 910 may include one or more processing units; preferably, the processor 910 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It is to be appreciated that the modem processor described above may not be integrated into processor 910.

The electronic device 900 may further include a power supply 911 (e.g., a battery) for supplying power to various components, and preferably, the power supply 911 may be logically connected to the processor 910 through a power management system, so as to manage charging, discharging, and power consumption management functions through the power management system.

In addition, the electronic device 900 includes some functional modules that are not shown, and are not described in detail here.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method of the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.

While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for locating a target object, comprising:

acquiring environmental data and motion data acquired by a target object;

solving the target Hessian matrix to obtain a plurality of sub-matrices;

performing accelerated calculation according to the plurality of sub-matrices to obtain a Jacobian matrix and a residual matrix, wherein the Jacobian matrix is a matrix used for representing state information of the visual inertial fusion system, and the residual matrix is a matrix used for optimizing variable parameters;

2. The method according to claim 1, wherein the sub-matrices include at least a first sub-matrix, a second sub-matrix and a third sub-matrix, and the accelerated computation according to the sub-matrices to obtain a jacobian matrix and a residual matrix comprises:

3. The method of claim 2, wherein performing a matrix operation according to the first feature information of the first target matrix to obtain a first inverse matrix of the first target matrix comprises:

4. The method according to claim 2, wherein performing a matrix operation according to the second feature information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix comprises:

5. The method according to claim 2, wherein performing a matrix operation using the third feature information of the diagonal matrix to obtain a jacobian matrix comprises:

acquiring a third eigenvector of the diagonal matrix;

6. The method of claim 2, wherein performing a matrix operation using the second inverse matrix and the third sub-matrix to obtain a residual matrix comprises:

acquiring a fourth eigenvector of the second inverse matrix;

7. The method according to claim 1, wherein the dimension information includes first dimension information that needs to be rimmed and second dimension information that needs to be retained, and the rimming processing the original hessian matrix according to the dimension information to obtain a target hessian matrix includes:

8. The visual inertial fusion system is characterized by comprising a visual and inertial measurement unit, a control instruction memory, a tensor data memory, a tensor arithmetic logic operation array and a tensor acceleration calculation control unit; the tensor data memory stores environment data and motion data acquired by the vision and inertia measurement unit; wherein the content of the first and second substances,

9. The system of claim 8, wherein the array of tensor arithmetic logic operations comprises a compute instruction resolution unit, a number of arithmetic logic units, and a tree of adders;

the arithmetic logic unit is used for adding the first sub-matrix and the transposed first sub-matrix to obtain a first target matrix; performing matrix operation according to first characteristic information of the first target matrix to obtain a first inverse matrix of the first target matrix; multiplying the first inverse matrix with the second sub-matrix and the third sub-matrix respectively to obtain a second target matrix corresponding to the second sub-matrix and a third target matrix corresponding to the third sub-matrix; performing matrix operation according to second characteristic information of the second target matrix to obtain a diagonal matrix corresponding to the second target matrix and a second inverse matrix corresponding to the diagonal matrix; performing matrix operation by adopting the third characteristic information of the diagonal matrix to obtain a Jacobian matrix; performing matrix operation by adopting the second inverse matrix and the third sub-matrix to obtain a residual error matrix;

the adder tree is used for accelerating operation when matrix multiplication is carried out.

10. The system of claim 9, wherein the arithmetic logic unit comprises a first arithmetic logic unit and a number of second arithmetic logic units; the first arithmetic logic unit is used for executing at least one operation of addition, subtraction, multiplication, division and evolution, and the second arithmetic logic unit is used for executing at least one operation of addition, subtraction and multiplication.

11. An apparatus for locating a target object, comprising:

12. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

the memory is used for storing a computer program;

the processor, when executing a program stored on the memory, implementing the method of any of claims 1-7.

13. A computer-readable storage medium having stored thereon instructions, which when executed by one or more processors, cause the processors to perform the method of any one of claims 1-7.