WO2022000713A1

WO2022000713A1 - Augmented reality self-positioning method based on aviation assembly

Info

Publication number: WO2022000713A1
Application number: PCT/CN2020/108443
Authority: WO
Inventors: 叶波; 唐健钧; 丁晓; 常壮; 金莹莹
Original assignee: 南京翱翔信息物理融合创新研究院有限公司
Priority date: 2020-06-28
Filing date: 2020-08-11
Publication date: 2022-01-06
Also published as: CN111968228B; CN111968228A

Abstract

Disclosed is an augmented reality self-positioning method based on an aviation assembly. The method comprises: designing a system framework, building an assembly scenario, constructing a high-precision three-dimensional map of the assembly scenario, building self-positioning scenario information, designing a self-positioning visual system, and performing a timing positioning process. By means of the present invention, the degree of understanding of an operator regarding a task is effectively improved, an operation threshold of the operator is reduced, the efficient and reliable completion of an assembly task is guaranteed, and precise positioning can also be achieved in a blank area with fewer feature points.

Description

An Augmented Reality Self-Location Method Based on Aeronautical Assembly

technical field

The invention relates to the technical field of self-positioning, in particular to an augmented reality self-positioning method based on aviation assembly.

Background technique

At present, in the assembly process of complex aviation products, there are problems such as large number of assembly parts, many coordination relationships, and high operational complexity. Operators must frequently switch between the virtual two-dimensional environment (computer terminal) and the real assembly environment, making the Assembly operation instructions cannot be integrated with the real environment, resulting in inefficient operation and high error rate of traditional assembly methods, and prolonging the equipment delivery cycle.

With the virtual reality guided assembly has been widely used in the field of complex product assembly, but the virtual reality equipment can only provide a single virtual information, there is no information in the real environment, the sense of substitution is not strong, so the use of augmented reality equipment for aviation products Assembly guidance avoids the disadvantage that virtual reality equipment can only provide a single virtual scene information. At the same time, the core of augmented reality is multi-sensor fusion self-positioning technology, which is widely used in unmanned driving, sweeping robots, logistics robots and augmented reality. middle. Through sensors such as the camera and inertial measurement unit carried by the carrier, the position and attitude of the carrier relative to the environment can be obtained in real time. Combined with the inertial measurement unit, it has the advantages of good position and attitude estimation in a short time, and the camera will cause problems when the camera moves rapidly. Due to the shortcomings of fuzzy defects, the positioning accuracy of multi-sensor fusion is greatly improved. However, limited by the principle of visual positioning, in the blank area with fewer feature points, the device cannot be positioned and the positioning accuracy is poor.

technical problem

The purpose of the present invention is to provide an augmented reality self-positioning method based on aviation assembly, which improves the long delivery cycle of aviation product assembly, the complex operation and the weak sense of substitution, aiming at the above-mentioned shortcomings of the prior art.

technical solutions

The technical scheme adopted in the present invention is as follows:

An augmented reality self-positioning method based on aviation assembly, which includes designing a system framework, building an assembly scene, constructing a high-precision three-dimensional map of the assembly scene, building self-positioning scene information, designing a self-positioning vision system and a timing positioning process, specifically including the following step:

Step 1: The design system framework adopts the client-server parallel development mode to receive and send data. The client is connected to the server wirelessly, and is used to transmit the assembly scene and assembly process information to the server. The server is wirelessly connected to the client, and is used to transmit the parsed pose of the assembly scene feature points and label information to the client;

Step 2: After completing the system framework design of the first step, build an assembly scene. The assembly scene includes a parts area, a to-be-assembled area, and a label area. For assembling the original assembly, the label area includes a plurality of labels, which are used to associate the position and attitude relationship between the plurality of labels and transmit them to the server;

Step 3: After completing the construction of the assembly scene in the second step, construct a high-precision three-dimensional map of the assembly scene. The construction of the high-precision three-dimensional map of the assembly scene first uses the distance information provided by the depth camera and the inertial measurement unit to obtain the density of the assembly scene. 3D map, then use the Apriltag tag to fill the dense 3D map of the assembly scene with information to create a discrete map, and then fuse the dense 3D map and the discrete map to form a high-precision 3D map of the assembly scene, and transmit it to the server;

Step 4: The high-precision three-dimensional map of the three-dimensional feature map of the assembly scene constructed in the step three is transmitted to the construction self-positioning scene information, and the construction self-positioning scene information is first analyzed. Apriltag tags are attached to less areas to form a tag set of the assembly scene, and then the relative pose relationship between the tag sets is measured, and then the spatial position relationship of the assembly parts is established according to the assembly process and assembly manual, and transmitted to the server;

Step 5: The spatial position relationship of the construction self-positioning scene information in the step 4 is transmitted to the design self-positioning vision system, and the designing the self-positioning vision system includes creating a virtual model, real-time computing equipment pose and virtual and real scene fusion, The created virtual model is connected to the pose of the real-time computing device, and is used for the AR development platform to build a three-dimensional scene, and the three-dimensional space coordinates of the virtual model are set according to the spatial positional relationship of the assembly parts, and then the augmented reality device is placed in the scene. , calculates the pose of the depth camera in the device in real time, and the real-time computing device pose is connected to the virtual and real scene fusion, and is used to load the virtual object to the client to realize the fusion display of the virtual object and the assembly scene;

Step 6: After completing the design of the self-positioning vision system in the fifth step, the timing positioning process is performed. The timing positioning process first completes the initialization of the self-positioning vision system in the to-be-assembled area of the part, then loads the high-precision three-dimensional map and opens two thread, and then compare the poses of the two threads. If the error meets the set requirements, the self-positioning vision system outputs the fusion result positioning; The system outputs the corrected pose.

In step 1, the client includes AR glasses, an inertial measurement unit and an industrial computer, the inertial measurement unit includes a sensor, and the industrial computer is connected to the sensor for controlling the sensor to transmit the calculated data to the server through a serial port.

In step 3, the depth camera is used to collect a week's video of the assembly scene, perform feature extraction and optical flow tracking on the collected video images, filter the extracted video features, and then extract feature frames for feature point retention.

In step 3, the information filling includes the key frame of the Apriltag label and the label corner point information corresponding to the key frame.

In step 6, the loading of the high-precision three-dimensional map is divided into two threads, one thread is to detect the Apriltag label information in real time, then estimate the relative label pose of the depth camera according to the Apriltag label, and then convert the label and the spatial position relationship of the self-positioning scene. The pose relative to the world coordinate; another thread is to fuse the inertial measurement unit according to the feature points in the assembly scene for fusion positioning, and obtain the pose of the depth camera relative to the world coordinate system in real time.

The specific steps of step 5 are: (1) Calculate the pose of the Apriltag tag; (2) Calculate the IMU pose; (3) Calculate the VSLAM pose; (4) Transfer the calculated pose to the server and fuse the virtual model. The three-dimensional space coordinates are then transmitted to the client for fusion display.

In step 5, the device pose includes the pose of the Apriltag tag, the IMU pose, and the VSLAM pose.

The beneficial effects of the invention are as follows: the operator wears the augmented reality device, the server interprets the assembly instructions, and at the same time, the assembly instructions are presented in front of the operator in the form of virtual information, guiding the operator to go to the parts area to find parts, and guiding the operator to the area to be assembled. , instruct the operator to assemble the precautions, effectively improve the operator's understanding of the task, lower the operator's operating threshold, ensure the efficient and reliable completion of the assembly task, and can also accurately locate in the blank area with fewer feature points.

beneficial effect

Description of drawings

1 is a frame diagram of an augmented reality system of the present invention;

Fig. 2 is the assembling scene frame diagram of the present invention;

Fig. 3 is the collection flow chart of the high-precision three-dimensional map of the assembly scene of the present invention;

FIG. 4 is a flow chart of the real-time positioning technology of the assembly process and equipment of the present invention.

Embodiments of the present invention

Below in conjunction with accompanying drawing, the present invention is further described:

As shown in Figures 1-4, the present invention includes designing a system framework, building an assembly scene, building a high-precision three-dimensional map of the assembly scene, building self-positioning scene information, designing a self-positioning vision system and a timing positioning process, and specifically includes the following steps:

Step 2: After completing the system framework design of the first step, build an assembly scene. The assembly scene includes a parts area, a to-be-assembled area, and a label area. For assembling the original assembly, the label area includes a plurality of labels, which are used to associate the position and attitude relationship between the plurality of labels and transmit them to the server; wherein the position and posture relationship between the labels is obtained through the multiple labels. Select any label as the start label, set its position as the origin (0, 0, 0), the initial rotation attitude as (0, 0, 0), and the rest of the labels do displacement rotation relative to the start label, and the displacement rotation is as Position and rotation initial pose.

Step 4: The high-precision three-dimensional map of the three-dimensional feature map of the assembly scene constructed in the step three is transmitted to the construction self-positioning scene information, and the construction self-positioning scene information is first analyzed. Apriltag tags are attached to fewer areas to form a tag set of the assembly scene, and then the relative pose relationship between the tag sets is measured. Calculate the relative pose of the assembly scene, and then establish the spatial position relationship of the assembly parts according to the assembly process and assembly manual, and transmit them to the server;

Step 5: The spatial position relationship of the construction self-positioning scene information in the step 4 is transmitted to the design self-positioning vision system, and the designing the self-positioning vision system includes creating a virtual model, real-time computing equipment pose and virtual and real scene fusion, The created virtual model is connected to the pose of the real-time computing device, and is used for the AR development platform to build a three-dimensional scene, and the three-dimensional space coordinates of the virtual model are set according to the spatial positional relationship of the assembly parts, and then the augmented reality device is placed in the scene. , calculates the pose of the depth camera in the device in real time, the real-time computing device pose is fused with the virtual and real scene, and is used to load the virtual object into the AR glasses on the client to realize the fusion display of the virtual object and the assembly scene, The device pose includes the pose of the Apriltag tag, the IMU pose, and the VSLAM pose.

Step 6: After completing the design of the self-positioning vision system in the fifth step, the timing positioning process is performed. The timing positioning process first completes the initialization of the self-positioning vision system in the to-be-assembled area of the part, then loads the high-precision three-dimensional map and opens two thread, and then compare the poses of the two threads. If the error meets the set requirements, the self-positioning vision system outputs the fusion result positioning; The system outputs the corrected pose, where the label pose is detected by the depth camera on the augmented reality device, and then calculates the position and pose of the label relative to the augmented reality device.

The specific steps of step five are:

(1) Calculate the pose of the Apriltag tag: the coordinate system of the tag code is

, the depth camera coordinate system is

, any point on the label code

The coordinates in the depth camera coordinate system are

, the corresponding relationship between the two is:

(1)

In formula (1), R is the rotation matrix representing the rotation of the depth camera coordinate system relative to the label code coordinate system, and T is the translation vector representing the translation of the depth camera coordinate system relative to the label code;

is the depth camera external parameter matrix, which can be used to convert the label code coordinate system to the depth camera coordinate system.

where the image coordinate system is

, the pixel coordinate system is

, the point on the label code

Imaging points in the image plane with the depth camera

The corresponding relationship between them is:

(2)

In formula (2),

is the center of the image plane,

are the normalized focal lengths of the x and y axes,

is the depth camera internal parameter matrix, which can be used to convert the depth camera coordinate system to the image plane coordinate system; set

, use the least squares method to solve formulas (1) and (2) to obtain the internal parameter matrix of the depth camera

; When the depth camera detects the tag, use the Apriltag algorithm to get R and T.

(2) Calculate the pose of the IMU: use the IMU to collect data during the movement of the device; at any time

and

In between, using formula (3) to integrate the angular rate of the inertial measurement unit, the angle increment of the device along the three axes can be obtained, which is recorded as

; at any time

and

In between, using formula (4) to double-integrate the acceleration of the inertial measurement unit, the displacement of the device during this period can be obtained

.

(3)

(4)

(3) Calculate the VSLAM pose: use the depth camera to collect a three-dimensional map of the scene, where n three-dimensional points in space are recorded as

, while the projected pixel coordinates are

, the two satisfy the following relationship:

(5)

in

It is the Lie algebra representation of the depth camera pose, and then uses Bundle Adjustment to minimize the projection error, sum the errors and construct the least squares, and then find the most accurate depth camera pose, so that the formula (6) Minimize and obtain R and T.

(6)

(4) Transfer the calculated pose to the server and fuse the three-dimensional space coordinates of the virtual model, and then transmit it to the client for fusion display: After obtaining the Apriltag label pose, IMU pose and VSLAM pose, use the traditional optimization The zero bias of the IMU is added to the state variable, and then the pose, velocity, and the target state equation constructed by the zero bias of the depth camera are estimated in a tightly coupled manner, as shown in formula (7), the ten The five-dimensional state variable is expressed as:

(7)

In formula (7),

,

and

are the rotation, translation and speed of the depth camera, respectively,

and

are the zero offsets of the accelerometer and gyroscope of the IMU, respectively; while this system adopts the strategy of local Apriltag tag-assisted positioning, and the tightly coupled system state variables are expressed as:

(8)

In formula (8),

Represents the positioning pose of the Apriltag tag,

Represents the positioning pose of VSLAM,

Represents the pose difference between the Apriltag tag and VSLAM, so there are system variables in two cases, when

, that is, the cumulative error of visual positioning exceeds the set threshold, and the visual fusion inertial navigation positioning continues; when

, that is, the cumulative error of visual positioning is large, and then the local positioning of the Apriltag label is used to integrate the inertial navigation positioning.

Example:

Step 1: Design the system framework: use the client-server parallel development mode to receive and send data. First, connect the AR glasses, inertial measurement unit and industrial computer in the client to the server wirelessly to connect the assembly scene. And the assembly process information is transmitted to the server, and then the server is wirelessly connected to the client to transmit the parsed pose of the assembly scene feature points and label information to the client;

Step 2, build an assembly scene: after completing the design system framework of step 1, assemble the parts to be assembled for aviation placed in the parts area in the to-be-assembled area, and then select label 1 in the 8 label areas arranged As the starting label, the position of label 1 is set as the origin (0, 0, 0), and the initial rotation attitude is (0, 0, 0). At the same time, label 2 performs displacement rotation relative to label 1, and its displacement rotates As its position (Position) and rotation attitude (Rotation), the rest of the labels are analogous, in which the rotation posture of each label is set to (0, 0, 0), that is, the spatial position of each label is adjusted to ensure that its orientation is consistent, such as Table 1 shows:

Table 1 Spatial Position Relationship of Assembly Scene Labels

标签号label number	位置 Location	旋转位姿Rotation pose
11	（0，0，0）(0, 0, 0)	（0，0，0）(0, 0, 0)
22	（0，8，0）(0, 8, 0)	（0，0，0）(0, 0, 0)
33	（4，8，0）(4, 8, 0)	（0，0，0）(0, 0, 0)
44	（8，8，0）(8, 8, 0)	（0，0，0）(0, 0, 0)
55	（10，6，0）(10, 6, 0)	（0，0，0）(0, 0, 0)
66	（10，0，0）(10, 0, 0)	（0，0，0）(0, 0, 0)
77	（8，-2，0）(8, -2, 0)	（0，0，0）(0, 0, 0)
88	（4，-2，0）(4, -2, 0)	（0，0，0）(0, 0, 0)

Step 3: Build a high-precision 3D map of the assembly scene: After completing the construction of the assembly scene in the second step, then use the depth camera to complete the initialization at label 1, and then perform video collection around the assembly scene, and the collected video images are processed. Feature extraction and optical flow tracking, filter the extracted video features, extract feature frames for feature point retention, and then combine with the distance information provided by the inertial measurement unit to obtain a dense three-dimensional map of the assembly scene; The dense 3D map of the assembly scene is filled with key frames and the label corner information corresponding to the key frames, and then a discrete map is established and fused with the dense 3D map to form a high-precision 3D map of the assembly scene;

Step 4: Build the self-positioning scene information: transfer the high-precision 3D map of the three-dimensional feature map of the assembly scene in step 3 to the building self-positioning scene information, then analyze the high-precision 3D map, and analyze the high-precision 3D map in the area with fewer feature points. Attach the artificial label Apriltag to form the label set of the assembly scene, then measure the relative pose relationship between the label sets, and then establish the spatial position relationship of the assembly parts according to the assembly process and assembly manual;

Step 5: Design a self-positioning vision system: transfer the spatial positional relationship of the building self-positioning scene information in step 4 to the design self-positioning vision system, which includes creating a virtual model, real-time computing device pose and virtual reality. Scene fusion, which connects the created virtual model with the pose of the real-time computing device, is used for the AR development platform to build a three-dimensional scene, and sets the three-dimensional spatial coordinates of the virtual model according to the spatial positional relationship of the assembled parts, and then places the augmented reality device on the In the scene, the pose of the depth camera in the device is calculated in real time, and then the real-time computing device pose is fused with the virtual and real scene to load the virtual object onto the AR glasses to realize the fusion display of the virtual object and the assembly scene;

Step 6, timing positioning process: After completing the design of the self-positioning vision system in the fifth step, the timing positioning process is carried out. The timing positioning process first completes the initialization of the self-positioning vision system in the part to be assembled area, and then loads a high-precision three-dimensional map and Open two threads, and then compare the poses of the two threads. If the error meets the set requirements, the self-positioning vision system will output the fusion result positioning; if the error is too large, use the label pose to correct the fusion pose. Finally, the self-positioning vision system outputs the corrected pose, thereby completing the self-positioning of the aviation assembly.

Other unexplained parts involved in the present invention are the same as those in the prior art.

Claims

An augmented reality self-positioning method based on aviation assembly is characterized in that it includes designing a system framework, constructing an assembly scene, constructing a high-precision three-dimensional map of the assembly scene, constructing self-positioning scene information, designing a self-positioning vision system and a timing positioning process, Specifically include the following steps:

Step 1: The design system framework adopts the client-server parallel development mode to receive and send data. The client is connected to the server wirelessly, and is used to transmit the assembly scene and assembly process information to the server. The server is wirelessly connected to the client, and is used to transmit the parsed pose of the assembly scene feature points and label information to the client;

Step 2: After completing the system framework design of the first step, build an assembly scene. The assembly scene includes a parts area, a to-be-assembled area, and a label area. For assembling the original assembly, the label area includes a plurality of labels, which are used to associate the position and attitude relationship between the labels and transmit them to the server;

Step 3: After completing the construction of the assembly scene in the second step, construct a high-precision three-dimensional map of the assembly scene. The construction of the high-precision three-dimensional map of the assembly scene first uses the distance information provided by the depth camera and the inertial measurement unit to obtain the density of the assembly scene. 3D map, then use the Apriltag tag to fill the dense 3D map of the assembly scene with information to create a discrete map, and then fuse the dense 3D map and the discrete map to form a high-precision 3D map of the assembly scene, and transmit it to the server;

Step 4: The high-precision three-dimensional map of the three-dimensional feature map of the assembly scene constructed in the step three is transmitted to the construction self-positioning scene information, and the construction self-positioning scene information is first analyzed. Apriltag tags are attached to less areas to form a tag set of the assembly scene, and then the relative pose relationship between the tag sets is measured, and then the spatial position relationship of the assembly parts is established according to the assembly process and assembly manual, and transmitted to the server;

Step 5: The spatial position relationship of the construction self-positioning scene information in the step 4 is transmitted to the design self-positioning vision system, and the designing the self-positioning vision system includes creating a virtual model, real-time computing equipment pose and virtual and real scene fusion, The created virtual model is connected to the pose of the real-time computing device, and is used for the AR development platform to build a three-dimensional scene, and the three-dimensional space coordinates of the virtual model are set according to the spatial positional relationship of the assembly parts, and then the augmented reality device is placed in the scene. , calculates the pose of the depth camera in the device in real time, and the real-time computing device pose is connected to the virtual and real scene fusion, and is used to load the virtual object to the client to realize the fusion display of the virtual object and the assembly scene;

Step 6: After completing the design of the self-positioning vision system in the fifth step, the timing positioning process is performed. The timing positioning process first completes the initialization of the self-positioning vision system in the to-be-assembled area of the part, then loads the high-precision three-dimensional map and opens two thread, and then compare the poses of the two threads. If the error meets the set requirements, the self-positioning vision system outputs the fusion result positioning; The system outputs the corrected pose.
The augmented reality self-positioning method based on aviation assembly according to claim 1, wherein in step 1, the client includes AR glasses, an inertial measurement unit and an industrial computer, the inertial measurement unit includes a sensor, and the The industrial computer is connected with the sensor and is used to control the sensor to transmit the calculated data to the server through the serial port.
The augmented reality self-positioning method based on aviation assembly according to claim 1, wherein in step 3, the depth camera is used to collect a video of the assembly scene for a week, and feature extraction and optical flow tracking are performed on the collected video images. , filter the extracted video features, and then extract feature frames for feature point retention.
The augmented reality self-positioning method based on aviation assembly according to claim 1, wherein in step 3, the information filling includes the key frame of the Apriltag label and the label corner point information corresponding to the key frame.
The augmented reality self-positioning method based on aviation assembly according to claim 1, wherein in step 6, the loading of the high-precision three-dimensional map is divided into two threads, one thread is to detect Apriltag label information in real time, and then according to Apriltag The label estimates the pose of the depth camera relative to the label, and then converts the spatial position relationship between the label and the self-positioning scene to the pose relative to the world coordinates; another thread fuses the inertial measurement unit according to the feature points in the assembly scene for fusion positioning, and obtains real-time results. The pose of the depth camera relative to the world coordinate system.
The augmented reality self-positioning method based on aviation assembly according to claim 1, wherein the specific steps of step 5 are: (1) calculating the pose of the Apriltag tag; (2) calculating the pose of the IMU; (3) calculating VSLAM pose; (4) transmit the calculated pose to the server and fuse the three-dimensional space coordinates of the virtual model, and then transmit it to the client for fusion display.
The augmented reality self-positioning method based on aviation assembly according to claim 1, wherein in step 5, the device pose includes the pose of the Apriltag tag, the pose of the IMU, and the pose of the VSLAM.