WO2022036634A1

WO2022036634A1 - Assembly/disassembly operation-oriented augmented reality guidance and remote collaboration development system

Info

Publication number: WO2022036634A1
Application number: PCT/CN2020/110258
Authority: WO
Inventors: 陈成军; 孙振武; 李东年; 洪军
Original assignee: 青岛理工大学
Priority date: 2020-08-20
Filing date: 2020-08-20
Publication date: 2022-02-24

Abstract

An assembly/disassembly operation-oriented augmented reality guidance and remote collaboration development system, comprising a scenario planning module and an augmented reality script parsing and displaying module. The scenario planning module comprises a motion planning unit, a data instrumentation configuring unit, and a hypertext planning unit. The motion planning unit is used for configuring a motion attribute of a node in a product virtual assembly model and generating a corresponding motion planning script and a three-dimensional scenario model file. The data instrumentation configuring unit is used for providing an instrumentation interface displayed in a virtual space. The hypertext planning unit is used for providing a hypertext interface for guiding an assembly/disassembly process in the virtual space. The augmented reality script parsing and display module comprises an augmented reality displaying unit and a script parsing unit. The augmented reality displaying unit is used for carrying out a coordinate conversion between the virtual space and the physical space and superimposing the product virtual assembly model in the virtual space.

Description

An augmented reality induction and remote collaborative development system for disassembly and assembly operations

technical field

The invention relates to an augmented reality induction and remote collaborative development system for disassembly and assembly operations, and belongs to the technical field of augmented reality applications.

Background technique

Disassembly and assembly (disassembly and assembly) is an important step in industrial production and maintenance. A reasonable disassembly and assembly process and operation process can effectively improve production efficiency. At present, with the changes in the market, the individual needs of product users also tend to be diversified, but in the traditional disassembly and assembly process, the disassembly and assembly staff spend a lot of time reading assembly manuals and It takes a lot of time and cost in the whole process to get started, and in mass customization production, due to the diverse needs of products, even two similar products may have completely different disassembly and assembly processes, so it is difficult to quickly Formation of training that has a significant effect on workers.

Augmented reality technology is a technology that can realize the fusion and superposition of virtual scenes and the real world in the user's field of vision. It has been widely used in product design, disassembly, maintenance and other fields. With the development of computer technology, the development of intelligent mobile terminals The computing power has been greatly improved, making it possible to complete the visual guidance process of the complete augmented reality disassembly process directly on mobile terminals such as tablet computers, smart phones, and head-mounted smart glasses.

Some existing augmented reality disassembly and assembly process guidance systems are often only for some special products. For the disassembly and assembly process of new products, it often takes a lot of effort to redesign the program, which will greatly increase the time and economic cost, and it is difficult to meet the actual production needs. In addition, when local experts guide remote equipment maintenance, an augmented reality development system is also required to quickly generate animations and superimpose them on the remote equipment.

SUMMARY OF THE INVENTION

In order to solve the above problems in the prior art, the present invention provides an augmented reality induction and remote collaborative development system for disassembly and assembly operations, which has strong versatility, does not need to redesign and develop new programs for new products, and saves a lot of time and energy. manpower.

The technical scheme of the present invention is as follows:

An augmented reality induction and remote collaborative development system for disassembly and assembly operations, including a scene planning module and an augmented reality script parsing and display module;

The scene planning module includes a motion planning unit, a data meter configuration unit and a hypertext planning unit; the motion planning unit is used to set the motion attributes of the nodes in the product virtual assembly model, and the motion attributes include motion types, motion parameters, motion trigger conditions, and generate corresponding motion planning script files and three-dimensional scene model files; the data meter configuration unit is used to set the meter interface displayed in the virtual space, and generate the corresponding meter interface script file; the hypertext planning unit It is used to set a hypertext interface displayed in the virtual space to guide the disassembly process, and generate corresponding hypertext script files and hypertext files; the augmented reality script parsing and display module includes an augmented reality display unit and a script parsing unit; The augmented reality display unit is used to complete the coordinate conversion between the virtual space and the physical space, and superimpose the product virtual assembly model in the virtual space;

The script parsing unit is connected in communication with the three units in the scene planning module and the augmented reality display unit, and is used for parsing and executing motion planning script files, instrument interface script files, and hypertext script files.

Further, the working steps of the motion planning unit are as follows:

Import product virtual assembly model;

Check whether the product virtual assembly model contains DOF nodes, LOD nodes and lighting nodes; if not, add several DOF nodes, LOD nodes and lighting nodes according to the structure level and motion relationship of the product virtual assembly model node;

According to different motion time sequences, several motion planning script files are generated, and the motion planning script files include all degree-of-freedom nodes whose motion attributes are set in the motion time sequence;

The product virtual assembly model processed according to the above steps is output as a 3D scene model file.

Further, the working steps of the data meter configuration unit are as follows:

Select instrument type;

Binding a data source, the data source contains several data that need to be displayed on the instrument interface;

Set the display trigger mode of the instrument interface;

Set the display parameters of the instrument interface;

Save all the information generated by the above steps to the instrument interface script file.

Further, the working steps of the hypertext planning unit are as follows:

Set the assembly instruction information to be displayed;

Set the hypertext interface display trigger mode;

Set hypertext interface display parameters;

Save all the information generated by the above steps into a hypertext script file; output the assembly instruction information as a hypertext file.

Further, the augmented reality script parsing and display module further includes a guidance interaction unit; the guidance interaction unit is connected in communication with the script parsing unit, and includes several interaction scenes and several interaction instructions, and the interaction scene is one with the motion planning script file. One-to-one correspondence, including at least the disassembly scene and the assembly scene.

Further, a working mode of the augmented reality display unit is an identification card AR mode, including a display device, an augmented reality registration card arranged in a physical space, and an image acquisition device fixedly connected to the display device; the augmented reality The registration card includes at least 3 non-collinear feature points;

Use the PnP algorithm to solve the transformation matrix of the coordinate system of the image acquisition device and the coordinate system of the augmented reality registration card; set the position of the virtual image acquisition device in the virtual space according to the transformation matrix, and superimpose the virtual assembly model of the product on the image collected by the image acquisition device Location of the Augmented Reality Registration Card.

Further, the augmented reality registration card further includes an information coding area; the information coding area is used to query the motion planning script file, the hypertext planning script file and the instrument interface script file.

Further, a working mode of the augmented reality display unit is viSLAM-AR mode, including a display device, an image acquisition device and a gyroscope sensor; the image acquisition device and the gyroscope sensor are fixedly connected; the display device is respectively connected to the gyroscope sensor. Gyro sensor and image acquisition device communication connection;

The working steps of the viSLAM-AR mode are as follows: using SLAM technology, according to the image collected by the image acquisition device and the rotation signal of the gyroscope sensor, a map is constructed and the product virtual assembly model is superimposed on the corresponding map points.

Further, it also includes inputting the 3D scene model file into a 3D engine to display a product virtual assembly model; inputting the hypertext file into a network engine to display a hypertext interface; calling an existing instrument interface description file and inputting it to the network Engine, display instrument interface;

The working steps of the augmented reality script parsing and display module are as follows:

Select the working mode of the augmented reality display unit, and automatically switch to the ID card AR mode when the augmented reality registration card is recognized, otherwise switch to the viSLAM-AR mode; the augmented reality display unit superimposes the product virtual assembly model in the virtual space middle;

Input motion planning script file, hypertext script file, instrument interface script file to script parsing unit;

instructing the interaction unit to transmit the interaction scene and interaction instruction selected by the user to the script parsing unit, and the script parsing unit selects and executes the corresponding motion planning script;

The script parsing unit parses and executes the hypertext script file and the instrument interface script file, and makes corresponding adjustments to the above hypertext interface and instrument interface displayed through the network engine.

The present invention has the following beneficial effects:

1. The present invention provides a general augmented reality induction and remote collaborative development system for disassembly and assembly operations, without the need to redesign and develop new programs for new products, saving a lot of time and manpower;

2. The present invention can quickly generate animation and superimpose it on the device by scanning the information coding area. Users can intuitively learn a series of processes such as product disassembly and assembly. The user experience is good, the operation is simple, and it is also conducive to remote local experts to guide remote devices. repair;

3. The present invention utilizes the SLAM technology to perform real-time map reconstruction of the working area, and directly display the disassembly and assembly animation in the set area, thereby breaking away from the restriction of the identification card, simplifying the operation and optimizing the user experience.

Description of drawings

Fig. 1 is the overall structure of the system of the present invention;

Fig. 2 is the working schematic diagram of motion planning unit;

Fig. 3 is the working schematic diagram of the AR mode of the identification card;

Figure 4 is a schematic diagram of the working of viSLAM-AR mode;

FIG. 5 is a working schematic diagram of an augmented reality script parsing display unit.

detailed description

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Example 1

Referring to Figure 1, an augmented reality induction and remote collaborative development system for disassembly and assembly operations, including a scene planning module and an augmented reality script parsing and display module;

The scene planning module includes a motion gauge unit, a data meter configuration unit and a hypertext planning unit; the motion planning unit is used to set the motion attributes of nodes in the product virtual assembly model, and the motion attributes include motion types, motion parameters, motion trigger conditions, and generate corresponding motion planning script files and three-dimensional scene model files; the data meter configuration unit is used to set the meter information displayed in the virtual space, and generate corresponding meter interface script files; the hypertext planning unit It is used to set the hypertext information displayed in the virtual space to guide the disassembly process, and generate corresponding hypertext script files and hypertext files;

The augmented reality script parsing and display module includes an augmented reality display unit and a script parsing unit; the augmented reality display unit is used to complete the coordinate conversion between the virtual space and the physical space, and superimpose and display the product virtual assembly model in the virtual space The script parsing unit is connected in communication with the three units in the scene planning module and the augmented reality display unit, and is used for parsing and executing the motion planning script file, the instrument interface script file and the hypertext script file.

Further, referring to Fig. 2, the working steps of the motion planning unit are as follows:

Import product virtual assembly model;

According to different motion time sequences, several motion planning script files (such as files in XML format) are generated, and the motion planning script files contain (with the degree of freedom of the node as the dimension) all the degree of freedom nodes that set the motion attributes in the motion time sequence ;The movement of the virtual assembly model of the product includes translation, rotation, zoom, display and concealment, highlighting, blurring, etc. The movement of each part of the model in different movements and the movement time sequence are different;

The product virtual assembly model processed in the above steps is output as a 3D scene model file (eg Openflight format).

Further, the working steps of the data meter configuration unit are as follows:

Select the instrument type, such as circular dial, progress bar, digital tube, etc.;

Set the display trigger mode of the instrument interface, such as scanning QR code trigger, button signal trigger, etc.;

Set the display parameters of the instrument interface, including the position, size, duration and other parameters displayed on the augmented reality registration card;

Further, the working steps of the hypertext planning unit are as follows:

Set the assembly guidance information to be displayed, such as text, pictures or indicators;

Set the trigger mode of hypertext interface display, such as scanning QR code, button signal, etc.;

Set the display parameters of the hypertext interface, including the position, size, duration and other parameters displayed on the augmented reality registration card;

Further, the augmented reality script parsing and displaying module further includes a guidance interaction unit; the guidance interaction unit is connected in communication with the script parsing unit, and includes several interaction scenes and several interaction instructions. The interactive scene corresponds to the motion planning script file one by one, including the initial scene, the disassembly scene, the assembly scene and the finished virtual use scene, wherein the interactive instructions in the initial scene are: operation instructions, disassembly demonstration, assembly demonstration, finished product running demonstration; The interactive instructions in the disassembly and assembly scene are: previous step, next step, reset, and exit.

The working process of the AR mode of the identification card is as follows: the display device (the coordinate system is O _s -X _s Y _s ) reads the coordinates of each feature point in real time, denoted as Z _i ( _u _i ,vi ); The three-dimensional coordinates in the coordinate system of the image acquisition device (O _c -X _c Y _c Z _c ) are P _i (x _i , y _i , z _i ), which can be solved according to the following formula, where K is the internal parameter matrix of the image acquisition device:

Let the coordinates of a point in the augmented reality registration card coordinate system (O _p -X _p Y _p Z _p ) be P _w (x _w , y _w , z _w ), then it corresponds to the coordinate system O _c -X _c Y _c Z _c is P _w (x _c , y _c , z _c ), their conversion relationship is as follows, where [R: t] is the conversion matrix:

Knowing the coordinates of each feature point in the coordinate system O _s -X _s Y _s and the coordinate system O _c -X _c Y _c Z _c , use the PnP algorithm to solve [R: t] according to formula (2); according to [R: t ] Set the position of the virtual image acquisition device in the virtual space, and superimpose the virtual assembly model of the product on the position of the augmented reality registration card in the image collected by the image acquisition device to complete the augmented reality registration.

The beneficial effect of this embodiment is to provide a general disassembly and assembly-oriented augmented reality induction and remote collaborative development system, without the need to redesign and develop new programs for new products, saving a lot of time and manpower.

Embodiment 2

As shown in Figure 3, take the QR code as an augmented reality registration card as an example: use the camera to shoot the QR code, detect the four corner points (ie feature points) through the image processing algorithm, and the display device reads the coordinates of the four corner points ; Obtain the coordinates of the four corners in the camera coordinate system according to the above formula (1), and then use the PnP algorithm to solve [R: t] according to the formula (2); set the virtual image acquisition device in the virtual space according to [R: t] s position. Use the two-dimensional code decoding algorithm to decode the two-dimensional code information. According to the two-dimensional code information, query the three-dimensional scene model file, motion planning script file, hypertext planning script file and instrument interface script file, and determine the superimposed display on the two-dimensional code. Content, such as product virtual assembly model, instrument interface, hypertext interface.

The improvement of this embodiment is that by scanning the information coding area, an animation can be quickly generated and superimposed on the device, and the user can intuitively learn a series of processes such as product disassembly and assembly. The user experience is good, the operation is simple, and it is also conducive to remote localization. Expert guidance on remote equipment repairs.

Embodiment 3

As shown in Figure 4, the working steps of the viSLAM-AR mode are as follows: using SLAM technology, according to the image collected by the image acquisition device and the rotation signal of the gyroscope sensor, build a map and superimpose the product virtual assembly model on the corresponding map point to complete the augmented reality registration.

In this embodiment, the steps of constructing a map are: firstly, rotate and move the image acquisition device and the gyro sensor, and initialize the map by using the multi-view geometry principle. After the initialization is completed, the mobile image acquisition device and the gyroscope sensor are used for map expansion to construct a simple reconstruction of the real scene.

The improvement of this embodiment lies in the use of SLAM technology to reconstruct the real-time map of the work area, and directly display the disassembly and assembly animation in the set area, thereby breaking away from the restriction of the identification card, simplifying the operation and optimizing the user experience.

Embodiment 4

Further, it also includes inputting the 3D scene model file into a 3D engine to display a product virtual assembly model; inputting the hypertext file into a network engine to display a hypertext interface; calling an existing instrument interface description file (echart) Input to network engine, display instrument interface;

Select the working mode of the augmented reality display unit, and automatically switch to the ID card AR mode when the augmented reality registration card is recognized, otherwise switch to the viSLAM-AR mode; the augmented reality display unit superimposes the product virtual assembly model in the virtual space , and make the product virtual assembly model produce corresponding pose changes with the change of the position of the image acquisition equipment;

Instruct the interaction unit to transmit the interaction scene and interaction instruction selected by the user to the script parsing unit (the initial scene is entered by default before the user selects); the script parsing unit selects and executes the corresponding motion planning script, so that the product virtual assembly model performs different movements;

The above descriptions are only the embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied to other related technologies Fields are similarly included in the scope of patent protection of the present invention.

Claims

An augmented reality induction and remote collaborative development system for disassembly and assembly operations, characterized in that it includes a scene planning module and an augmented reality script parsing and display module;

The scene planning module includes a motion planning unit, a data meter configuration unit and a hypertext planning unit; the motion planning unit is used to set the motion attributes of the nodes in the product virtual assembly model, and the motion attributes include motion types, motion parameters, motion trigger conditions, and generate corresponding motion planning script files and three-dimensional scene model files; the data meter configuration unit is used to set the meter interface displayed in the virtual space, and generate the corresponding meter interface script file; the hypertext planning unit It is used to set the hypertext interface displayed in the virtual space to guide the disassembly process, and generate corresponding hypertext script files and hypertext files;

The augmented reality script parsing display module includes an augmented reality display unit and a script parsing unit; the augmented reality display unit is used to complete the coordinate conversion between the virtual space and the physical space, and superimpose the product virtual assembly model in the virtual space; The script parsing unit is connected in communication with the three units in the scene planning module and the augmented reality display unit, and is used for parsing and executing motion planning script files, instrument interface script files, and hypertext script files.
The augmented reality guidance and remote collaborative development system for disassembly and assembly operations according to claim 1, wherein the working steps of the motion planning unit are as follows:

Import product virtual assembly model;

Check whether the product virtual assembly model contains DOF nodes, LOD nodes and lighting nodes; if not, add several DOF nodes, LOD nodes and lighting nodes according to the structure level and motion relationship of the product virtual assembly model node;

According to different motion time sequences, several motion planning script files are generated, and the motion planning script files include all the degree-of-freedom nodes for setting motion attributes in the motion time sequence;

The product virtual assembly model processed according to the above steps is output as a 3D scene model file.
The augmented reality induction and remote collaborative development system for disassembly and assembly operations according to claim 1, wherein the working steps of the data meter configuration unit are as follows:

Select instrument type;

Binding a data source, the data source contains several data that need to be displayed on the instrument interface;

Set the display trigger mode of the instrument interface;

Set the display parameters of the instrument interface;

Save all the information generated by the above steps to the instrument interface script file.
The augmented reality induction and remote collaborative development system for disassembly and assembly operations according to claim 1, wherein the working steps of the hypertext planning unit are as follows:

Set the assembly instruction information to be displayed;

Set the hypertext interface display trigger mode;

Set hypertext interface display parameters;

Save all the information generated by the above steps into a hypertext script file; output the assembly instruction information as a hypertext file.
The augmented reality guidance and remote collaborative development system for disassembly and assembly operations according to claim 1, wherein the augmented reality script parsing and displaying module further comprises a guidance interaction unit; the guidance interaction unit and the script parsing unit The communication connection includes several interactive scenarios and several interactive instructions, and the interactive scenarios are in one-to-one correspondence with the motion planning script file, including at least a disassembly scenario and an assembly scenario.
The augmented reality induction and remote collaborative development system for disassembly and assembly operations according to claim 1, wherein a working mode of the augmented reality display unit is an AR mode of an identification card, comprising a display device, a an augmented reality registration card in a physical space, and an image acquisition device fixedly connected to the display device; the augmented reality registration card includes at least 3 non-collinear feature points;

Use the PnP algorithm to solve the transformation matrix of the coordinate system of the image acquisition device and the coordinate system of the augmented reality registration card; set the position of the virtual image acquisition device in the virtual space according to the transformation matrix, and superimpose the virtual assembly model of the product on the image collected by the image acquisition device Location of the Augmented Reality Registration Card.
The augmented reality guidance and remote collaborative development system for disassembly and assembly operations according to claim 6, wherein the augmented reality registration card further comprises an information coding area; the information coding area is used for querying motion planning scripts files, hypertext planning script files, and instrument interface script files.
The augmented reality induction and remote collaborative development system for disassembly and assembly operations according to claim 1, wherein a working mode of the augmented reality display unit is viSLAM-AR mode, including a display device, an image capture a device and a gyroscope sensor; the image acquisition device is fixedly connected with the gyroscope sensor; the display device is connected in communication with the gyroscope sensor and the image acquisition device respectively;

The working steps of the viSLAM-AR mode are as follows: using SLAM technology, according to the image collected by the image acquisition device and the rotation signal of the gyroscope sensor, a map is constructed and the product virtual assembly model is superimposed on the corresponding map points.
The augmented reality induction and remote collaborative development system for disassembly and assembly operations according to any one of claims 1 to 8, further comprising inputting the three-dimensional scene model file into a 3D engine to display the virtual product of the product. Assembling the model; inputting the hypertext file into the network engine to display the hypertext interface; calling the existing instrument interface description file and inputting it to the network engine to display the instrument interface;

The working steps of the augmented reality script parsing and display module are as follows:

Select the working mode of the augmented reality display unit, and automatically switch to the ID card AR mode when the augmented reality registration card is recognized, otherwise switch to the viSLAM-AR mode; the augmented reality display unit superimposes the product virtual assembly model in the virtual space middle;

Input motion planning script file, hypertext script file, instrument interface script file to script parsing unit;

instructing the interaction unit to transmit the interaction scene and interaction instruction selected by the user to the script parsing unit, and the script parsing unit selects and executes the corresponding motion planning script;

The script parsing unit parses and executes the hypertext script file and the instrument interface script file, and makes corresponding adjustments to the above hypertext interface and instrument interface displayed through the network engine.