CN115877736B - Digital twinning-based multi-robot collaborative operation simulation monitoring method - Google Patents

Abstract

The present invention relates to the field of industrial robot technology, and in particular to a multi-robot collaborative operation simulation monitoring method based on digital twins. The method includes the following steps: S1. Based on digital twin technology, build a digital twin model of a multi-robot collaborative operation environment, The multi-robot collaborative working environment includes robots and control equipment; S2. Build a multi-robot collaborative working simulation monitoring system, and encapsulate the digital twin model into the multi-robot collaborative working simulation monitoring system; S3. Obtain the multi-robot program files and run them on multiple robots. The robot collaborative operation simulation monitoring system runs and the offline simulation results are obtained; S4. Connect the multi-robot collaborative operation environment and the multi-robot collaborative operation simulation monitoring system through the communication protocol, and monitor them in the multi-robot collaborative operation simulation monitoring system. The invention analyzes interference collisions in advance through motion, thereby improving the real-time and accuracy of robot operation monitoring.

Description

Multi-robot collaborative operation simulation monitoring method based on digital twin

Technical field

The invention relates to the technical field of industrial robots, and in particular to a multi-robot collaborative operation simulation monitoring method based on digital twins.

Background technique

With the continuous development of intelligent manufacturing, robots replacing manual production have become the development trend of the future manufacturing industry, and are also the guarantee for the future realization of industrial automation, digitization, and intelligence. Industrial robots are widely used in multi-joint manipulators or multi-degree-of-freedom machine devices in the industrial field. They have a certain degree of automation and can rely on their own power energy and control capabilities to achieve various industrial processing and manufacturing functions.

In a highly automated production line, a certain process may be completed by multiple industrial robots, or multiple industrial robots may be deployed on the same machine. Due to various restrictions in the production environment, the work spaces of multiple industrial robots may overlap. In this case, avoiding interference and collisions between multiple industrial robots, as well as significant economic losses caused by interference and collision between a single industrial robot and the working environment, has become a major problem in multi-robot collaborative programming.

On the one hand, industrial robots in the existing technology mainly use external torque feedback type or electronic skin type for collision detection. The external torque feedback method usually estimates the external torque based on the power loop feedback and the robot system dynamics equation. The external torque can also be estimated by adding torque sensor feedback at the joints. However, it is difficult to accurately estimate the robot joint friction when using the power loop feedback method. Modeling and identification lead to limited accuracy in detecting collision torque; using a torque sensor or adding electronic skin sensing to the robot has high sensitivity, but the cost is also greatly increased. In recent years, robot collision detection has also appeared using a combination of virtual robots and bounding boxes. However, because the bounding boxes used are mostly AABB bounding boxes or OBB bounding boxes, the accuracy of collision detection is not high, and it is easy to cause errors due to excessively large bounding boxes. shutdown phenomenon.

On the other hand, most of the existing robot offline programming software only supports simulation verification of the robot workstation after offline programming, and does not support status monitoring and collision warning during the actual operation of the robot.

To sum up, the existing technology lacks a low-cost, high-precision, fast and efficient, integrated robot offline verification and online monitoring method.

Contents of the invention

The present invention provides a multi-robot collaborative operation simulation monitoring method based on digital twins, aiming to solve the technical problem that the existing technology cannot efficiently realize offline verification and online monitoring of robots and control equipment.

Specifically, embodiments of the present invention provide a digital twin-based multi-robot collaborative operation simulation monitoring method, which method includes the following steps:

S1. Based on digital twin technology, build a digital twin model of a multi-robot collaborative working environment, which includes robots and control equipment;

S2. Construct a multi-robot collaborative operation simulation and monitoring system, and encapsulate the digital twin model into the multi-robot collaborative operation simulation and monitoring system;

S3. Obtain the multi-robot program files and run them in the multi-robot collaborative operation simulation monitoring system to obtain offline simulation results;

S4. Connect the multi-robot collaborative working environment and the multi-robot collaborative working simulation and monitoring system through a communication protocol, and monitor the multi-robot collaborative working simulation and monitoring system in the multi-robot collaborative working simulation and monitoring system.

Furthermore, step S1 includes the following sub-steps:

S11. Model the robot and control equipment to obtain a robot model and a control equipment model;

S12. Based on the motion mechanism of the robot and the control device, construct the robot model and the control device model into a robot mechanism model and a control device mechanism model respectively;

S13. Construct a data interaction interface for different motion states in the robot model and the control device model;

S14. Construct an enclosing volume including the robot mechanism model and the control equipment mechanism model, and build a collision group based on the enclosing volume for collision detection of any combination of the robot mechanism model and the control equipment mechanism model, Thus, the construction of the digital twin model is completed, wherein the bounding volume includes a rough bounding volume and a corresponding fine bounding volume.

Furthermore, the multi-robot collaborative operation simulation monitoring system includes:

The display layer includes a user-oriented display interface for interaction between the user and the multi-robot collaborative operation simulation monitoring system;

A simulation layer used to encapsulate the digital twin model;

A business layer, used to link the data interaction interface and determine whether a collision occurs between the robot mechanism model and the control device mechanism model based on the collision group;

The data layer is used to implement data communication with the multi-robot collaborative working environment.

Furthermore, step S3 includes the following sub-steps:

S31. Obtain the multi-robot program file and parse it into a simulation execution file. The multi-robot program file is used to simulate the motion state of the robot mechanism model and the control device mechanism model;

S32. Execute the simulation execution file and start offline simulation;

S33. In the collision group, determine whether a collision occurs between the rough surrounding bodies. If not, execute step S35; if yes, execute step S34;

S34. Determine whether a collision occurs between the fine bounding volumes corresponding to the rough bounding volumes. If not, execute step S35; if yes, record the collision information and execute step S35;

S35. Determine whether the control command is received to end the simulation. If not, return to step S33; if yes, output all the collision information as the offline simulation result.

Furthermore, step S4 includes the following sub-steps:

S41. Connect the multi-robot collaborative working environment and the multi-robot collaborative working simulation monitoring system through a communication protocol;

S42. Obtain the real-time status of the robot and the control device in the multi-robot collaborative working environment;

S43. Write data to the robot mechanism model and the control device mechanism model according to the real-time state, and use the real-time state as the simulation execution file in the multi-robot collaborative operation simulation monitoring system. simulation;

S44. Determine whether the simulation result performed in the real-time state contains the collision information. If so, stop the operation of the robot and the control device; if not, maintain normal operation.

Furthermore, before step S43, there are also steps:

The surrounding volume is enlarged in equal proportions according to a preset ratio as a collision detection redundant zone.

Furthermore, if the simulation result contains the collision information, the positions of the robot and the control device are adjusted based on the simulation result.

The beneficial effect achieved by the present invention is to propose a multi-robot collaborative operation simulation monitoring method based on digital twins. This method is based on digital twin technology to build a digital twin of a multi-robot collaborative operation environment to ensure that the simulation environment and the real environment are consistent. Consistency; secondly, by building a monitoring platform and conducting offline simulation verification on multiple industrial robots, it is possible to analyze whether interference occurs between the robot and the operating environment before operation, so as to avoid interference and collision during the debugging and operation of the physical robot; finally, combined with digital Twin technology monitors robot operations online and analyzes interference and collisions in advance through motion, improving the real-time and accuracy of robot operation monitoring.

Attached pictures

Figure 1 is a schematic flowchart of the steps of a multi-robot collaborative work simulation and monitoring method based on digital twins provided by an embodiment of the present invention;

Figure 2 is a schematic diagram of a rough surrounding body provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of a fine enclosure provided by an embodiment of the present invention;

Figure 4 is a schematic structural diagram of a multi-robot collaborative operation simulation and monitoring system provided by an embodiment of the present invention;

Figure 5 is a schematic sub-flow diagram of step S3 in the digital twin-based multi-robot collaborative operation simulation and monitoring method provided by the embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

Please refer to Figure 1. Figure 1 is a schematic flow chart of a multi-robot collaborative operation simulation and monitoring method based on digital twins provided by an embodiment of the present invention. The method includes the following steps:

S1. Based on digital twin technology, construct a digital twin model of a multi-robot collaborative working environment. The multi-robot collaborative working environment includes robots and control equipment.

Digital twin technology is a simulation process that uses physical models, sensor updates, operation history and other data to integrate multi-disciplines, multi-physical quantities, multi-scales, and multi-probabilities. It can complete mapping in virtual space to reflect the corresponding physical equipment. full life cycle process.

Furthermore, step S1 includes the following sub-steps:

S11. Model the robot and control equipment to obtain a robot model and a control equipment model.

Illustratively, in the embodiment of the present invention, based on the real scene of multi-robot collaborative operation, three-dimensional modeling software such as SolidWorks, Unigraphics NX, 3D Studio Max, etc. is used to establish a high-precision model of the working environment. The model should maintain the kinematics corresponding to the real object. relationship, among which, for the modeling of robots, each moving joint needs to be modeled independently, and finally combined in the form of an assembly, and the degree of freedom of the model must be guaranteed according to its correct kinematic constraints; for the modeling of control equipment in the operating environment, if the equipment If motion is required in offline verification simulation, the moving parts also need to be modeled separately and combined in the form of an assembly. Other operating environment objects that do not require movement can be directly expressed as an overall model.

S12. Based on the motion mechanisms of the robot and the control device, construct the robot model and the control device model into a robot mechanism model and a control device mechanism model respectively.

Specifically, for the packaging of the robot mechanism model, first, the robot model kinematic chain is constructed, that is, based on the mechanical structure and kinematic constraints of the physical robot, the child-parent relationship between the joints of the robot is obtained. After the kinematic chain is constructed, the kinematic chain is corrected according to the standard size parameters or DH parameters of the robot so that the relative positions between the joints fully meet the requirements and reduce the dimensional error between the virtual robot and the real robot; secondly, construct the robot kinematics algorithm , that is, based on the robot DH parameters, the forward kinematics and inverse kinematics algorithms of the robot are constructed. Among them, forward kinematics is used to solve the end pose of the robot, and inverse kinematics is used to solve the robot joint pose. Inverse kinematics algorithms for robots include, but are not limited to, algebraic, geometric, analytical, and numerical methods, but analytical methods should be used first to achieve faster solution speeds. Finally, common robot motion methods such as linear motion, arc motion, and point-to-point motion are encapsulated so that instructions can be used to directly control the virtual robot.

For the encapsulation of the control equipment mechanism model in the operating environment, first, the equipment model kinematic chain is constructed, that is, based on the mechanical structure, dynamic form and kinematic constraints of the physical equipment, the child-parent class relationship between the various mechanisms of the equipment is derived, thereby constructing the equipment The motion chain of the model; secondly, encapsulate the motion method according to the control method of the physical device.

S13. Construct a data interaction interface for different motion states in the robot model and the control device model.

The data interaction interface in the embodiment of the present invention is used for robots to update the joint angles of the virtual robot or send corresponding simulation events after receiving new joint angle data and signal data; for control equipment and other operating equipment in the operating environment, Used to trigger corresponding simulation events.

S14. Construct an enclosing volume including the robot mechanism model and the control equipment mechanism model, and build a collision group based on the enclosing volume for collision detection of any combination of the robot mechanism model and the control equipment mechanism model, Thus, the construction of the digital twin model is completed, wherein the bounding volume includes a rough bounding volume and a corresponding fine bounding volume.

For example, please refer to Figures 2 and 3. Figure 2 is a schematic diagram of a rough bounding volume provided by an embodiment of the present invention. Figure 3 is a schematic diagram of a fine bounding volume provided by an embodiment of the present invention. In step S14, each The rough bounding box of the element, that is, the OBB bounding box is generated for each model, which is used for rough collision detection to reduce the calculation amount of collision detection; secondly, the fine bounding volume of each element is constructed, that is, the triangular patch bounding volume is generated for each model. , for fine collision detection; finally, all models are classified into collision groups. In the operating environment, all bounding bodies within a single robot model are classified into one collision group, and all surrounding bodies in all models of other equipment in the operating environment are classified into another collision group. Group, when performing collision detection, only collision detection between collision groups is performed, and collision detection of models within the collision group is not performed, thereby reducing the calculation amount of collision detection and improving calculation efficiency.

S2. Construct a multi-robot collaborative operation simulation and monitoring system, and encapsulate the digital twin model into the multi-robot collaborative operation simulation and monitoring system.

Furthermore, the multi-robot collaborative operation simulation monitoring system includes:

The display layer includes a user-oriented display interface for interaction between the user and the multi-robot collaborative operation simulation monitoring system;

A simulation layer used to encapsulate the digital twin model;

A business layer, used to link the data interaction interface and determine whether a collision occurs between the robot mechanism model and the control device mechanism model based on the collision group;

The data layer is used to implement data communication with the multi-robot collaborative working environment.

For example, please refer to Figure 4, which is a schematic structural diagram of a multi-robot collaborative work simulation monitoring system provided by an embodiment of the present invention. In actual implementation:

The display layer is mainly for direct viewing and operation interaction, including the front-end UI interface and three-dimensional rendering engine. The front-end UI interface is responsible for the configuration and display of test items and test data, and can be written in Java, Html5, JavaScript and other languages. The 3D rendering engine is responsible for rendering the 3D scene images of the working environment and interacting with the scene. Mature 3D rendering engines such as Unreal Engine, JMonkey Engine, Unity 3D, and threes.js can be used to reduce platform development work.

The simulation layer is the virtual mechanism model of each element in the digital twin of the operating environment. When implemented, it includes but is not limited to robots, equipment, etc.

The business layer is mainly the core business module of the test platform, including the model basic motion module, physics engine module, communication module, front-end UI interface server module, front-end event processing module, test report module, program semantic analysis module, collision detection module, etc. The model motion module is responsible for the interpolation of basic motions such as straight lines, curves, and rotations of the model in the three-dimensional scene to accurately control the model motion; the physics engine module is responsible for the calculation of the physical properties of the model in the three-dimensional scene; the communication module is responsible for establishing the relationship with the physical operating environment Communication, including communication protocol definition, communication server, data transceiver interface definition, etc., enables the test platform to receive real-time data of the physical operating environment; the front-end UI interface server module is responsible for starting the lightweight server for loading the front-end UI interface; the front-end event processing module It is responsible for responding to events triggered by front-end buttons and pushing and displaying online monitoring data; the test report module is responsible for recording and exporting offline simulation verification results and related data into PDF format documents; the program semantic analysis module is responsible for parsing robot code files and based on the code files. Offline simulation is performed on the virtual robot; the collision detection module is responsible for detecting whether interference occurs between collision groups in the scene.

The data layer is mainly responsible for data interaction between various modules, including data sending and receiving modules, data processing modules, and variable reading and writing modules. The data transceiver module is responsible for receiving or sending relevant data to the physical operating environment through the communication module; the data processing module is responsible for parsing the received data, or encoding the relevant data and sending it to the control system; the variable reading and writing module is responsible for writing data to the model. variables, or read model-related variables.

S3. Obtain multi-robot program files, run them in the multi-robot collaborative operation simulation monitoring system, and obtain offline simulation results.

Further, please refer to Figure 5. Figure 5 is a schematic sub-flow diagram of step S3 in the multi-robot collaborative work simulation monitoring method based on digital twins provided by an embodiment of the present invention. Step S3 includes the following sub-steps:

S31. Obtain the multi-robot program file and parse it into a simulation execution file. The multi-robot program file is used to simulate the motion state of the robot mechanism model and the control device mechanism model;

S32. Execute the simulation execution file and start offline simulation;

S33. In the collision group, determine whether a collision occurs between the rough surrounding bodies. If not, execute step S35; if yes, execute step S34;

S34. Determine whether a collision occurs between the fine bounding volumes corresponding to the rough bounding volumes. If not, execute step S35; if yes, record the collision information and execute step S35;

S35. Determine whether the control command is received to end the simulation. If not, return to step S33; if yes, output all the collision information as the offline simulation result.

S4. Connect the multi-robot collaborative working environment and the multi-robot collaborative working simulation and monitoring system through a communication protocol, and monitor the multi-robot collaborative working simulation and monitoring system in the multi-robot collaborative working simulation and monitoring system.

The ultimate purpose of step S4 is to use digital twin technology to monitor the robot's operations online during the debugging and operation of the physical robot. At the same time, it can analyze in advance whether an interference collision will occur through the movement of the robot's mechanism model, and stop the operation of the physical robot in a timely manner.

Furthermore, step S4 includes the following sub-steps:

S41. Connect the multi-robot collaborative working environment and the multi-robot collaborative working simulation monitoring system through a communication protocol;

S42. Obtain the real-time status of the robot and the control device in the multi-robot collaborative working environment;

S43. Write data to the robot mechanism model and the control device mechanism model according to the real-time state, and use the real-time state as the simulation execution file in the multi-robot collaborative operation simulation monitoring system. simulation;

S44. Determine whether the simulation result performed in the real-time state contains the collision information. If so, stop the operation of the robot and the control device; if not, maintain normal operation.

Specifically, between different collision groups, the rough bounding volume of the model is used for collision detection; if interference occurs in the rough collision detection, fine collision detection is performed on the model corresponding to the rough bounding volume that interferes, that is, the fine bounding volume of the model is used Carry out collision detection. If interference still occurs, it means that a collision has occurred between the two models, that is, there is a risk of collision with the physical equipment. A shutdown command needs to be sent to the physical operating environment and waits for manual processing until the online monitoring is completed, then the collision detection is completed. .

Furthermore, before step S43, there are also steps:

The surrounding volume is enlarged in equal proportions according to a preset ratio as a collision detection redundant zone. Specifically, during the simulation, the rough and fine bounding volumes of the model are first adjusted, with the centroid as the center, and enlarged in equal proportions. The space formed by the bounding volume and the model surface is the collision detection redundant zone, so in practice In the environment, the amplification factor needs to be comprehensively determined based on the maximum operating speed of the robot, data collection communication delay, safety factor, etc. The higher the operating speed, the longer the data collection communication delay, and the greater the safety factor, the greater the amplification factor. The beneficial effect achieved by the present invention is to propose a multi-robot collaborative operation simulation monitoring method based on digital twins. This method is based on digital twin technology to build a digital twin of a multi-robot collaborative operation environment to ensure that the simulation environment and the real environment are consistent. Consistency; secondly, by building a monitoring platform and conducting offline simulation verification on multiple industrial robots, it is possible to analyze whether interference occurs between the robot and the operating environment before operation, so as to avoid interference and collision during the debugging and operation of the physical robot; finally, combined with digital Twin technology monitors robot operations online and analyzes interference and collisions in advance through motion, improving the real-time and accuracy of robot operation monitoring.

Furthermore, if the simulation result contains the collision information, the positions of the robot and the control device are adjusted based on the simulation result.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. During execution, the process may include the processes of the embodiments of each of the above methods. The storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM for short), etc.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including several instructions to cause a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only the preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative. It is not intended to be limiting. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms and equivalent changes without departing from the spirit of the present invention and the scope protected by the claims, which all belong to the present invention. Within protection.

Claims (2)

1. A digital twin-based multi-robot collaborative operation simulation monitoring method, characterized in that the method includes the following steps: S1. Based on digital twin technology, build a digital twin model of a multi-robot collaborative working environment. The multi-robot collaborative working environment includes robots and control equipment. Step S1 includes the following sub-steps: S11. Model the robot and control equipment to obtain a robot model and a control equipment model; S12. Based on the motion mechanism of the robot and the control device, construct the robot model and the control device model into a robot mechanism model and a control device mechanism model respectively; S13. Construct a data interaction interface for different motion states in the robot model and the control device model; S14. Construct an enclosing volume including the robot mechanism model and the control equipment mechanism model, and build a collision group based on the enclosing volume for collision detection of any combination of the robot mechanism model and the control equipment mechanism model, Thus, the construction of the digital twin model is completed, wherein the bounding volume includes a rough bounding volume and a corresponding fine bounding volume; S2. Construct a multi-robot collaborative operation simulation and monitoring system, and encapsulate the digital twin model into the multi-robot collaborative operation simulation and monitoring system. The multi-robot collaborative operation simulation and monitoring system includes: The display layer includes a user-oriented display interface for interaction between the user and the multi-robot collaborative operation simulation monitoring system; A simulation layer used to encapsulate the digital twin model; A business layer, used to link the data interaction interface and determine whether a collision occurs between the robot mechanism model and the control device mechanism model based on the collision group; A data layer is used to implement data communication with the multi-robot collaborative working environment; S3. Obtain the multi-robot program files and run them in the multi-robot collaborative operation simulation monitoring system to obtain offline simulation results. Step S3 includes the following sub-steps: S31. Obtain the multi-robot program file and parse it into a simulation execution file. The multi-robot program file is used to simulate the motion state of the robot mechanism model and the control device mechanism model; S32. Execute the simulation execution file and start offline simulation; S33. In the collision group, determine whether a collision occurs between the rough surrounding bodies. If not, execute step S35; if yes, execute step S34; S34. Determine whether a collision occurs between the fine bounding volumes corresponding to the rough bounding volumes. If not, execute step S35; if yes, record the collision information and execute step S35; S35. Determine whether the control command is received to end the simulation. If not, return to step S33; if yes, output all the collision information as the offline simulation result; S4. Connect the multi-robot collaborative working environment and the multi-robot collaborative working simulation monitoring system through a communication protocol, and monitor in the multi-robot collaborative working simulation monitoring system. Step S4 includes the following sub-steps: S41. Connect the multi-robot collaborative working environment and the multi-robot collaborative working simulation monitoring system through a communication protocol; S42. Obtain the real-time status of the robot and the control device in the multi-robot collaborative working environment; S43. Write data to the robot mechanism model and the control device mechanism model according to the real-time state, and use the real-time state as the simulation execution file in the multi-robot collaborative operation simulation monitoring system. simulation; S44. Determine whether the simulation result performed in the real-time state contains the collision information. If so, stop the operation of the robot and the control device; if not, maintain normal operation; Among them, before step S43, there are also steps: The surrounding volume is enlarged in equal proportions according to a preset ratio as a collision detection redundant zone. 2. The digital twin-based multi-robot collaborative operation simulation monitoring method as claimed in claim 1, characterized in that, after step S4, it also includes the following sub-steps: If the simulation result contains the collision information, the positions of the robot and the control device are adjusted based on the simulation result.