CN116500901A - Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention - Google Patents
Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention Download PDFInfo
- Publication number
- CN116500901A CN116500901A CN202310613961.9A CN202310613961A CN116500901A CN 116500901 A CN116500901 A CN 116500901A CN 202310613961 A CN202310613961 A CN 202310613961A CN 116500901 A CN116500901 A CN 116500901A
- Authority
- CN
- China
- Prior art keywords
- task
- user
- robot
- twin
- assembly
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000013528 artificial neural network Methods 0.000 claims abstract description 37
- 239000011159 matrix material Substances 0.000 claims abstract description 25
- 238000013507 mapping Methods 0.000 claims abstract description 14
- 238000012549 training Methods 0.000 claims abstract description 8
- 238000005070 sampling Methods 0.000 claims abstract description 6
- 239000013598 vector Substances 0.000 claims description 24
- 230000009471 action Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 15
- 230000006399 behavior Effects 0.000 claims description 13
- 230000033001 locomotion Effects 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 239000004973 liquid crystal related substance Substances 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000007704 transition Effects 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 238000003780 insertion Methods 0.000 claims description 2
- 230000002452 interceptive effect Effects 0.000 claims description 2
- 230000008901 benefit Effects 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000002787 reinforcement Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B13/00—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion
- G05B13/02—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
- G05B13/04—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
- G05B13/042—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators in which a parameter or coefficient is automatically adjusted to optimise the performance
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention relates to a digital twin-driven man-machine cooperation task planning method with unknown user intention, which comprises the following steps: establishing a twin scene according to a man-machine cooperation working platform of the physical space, and periodically sampling data by a sensor to realize real-time mapping of the twin space and the physical space; identifying a product assembly drawing based on a scene data set generator in a twin space, constructing a human-computer cooperation product assembly scene, and generating a virtual data set; constructing a task planner based on a graph neural network and with unknown user intention in a twin space, training by a virtual data set generated in the twin space, and outputting a task planning sequence matrix according to a current product assembly scene and a product assembly graph; and controlling and executing the man-machine cooperation tasks in the physical space according to the task planning sequence matrix. Compared with the prior art, the method has the advantages of being capable of achieving man-machine cooperation task planning with unknown human intention and the like.
Description
Technical Field
The invention relates to the technical field of man-machine cooperation, in particular to a man-machine cooperation task planning method under the condition that the intention of a user of digital twin driving is unknown.
Background
With the development of manufacturing industry and the progress of scientific technology, intelligent manufacturing is increasingly emphasized. The appearance of robots greatly improves the efficiency of industrial production. However, robots have limited understanding capabilities and are difficult to accomplish. Human labor costs are high and stability is poor relative to robots, but in some complex tasks, humans still have the advantage of being irreplaceable. Therefore, the efficient man-machine cooperation method is an ideal intelligent manufacturing scheme.
The digital twin technology is a technology combining a real physical world and a virtual information world, and realizes mapping between twin space and physical space through geometric modeling and abstract physical attributes. In human-computer collaboration, digital twinning can be used for visualization of interactive behavior, security pre-warning, or by simulation to provide a large number of data sets for human-computer collaboration related algorithms.
One key problem in human-machine collaboration is the task allocation method. Some existing human-computer collaborative task allocation methods include optimization, reinforcement learning, neural network methods and the like. The optimization method is based on fixed rules, and is easy to cause fatigue of workers; the reinforcement learning method has difficulty in setting a bonus function; the neural network method has the defects of difficult acquisition of the data set and the like.
CN110717381a discloses a human intention understanding method facing man-machine cooperation, but does not relate to a task allocation method in man-machine cooperation; CN115439101a discloses a human-computer collaborative task allocation method, but the method is based on fixed allocation rules, and cannot adjust allocation schemes in real time according to human intention.
Disclosure of Invention
The invention aims to provide a man-machine cooperation task planning method under the condition of unknown user intention of digital twin driving, which is characterized in that a three-dimensional model of a robot, a user and a product to be assembled is constructed, point cloud information obtained by a depth camera is utilized for carrying out model matching and fusion, a multi-source model man-machine cooperation equivalent digital twin scene which is mapped with a physical space in real time is generated, a man-machine cooperation virtual-real synthesis knowledge base based on the twin model is constructed in the twin space, a man-machine cooperation task planner under the condition of unknown user intention is constructed based on the knowledge base, man-machine cooperation task planning is realized, and planning efficiency is improved.
The aim of the invention can be achieved by the following technical scheme:
a digital twin-driven man-machine cooperation task planning method under the condition of unknown user intention comprises the following steps:
establishing a twin scene according to a man-machine cooperation working platform of the physical space, and periodically sampling data by a sensor to realize real-time mapping of the twin space and the physical space;
identifying a product assembly drawing based on a scene data set generator in a twin space, constructing a human-computer cooperation product assembly scene, and generating a virtual data set;
constructing a task planner based on a graph neural network and with unknown user intention in a twin space, training by a virtual data set generated in the twin space, and outputting a task planning sequence matrix according to a current product assembly scene and a product assembly graph;
and controlling and executing the man-machine cooperation tasks in the physical space according to the task planning sequence matrix.
The man-machine cooperation platform comprises a user, a robot, a working environment, a man-machine interaction interface for guiding user behaviors in real time, a controller for controlling the operation of the robot and a state monitoring sensor, wherein the state monitoring sensor comprises a depth camera and a built-in sensor of the robot.
The twin space comprises a twin model, a preprocessing module and a decision module, wherein the twin model comprises a digital person corresponding to three entities of a real physical space, a robot and a twin body of a working environment, and the twin body is a digital object formed by combining a geometric model and physical attributes; the preprocessing module comprises a scene data set generator for generating a virtual data set; the decision module includes a mission planner for performing mission planning.
The physical space is provided with a working table top, a robot and products to be assembled, three depth cameras are respectively arranged in front of, behind and above the working table top, a display is arranged on the table top, and one depth camera is arranged in front of a user and used for detecting the behavior state of the user.
The implementation process of the real-time mapping of the twin space and the physical space comprises the following steps: acquiring joint angle and angular velocity information through a built-in sensor of the robot, creating a virtual robot, and periodically updating a virtual robot model according to the built-in sensor data; obtaining point cloud information of a user and a product part in a physical space through a depth camera, carrying out point cloud registration by adopting an iterative nearest point algorithm, extracting contour information of the product part, sampling to obtain a shape curve of a three-dimensional model, and searching a similar curve in a model information base and carrying out part model matching; for mapping of digital people, the camera identifies the position information of the preset number of key joints of the human body, the overall motion of the user is estimated under the constraint range, and meanwhile, the mark positions defined by the surface of the digital people are fitted to the positions of the key points captured by the camera, so that real-time mapping between the real user and the digital people is realized.
The identification of the product assembly drawing is realized based on a drawing convolution neural network, the node characteristics of the drawing convolution neural network are part numbers, assembly priorities and types, the characteristics of directed edges are assembly priorities and task types, the drawing convolution neural network performs super-pixel segmentation on the image according to the outline of each part, and the part information, attribute information and task types of subtasks in the image are marked, wherein each task corresponds to one task complexity; and outputting the assembly tasks, the assembly sequence and the task complexity of the subtasks by using the training completed graph convolution neural network according to the product assembly graph.
The task categories include part movement, part assembly, screwing, and wire insertion operations.
Classifying the subtasks according to the task completeness and the multi-stage task complexity threshold, and dividing the subtasks into four types: the method comprises the following steps of independently completing subtasks of a user and a robot, completing subtasks of the user, completing subtasks of man-machine cooperation, wherein the subtasks with lowest complexity level are completed by the robot, the subtasks with medium complexity level are completed by the user, and the subtasks with highest complexity level are completed by the man-machine cooperation.
The construction of the man-machine cooperation product assembly scene specifically comprises the following steps:
randomly generating the position of the product part on the working desktop according to the product number; the robot controller and the human body controller guide the digital person and the robot to carry out the assembly task, and various subtask assembly scenes are generated according to the multiple resolvability of inverse kinematics; in the subtasks which can be independently completed by a user and a robot, the robot is set to complete, and meanwhile, the user behavior is used as a random variable, and the user selects to randomly complete a certain subtask, so that a plurality of subtask assembly scenes with unknown user intention are generated; taking priority among a plurality of subtask assembly scenes into consideration, and sequentially connecting the subtask assembly scenes with the aim of minimizing the moving distance of the product parts to generate the assembly scenes.
The task planner with unknown user intention executes the following man-machine cooperation task planning steps:
step 1) obtaining a product assembly scene;
step 2) representing the product assembly scene by a graph G= { V, E }, wherein V is a set of nodes, E is a set of edges, the edges connect the nodes, and each node n i Corresponds to a feature vector X with dimension m i The edges correspond to different weights, and the set of adjacent nodes of the node i is represented as N i M-dimensional initial feature vector X for each node i Is converted intoTarget feature vector of dimension->Wherein the new feature vector X is obtained by one conversion i ′ Expressed as:
wherein Θ is 1 And theta (theta) 2 Is a weight matrix of size (m ′ M), the feature vectors of all received neighbors j are multiplied by the edge weight e between the current node i and the transmitting node j jni Then summing the elements, and multiplying the eigenvectors after summation by a weight matrix theta 2 The two transformed vectors are added to form a final updated feature vector X i ′ The method comprises the steps of carrying out a first treatment on the surface of the The updated feature vector is sent through an activation function;
each product part is represented by a node in the graph, the characteristics of the object consist of an initial position < x, y, z >, a target position < x ', y ', z ' > and a product assembly priority {0,1,2,3,4, … … }, part nodes are represented by directed edges, the characteristics of the directed edges represent operations to be performed, task complexity, task completion and current time, and the graph convolution neural network is constructed by:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the output of the first layer of the neural network, N (i) being the neighbor set of nodes,product representing square root of node degree, sigma is an activation function, b (l) Is the bias parameter matrix of the first layer of the neural network, W (l) Weight parameter matrix representing layer I, < ->An input representing a first layer;
step 3) identifying the user's intent and adding it to the mission planner:
acquiring a motion skeleton sequence of the digital person in the twin space according to the joint information of the digital person;
adding nodes and edges in the constructed graph neural network, wherein the nodes are characterized by position information of joints, the undirected edges are natural connection of a skeleton structure and time, the joint nodes are connected with each product node through undirected edges, and the edges are characterized by the current time;
in the process of message transmission of the graph convolution neural network, the relative relation between the joint position and the product part position is evaluated through message aggregation, the relation between the user action information and the product part is obtained, whether a user is executing a specific operation is judged, the operation which is not completed when the user is supposed to complete at present is executed by the robot, the planner adjusts the distribution sequence, the robot executes the operation, if the robot can not complete the action, the planner outputs prompt information, and the user is prompted to execute the operation;
step 4) use a triplet<A,s 0 G > represents different tasks in human-machine collaboration, where A is the set of subtasks that need to be performed between the initial and target states of the part, s 0 Is the initial state of the part, g is the target state of the object, and the state transition process of the object is expressed as s t+1 ＝f(s t ,a t ) By performing operation a according to state transition equation f t The state of the part is defined by s t Conversion to s t+1 The solution of the mission planner is an action sequence matrix; product assembly has a pre-defined priority constraint, and when one node of the two nodes has a relatively high priority and the task represented by the edge feature corresponding to the node is not completed, operation a is performed t Otherwise, the operation is not executed, and the iteration process is repeated until all the subtasks are completed.
Compared with the prior art, the invention has the following beneficial effects:
(1) According to the invention, the product assembly image is used as the input of the man-machine cooperation planner, and the decomposition of the sub-tasks and the arrangement of the assembly priority sequence in the task allocation process can be completed only by providing the product assembly image.
(2) According to the invention, a twin digital human model and a robot model are established, a large number of virtual assembly scenes are generated according to the design of the product assembly drawing, a data set is easy to establish, and the task planner can be trained better through the large number of data sets.
(3) According to the invention, the user intention recognition technology is integrated into the task planning process, and the man-machine cooperation task planner for generating the work sequence in real time under the condition that the user intention is unknown is designed, so that the applicability of the method is strong, and the method is more suitable for actual application scenes.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of a human-computer collaborative task planner with unknown user intent;
fig. 3 is a schematic diagram of a sequence of results of human-computer collaborative task planning in an embodiment.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples. The present embodiment is implemented on the premise of the technical scheme of the present invention, and a detailed implementation manner and a specific operation process are given, but the protection scope of the present invention is not limited to the following examples.
The invention adopts a graph neural network method to distribute tasks between the human and the robot. The graph neural network method can complete assembly task planning between the human body and the robot by adjusting the strategy of nodes and edges of the graph. In order to achieve dynamic task allocation in human-machine collaboration, robots are required to be able to understand the meaning of human actions, and therefore cameras are set up in the environment to monitor user behavior and to periodically communicate the behavior information to the planner. In the implementation process, a twin scene is firstly established, the scene mainly comprises a product to be assembled, a digital person and a robot, and the sensor periodically samples data to realize real-time mapping of the twin scene and a physical space. And then, constructing a man-machine cooperation product assembly scene in the twin scene, and synthesizing a large number of virtual data sets. And then training by a task planner with unknown user intention based on the graph neural network by using a large number of data sets generated in the twin space, and outputting a task planning sequence matrix according to the current product assembly scene and the product assembly graph.
Specifically, the present embodiment provides a method for planning a human-computer cooperation task under the condition of unknown user intention of digital twin driving, as shown in fig. 1, including the following steps:
s1: and a twin scene is established according to the man-machine cooperation working platform of the physical space, and the sensor periodically samples data to realize real-time mapping of the twin space and the physical space.
The physical space comprises a man-machine cooperation platform and a state monitoring sensor. The man-machine cooperation platform comprises three main bodies of a user, a robot and a working environment, and further comprises a man-machine interaction interface for guiding the behavior of the user in real time and a controller for controlling the operation of the robot; the state monitoring sensor comprises a depth camera and a built-in sensor of the robot.
The twin space comprises a twin model, a preprocessing module and a decision module, wherein the twin model comprises a digital person corresponding to three entities of a real physical space, a robot and a twin body of a working environment, and the twin body is a digital object formed by combining a geometric model and physical attributes; the preprocessing module comprises a scene data set generator for generating a virtual data set; the decision module includes a mission planner for performing mission planning.
The embodiment presets a human-computer cooperation product assembly scene, a working table top, a robot and products to be assembled (such as an engine, a clutch, a motor and the like) are arranged in a physical space, three depth cameras are respectively arranged in front of, behind and above the working table top, and a display is arranged on the table top. One of the depth cameras is placed in front of the user and detects the behavior state of the user.
The assembly product and the robot can generate a three-dimensional model in the design process, and the three-dimensional model of the parts can be obtained easily. The depth camera can obtain color RGB images of the object and distance values of each point to a vertical plane where the depth camera is located, and further obtain a depth image, so that a point cloud model of the object in a camera coordinate system is obtained. Meanwhile, a digital person modeling method is adopted, and actions generated when the digital person reaches a certain point are obtained through derivation of the kinematics and inverse kinematics of main joints of the human body. Thus, a large number of man-machine cooperation assembly scene graphs are further obtained.
When constructing a scene in a twin space, firstly, a three-dimensional model of a product to be assembled and a robot is imported into simulation software (such as Unity 3D), and a product part database is created according to a model provided by a manufacturer. Physical attributes such as mass, surface texture, coefficient of friction, rigid body properties, etc. of the individual parts are added to the model of the assembled product and robot. A digital human model is introduced, and the digital human model consists of 48-degree-of-freedom connecting rods and a surface grid changing along with the angle of the joints, and comprises 15 joints. And then the quality, height and other attributes of the digital person are set.
Next, a digital robot controller and a digital human controller are designed. The control of the robot can be realized by establishing a communication mechanism between the existing robot control software and simulation software. And deducing a transformation matrix according to the connecting rod information of the digital person, establishing a kinematic model of the digital person, further deducing an inverse kinematic model and a kinetic model, adding constraint conditions for the movement of each joint of the human body, calculating the joint moment of the trunk of the human body, the shoulder moment and the joint moment of each arm, and realizing the control of the digital person.
Again, a twin space is created that maps to the physical space. Acquiring joint angle and angular velocity information through a built-in sensor of the robot, creating a virtual robot, and periodically updating a virtual robot model according to the built-in sensor data; obtaining point cloud information of a user and a product part in a physical space through a depth camera, carrying out point cloud registration by adopting an iterative nearest point algorithm, extracting contour information of the product part, sampling to obtain a shape curve of a three-dimensional model, and searching a similar curve in a model information base and carrying out part model matching; for mapping of digital people, the camera identifies the position information of the preset number of key joints of the human body, the overall motion of the user is estimated under the constraint range, and meanwhile, the mark positions defined by the surface of the digital people are fitted to the positions of the key points captured by the camera, so that real-time mapping between the real user and the digital people is realized.
S2: and identifying a product assembly drawing based on a scene data set generator in the twin space, constructing a human-computer cooperation product assembly scene, and generating a virtual data set.
The product assembly drawing is generally a structured drawing with no interference in the background and is easier to identify. The identification of the product assembly drawing is realized based on a drawing convolution neural network, wherein the node characteristics of the drawing convolution neural network are part numbers, assembly priorities and types, and the characteristics of the directed edges are assembly priorities and task types. The graph convolution neural network performs super-pixel segmentation on the image according to the outline of each part, marks the part information, attribute information and task types of subtasks (such as part movement, part assembly, screwing, line embedding operation and the like) in the image, and each task corresponds to one task complexity; and outputting the assembly tasks, the assembly sequence and the task complexity of the subtasks by using the training completed graph convolution neural network according to the product assembly graph.
Classifying the subtasks according to the task completeness and the multi-stage task complexity threshold, and dividing the subtasks into four types: the method comprises the following steps of independently completing subtasks of a user and a robot, completing subtasks of the user, completing subtasks of man-machine cooperation, wherein the subtasks with lowest complexity level are completed by the robot, the subtasks with medium complexity level are completed by the user, and the subtasks with highest complexity level are completed by the man-machine cooperation.
A large number of man-machine cooperation product assembly scenes are constructed, specifically:
randomly generating the position of the product part on the working desktop according to the product number; the robot controller and the human body controller guide the digital person and the robot to carry out the assembly task, and various subtask assembly scenes can be generated due to the multiple resolvability of inverse kinematics; in the subtasks which can be independently completed by a user and a robot, the robot is set to complete, and meanwhile, the user behavior is used as a random variable, and the user selects to randomly complete a certain subtask, so that a plurality of subtask assembly scenes with unknown user intention are generated; taking priority among a plurality of subtask assembly scenes into consideration, aiming at minimizing the moving distance of the product parts, and sequentially connecting the subtask assembly scenes to generate a large number of assembly scenes. Meanwhile, the assembly scene is generated according to the product assembly drawing, and contains information such as product position, assembly priority order, task complexity and the like, so that the data set does not need to be manually marked.
S3: and constructing a task planner based on the graphic neural network under the unknown user intention in the twin space, training by a virtual data set generated in the twin space, and outputting a task planning sequence matrix according to the current product assembly scene and the product assembly diagram.
The task planner in the decision module is the part of the invention that is of great interest, where the user's intention understanding part is added. And inputting an initial scene image and a product assembly drawing, generating a group of action sequences based on time sequence by a task planner, and dividing the action sequences into three groups of people, robots and man-machine cooperation. The results are then displayed in a display in physical space through which the user can see what should be done currently. The depth camera placed in front of the user monitors the user behavior in real time, and the planner periodically plans a new action sequence according to the user action and updates the result to the display.
Specifically, the task planner with unknown user intention executes the following man-machine cooperation task planning steps:
step 1) obtaining a product assembly scene.
In the previous step, a method of generating a twin space mapped in real time with a real assembly space has been described. In the twin space, information such as the position of the current product, the product assembly priority and the like can be obtained.
Step 2) representing the product assembly scene by a graph G= { V, E }, wherein V is a set of nodes, E is a set of edges, and the edges connect the nodes. Each node n i Corresponds to a feature vector X with dimension m i . Edges correspond to different weights and can be understood as the cost of operation between different nodes. The set of neighboring nodes of node i is denoted as N i M-dimensional initial feature vector X for each node i Is converted intoTarget feature vector of dimension->Novel feature vector X obtained by once conversion of node i i ′ Can be expressed as:
wherein Θ is 1 And theta (theta) 2 Is a weight matrix of size (m ′ M), the feature vectors of all received neighbors j are multiplied by the edge weight e between the current node i and the transmitting node j jni Then summing the elements, and multiplying the eigenvectors after summation by a weight matrix theta 2 The two transformed vectors are added to form a final updated feature vector X i ′ . The updated feature vector is sent through the activation function.
Each product part is represented by a node in the graph, the characteristics of the object consist of an initial position < x, y, z >, a target position < x ', y ', z ' > and a product assembly priority {0,1,2,3,4, … … }, part nodes are represented by directed edges, the characteristics of the directed edges represent operations to be performed, task complexity, task completion and current time, and the graph convolution neural network is constructed by:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the output of the first layer of the neural network, N (i) being the neighbor set of nodes,product representing square root of node degree, sigma is an activation function, b (l) Is the bias parameter matrix of the first layer of the neural network, W (l) Weight parameter matrix representing layer I, < ->Representing the input of the first layer.
Step 3) identify the user's intent and add it to the mission planner.
Step 3-1) according to the joint information of the digital person, the skeleton of the digital person in the twin space is composed of 15 joint parts, and the motion skeleton sequence of the digital person in the twin space can be obtained.
And 3-2) adding nodes and edges in the constructed graph neural network, wherein the nodes are characterized by position information of joints, the undirected edges are natural connection of a skeleton structure and time, the joint nodes are connected with each product node through undirected edges, and the edges are characterized by the current time.
Step 3-3) in the process of message transmission of the graph convolution neural network, the relative relation between the joint position and the product part position is evaluated through message aggregation, the relation between the user action information and the product part is further obtained, whether the user is executing a specific operation is judged, the planner adjusts the distribution sequence for the operation which is not completed by the user at present, the robot executes the operation, and if the robot cannot complete the action, the planner outputs prompt information to prompt the user to execute the operation. In this embodiment, the graph neural network uses 4 GraphConv layers as convolutional layers, each followed by a ReLU. The network uses cross entropy loss as a loss function, which is then classified by a standard SoftMax classifier.
Step 4) Using a triplet < A, s 0 G > represents different tasks in human-machine collaboration, where A is the set of subtasks that need to be performed between the initial and target states of the part, s 0 Is the initial state of the part and g is the target state of the object. The state transition process of an object can be expressed as s t+1 ＝f(s t ,a t ) By performing operation a according to state transition equation f t The state of the part is defined by s t Conversion to s t+1 . The solution of the mission planner is an action sequence matrix. Product assembly has a pre-defined priority constraint when the assembly of a certain product is in orderThe priority is high when the order is in the front, and the corresponding priority is low when the assembly order is in the rear. When one node of the two nodes has a relatively high priority and the task represented by the edge feature corresponding to the node is not completed, performing operation a t Otherwise, the operation is not executed, and the iteration process is repeated until all the subtasks are completed.
S4: and controlling and executing the man-machine cooperation tasks in the physical space according to the task planning sequence matrix.
The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention by one of ordinary skill in the art without undue burden. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by a person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.
Claims (10)
1. The digital twin-driven man-machine cooperation task planning method under the condition of unknown user intention is characterized by comprising the following steps of:
establishing a twin scene according to a man-machine cooperation working platform of the physical space, and periodically sampling data by a sensor to realize real-time mapping of the twin space and the physical space;
identifying a product assembly drawing based on a scene data set generator in a twin space, constructing a human-computer cooperation product assembly scene, and generating a virtual data set;
constructing a task planner based on a graph neural network and with unknown user intention in a twin space, training by a virtual data set generated in the twin space, and outputting a task planning sequence matrix according to a current product assembly scene and a product assembly graph;
and controlling and executing the man-machine cooperation tasks in the physical space according to the task planning sequence matrix.
2. The digital twin-driven human-computer collaborative task planning method with unknown user intention according to claim 1, wherein the human-computer collaborative platform comprises three entities of a user, a robot and a working environment, a human-computer interactive interface for guiding user behaviors in real time, and a controller and a state monitoring sensor for controlling the operation of the robot, wherein the state monitoring sensor comprises a depth camera and a built-in sensor of the robot.
3. The method for planning the human-computer collaborative task under the unknown user intention, which is driven by digital twinning according to claim 2, is characterized in that the twinning space comprises a twinning model, a preprocessing module and a decision module, wherein the twinning model comprises a digital person, a robot and a twinning body of a working environment corresponding to three entities of a real physical space, and the twinning body is a digital object formed by combining a geometric model and physical attributes; the preprocessing module comprises a scene data set generator for generating a virtual data set; the decision module includes a mission planner for performing mission planning.
4. The method for planning a human-computer collaborative task with unknown user intention according to claim 1, wherein a working desktop, a robot and a product to be assembled are arranged in the physical space, three depth cameras are respectively arranged in front of, behind and above the working desktop, a display is arranged on the desktop, and one of the depth cameras is arranged in front of the user for detecting the behavior state of the user.
5. The method for planning a human-computer collaborative task with unknown user intention by digital twin driving according to claim 2, wherein the implementation process of the real-time mapping of the twin space and the physical space is as follows: acquiring joint angle and angular velocity information through a built-in sensor of the robot, creating a virtual robot, and periodically updating a virtual robot model according to the built-in sensor data; obtaining point cloud information of a user and a product part in a physical space through a depth camera, carrying out point cloud registration by adopting an iterative nearest point algorithm, extracting contour information of the product part, sampling to obtain a shape curve of a three-dimensional model, and searching a similar curve in a model information base and carrying out part model matching; for mapping of digital people, the camera identifies the position information of the preset number of key joints of the human body, the overall motion of the user is estimated under the constraint range, and meanwhile, the mark positions defined by the surface of the digital people are fitted to the positions of the key points captured by the camera, so that real-time mapping between the real user and the digital people is realized.
6. The method for planning the human-computer collaborative task under the unknown user intention, which is characterized in that the identification of the product assembly drawing is realized based on a drawing convolution neural network, the node characteristics of the drawing convolution neural network are part numbers, assembly priorities and types, the characteristics of directed edges are assembly priorities and task types, the drawing convolution neural network performs super-pixel segmentation on an image according to the outline of each part, marks the part information, attribute information and task types of subtasks in the image, and the task of each type corresponds to one task complexity; and outputting the assembly tasks, the assembly sequence and the task complexity of the subtasks by using the training completed graph convolution neural network according to the product assembly graph.
7. The digital twin driven user intent blind man-machine collaboration task planning method of claim 6, wherein the task categories include part movement, part assembly, screw-on, wire-insertion operations.
8. The digital twin-driven human-computer collaborative task planning method with unknown user intent of claim 6, wherein the classification of subtasks is performed by task completability and multi-level task complexity thresholds, dividing the subtasks into four categories: the method comprises the following steps of independently completing subtasks of a user and a robot, completing subtasks of the user, completing subtasks of man-machine cooperation, wherein the subtasks with lowest complexity level are completed by the robot, the subtasks with medium complexity level are completed by the user, and the subtasks with highest complexity level are completed by the man-machine cooperation.
9. The digital twin-driven human-computer collaborative task planning method with unknown user intention according to claim 8, wherein the construction of the human-computer collaborative product assembly scene is specifically as follows:
randomly generating the position of the product part on the working desktop according to the product number; the robot controller and the human body controller guide the digital person and the robot to carry out the assembly task, and various subtask assembly scenes are generated according to the multiple resolvability of inverse kinematics; in the subtasks which can be independently completed by a user and a robot, the robot is set to complete, and meanwhile, the user behavior is used as a random variable, and the user selects to randomly complete a certain subtask, so that a plurality of subtask assembly scenes with unknown user intention are generated; taking priority among a plurality of subtask assembly scenes into consideration, and sequentially connecting the subtask assembly scenes with the aim of minimizing the moving distance of the product parts to generate the assembly scenes.
10. The digital twinned driven human-computer collaborative task planning method with unknown user intent of claim 1, wherein the task planner with unknown user intent performs the following human-computer collaborative task planning steps:
step 1) obtaining a product assembly scene;
step 2) representing the product assembly scene by a graph G= { V, E }, wherein V is a set of nodes, E is a set of edges, the edges connect the nodes, and each node n i Corresponds to a feature vector X with dimension m i The edges correspond to different weights, and the set of adjacent nodes of the node i is represented as N i M-dimensional initial feature vector X for each node i Is converted intoTarget feature vector of dimensionWherein the new feature vector X is obtained by one conversion i ′ Expressed as:
wherein Θ is 1 And theta (theta) 2 Is a weight matrix of size (m ′ M), the feature vectors of all received neighbors j are multiplied by the edge weight e between the current node i and the transmitting node j jni Then summing the elements, and multiplying the eigenvectors after summation by a weight matrix theta 2 The two transformed vectors are added to form a final updated feature vector X i ′ The method comprises the steps of carrying out a first treatment on the surface of the The updated feature vector is sent through an activation function;
each product part is represented by a node in the graph, the characteristics of the object consist of an initial position < x, y, z >, a target position < x ', y ', z ' > and a product assembly priority {0,1,2,3,4, … … }, part nodes are represented by directed edges, the characteristics of the directed edges represent operations to be performed, task complexity, task completion and current time, and the graph convolution neural network is constructed by:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the output of the first layer of the neural network, N (i) being the neighbor set of nodes, ++>Product representing square root of node degree, sigma is an activation function, b (l) Is the bias parameter matrix of the first layer of the neural network, W (l) Weight parameter matrix representing layer I, < ->An input representing a first layer;
step 3) identifying the user's intent and adding it to the mission planner:
acquiring a motion skeleton sequence of the digital person in the twin space according to the joint information of the digital person;
adding nodes and edges in the constructed graph neural network, wherein the nodes are characterized by position information of joints, the undirected edges are natural connection of a skeleton structure and time, the joint nodes are connected with each product node through undirected edges, and the edges are characterized by the current time;
in the process of message transmission of the graph convolution neural network, the relative relation between the joint position and the product part position is evaluated through message aggregation, the relation between the user action information and the product part is obtained, whether a user is executing a specific operation is judged, the operation which is not completed when the user is supposed to complete at present is executed by the robot, the planner adjusts the distribution sequence, the robot executes the operation, if the robot can not complete the action, the planner outputs prompt information, and the user is prompted to execute the operation;
step 4) Using a triplet < A, s 0 G > represents different tasks in human-machine collaboration, where A is the set of subtasks that need to be performed between the initial and target states of the part, s 0 Is the initial state of the part, g is the target state of the object, and the state transition process of the object is expressed as s t+1 ＝f(s t ,a t ) By performing operation a according to state transition equation f t The state of the part is defined by s t Conversion to s t+1 The solution of the mission planner is an action sequence matrix; product assembly has a pre-defined priority constraint, and when one node of the two nodes has a relatively high priority and the task represented by the edge feature corresponding to the node is not completed, operation a is performed t Otherwise, the operation is not executed, and the iteration process is repeated until all the subtasks are completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310613961.9A CN116500901A (en) | 2023-05-29 | 2023-05-29 | Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310613961.9A CN116500901A (en) | 2023-05-29 | 2023-05-29 | Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116500901A true CN116500901A (en) | 2023-07-28 |
Family
ID=87318413
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310613961.9A Pending CN116500901A (en) | 2023-05-29 | 2023-05-29 | Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116500901A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117078866A (en) * | 2023-10-12 | 2023-11-17 | 北京市市政工程研究院 | Bridge modeling updating method, system and storage medium based on digital twin |
-
2023
- 2023-05-29 CN CN202310613961.9A patent/CN116500901A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117078866A (en) * | 2023-10-12 | 2023-11-17 | 北京市市政工程研究院 | Bridge modeling updating method, system and storage medium based on digital twin |
| CN117078866B (en) * | 2023-10-12 | 2024-01-02 | 北京市市政工程研究院 | Bridge modeling updating method, system and storage medium based on digital twin |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280856B (en) | Unknown object grabbing pose estimation method based on mixed information input network model | |
| CN111079561B (en) | Robot intelligent grabbing method based on virtual training | |
| CN108656107B (en) | Mechanical arm grabbing system and method based on image processing | |
| CN109800864B (en) | Robot active learning method based on image input | |
| CN108875935B (en) | Natural image target material visual characteristic mapping method based on generation countermeasure network | |
| JP2021176109A (en) | Room layout estimating method and technique | |
| CN109816725A (en) | A kind of monocular camera object pose estimation method and device based on deep learning | |
| CN106874914A (en) | A kind of industrial machinery arm visual spatial attention method based on depth convolutional neural networks | |
| CN107150347A (en) | Robot perception and understanding method based on man-machine collaboration | |
| CN111098301A (en) | Control method of task type robot based on scene knowledge graph | |
| Wilson et al. | How to generate a thousand master plans: A framework for computational urban design | |
| CN111260649B (en) | Close-range mechanical arm sensing and calibrating method | |
| CN116500901A (en) | Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention | |
| CN113370217B (en) | Object gesture recognition and grabbing intelligent robot method based on deep learning | |
| CN111161364A (en) | Real-time shape completion and attitude estimation method for single-view depth map | |
| CN114117926A (en) | Robot cooperative control algorithm based on federal learning | |
| CN111152227A (en) | Mechanical arm control method based on guided DQN control | |
| Yandun et al. | Visual 3d reconstruction and dynamic simulation of fruit trees for robotic manipulation | |
| CN114851201A (en) | Mechanical arm six-degree-of-freedom vision closed-loop grabbing method based on TSDF three-dimensional reconstruction | |
| Orsula et al. | Learning to Grasp on the Moon from 3D Octree Observations with Deep Reinforcement Learning | |
| Hoyos et al. | Incremental learning of skills in a task-parameterized gaussian mixture model | |
| CN111496794B (en) | Kinematics self-grabbing learning method and system based on simulation industrial robot | |
| CN113313176A (en) | Point cloud analysis method based on dynamic graph convolution neural network | |
| CN109493279B (en) | Large-scale unmanned aerial vehicle image parallel splicing method | |
| CN114131603B (en) | Deep reinforcement learning robot grabbing method based on perception enhancement and scene migration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |