CN117806335A - Intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation - Google Patents
Intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation Download PDFInfo
- Publication number
- CN117806335A CN117806335A CN202410235805.8A CN202410235805A CN117806335A CN 117806335 A CN117806335 A CN 117806335A CN 202410235805 A CN202410235805 A CN 202410235805A CN 117806335 A CN117806335 A CN 117806335A
- Authority
- CN
- China
- Prior art keywords
- assembler
- digital twin
- obstacle avoidance
- model
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 20
- 230000008859 change Effects 0.000 claims abstract description 5
- 238000001514 detection method Methods 0.000 claims description 40
- 238000003062 neural network model Methods 0.000 claims description 27
- 238000010586 diagram Methods 0.000 claims description 15
- 239000012636 effector Substances 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 13
- 210000003423 ankle Anatomy 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 8
- 210000001145 finger joint Anatomy 0.000 claims description 6
- 210000003127 knee Anatomy 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 5
- 238000012549 training Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 3
- 238000005315 distribution function Methods 0.000 claims description 3
- 210000003414 extremity Anatomy 0.000 claims description 3
- 210000004936 left thumb Anatomy 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 210000004935 right thumb Anatomy 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 6
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Abstract
The invention belongs to the technical field of obstacle avoidance of man-machine cooperation scenes, and solves the problem of combining a digital twin technology with an intelligent robot obstacle avoidance technology in a complex dynamic environment. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation comprises the following steps: setting the space position of a physical entity; constructing an assembly robot digital twin model and an assembler digital twin model; establishing an assembly robot digital twin model and an obstacle avoidance strategy of an assembler digital twin model in a digital space; the improved Blazepost algorithm is adopted to detect the posture of the assembler in real time, and real-time posture information is returned to the corresponding assembler digital twin model to carry out virtual-real synchronization; the digital twin model of the assembly robot carries out obstacle avoidance according to the posture change of the digital twin model of the assembler and the obstacle avoidance strategy, and feeds back the obstacle avoidance to the physical space in real time. The invention can dynamically avoid the obstacle in a complex dynamic environment and improve the efficiency and safety of man-machine cooperation.
Description
Technical Field
The invention belongs to the technical field of obstacle avoidance of man-machine cooperation scenes, and particularly relates to an intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation.
Background
In a complex assembly environment, the intelligent robot needs to have intelligent sensing, decision making and action capabilities so as to adapt to dynamically changing scenes, such as avoiding moving obstacles, quickly responding and the like, and a dynamic obstacle avoidance technology has become an important problem in intelligent robot research.
The digital twin technology can provide real-time environmental data, simulation and decision support, and provides a new method and tool for dynamic obstacle avoidance.
Therefore, how to combine the digital twin technology to solve the obstacle avoidance problem of the intelligent robot in the complex dynamic environment becomes the current problem to be solved urgently.
Disclosure of Invention
The invention provides an intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation, which aims to solve at least one technical problem in the prior art.
The invention is realized by adopting the following technical scheme: an intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation comprises the following steps:
s1: setting the spatial positions of physical entities, including the positions of a depth camera, an assembly robot and an assembler;
s2: constructing an assembly robot digital twin model and an assembler digital twin model;
s3: establishing an assembly robot digital twin model and an obstacle avoidance strategy of an assembler digital twin model in a digital space;
s4: the improved Blazepost algorithm is adopted to detect the posture of the assembler in real time, and real-time posture information is returned to the corresponding assembler digital twin model to carry out virtual-real synchronization;
s5: the digital twin model of the assembly robot carries out obstacle avoidance according to the posture change of the digital twin model of the assembler and the obstacle avoidance strategy, and feeds back the obstacle avoidance to the physical space in real time.
Preferably, the assembler digital twin model construction includes the followingThe steps are as follows: constructing a virtual model of a basic human body framework through three-dimensional modeling software, and setting the virtual modelKey point information; and (3) using a hinge to connect and restrict key points according to human body structures, and constructing the digital twin model of the assembler.
Preferably, the key points in the assembler digital twin model include nose, left eye inside, left eye outside, right eye inside, right eye outside, left ear, right ear, left side of mouth, right side of mouth; left shoulder, right shoulder, left elbow, right elbow, left ankle, right ankle, left thumb joint, right thumb joint, left index finger joint, right index finger joint, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right toe.
Preferably, the obstacle avoidance strategy comprises the steps of:
s31: defining a minimum contact range of the assembler digital twin model, and adding a minimum contact cylinder at each joint of the assembler digital twin model;
s32: respectively establishing a space coordinate system of an assembly robot digital twin model and an assembler digital twin model; establishing a space coordinate system of an end effector in the assembly robot digital twin model, and establishing a space coordinate system of a hand in the assembler digital twin model;
s33: establishing a D-H relation matrix of a space coordinate system of the digital twin model of the assembly robot and a space coordinate system of the end effector;
s34: and setting a touch relation in a digital space, and automatically avoiding a minimum contact cylinder of an area where an assembler is located by using the digital twin model of the assembly robot.
Preferably, the minimum contact range of the head in the assembler digital twin model is greater than 5cm and less than 10cm; the minimum contact range of the limbs in the assembler digital twin model is more than 3cm and less than 5cm; the radius of the smallest contact cylinder corresponds to the value of the smallest contact range at the corresponding position.
Preferably, in step S34, the end effector trajectory of the assembly robot is set as a spatial point-to-point control, and the obstacle avoidance trajectory is planned through a B-spline curve.
Preferably, the step of assembler gesture detection using the modified blazepost algorithm comprises:
s41: acquiring human body posture images of an assembler in an assembly site through a depth camera, manually marking key point positions in the acquired images, and manufacturing an assembler posture detection data set;
s42: the method comprises the steps of constructing an improved Blazepost neural network model, wherein the improved Blazepost neural network model comprises a trunk module, a thermodynamic diagram module, a detection module, a positioning module and a three-dimensional position adjustment module; the human body posture image of the assembler is used as an input layer of the improved Blazepost neural network model, and an output layer is key point information of the posture of the assembler;
s43: inputting the assembler gesture detection data set into an improved Blazepost neural network model for model training; arranging the trained Blazepore neural network model in a Python environment;
s44: and shooting an assembly area by a depth camera, and outputting key point information of the posture of the assembler to a digital twin model of the assembler by using the improved Blazepost neural network model to realize virtual-real synchronization between the two.
Preferably, the trunk module of the improved Blazepost neural network model is used for extracting characteristic information of a human body posture image of an assembler of the input layer and inputting the adjusted image to the thermodynamic diagram module; the thermodynamic diagram module is used for outputting the local position of the assembler in the shooting area and inputting the thermodynamic diagram into the detection module; the positioning module is used for extracting characteristic information of a human body posture image of an assembler of the input layer and inputting the characteristic information to the detection module through the three-dimensional position adjustment module; the three-dimensional position adjustment module is used for adjusting the position information of the assembler extracted from the original detection module by combining the position information of the assembler in the positioning module; the detection module extracts the region where the assembler is located through a software-NMS non-maximum algorithm, and outputs the visualized key point information of the assembler gesture.
Preferably, the positioning module andthe first assembler gesture obtained by the original detection modulePositioning spatial error of individual key points->The expression of (2) is:
in the method, in the process of the invention,the first person of the assembler gesture detected for the positioning module>Positioning coordinates of the key points, < >>The first person is the assembler gesture detected by the original detection module>Positioning coordinates of the key points;
when (when)When the method is used, the positioning coordinates of the posture of the assembler detected by the original detection module are taken as output references; when->And when the positioning module detects the positioning coordinates of the posture of the assembler, the positioning module takes the positioning coordinates of the posture of the assembler detected by the positioning module as an output reference, and the positioning coordinates of the posture of the assembler detected by the positioning module are fed back to the detection module through an adjustment matrix in the three-dimensional position adjustment module.
Preferably, the modified Blazepost neural network model has a loss functionThe expression of (2) is:
in the method, in the process of the invention,a probability for the presence of a target in the pre-selected box; />Probability of actually existing target; />Predicting an offset distance from the pre-selected frame to the real target; />Predicted offset distance for calibration frame from real target, +.>Is->Is required to adjust, is->Is->Positioning distance to be adjusted; />Is a classification parameter; />Is a regression parameter;is a positioning parameter; />To adjust parameters; />Is a classification loss; />Is regression loss; />Adding new positioning loss into the positioning module; />The number of human body posture images for the assembler;
adding new positioning loss in the positioning moduleThe expression of (2) is:
in the method, in the process of the invention,is a distance scale factor; />The first position of the assembler is obtained for the positioning module and the original detection modulePositioning space errors of the key points; />Is a normal distribution function.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, the digital twin model of the assembly robot, the digital twin model of the assembler and the obstacle avoidance strategy are constructed, the real-time detection of the posture of the assembler is carried out by utilizing a depth camera and an improved Blazepost algorithm, real-time posture information is returned to the digital twin model of the assembler for virtual-real synchronization, the digital twin model of the assembly robot carries out obstacle avoidance according to the posture change of the digital twin model of the assembler and the obstacle avoidance strategy and feeds back to a physical space in real time, the purpose of dynamic obstacle avoidance under a complex dynamic environment is achieved, and the efficiency and safety of man-machine cooperation are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a physical spatial location layout of an object of the present invention;
FIG. 2 is a schematic illustration of the location of key points in an assembler digital twinning model of the present invention;
FIG. 3 is a block diagram of the improved Blazepost algorithm of the present invention;
FIG. 4 is a block diagram of the Blazepost algorithm prior to modification;
FIG. 5 is a graph showing the comparison of the loss functions of the Blazepost algorithm before and after modification in the present invention;
FIG. 6 is a comparison of the accuracy of the Blazepost algorithm before and after modification in the present invention;
FIG. 7 is a connection block diagram of the human-machine collaboration of the present invention;
FIG. 8 is a schematic representation of the location of the minimum contact cylinder at the assembler digital twinning model of the present invention;
FIG. 9 is a plot of the coordinate relationship in the assembler digital twin model of the present invention;
FIG. 10 is a plot of the coordinate relationship in the digital twin model of the assembly robot of the present invention;
FIG. 11 is a graph of obstacle avoidance trajectory planning for B-spline curves in the present invention.
In the figure: 1-assembler; 2-an assembly robot; 3-depth camera; 4-minimum contact cylinder; 5-an end effector; and 6-B spline curve obstacle avoidance track.
Detailed Description
Technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the examples of this invention without making any inventive effort, are intended to fall within the scope of this invention.
It should be understood that the structures, proportions, sizes, etc. shown in the drawings are merely for the purpose of understanding and reading the disclosure, and are not intended to limit the scope of the invention, which is defined by the appended claims, and any structural modifications, proportional changes, or dimensional adjustments, which may be made by those skilled in the art, should fall within the scope of the present disclosure without affecting the efficacy or the achievement of the present invention, and it should be noted that, in the present disclosure, relational terms such as first and second are used solely to distinguish one entity from another entity without necessarily requiring or implying any actual relationship or order between such entities.
The present invention provides an embodiment:
as shown in fig. 1 and 7, the intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation comprises the following steps:
s1: setting the spatial positions of physical entities, including the positions of the depth camera 3, the assembly robot 2 and the assembler 1;
s2: constructing an assembly robot digital twin model and an assembler digital twin model;
s3: establishing an assembly robot digital twin model and an obstacle avoidance strategy of an assembler digital twin model in a digital space;
s4: the improved Blazepost algorithm is adopted to detect the posture of the assembler in real time, and real-time posture information is returned to the digital twin model of the assembler to carry out virtual-real synchronization;
s5: the digital twin model of the assembly robot carries out obstacle avoidance according to the posture change of the digital twin model of the assembler and the obstacle avoidance strategy, and feeds back the obstacle avoidance to the physical space in real time.
In this embodiment, the assembler digital twin model building packageThe method comprises the following steps: constructing a virtual model of a basic human body framework through three-dimensional modeling software Solidworks, and setting in the virtual modelKey point information; the key points are connected and restrained by hinges according to the human body structure, and the digital twin model of the assembler is built; the construction of the digital twin model of the assembly robot comprises the following steps: and constructing a virtual model of the assembly robot 2 through three-dimensional modeling software Solidworks, and constructing a digital twin model of the assembly robot by using a virtual physical engine Unity3D given the rod member constraint and the kinematic constraint of the robot.
As shown in fig. 2, the 33 key points in the assembler digital twin model include nose, left-eye inner side, left-eye outer side, right-eye inner side, right-eye outer side, left ear, right ear, left mouth side, right mouth side; a left shoulder, a right shoulder, a left elbow, a right elbow, a left ankle, a right ankle, a left thumb joint, a right thumb joint, a left index finger joint, a right index finger joint, a left hip, a right hip, a left knee, a right knee, a left ankle, a right ankle, a left heel, a left toe, a right toe, and a right toe correspond to a0, a1, a2, a3, a4, a5, a6, a7, a8, a9, a10, a11, a12, a13, a14, a15, a16, a17, a18, a19, a20, a21, a22, a23, a24, a25, a26, a27, a28, a29, a30, a31, and a32 in the sequence shown in fig. 2.
The obstacle avoidance strategy comprises the following steps:
s31: defining a minimum contact range of the assembler digital twin model, and adding a minimum contact cylinder 4 at each joint of the assembler digital twin model, as shown in fig. 8; the minimum contact range of the head part in the assembler digital twin model is more than 5cm and less than 10cm; the minimum contact range of the limbs in the assembler digital twin model is more than 3cm and less than 5cm; the radius of the minimum contact cylinder 4 is consistent with the value of the minimum contact range at the corresponding position;
s32: respectively establishing a space coordinate system of an assembly robot digital twin model and an assembler digital twin model in a Unity3D physical engineAnd->The method comprises the steps of carrying out a first treatment on the surface of the Establishing a spatial coordinate system of the end effector 5 in the assembly robot digital twin model>Establishing a spatial coordinate system of the hand in the assembler digital twin model>. As shown in fig. 9 and 10; assembler digital twin model space coordinate system +.>Spatial coordinate system with hand->Both may be co-ordinate determined by Unity 3D.
S33: establishing a rod D-H matrix relation of the assembly robot 2, and obtaining a D-H relation matrix of a space coordinate system of the digital twin model of the assembly robot and a space coordinate system of the end effector 5 through the rod relation;
in the method, in the process of the invention,indicate the assembly robot->Individual joints to->A transformation matrix of the individual joints; />Is->Torsion angle of individual joints; />Respectively represent +.>The torsion angle, deflection angle, rotation angle and length of the assembly robot rod piece of each joint.
S34: setting a touch relation in a digital space, and automatically avoiding a minimum contact cylinder 4 of an area where an assembler is located by using a digital twin model of the assembly robot;
in step S34, setting the trajectory of the end effector 5 of the assembly robot 2 as a spatial point-to-point control, and planning an obstacle avoidance trajectory through a B-spline curve; to solve the positional relationship between the current B-spline track curve and the end effector 5, a basis function of the B-spline curve during the motion track of the end effector 5 needs to be calculated. When the human body obstacle is faced, curve smooth connection is carried out on coordinate points in the track, so that the position of the end effector 5 is changed, the position points where the track of the end effector 5 passes are connected, and finally, the B-spline curve obstacle avoidance track 6 is obtained, and the track obstacle avoidance operation of the end effector 5 of the assembly robot 2 is realized, as shown in fig. 11. The means for avoiding the obstacle through the B-spline curve is the prior art, and is not described in detail.
As shown in fig. 3 and 4, the steps of using the modified blazepost algorithm for assembler gesture detection include:
s41: constructing a neural network framework in Pycharm programming software by using Python language, connecting to Unity3D by using a camera communication protocol, and enabling the depth camera 3 to communicate with the virtual physical engine Unity3D in a video data stream; acquiring 500 complete human body posture images of an assembler 1 in an assembly site through a depth camera 3, manually marking key point positions in the acquired images by using a Labelimg marking tool, and manufacturing an assembler posture detection data set;
s42: the method comprises the steps of constructing an improved Blazepost neural network model, wherein the improved Blazepost neural network model comprises a trunk module, a thermodynamic diagram module, a detection module, a positioning module and a three-dimensional position adjustment module; the human body posture image of the assembler 1 is used as an input layer of the improved Blazepost neural network model, and an output layer is key point information of the assembler posture;
s43: inputting the assembler gesture detection data set into an improved Blazepost neural network model for model training; arranging the trained Blazepore neural network model in a Python environment;
s44: the depth camera 3 shoots an assembly area, the improved Blazepost neural network model outputs key point information of the posture of an assembler to the digital twin model of the assembler, and virtual-real synchronization between the two is achieved.
The Blazepost neural network model input layer inputs real-time human body posture images of the assembler 1, and the image size isThe method comprises the steps of carrying out a first treatment on the surface of the The trunk module of the improved Blazepost neural network model is provided with 5 convolution layers, and the characteristic information of the human body posture image of the assembler 1 of the input layer is extracted and the image size is changed from +.>Become->The channel number of the image is changed to 32 and then is input to a thermodynamic diagram module; the thermodynamic diagram module consists of 4 convolution layers and 1 output layer, wherein the convolution layers output the local position of the assembler 1 in the shooting area through convolution calculation and input the thermodynamic diagram to the detection module; the detection module consists of 6 convolution layers, extracts characteristic information in an image, and outputs the image with the size of +.>The method comprises the steps of carrying out a first treatment on the surface of the The positioning module consists of 7 depth convolution layers, extracts characteristic information of human body posture images of an assembler 1 of the input layer and converts the image size fromBecome->The three-dimensional position adjustment module is used for inputting the three-dimensional position adjustment module into the detection module; the three-dimensional position adjustment module is used for adjusting the position information of the assembler extracted from the original detection module by combining the position information of the assembler in the positioning module; the detection module extracts the region where the assembler 1 is located through a software-NMS non-maximum algorithm, and outputs 33 pieces of visualized key point information of the assembler gesture.
The positioning module and the original detection module obtain the assembler gesturePositioning spatial error of individual key points->The expression of (2) is:
in the method, in the process of the invention,the first person of the assembler gesture detected for the positioning module>Positioning coordinates of the key points, < >>The first person is the assembler gesture detected by the original detection module>Positioning coordinates of the key points;
when (when)When the method is used, the positioning coordinates of the posture of the assembler detected by the original detection module are taken as output references; when->When the positioning module is used, the positioning coordinates of the posture of the assembler detected by the positioning module are taken as output references and pass through the three-dimensional positionAnd the adjusting matrix in the adjusting module feeds back the positioning coordinates of the assembler gesture detected by the positioning module to the detecting module.
Defining a loss function of an improved Blazepost neural network modelThe expression of (2) is:
in the method, in the process of the invention,a probability for the presence of a target in the pre-selected box; />Probability of actually existing target; />Predicting an offset distance from the pre-selected frame to the real target; />Predicted offset distance for calibration frame from real target, +.>Is->Is required to adjust, is->Is->Positioning distance to be adjusted; />Is a classification parameter; />Is a regression parameter;is a positioning parameter; />To adjust parameters; />Is a classification loss; />Is regression loss; />Adding new positioning loss into the positioning module; />The number of human body posture images for the assembler;
adding new positioning loss in the positioning moduleThe expression of (2) is:
in the method, in the process of the invention,is a distance scale factor; />The first position of the assembler is obtained for the positioning module and the original detection modulePositioning space errors of the key points; />Is a normal distribution function;
classification lossThe expression of (2) is:
regression lossThe expression of (2) is:
in the method, in the process of the invention,is an absolute value; />Is the standard deviation of the function; />Is a variable; />As a conditional function.
Definition of accuracy of improved Blazepost neural network modelThe formula of (2) is:
in the method, in the process of the invention,to accurately predict the proportion of fitters present in the photographing region and actually entering the photographing region; />The proportion of the fitter is judged to be present in the shooting area when the fitter is not present.
The training process to define the improved blazepost neural network model is: inputting 500 images of the assembler gesture detection data set to perform data enhancement operation, wherein the original image is changed into 800 images through rotation and translation operation; setting basic parameters: training iteration times are 50 times, learning rate is 0.01, attenuation coefficient is 0.005, and batch is 8; calling an original Blazepore neural network algorithm by using a Meidappipe function interface in Pytorch, and embedding an improved module into the original network;
as shown in fig. 5 and 6, the loss value of the original blazepore algorithm becomes 0.075 after 50 iterations, and the loss value of the modified blazepore algorithm is 0.06; the accuracy of the original Blazepore algorithm is 86%, and the improved Blazepore algorithm is 92%. And arranging the trained network in a Python environment, and opening the depth camera 3 to shoot a designated area to obtain 33 key point information of the human body frame.
As shown in fig. 7, in step S5, an assembler digital twin virtual-real collaboration is established, video data stream collaboration is performed by the depth camera 3, and 33 pieces of key point information of the detected assembler gesture are given to the assembler digital twin model, which moves along with the movement of the assembler 1; establishing digital twin virtual-real cooperation of the assembly robot, wherein the physical entity of the assembly robot 2 is in communication cooperation with the PLC protocol communication, and the PLC is reflected to the digital twin model of the assembly robot through the input of the physical entity of the assembly robot 2, and the digital twin model of the assembly robot is driven along with the driving of the physical entity.
The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the technical scope of the present invention should be included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation is characterized by comprising the following steps of:
s1: setting the spatial positions of physical entities, including the positions of a depth camera (3), an assembly robot (2) and an assembler (1);
s2: constructing an assembly robot digital twin model and an assembler digital twin model;
s3: establishing an assembly robot digital twin model and an obstacle avoidance strategy of an assembler digital twin model in a digital space;
s4: the improved Blazepost algorithm is adopted to detect the posture of the assembler in real time, and real-time posture information is returned to the digital twin model of the assembler to carry out virtual-real synchronization;
s5: the digital twin model of the assembly robot carries out obstacle avoidance according to the posture change of the digital twin model of the assembler and the obstacle avoidance strategy, and feeds back the obstacle avoidance to the physical space in real time.
2. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation of claim 1, wherein the method comprises the following steps: the assembler digital twin model construction comprises the following steps: constructing a virtual model of a basic human body framework through three-dimensional modeling software, and setting the virtual modelKey point information; and (3) using a hinge to connect and restrict key points according to human body structures, and constructing the digital twin model of the assembler.
3. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation as claimed in claim 2, wherein the method comprises the following steps: key points in the assembler digital twin model include nose, left eye inner side, left eye outer side, right eye inner side, right eye outer side, left ear, right ear, left side of mouth, right side of mouth; left shoulder, right shoulder, left elbow, right elbow, left ankle, right ankle, left thumb joint, right thumb joint, left index finger joint, right index finger joint, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right toe.
4. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation as claimed in claim 2, wherein the method comprises the following steps: the obstacle avoidance strategy comprises the following steps:
s31: defining a minimum contact range of the assembler digital twin model, and adding a minimum contact cylinder (4) at each joint of the assembler digital twin model;
s32: respectively establishing a space coordinate system of an assembly robot digital twin model and an assembler digital twin model; establishing a space coordinate system of an end effector (5) in the assembly robot digital twin model, and establishing a space coordinate system of a hand in the assembler digital twin model;
s33: establishing a D-H relation matrix of a space coordinate system of the digital twin model of the assembly robot and a space coordinate system of the end effector (5);
s34: and setting a touch relation in a digital space, and automatically avoiding a minimum contact cylinder (4) in the area where an assembler (1) is located by using the digital twin model of the assembly robot.
5. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation as set forth in claim 4, wherein: the minimum contact range of the head part in the assembler digital twin model is more than 5cm and less than 10cm; the minimum contact range of the limbs in the assembler digital twin model is more than 3cm and less than 5cm; the radius of the smallest contact cylinder (4) is consistent with the value of the smallest contact range at the corresponding position.
6. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation as set forth in claim 4, wherein: in step S34, the trajectory of the end effector (5) of the assembly robot (2) is set to be a spatial point-to-point control, and the obstacle avoidance trajectory is planned by a B-spline curve.
7. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation as set forth in claim 4, wherein: the step of assembler gesture detection using the modified blazepost algorithm includes:
s41: acquiring a human body posture image of an assembler (1) in an assembly site through a depth camera (3), manually marking the positions of key points in the acquired image, and manufacturing an assembler posture detection data set;
s42: the method comprises the steps of constructing an improved Blazepost neural network model, wherein the improved Blazepost neural network model comprises a trunk module, a thermodynamic diagram module, a detection module, a positioning module and a three-dimensional position adjustment module; the human body posture image of the assembler (1) is used as an input layer of the improved Blazepost neural network model, and an output layer is key point information of the assembler posture;
s43: inputting the assembler gesture detection data set into an improved Blazepost neural network model for model training; arranging the trained Blazepore neural network model in a Python environment;
s44: and shooting an assembly area by a depth camera (3), and outputting key point information of the posture of an assembler to a digital twin model of the assembler by using the improved Blazepost neural network model to realize virtual-real synchronization between the two.
8. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation of claim 7, wherein the method comprises the following steps: the trunk module of the improved Blazepost neural network model is used for extracting characteristic information of a human body posture image of an assembler (1) of the input layer and inputting the adjusted image to the thermodynamic diagram module; the thermodynamic diagram module is used for outputting the local position of the assembler (1) in the shooting area and inputting the thermodynamic diagram into the detection module; the positioning module is used for extracting characteristic information of a human body posture image of the assembler (1) of the input layer and inputting the characteristic information to the detection module through the three-dimensional position adjustment module; the three-dimensional position adjustment module is used for adjusting the position information of the assembler extracted from the original detection module by combining the position information of the assembler in the positioning module; the detection module extracts the region where the assembler (1) is located through a software-NMS non-maximum algorithm, and outputs the visualized key point information of the assembler gesture.
9. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation of claim 8, wherein the method comprises the following steps: the positioning module and the original detection module obtain the assembler gesturePositioning spatial error of key pointsThe expression of (2) is:
in the method, in the process of the invention,the first person of the assembler gesture detected for the positioning module>The location coordinates of the individual keypoints are,the first person is the assembler gesture detected by the original detection module>Positioning coordinates of the key points;
when (when)When the method is used, the positioning coordinates of the posture of the assembler detected by the original detection module are taken as output references; when (when)When the positioning module is used for detecting the positioning coordinate of the posture of the assembler, the positioning coordinate of the posture of the assembler detected by the positioning module is taken as an output reference, and the positioning coordinate of the posture of the assembler detected by the positioning module is fed back to the detection module through the adjustment matrix in the three-dimensional position adjustment module。
10. The intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation of claim 9, wherein the method comprises the following steps: improved Blazepost neural network model loss functionThe expression of (2) is:
in the method, in the process of the invention,a probability for the presence of a target in the pre-selected box; />Probability of actually existing target; />Predicting an offset distance from the pre-selected frame to the real target; />Predicted offset distance for calibration frame from real target, +.>Is->Is required to adjust, is->Is->Positioning distance to be adjusted; />Is a classification parameter; />Is a regression parameter; />Is a positioning parameter; />To adjust parameters; />Is a classification loss; />Is regression loss; />Adding new positioning loss into the positioning module; />The number of human body posture images for the assembler;
adding new positioning loss in the positioning moduleThe expression of (2) is:
in the method, in the process of the invention,is a distance scale factor; />The first step of the assembler gesture obtained for the positioning module and the original detection module>Positioning space errors of the key points; />Is a normal distribution function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410235805.8A CN117806335A (en) | 2024-03-01 | 2024-03-01 | Intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410235805.8A CN117806335A (en) | 2024-03-01 | 2024-03-01 | Intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117806335A true CN117806335A (en) | 2024-04-02 |
Family
ID=90432345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410235805.8A Pending CN117806335A (en) | 2024-03-01 | 2024-03-01 | Intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117806335A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015275A (en) * | 2020-08-29 | 2020-12-01 | 南京翱翔智能制造科技有限公司 | Digital twin AR interaction method and system |
| CN112115607A (en) * | 2020-09-16 | 2020-12-22 | 同济大学 | Mobile intelligent digital twin system based on multidimensional Sayboat space |
| CN113673894A (en) * | 2021-08-27 | 2021-11-19 | 东华大学 | Multi-person cooperation AR assembly method and system based on digital twin |
| US20220105635A1 (en) * | 2021-12-17 | 2022-04-07 | Intel Corporation | Repetitive task and contextual risk analytics for human-robot collaboration |
| CN116460857A (en) * | 2023-05-30 | 2023-07-21 | 郑州轻工业大学 | Personalized man-machine cooperation assembly safety detection and early warning method based on digital twinning |
| CN117021118A (en) * | 2023-10-08 | 2023-11-10 | 中北大学 | Dynamic compensation method for digital twin track error of parallel robot |
| CN117454288A (en) * | 2023-09-25 | 2024-01-26 | 江苏省特种设备安全监督检验研究院 | Numerical twinning-based escalator passenger abnormal behavior monitoring method |
-
2024
- 2024-03-01 CN CN202410235805.8A patent/CN117806335A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015275A (en) * | 2020-08-29 | 2020-12-01 | 南京翱翔智能制造科技有限公司 | Digital twin AR interaction method and system |
| CN112115607A (en) * | 2020-09-16 | 2020-12-22 | 同济大学 | Mobile intelligent digital twin system based on multidimensional Sayboat space |
| CN113673894A (en) * | 2021-08-27 | 2021-11-19 | 东华大学 | Multi-person cooperation AR assembly method and system based on digital twin |
| US20220105635A1 (en) * | 2021-12-17 | 2022-04-07 | Intel Corporation | Repetitive task and contextual risk analytics for human-robot collaboration |
| CN116460857A (en) * | 2023-05-30 | 2023-07-21 | 郑州轻工业大学 | Personalized man-machine cooperation assembly safety detection and early warning method based on digital twinning |
| CN117454288A (en) * | 2023-09-25 | 2024-01-26 | 江苏省特种设备安全监督检验研究院 | Numerical twinning-based escalator passenger abnormal behavior monitoring method |
| CN117021118A (en) * | 2023-10-08 | 2023-11-10 | 中北大学 | Dynamic compensation method for digital twin track error of parallel robot |
Non-Patent Citations (8)
| Title |
|---|
| KLAUS DRÖDER ETAL.: "A machine learning-enhanced digital twin approach for human-robot-collaboration", 《7TH CIRP CONFERENCE ON ASSEMBLY TECHNOLOGIES AND SYSTEMS》, 31 December 2018 (2018-12-31), pages 187 - 192 * |
| MUHAMMAD USMAN SHOUKAT ETAL.: "Digital Twin-Driven Virtual Control Technology of Home-Use Robot: Human-Cyber-Physical System", 《INTERNATIONAL CONFERENCE ON EMERGING TECHNOLOGIES (ICET) 2022》, 31 December 2022 (2022-12-31), pages 240 - 246 * |
| YEPENG FAN ETAL.: "A digital-twin visualized architecture for Flexible Manufacturing System", 《JOURNAL OF MANUFACTURING SYSTEMS》, 31 December 2021 (2021-12-31), pages 176 - 201 * |
| 凌振光等: "人机协作与数字孪生技术在智能实验室中的前瞻性研究", 《广州建筑》, 31 December 2023 (2023-12-31), pages 118 - 120 * |
| 张旭辉等: "数字孪生驱动的掘进机器人决策控制系统研究", 《煤炭科学技术》, 31 July 2022 (2022-07-31), pages 36 - 49 * |
| 杨赓等: "数字孪生驱动的智能人机协作：理论、技术与应用", 《机械工程学报》, 30 September 2022 (2022-09-30), pages 279 - 291 * |
| 纪慧君: "面向数字孪生的工业机械臂抓取系统关键技术的研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, 15 January 2023 (2023-01-15), pages 140 - 1095 * |
| 鲍劲松等: "面向人-机-环境共融的数字孪生协同技术", 《机械工程学报》, 30 September 2022 (2022-09-30), pages 103 - 115 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108876815B (en) | Skeleton posture calculation method, character virtual model driving method and storage medium | |
| CN109344882B (en) | Convolutional neural network-based robot control target pose identification method | |
| CN105252532B (en) | The method of the flexible gesture stability of motion capture robot collaboration | |
| CN108829232B (en) | Method for acquiring three-dimensional coordinates of human skeletal joint points based on deep learning | |
| CN111152218B (en) | Action mapping method and system of heterogeneous humanoid mechanical arm | |
| CN105652871A (en) | Repositioning method for mobile robot | |
| CN108346165A (en) | Robot and three-dimensional sensing components in combination scaling method and device | |
| CN107627303A (en) | A kind of PD SMC control methods based on vision servo system of the eye in structure on hand | |
| CN110232706B (en) | Multi-person follow shooting method, device, equipment and storage medium | |
| CN109940626B (en) | Control method of eyebrow drawing robot system based on robot vision | |
| CN106945043A (en) | A kind of master-slave mode telesurgery robot multi-arm cooperative control system | |
| CN111085996B (en) | Control method, device and system of live working robot | |
| CN109325995B (en) | Low-resolution multi-view hand reconstruction method based on hand parameter model | |
| CN112749512B (en) | Gesture estimation optimization method, system and electronic device | |
| CN111152220B (en) | Mechanical arm control method based on man-machine fusion | |
| CN112276914B (en) | Industrial robot based on AR technology and man-machine interaction method thereof | |
| JP3780017B2 (en) | Time-series image analysis apparatus and analysis method thereof | |
| CN115454240A (en) | Meta universe virtual reality interaction experience system and method | |
| CN104858852A (en) | Optimization and constraint method of humanoid robot for real-time imitating human upper limb movement | |
| CN113146634A (en) | Robot attitude control method, robot and storage medium | |
| CN113822251A (en) | Ground reconnaissance robot gesture control system and control method based on binocular vision | |
| CN117806335A (en) | Intelligent robot digital twin dynamic obstacle avoidance method based on man-machine cooperation | |
| WO2021000327A1 (en) | Hand model generation method, apparatus, terminal device, and hand motion capture method | |
| Xiong et al. | Predictive display and interaction of telerobots based on augmented reality | |
| Sung et al. | Arm orientation estimation method with multiple devices for NUI/NUX |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |