CN111438499A - 5G + industrial AR-based assembly method using constraint-free force feedback - Google Patents

5G + industrial AR-based assembly method using constraint-free force feedback Download PDF

Info

Publication number
CN111438499A
CN111438499A CN202010239148.6A CN202010239148A CN111438499A CN 111438499 A CN111438499 A CN 111438499A CN 202010239148 A CN202010239148 A CN 202010239148A CN 111438499 A CN111438499 A CN 111438499A
Authority
CN
China
Prior art keywords
hand
force
user
virtual
force feedback
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010239148.6A
Other languages
Chinese (zh)
Inventor
杜广龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN202010239148.6A priority Critical patent/CN111438499A/en
Publication of CN111438499A publication Critical patent/CN111438499A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B23—MACHINE TOOLS; METAL-WORKING NOT OTHERWISE PROVIDED FOR
    • B23P—METAL-WORKING NOT OTHERWISE PROVIDED FOR; COMBINED OPERATIONS; UNIVERSAL MACHINE TOOLS
    • B23P19/00—Machines for simply fitting together or separating metal parts or objects, or metal and non-metal parts, whether or not involving some deformation; Tools or devices therefor so far as not provided for in other classes
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/04—Manufacturing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Abstract

The invention discloses an assembling method using unconstrained force feedback based on 5G + industrial AR. The method comprises the following steps: dynamically tracking and capturing gesture data of a user by using a mobile operation platform, and transmitting the captured gesture data to a computer by using a 5G technology; according to the captured gesture data, the computer models the hand of the user and projects the hand into a three-dimensional space of virtual reality established in advance, and then interactive detection is carried out on a virtual hand model and a part model in the three-dimensional space, so that the user affects the part model; by controlling the current and the position, the electromagnet on the mobile operation platform is used for generating corresponding electromagnetic force feedback, so that unlimited force feedback is realized; the method is characterized in that an artificial potential field method and a constraint identification technology are used for carrying out force assistance on the operation of a user, and the user is helped to realize natural interactive assembly by using unconstrained force feedback. The invention allows an operator to directly use gestures to perform virtual assembly, and can help the operator to improve the operation precision.

Description

5G + industrial AR-based assembly method using constraint-free force feedback
Technical Field
The invention belongs to the field of virtual assembly, and particularly relates to an assembly method using unconstrained force feedback based on 5G + industrial AR.
Background
For workers in a factory, assembly is their daily work, but conventional assembly training requires a large expenditure of financial resources and material resources. Meanwhile, for a manufacturer, after a part is designed, the part needs to be produced to check the quality of the part, and the process and the design period consume certain financial and material resources. Therefore, virtual assembly techniques have been created to solve these problems. The learning efficiency of tasks such as machine assembly can be improved by introducing virtual training. For manufacturers, virtual assembly techniques help assess potential problems with assembled parts to reduce design cycle time and improve product quality before deployment. Virtual assembly also helps to estimate manufacturing costs and risks and to train workers to improve assembly skills.
some methods of operation (S.Hassan, J.Yoon, "aided by vertical optimization on searching and designing field application," Advances in Engineering Software, vol.69, pp.18-25, March.Germao Gonzalez-Badillo, Hugo media-casting, Theodere L im, Jame research, Samir Garbaya, "The analysis of a physics and construction-basic analysis system," analysis to, Vol.34susurue: 1, pp.41-55,2014. geological Software virtual learning system, "analysis to, virtual learning of analysis of, virtual learning of, simulation of, virtual learning of, quality of, simulation of, sample of, simulation of, sample of, simulation of, and simulation of, experiment of, virtual learning of, cost of, and virtual learning of, quality learning, quality of, quality.
Therefore, virtual reality technology is introduced and applied to realize virtual assembly. However, a completely virtual environment May cause discomfort to the muscles of the operator (C.Pontonnier, A.Samani, M.Badawi, P.Madelein and G.Dumont, "assembling the activity of a VR-Based Assembly Task organization to EvaluatePhysica Risk organisms," IEEE Transactions on Visualization and computer graphics, vol.20, No.5, pp.664-674, May 2014.). Haptic-based virtual training is more similar to real training, so force feedback, a technique that can reduce the sensory and difficulty gap between real and virtual environments. However, after the force feedback device is used, the operator is restricted by the force feedback device and cannot move freely. Meanwhile, the current research finds that the platform of the industrial AR technology is more suitable for training. The use of industrial AR improves the accuracy and efficiency of task execution, significantly reducing the impact of proficiency differences between participants. The virtual assembly method using the industrial AR is superior in time to complete the assembly task, compared to the classical method.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and provides a novel virtual assembly method which can realize unlimited force feedback and simultaneously apply industrial AR technology to improve user experience. The mobile operation platform firstly dynamically tracks and captures gesture data of a user, then establishes a virtual hand model according to the data and projects the virtual hand model to virtual reality, the interaction between the user's direct bare hand and a virtual object is realized by utilizing an interaction detection technology, and a non-contact force of electromagnetic force is selected as a feedback force, so that non-limiting force feedback is realized. On the basis, in order to improve the success rate and the efficiency of assembly, an auxiliary technology is designed, and the precision and the speed of completing tasks by a user are improved.
The purpose of the invention is realized by at least one of the following technical solutions.
An assembly method using unconstrained force feedback based on 5G + industrial AR, as shown in fig. 1, comprises the following steps:
S1, dynamically tracking and capturing gesture data of a user by using the mobile operation platform, and transmitting the captured gesture data to a computer by using a 5G technology;
S2, according to the gesture data captured in the step S1, the computer models the hand of the user and projects the hand into a virtual reality three-dimensional space which is established in advance, the three-dimensional space comprises an assembly part model which needs to be operated by the user in the industry, and then the virtual hand model and the part model are subjected to interactive detection, so that the user affects the part model;
S3, controlling current and position, and using an electromagnet on the mobile operation platform to generate corresponding electromagnetic force feedback to realize unlimited force feedback;
S4, the user is assisted by force through an artificial potential field method and constraint identification technology, and natural interactive assembly with constraint-free force feedback is achieved.
Further, in step S1, the mobile operation platform includes a mobile cart and a mechanical arm, the mechanical arm is fixed on the mobile cart, a platform is disposed at a tail end of the mechanical arm, and an electromagnet and a camera for capturing gesture data are mounted on the platform; a user feels the electromagnetic force by wearing a glove attached with a magnet array; when a user operates the robot, the hand with the gloves is placed above the electromagnet of the platform, the camera fixed above the platform through the bracket captures gesture data, and when the hand needs to move in the operation process, the computer can presume the moving direction of the hand according to the returned gesture data so as to transmit information to the mechanical arm through the 5G network, so that the mechanical arm moves the tail end platform to track the hand; when the moving direction and distance of the hand are about to exceed the range which can be reached by the mechanical arm, the trolley is moved by an instruction, so that the mechanical arm can track the position of the hand again.
Further, the step S1 specifically includes the following steps:
S11, in order to realize mobile interaction, firstly, coordinates of different devices need to be unified; the calibration box for constructing the virtual reality space is fixed in position, the gesture data of the user is acquired by a camera on a mobile operation platform, the relative position relationship between the camera and the mobile operation platform is fixed, the mobile operation platform is arranged at the tail end of a mechanical arm of the mobile robot, and the mechanical arm of the mobile robot and the calibration box are in a world coordinate system, so that the position relationship between the hand of the user and a virtual hand model in the virtual space is obtained through a series of coordinate transformation;
S12, certain noise exists in the user gesture data during measurement, so that the gesture data obtained through measurement are subjected to denoising processing by using a Kalman filter, and the data are ensured to be more accurate;
S13, when dynamically tracking the hand of the user, in order to enhance robustness to sensor noise and equipment uncertainty, a proportional-integral-derivative control strategy is used for controlling the movement track of the mobile operation platform, so that the movement accuracy of the mobile operation platform is ensured, and meanwhile, a Kalman filter is also used for ensuring the smoothness of the movement track of the mobile operation platform.
Further, the step S2 specifically includes the following steps:
S21, constructing a virtual hand model according to the gesture data captured in the step S1;
S22, an industrial assembly part model which is established in advance is arranged in the constructed virtual reality space, after the hand model is projected into the virtual reality space, the virtual hand model and the part model are interactively detected, and the virtual hand model and the part model are surrounded by a cylinder surrounding box, so that the collision between the virtual hand and the virtual part is simplified into the collision between cylinders;
And S23, hiding the established virtual hand model when the assembly work is executed, so as to realize the effect that the bare hand of the user interacts with the virtual part.
Further, the step S3 specifically includes the following steps:
S31, moving a coil on the operation platform to generate electromagnetic force, and wearing a magnet array on the hand of an operator to sense electromagnetic force feedback provided by the coil;
S32, in order to eliminate the adverse effect caused by the external force, a series of sample forces are generated by using a proportional integral derivative control strategy for training;
S33, training the current and the position required by generating certain force at different positions by using an inverse neural network, and in order to prevent data errors caused by shaking of human hands in the training process, training by using a magnet worn by a dummy hand; the position of a hand and the force in the environment are used as input, so that the input layer comprises 6 elements which are three-dimensional data of the position and the force respectively, the output layer comprises 4 elements which are three-dimensional data of the current magnitude in the electromagnet and the position of the electromagnet, a neural network comprising two hidden layers is determined through continuous comparison, and the reverse neural network with the structure of 6-14-8-4 is finally obtained.
Further, in step S33, by setting a desired force and a corresponding hand position, the computer adjusts the current and the position of the electromagnet according to the given force and the corresponding hand position so that the hand can feel a desired amount of force at the position; the expected force and the generated force are compared and training is completed when the difference between the two forces reaches a user preset threshold.
Further, the step S4 specifically includes the following steps:
S41, using a constraint identification technology to help a user determine the position and posture relation between parts, so that the user can align and assemble more quickly and accurately; when relative motion exists between the two geometric bodies, the computer calculates the position and posture relation between the two geometric bodies in real time, namely compares the geometric relation between the planes of the two geometric bodies, and when the position and posture relation of the two geometric bodies meets the preset condition, the position and posture relation of the two geometric bodies is determined;
And S42, after the position and posture relation is determined, generating virtual force by using an artificial potential field method to guide a user to assemble the parts at the correct positions.
Further, said correct position acts as a gravitational pole in the potential field generating a gravitational field U gra(p), the specific formula is as follows:
Figure BDA0002431966800000051
where ξ represents the scale factor and ρ represents the part current position p to the target position p tarIn between Distance, the gravitational force generated in the gravitational field is the derivative of the gravitational field with respect to distance.
Further, to ensure that the user fits the component in the correct position, the wrong position in the potential field as a repulsive pole creates a repulsive field U rep(p), the specific formula is as follows:
Figure BDA0002431966800000052
where η represents the scale factor and μ represents the current position p to the error position p of the component unWhen the distance is greater than the threshold value mu 0In time, no repulsive force is generated; the repulsion force is the derivative of the repulsion force field to the distance; the resultant force of the parts is the sum of the repulsive force and the attractive force.
Compared with the prior art, the invention has the following advantages:
(1) By moving the gesture tracking platform, the gestures of the operator can be dynamically tracked and captured, and the operator is allowed to directly use the gestures for virtual assembly. The combination of industrial AR and mobile interaction makes the operation space and feedback space unlimited. In the interaction process, the operator can realize all-round perception. The movement of the operator does not affect the interaction.
(2) The characteristic that the electromagnetic force is non-contact force is utilized, and the limitation of a force feedback device on an operator is eliminated. Thus, the operator can feel force feedback without limitation when interacting. In addition, the proposed assistance technique can help the operator to improve the operation accuracy.
(3) Through hand modeling and interactive detection, an operator can directly touch and perceive a virtual object, so that the operator can focus more on the work in front without considering how to operate the device. The operation mode accords with the daily operation habit of people, thereby reducing the learning cost and improving the interaction efficiency.
Drawings
FIG. 1 is a flow chart of an assembly method using unconstrained force feedback based on 5G + industrial AR in accordance with the present invention.
Detailed Description
Specific implementations of the present invention will be further described with reference to the following examples and drawings, but the embodiments of the present invention are not limited thereto.
Example (b):
An assembly method using unconstrained force feedback based on 5G + industrial AR, comprising the steps of:
S1, dynamically tracking and capturing gesture data of a user by using the mobile operation platform, and transmitting the captured gesture data to a computer by using a 5G technology;
The mobile operation platform comprises a mobile trolley and a mechanical arm, the mechanical arm is fixed on the mobile trolley, a platform is arranged at the tail end of the mechanical arm, and an electromagnet and a camera used for capturing gesture data are mounted on the platform; a user feels the electromagnetic force by wearing a glove attached with a magnet array; when a user operates the robot, the hand with the gloves is placed above the electromagnet of the platform, the camera fixed above the platform through the bracket captures gesture data, and when the hand needs to move in the operation process, the computer can presume the moving direction of the hand according to the returned gesture data so as to transmit information to the mechanical arm through the 5G network, so that the mechanical arm moves the tail end platform to track the hand; when the moving direction and distance of the hand are about to exceed the range which can be reached by the mechanical arm, the trolley is moved by an instruction, so that the mechanical arm can track the position of the hand again.
Step S1 specifically includes the following steps:
S11, in order to realize mobile interaction, firstly, coordinates of different devices need to be unified; the calibration box for constructing the virtual reality space is fixed in position, the gesture data of the user is acquired by a camera on a mobile operation platform, the relative position relationship between the camera and the mobile operation platform is fixed, the mobile operation platform is arranged at the tail end of a mechanical arm of the mobile robot, and the mechanical arm of the mobile robot and the calibration box are in a world coordinate system, so that the position relationship between the hand of the user and a virtual hand model in the virtual space is obtained through a series of coordinate transformation;
S12, certain noise exists in the user gesture data during measurement, so that the gesture data obtained through measurement are subjected to denoising processing by using a Kalman filter, and the data are ensured to be more accurate;
And S13, when dynamically tracking the hand of the user, in order to enhance robustness to sensor noise and equipment uncertainty, controlling the movement track of the mobile operation platform by using a proportional-integral-derivative control (PID) strategy to ensure the movement accuracy of the mobile operation platform, and simultaneously ensuring the smoothness of the movement track of the mobile operation platform by using a Kalman filter.
S2, according to the gesture data captured in the step S1, the computer models the hand of the user and projects the hand into a virtual reality three-dimensional space which is established in advance, the three-dimensional space comprises an assembly part model which needs to be operated by the user in the industry, and then the virtual hand model and the part model are subjected to interactive detection, so that the user affects the part model; the method specifically comprises the following steps:
S21, constructing a virtual hand model according to the gesture data captured in the step S1;
S22, an industrial assembly part model which is established in advance is arranged in the constructed virtual reality space, after the hand model is projected into the virtual reality space, the virtual hand model and the part model are interactively detected, and the virtual hand model and the part model are surrounded by a cylinder surrounding box, so that the collision between the virtual hand and the virtual part is simplified into the collision between cylinders;
And S23, hiding the established virtual hand model when the assembly work is executed, so as to realize the effect that the bare hand of the user interacts with the virtual part.
S3, controlling current and position, and using an electromagnet on the mobile operation platform to generate corresponding electromagnetic force feedback to realize unlimited force feedback; the method specifically comprises the following steps:
S31, moving a coil on the operation platform to generate electromagnetic force, and wearing a magnet array on the hand of an operator to sense electromagnetic force feedback provided by the coil;
S32, in order to eliminate the adverse effect caused by the external force, a series of sample forces are generated by using a proportional integral derivative control strategy for training;
S33, training the current and the position required by generating certain force at different positions by using an inverse neural network, and in order to prevent data errors caused by shaking of human hands in the training process, training by using a magnet worn by a dummy hand; the position of a hand and the force in the environment are used as input, so that the input layer comprises 6 elements which are three-dimensional data of the position and the force respectively, the output layer comprises 4 elements which are three-dimensional data of the current magnitude in the electromagnet and the position of the electromagnet, a neural network comprising two hidden layers is determined through continuous comparison, and the reverse neural network with the structure of 6-14-8-4 is finally obtained.
By giving a desired force and corresponding hand position, the computer adjusts the current and the position of the electromagnet according to the given force and corresponding hand position so that the hand can feel the desired amount of force at that position; the expected force and the generated force are compared, and when the difference between the two forces reaches a user-set threshold, in this embodiment, the threshold is 0.01N, and the training is completed.
S4, performing force assistance on the operation of the user by using an artificial potential field method and a constraint identification technology, and helping the user to realize natural interactive assembly by using unconstrained force feedback; the method specifically comprises the following steps:
S41, using a constraint identification technology to help a user determine the position and posture relation between parts, so that the user can align and assemble more quickly and accurately; when relative motion exists between the two geometric bodies, the computer calculates the position and posture relation between the two geometric bodies in real time, namely compares the geometric relation between the planes of the two geometric bodies, and when the position and posture relation of the two geometric bodies meets the preset condition, the position and posture relation of the two geometric bodies is determined;
And S42, after the position and posture relation is determined, generating virtual force by using an artificial potential field method to guide a user to assemble the parts at the correct positions.
Said correct position acting as an attractive pole in the potential field to generate a gravitational field U gra(p), the specific formula is as follows:
Figure BDA0002431966800000091
where ξ represents the scale factor and ρ represents the part current position p to the target position p tarThe gravitational force generated in the gravitational field is the derivative of the gravitational field to the distance.
Further, to ensure that the user fits the component in the correct position, the wrong position in the potential field as a repulsive pole creates a repulsive field U rep(p), the specific formula is as follows:
Figure BDA0002431966800000092
where η represents the scale factor and μ represents the current position p to the error position p of the component unWhen the distance is greater than the threshold value mu 0In time, no repulsive force is generated; the repulsion force is the derivative of the repulsion force field to the distance; the resultant force of the parts is the sum of the repulsive force and the attractive force.

Claims (9)

1. An assembly method using unconstrained force feedback based on 5G + industrial AR, comprising the steps of:
S1, dynamically tracking and capturing gesture data of a user by using the mobile operation platform, and transmitting the captured gesture data to a computer by using a 5G technology;
S2, according to the gesture data captured in the step S1, the computer models the hand of the user and projects the hand into a virtual reality three-dimensional space which is established in advance, the three-dimensional space comprises an assembly part model which needs to be operated by the user in the industry, and then the virtual hand model and the part model are subjected to interactive detection, so that the user affects the part model;
S3, controlling current and position, and using an electromagnet on the mobile operation platform to generate corresponding electromagnetic force feedback to realize unlimited force feedback;
S4, the user is assisted by force through an artificial potential field method and constraint identification technology, and natural interactive assembly with constraint-free force feedback is achieved.
2. The assembly method using unconstrained force feedback based on 5G + industrial AR as claimed in claim 1, wherein in step S1, the mobile operation platform comprises a mobile cart and a robotic arm, the robotic arm is fixed on the mobile cart, a platform is arranged at the end of the robotic arm, and an electromagnet and a camera for capturing gesture data are mounted on the platform; a user feels the electromagnetic force by wearing a glove attached with a magnet array; when a user operates the robot, the hand with the gloves is placed above the electromagnet of the platform, the camera fixed above the platform through the bracket captures gesture data, and when the hand needs to move in the operation process, the computer can presume the moving direction of the hand according to the returned gesture data so as to transmit information to the mechanical arm through the 5G network, so that the mechanical arm moves the tail end platform to track the hand; when the moving direction and distance of the hand are about to exceed the range which can be reached by the mechanical arm, the trolley is moved by an instruction, so that the mechanical arm can track the position of the hand again.
3. The assembly method using unconstrained force feedback based on 5G + industrial AR as claimed in claim 1, wherein said step S1 specifically includes the following steps:
S11, in order to realize mobile interaction, firstly, coordinates of different devices need to be unified; the calibration box for constructing the virtual reality space is fixed in position, the gesture data of the user is acquired by a camera on a mobile operation platform, the relative position relationship between the camera and the mobile operation platform is fixed, the mobile operation platform is arranged at the tail end of a mechanical arm of the mobile robot, and the mechanical arm of the mobile robot and the calibration box are in a world coordinate system, so that the position relationship between the hand of the user and a virtual hand model in the virtual space is obtained through a series of coordinate transformation;
S12, certain noise exists in the user gesture data during measurement, so that the gesture data obtained through measurement are subjected to denoising processing by using a Kalman filter, and the data are ensured to be more accurate;
S13, when dynamically tracking the hand of the user, in order to enhance robustness to sensor noise and equipment uncertainty, a proportional-integral-derivative control strategy is used for controlling the movement track of the mobile operation platform, so that the movement accuracy of the mobile operation platform is ensured, and meanwhile, a Kalman filter is also used for ensuring the smoothness of the movement track of the mobile operation platform.
4. The assembly method using unconstrained force feedback based on 5G + industrial AR as claimed in claim 1, wherein said step S2 specifically includes the following steps:
S21, constructing a virtual hand model according to the gesture data captured in the step S1;
S22, an industrial assembly part model which is established in advance is arranged in the constructed virtual reality space, after the hand model is projected into the virtual reality space, the virtual hand model and the part model are interactively detected, and the virtual hand model and the part model are surrounded by a cylinder surrounding box, so that the collision between the virtual hand and the virtual part is simplified into the collision between cylinders;
And S23, hiding the established virtual hand model when the assembly work is executed, so as to realize the effect that the bare hand of the user interacts with the virtual part.
5. The assembly method using unconstrained force feedback based on 5G + industrial AR as claimed in claim 1, wherein said step S3 specifically includes the following steps:
S31, moving a coil on the operation platform to generate electromagnetic force, and wearing a magnet array on the hand of an operator to sense electromagnetic force feedback provided by the coil;
S32, in order to eliminate the adverse effect caused by the external force, a series of sample forces are generated by using a proportional integral derivative control strategy for training;
S33, training the current and the position required by generating certain force at different positions by using an inverse neural network, and in order to prevent data errors caused by shaking of human hands in the training process, training by using a magnet worn by a dummy hand; the position of a hand and the force in the environment are used as input, so that the input layer comprises 6 elements which are three-dimensional data of the position and the force respectively, the output layer comprises 4 elements which are three-dimensional data of the current magnitude in the electromagnet and the position of the electromagnet, a neural network comprising two hidden layers is determined through continuous comparison, and the reverse neural network with the structure of 6-14-8-4 is finally obtained.
6. The natural interactive assembling method using unconstrained force feedback based on industrial AR as recited in claim 5, wherein in step S33, by giving a desired force and a corresponding hand position, the computer adjusts the current and the position of the electromagnet according to the given force and the corresponding hand position so that the hand can feel the desired amount of force at the position; the expected force and the generated force are compared and training is completed when the difference between the two forces reaches a user preset threshold.
7. The industrial AR-based natural interactive assembling method using unconstrained force feedback according to claim 1, wherein said step S4 specifically comprises the steps of:
S41, using a constraint identification technology to help a user determine the position and posture relation between parts, so that the user can align and assemble more quickly and accurately; when relative motion exists between the two geometric bodies, the computer calculates the position and posture relation between the two geometric bodies in real time, namely compares the geometric relation between the planes of the two geometric bodies, and when the position and posture relation of the two geometric bodies meets the preset condition, the position and posture relation of the two geometric bodies is determined;
And S42, after the position and posture relation is determined, generating virtual force by using an artificial potential field method to guide a user to assemble the parts at the correct positions.
8. The industrial AR based natural interaction with unconstrained force feedback according to claim 7 The method of assembly being characterized in that said correct position generates a gravitational field U as a gravitational pole in the potential field gra(p), the specific formula is as follows:
Figure FDA0002431966790000031
where ξ represents the scale factor and ρ represents the part current position p to the target position p tarThe gravitational force generated in the gravitational field is the derivative of the gravitational field to the distance.
9. The industrial AR based natural interaction fitting method using unconstrained force feedback according to claim 7, wherein to ensure that the user fits the component in the correct position, the wrong position in the potential field as a repulsive pole creates a repulsive force field U rep(p), the specific formula is as follows:
Figure FDA0002431966790000041
where η represents the scale factor and μ represents the current position p to the error position p of the component unWhen the distance is greater than the threshold value mu 0In time, no repulsive force is generated; the repulsion force is the derivative of the repulsion force field to the distance; the resultant force of the parts is the sum of the repulsive force and the attractive force.
CN202010239148.6A 2020-03-30 2020-03-30 5G + industrial AR-based assembly method using constraint-free force feedback Pending CN111438499A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010239148.6A CN111438499A (en) 2020-03-30 2020-03-30 5G + industrial AR-based assembly method using constraint-free force feedback

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010239148.6A CN111438499A (en) 2020-03-30 2020-03-30 5G + industrial AR-based assembly method using constraint-free force feedback

Publications (1)

Publication Number Publication Date
CN111438499A true CN111438499A (en) 2020-07-24

Family

ID=71651217

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010239148.6A Pending CN111438499A (en) 2020-03-30 2020-03-30 5G + industrial AR-based assembly method using constraint-free force feedback

Country Status (1)

Country Link
CN (1) CN111438499A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115346413A (en) * 2022-08-19 2022-11-15 南京邮电大学 Assembly guidance method and system based on virtual-real fusion

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8169306B2 (en) * 2009-03-23 2012-05-01 Methode Electronics, Inc. Touch panel assembly with haptic effects and method of manufacturing thereof
CN204288637U (en) * 2014-12-19 2015-04-22 高晓瑞 The multiplex apparatus for demonstrating of a kind of electromagnetic induction
CN204374672U (en) * 2014-10-11 2015-06-03 华南理工大学 The grand dynamic parallel positioning system of precision under a kind of micro-nano operating environment
CN108406725A (en) * 2018-02-09 2018-08-17 华南理工大学 Force feedback man-machine interactive system and method based on electromagnetic theory and mobile tracking
CN109521868A (en) * 2018-09-18 2019-03-26 华南理工大学 A kind of dummy assembly method interacted based on augmented reality and movement
CN110515455A (en) * 2019-07-25 2019-11-29 山东科技大学 It is a kind of based on the dummy assembly method cooperateed in Leap Motion and local area network
CN110794969A (en) * 2019-10-30 2020-02-14 华南理工大学 Non-contact force feedback-oriented natural man-machine interaction method
CN110815258A (en) * 2019-10-30 2020-02-21 华南理工大学 Robot teleoperation system and method based on electromagnetic force feedback and augmented reality

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8169306B2 (en) * 2009-03-23 2012-05-01 Methode Electronics, Inc. Touch panel assembly with haptic effects and method of manufacturing thereof
CN204374672U (en) * 2014-10-11 2015-06-03 华南理工大学 The grand dynamic parallel positioning system of precision under a kind of micro-nano operating environment
CN204288637U (en) * 2014-12-19 2015-04-22 高晓瑞 The multiplex apparatus for demonstrating of a kind of electromagnetic induction
CN108406725A (en) * 2018-02-09 2018-08-17 华南理工大学 Force feedback man-machine interactive system and method based on electromagnetic theory and mobile tracking
CN109521868A (en) * 2018-09-18 2019-03-26 华南理工大学 A kind of dummy assembly method interacted based on augmented reality and movement
CN110515455A (en) * 2019-07-25 2019-11-29 山东科技大学 It is a kind of based on the dummy assembly method cooperateed in Leap Motion and local area network
CN110794969A (en) * 2019-10-30 2020-02-14 华南理工大学 Non-contact force feedback-oriented natural man-machine interaction method
CN110815258A (en) * 2019-10-30 2020-02-21 华南理工大学 Robot teleoperation system and method based on electromagnetic force feedback and augmented reality

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115346413A (en) * 2022-08-19 2022-11-15 南京邮电大学 Assembly guidance method and system based on virtual-real fusion

Similar Documents

Publication Publication Date Title
US20210205986A1 (en) Teleoperating Of Robots With Tasks By Mapping To Human Operator Pose
Wang et al. Real-virtual components interaction for assembly simulation and planning
Neto et al. High‐level robot programming based on CAD: dealing with unpredictable environments
Pan et al. Augmented reality-based robot teleoperation system using RGB-D imaging and attitude teaching device
CN113829343B (en) Real-time multitasking and multi-man-machine interaction system based on environment perception
CN110385694A (en) Action teaching device, robot system and the robot controller of robot
Li et al. An enhanced teaching interface for a robot using DMP and GMR
Liang et al. An augmented discrete-time approach for human-robot collaboration
CN113103230A (en) Human-computer interaction system and method based on remote operation of treatment robot
Itauma et al. Gesture imitation using machine learning techniques
Ben Abdallah et al. Kinect-based sliding mode control for Lynxmotion robotic arm
Yang et al. Research on virtual haptic disassembly platform considering disassembly process
Chen et al. A human–robot interface for mobile manipulator
Nasim et al. Physics-based interactive virtual grasping
Lambrecht et al. Markerless gesture-based motion control and programming of industrial robots
Mower et al. ROS-PyBullet Interface: A framework for reliable contact simulation and human-robot interaction
Xiong et al. Predictive display and interaction of telerobots based on augmented reality
Du et al. A gesture-and speech-guided robot teleoperation method based on mobile interaction with unrestricted force feedback
CN111438499A (en) 5G + industrial AR-based assembly method using constraint-free force feedback
Das et al. GeroSim: A simulation framework for gesture driven robotic arm control using Intel RealSense
Du et al. Natural human–machine interface with gesture tracking and cartesian platform for contactless electromagnetic force feedback
Kim et al. Adaptation of human motion capture data to humanoid robots for motion imitation using optimization
CN115481489A (en) System and method for verifying suitability of body-in-white and production line based on augmented reality
Du et al. An offline-merge-online robot teaching method based on natural human-robot interaction and visual-aid algorithm
Al-Junaid ANN based robotic arm visual servoing nonlinear system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200724

RJ01 Rejection of invention patent application after publication