KR102094360B1 - System and method for predicting force based on image - Google Patents

Abstract

An image-based force prediction system and method are disclosed. A method of predicting a force of a robot arm of an image-based force prediction system includes acquiring real images generated by continuously photographing an operation of a robot arm interacting with an object; Obtaining time-series motion information related to a change in motion of the robot arm over time; And predicting an interaction force of the robot arm based on the actual images and the time-series motion information.

Description

Image-based force prediction system and its method {SYSTEM AND METHOD FOR PREDICTING FORCE BASED ON IMAGE}

The present invention relates to a system and method for predicting the interaction force of a robot arm using an image.

Since the interaction force of the robot arm includes the force of the robot arm applied to the object and the force fed back to the robot arm by the physical properties of the object, when the user interacts with the object, the force felt through tactile sense similar. Therefore, in order to precisely control the arm of the robot, it is necessary to measure the interaction force of the robot arm.

Conventional apparatus for measuring the interaction force of the robot arm measures the interaction force of the robot arm using a force sensor or a torque sensor attached to the robot. However, since the force sensor and the torque sensor are expensive devices, there is a problem that the price increases of the robot arm.

In addition, depending on the structure and size of the robot arm, there is a limit that it is difficult to attach a force sensor or a torque sensor.

In addition, the force sensor or the torque sensor measures the interaction force of the robot arm after the robot arm contacts the object, so when the strength of the object is weak, the force sensor or the torque sensor is the robot after the robot arm contacts the object. There was also the possibility of the object breaking before measuring the interaction force of the arm.

Therefore, there is a need for a method of measuring or predicting the interaction force of the robot arm to interact with the object before the robot arm contacts the object without a sensor.

The present invention learns the correlation between the real image of the robot's motion and time series motion information, the object's physical properties and the interaction force of the robot's arm, and applies the real image and the time series motion information of the robot's motion to the learning results. By doing so, it is possible to provide a system and method for predicting the interaction force of the robot arm without a sensor for measuring the interaction force of the robot arm.

In addition, the present invention provides an apparatus and method for reproducing a physical sense to a user using only a camera without configuring a complex sensor network by providing feedback to a user who controls the robot arm according to the predicted interaction force of the robot arm can do.

In addition, since the present invention can predict the interaction force of the robot arm using a real image, when an error occurs in the sensor, the interaction force of the robot arm measured by the sensor and the robot arm predicted using the real image It is possible to provide an apparatus and method capable of determining whether a sensor is abnormal by comparing operating forces.

A method for predicting a force of a robot arm according to an embodiment of the present invention includes: acquiring real images generated by continuously photographing an operation of a robot arm interacting with an object; Obtaining time-series motion information related to a change in motion of the robot arm over time; And predicting an interaction force of the robot arm based on the actual images and the time-series motion information.

The predicting step of the method for predicting the force of the robot arm according to an embodiment of the present invention may include learning results of a correlation between the actual images, the time series motion information, the physical properties of the object, and the interaction force of the robot arm. It is possible to predict the interaction force of the robot arm by applying the actual images, the time series motion information, and the physical properties of the object to a database.

Correlation of the method for predicting the force of a robot arm according to an embodiment of the present invention includes inputting time series motion information of the robot and the real images into a deep learning algorithm of a neural network structure, and predicting the interaction force of the robot arm and the object It can be learned by comparing the physical properties of the object and the interaction force of the robot arm measured using the sensor.

The deep learning algorithm of the method for predicting the force of a robot arm according to an embodiment of the present invention extracts a region related to the motion of the robot from the real images by using a convolutional neural network (CNN) in which the real images are input. ; Calculating class scores of the time-series motion information of the robot using a first fully-connected (FC) layer into which the time-series motion information of the robot is input; Learning the relationship between the region and the time-series operation information that changes over time by inputting the extracted region and the class scores of the time-series operation information into a recurrent neural network (RNN); And calculating a class score of the learning result by the second FC layer into which the learning result of the RNN is input, and predicting the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical property value of the object. By performing, it is possible to learn the correlation between the actual images and time-series motion information of the robot and the interaction force of the robot arm.

The deep learning algorithm of the method for predicting the force of a robot arm according to an embodiment of the present invention generates a virtual image representing a change in motion information over time based on the time series motion information in a graph form, and generates CNN (Convolutional Neural) Network) input to the first convolutional layer (convolutional layer); Inputting the actual images accumulated over time into a second convolutional layer different from the first convolutional layer; Matching a virtual image output from the first convolutional layer and a real image output from the second convolutional layer using a pooling layer; And classifying the matched information through a fully connected layer to perform the process of predicting the time series motion information and the interaction force of the robot arm corresponding to the real image, and the physical property value of the object. A correlation between time-series motion information of the robot and the interaction force of the robot arm may be learned.

Time-series operation information of the method for predicting the force of a robot arm according to an embodiment of the present invention includes a change in power used by the motor of the robot arm to perform the operation of the robot arm that changes over time, and the robot arm It may include at least one of the position change and the motion change of the robot arm.

A method for predicting a force of a robot arm according to an embodiment of the present invention includes predicting haptic sensing information using a material property of the object and a predicted interaction force of the robot arm; And providing feedback to the user of the robot arm according to the haptic sensing information.

A method for learning the force of a robot arm according to an embodiment of the present invention includes: acquiring real images generated by photographing a motion of a robot arm related to an object, and time series motion information related to the motion of the robot arm; Predicting the interaction force of the robot arm corresponding to the time series motion information and the real image and physical properties of the object by inputting the real images and the time series motion information into a deep learning algorithm; And comparing the predicted interaction force of the robot arm and the physical force of the object using the sensor and the physical force of the object, and comparing the actual images with the time series motion information, the physical property value of the object, and , Learning a correlation between the interaction forces of the robot arm.

The step of predicting the interaction force of the robot arm in the method of learning the force of the robot arm according to an embodiment of the present invention, extracts a region related to the motion of the robot from the real images using a CNN in which the real images are input. To do; Calculating class scores of the time-series motion information of the robot using a first fully-connected (FC) layer into which the time-series motion information of the robot is input; Learning the relationship between the region changing over time and the time series operation information by inputting the extracted region and the class scores of the time series operation information to the RNN; And calculating a class score of the learning result to a second FC layer into which the learning result of the RNN is input, and predicting the interaction force of the robot arm corresponding to the time series motion information and the actual image and the physical property value of the object. It can contain.

The step of predicting the interaction force of the robot arm of the robot arm force learning method according to an embodiment of the present invention is a virtual image representing a change in motion information over time based on the time series motion information in a graph form Generating and inputting the first convolutional layer of the CNN; Inputting the actual images accumulated over time into a second convolutional layer different from the first convolutional layer; Matching the virtual image output from the first convolutional layer and the actual image output from the second convolutional layer using a pooling layer; And classifying the matched information through a fully connected layer to predict the time series motion information and the interaction force of the robot arm corresponding to the real image and the physical properties of the object.

An apparatus for predicting a force of a robot arm according to an embodiment of the present invention includes an image acquiring unit for acquiring real images generated by continuously photographing an operation of a robot arm interacting with an object; A motion information acquisition unit that acquires time-series motion information related to a change in the motion of the robot arm over time; And a force predicting unit predicting an interaction force of the robot arm based on the actual images and the time-series motion information.

The force prediction unit of the robot arm force prediction apparatus according to an embodiment of the present invention is a database in which learning results of a correlation between the actual images, the time series motion information, the physical properties of the object, and the interaction force of the robot arm are stored. The actual images, the time series motion information, and the physical properties of the object may be applied to predict the interaction force of the robot arm.

Time-series motion information of the robot arm force prediction device according to an embodiment of the present invention is a change in power used by the motor of the robot arm to perform the operation of the robot arm that changes over time, the robot arm It may include at least one of the position change and the motion change of the robot arm.

An apparatus for predicting a force of a robotic arm according to an embodiment of the present invention includes a haptic sensing information predicting unit predicting haptic sensing information using the predicted interaction force between the object and the physical properties of the object; And a feedback unit providing feedback to the user of the robot arm according to the haptic sensing information.

The apparatus for learning the force of a robot arm according to an embodiment of the present invention includes a communicator for acquiring real images generated by photographing an operation of a robot arm related to an object and time series operation information related to the operation of the robot arm; A force predicting unit that inputs the real images and the time series motion information into a deep learning algorithm to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object; And comparing the predicted interaction force of the robot arm and the physical force of the object using the sensor and the physical force of the object, and comparing the actual images with the time series motion information, the physical property value of the object, and And, it may include a correlation learning unit for learning the correlation between the interaction force of the robot arm.

The force predicting unit of the force learning device of the robot arm according to an embodiment of the present invention extracts a region related to the robot's motion from the real images using the CNN to which the real images are input, and the time series motion information of the robot Calculates the class scores of the time-series motion information of the robot using the first FC layer to which is input, and inputs the extracted area and the class scores of the time-series motion information to the RNN to change the area and the time series that change over time. The relationship between the motion information and the interaction force of the robot arm corresponding to the time series motion information and the actual image by calculating the class scores of the learning result to the second FC layer into which the learning result of the RNN is input is calculated. The physical properties of the object can be predicted.

The force predicting unit of the force learning device of the robot arm according to an embodiment of the present invention generates a virtual image representing a change in motion information over time based on the time series motion information in a graph form, thereby generating the first convolution of the CNN A virtual image output from the first convolutional layer and the second inputted to a solution layer, the real images accumulated over time are input to a second convolutional layer different from the first convolutional layer. The actual images output from the convolution layer are matched using the integrated layer, and the matched information is classified through the FC layer, and the interaction force of the robot arm corresponding to the time series motion information and the actual image and the physical properties of the object Predictable.

According to an embodiment of the present invention, a correlation between a real image photographing a robot's motion and time series motion information, an object's physical properties, and an interaction force of a robot arm is learned, and a real image and a time series motion photographing the robot's motion By applying the information to the learning results, the interaction force of the robot arm can be predicted without a sensor for measuring the interaction force of the robot arm.

In addition, according to an embodiment of the present invention, by providing feedback to the user who controls the robot arm according to the predicted interaction force of the robot arm, the physical sense is reproduced to the user using only the camera without configuring a complex sensor network. I can do it.

In addition, according to an embodiment of the present invention, since the interaction force of the robot arm can be predicted using the actual image, when an error occurs in the sensor, the interaction force of the robot arm measured by the sensor and the actual image are used. By comparing the predicted interaction force of the robot arm, it is possible to determine whether the sensor is abnormal.

1 is a view showing an image-based force prediction system according to an embodiment of the present invention.
2 is a view showing the structure of a robotic force learning apparatus included in an image-based force prediction system according to an embodiment of the present invention.
3 is an example of a process in which a force learning device for a robot arm according to an embodiment of the present invention learns a correlation between an image, motion information, and interaction force of a robot arm.
4 is an example of a method for predicting an interaction force of a robot arm using CNN and RNN according to an embodiment of the present invention.
5 is an example of a method for predicting an interaction force of a robot arm using CNN according to an embodiment of the present invention.
6 is a view showing the structure of a robot arm force prediction device included in an image-based force prediction system according to an embodiment of the present invention.
7 is an example of a process for predicting an interaction force of a robot arm according to an embodiment of the present invention.
8 is another example of a process for predicting an interaction force of a robot arm according to an embodiment of the present invention.
9 is a flowchart illustrating a method for learning an interaction force of a robot arm according to an embodiment of the present invention.
10 is a flowchart illustrating a method for predicting an interaction force of a robot arm according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments, and the scope of the patent application right is not limited or limited by these embodiments. It should be understood that all modifications, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described on the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the embodiments, detailed descriptions thereof will be omitted.

The method for learning the interaction force of the robot arm according to an embodiment of the present invention may be performed by the force learning device of the robot arm included in the image-based force prediction system. In addition, the method for predicting the interaction force of the robot arm according to an embodiment of the present invention may be performed by the force prediction device of the robot arm included in the image-based force prediction system.

1 is a view showing an image-based force prediction system according to an embodiment of the present invention.

The image-based force prediction system may include a force learning device 110 of the robot arm, a database 120, and a force prediction device 130 of the robot arm, as shown in FIG. 1.

The force learning device 110 of the robot arm between the actual images generated by continuously photographing the motion of the robot arm related to the object, time-series motion information related to the motion of the robot arm, the physical properties of the object, and the interaction force of the robot arm Can learn correlation. And, the robot arm force learning device 110 may store the learned correlation in the database 120. At this time, the interaction force of the robot arm is the grip force of the robot arm holding the object, the contact force of the robot arm contacting the object, the force of the robot arm pulling the object (Pull force), the force of the robot arm pushing the object ( Push force), and at least one of the direction of the force of the robot arm holding or touching the object.

At this time, the physical property value of the object may include at least one of elasticity, rigidity, density, and specific gravity of the object. In addition, the time series motion information may include at least one of a change in power used by the motor of the robot arm, a change in position of the robot arm, and a change in motion of the robot arm to perform the operation of the robot arm that changes over time. have. Also, time-series motion information may be generated corresponding to each of the actual images. For example, when a real image is taken at an interval of 0.1 second, the robot arm can also generate motion information at a corresponding time at an interval of 0.1 second. Then, the robot arm force learning device 110 may match the motion information of the robot arm corresponding to each frame of the actual image generated at 0.1 second intervals.

The specific configuration and operation for the robot arm force learning device 110 to learn the correlation will be described in detail with reference to FIGS. 2 to 5 below.

The database 120 may store and manage correlations between actual images learned by the robot arm force learning apparatus 110 and time series motion information, object properties, and robot arm interaction forces.

The robot arm force predicting device 130 records at least one of real images generated by photographing the motion of the robot arm related to the object, time series motion information related to the motion of the robot arm, and physical properties of the object in the database 120. It can be applied to the loaded correlation to predict the interaction force of the robot arm.

The specific configuration and operation for the robot arm force prediction device 130 to predict the interaction force of the robot arm will be described in detail with reference to FIGS. 6 to 8 below.

The image-based force prediction system learns the correlation between the real image of the robot's motion and the time series motion information, the object's physical properties, and the interaction force of the robot's arm, and learns the real image and the time-series motion information of the robot's motion. By applying to the results, it is possible to predict the interaction force of the robot arm without a sensor for measuring the interaction force of the robot arm. In addition, the image-based force prediction system provides feedback to the user who controls the robot arm according to the predicted interaction force of the robot arm, thereby reproducing the physical sense to the user using only the camera without configuring a complex sensor network. .

In addition, since the image-based force prediction system can predict the interaction force of the robot arm using the actual image, when an error occurs in the sensor, the robot predicted by using the interaction force of the robot arm measured by the sensor and the actual image It is possible to determine whether the sensor is abnormal by comparing the interaction forces of the arms.

In addition, since the image-based force prediction system can predict the interaction force of the robot arm without an expensive and complex sensor network, it can be implemented at a lower cost than the conventional robot arm interaction force measurement system using a sensor. Therefore, it is possible to increase the prevalence of the system for measuring or predicting the interaction force of the robot arm.

2 is a view showing the structure of a robotic force learning apparatus included in an image-based force prediction system according to an embodiment of the present invention.

The force learning device 110 of the robot arm may include an image acquisition unit 210, a motion information acquisition unit 220, a force prediction unit 230, and a correlation learning unit 240 as illustrated in FIG. 1. have. In this case, the force prediction unit 230 and the correlation learning unit 240 may be respective modules for performing different processes or programs included in one process.

The image acquisition unit 210 may acquire an actual image generated by the camera 201 linked to the learning robot arm 202 photographing the operation of the robot arm 202 associated with the object. At this time, the image acquiring unit 210 may be a communicator for receiving an actual image wirelessly from the camera 201, or a port through which a wire for receiving the actual image from the camera 201 is coupled.

The motion information acquiring unit 220 may acquire time-series motion information related to the motion of the robot arm 202. At this time, the operation information acquisition unit 220 may be a communicator for receiving operation information wirelessly from the robot arm 202, or a port through which a wire for receiving operation information from the robot arm 202 is coupled.

In addition, in FIG. 2, although the image acquisition unit 210 and the operation information acquisition unit 220 are described in separate configurations, the force learning device 110 of the robot arm uses a single communicator to acquire actual image and operation information. It is also possible to perform all acquisitions.

The force predicting unit 230 may input actual images and time series motion information into a deep learning algorithm to predict the interaction force of the robot arm and the physical properties of the object corresponding to the time series motion information and the real image. At this time, the force prediction unit 230 may predict the interaction force of the robot arm using the Convolutional Neural Network (CNN) and the Recurrent Neural Network (RNN), or the interaction force of the robot arm using only the CNN.

A method of predicting the interaction force of the robot arm using CNN and RNN will be described in detail with reference to FIG. 4 below. In addition, a method of predicting the interaction force of the robot arm using only CNN will be described in detail with reference to FIG. 5 below.

The correlation learning unit 240 compares the interaction force and the object physical properties of the robot arm measured using the sensor with the physical force of the object and the interaction force of the robot arm predicted by the force prediction unit 230 and compares the actual images. You can learn the correlation between time-series motion information, physical properties of an object, and interaction force of a robot arm.

Specifically, in the correlation learning unit 240, the interaction force of the robot arm predicted by the force predicting unit 230 and the physical property value of the object are the same as the interaction force of the robot arm measured using a sensor and the physical property value of the object, , If there is a difference below the error range, it can be learned that there is a correlation between the actual images input by the force prediction unit 230 and time series motion information, the predicted object properties, and the interaction force of the robot arm. . In addition, the correlation learning unit 240 determines the interaction force of the robot arm predicted by the force predicting unit 230 and the physical force of the object and the error range of the interaction force of the robot arm measured using a sensor and the physical properties of the object. When there is an excess difference, it may be determined that there is no correlation between the actual images input by the force prediction unit 230 and time series motion information, the predicted object properties, and the interaction force of the robot arm. At this time, the force predicting unit 230 repeatedly performs a process of predicting the physical force of the object correlated with the actual images and time series motion information and the interaction force of the robot arm until the robot arm interaction force is predicted. can do.

3 is an example of a process in which a force learning device for a robot arm according to an embodiment of the present invention learns a correlation between an image, motion information, and interaction force of a robot arm.

The robot arm 320 connected to the force learning device 110 of the robot arm, or the surgical robot forceps 240 may include a sensor for measuring the interaction force of the robot arm. For example, the robot arm 320, or the surgical robot forceps 340 may include at least one of a force sensor, a torque sensor, and a pressure sensor.

The camera 310 corresponding to the robot arm 320 may transmit the actual images 311 photographing the operation of the robot arm 320 at regular time intervals to the force learning device 110 of the robot arm. In addition, the robot arm 320 may transmit time series motion information 321 corresponding to each of the frames of the actual image 311 to the force learning device 110 of the robot arm. At this time, the robot arm 320 may transmit the interaction force of the robot arm measured using a sensor to the force learning device 110 of the robot arm along with the time series motion information 321.

Then, the robot arm force learning device 110 contacts the robot arm 320 using the actual image 311 and time series motion information 321, or the interaction force of the robot arm held by the robot arm 320. Can predict. In addition, the force learning device 110 of the robot arm may predict the physical properties of the object according to the position and location of the object using the actual image 311. Next, the force learning device 110 of the robot arm compares the predicted interaction force of the robot arm with the interaction force of the robot arm received from the robot arm 320, and displays the actual images 311 and time series motion information ( 321), the correlation between the physical properties of the object and the interaction force of the robot arm can be learned.

In addition, the camera 330 corresponding to the surgical robot forceps 340 may transmit the actual images 331 of the operation of the surgical robot forceps 340 at regular time intervals to the force learning device 110 of the robot arm. . In addition, the surgical robot forceps 340 may transmit time-series motion information 341 corresponding to each of the frames of the actual image 313 to the force learning device 110 of the robot arm. At this time, the surgical robot forceps 340 may measure the interaction force of another robot arm contacting the object held by the surgical robot forceps 340 using a sensor. Then, the surgical robot forceps 340 may transmit the measured interaction force of the robot arm to the force learning device 110 of the robot arm along with the time series motion information 341.

In addition, the force learning device 110 of the robot arm uses the physical image 331 and the time series motion information 341 to interact with the physical properties of other objects in contact with the object held by the surgical robot forceps 340 and the interaction between the other robot arms. The force of action can be predicted. In addition, the force learning device 110 of the robot arm may predict physical properties of other objects according to positions and positions of other objects using the actual image 331. Next, the force learning device 110 of the robot arm compares the predicted interaction force of the robot arm with the interaction force of the robot arm received from the surgical robot forceps 340 and displays actual images 331 and time series motion information. (321), it is possible to learn the correlation between the physical properties of other objects and the interaction force of the robot arm.

At this time, when the shooting interval between the camera 310 and the camera 330 is 0.1 seconds, the robot arm 320 and the surgical robot forceps 340 also generate motion information at a corresponding time at 0.1 second intervals, respectively, to generate the force of the robot arm. It can be transmitted to the learning device 110. In addition, the robot arm force learning apparatus 110 may match the motion information of the robot arm corresponding to each of the actual images generated at 0.1 second intervals.

4 is an example of a method for predicting an interaction force of a robot arm using CNN and RNN according to an embodiment of the present invention.

First, the force predicting unit 230 extracts a region related to the motion of the robot from the actual images by using the CNN 411 into which the actual images obtained from each of at least one camera, such as camera 1 and camera 2, are input. You can. For example, the force predicting unit 230 extracts an area including the robot arm and the object from the actual image generated by photographing the robot arm as an area related to the robot's motion, thereby relating to an image to be processed by the RNN 420. The size of the information can be minimized.

In addition, when one robot arm is photographed with two cameras, the force predicting unit 230 displays a region related to the motion of the robot extracted from the CNN 411 corresponding to each of the cameras as shown in FIG. 4. It can be integrated in the FC layer 412 and delivered to the RNN 420. In this case, the fourth FC layer 412 may match and output regions related to the operation of the robot extracted from different images.

In addition, the force predicting unit 230 may process the time-series motion information of the robot to the first fully-connected (FC) layer 413 into which the time-series motion information of the robot is input, and transmit it to the RNN 420. At this time, the first FC layer 413 may be composed of a plurality of FC layers as shown in FIG. 4. In addition, the first FC layer 413 may calculate and output time-series motion information of the robot.

In addition, the force prediction unit 230 may generate an input network composed of the CNN 411 and the first FC layer 413 for each real image. For example, when there are N actual images, the force prediction unit 230 may generate the input networks 1 410 to N as shown in FIG. 4. At this time, the input network 2 to the input network N are only between the input real image and the motion information corresponding to the real image, and the structure for processing the real image and the motion information includes CNN 411 of the input network 1 410 and It may be the same as the first FC layer 413.

Next, the force predicting unit 230 may use the RNN 420 to learn the relationship between the region that changes over time and time series motion information. For example, as shown in FIG. 4, the RNN 420 includes an area related to the motion of a robot output from a plurality of input networks over time, and processed time series motion information composed of multiple layers of LSTM ( long short term memory networks) to learn the relationship between time-varying motion information and areas that change over time. Although the LSTM is used in FIG. 4, the force predicting unit 230 may also learn the relationship between the time-varying motion information and a region that changes over time by using another network corresponding to the RNN in addition to the LSTM.

Lastly, the force predicting unit 230 processes the learning result with the second FC layer 430 into which the learning result of the RNN is input, and the interaction force (Task1) and the object of the robot arm corresponding to the time series motion information and the actual image. The property value (Task2) can be predicted. At this time, the second FC layer 430 may calculate the class scores of each of the relationships between the region and the time-series operation information that change over time.

5 is an example of a method for predicting an interaction force of a robot arm using CNN according to an embodiment of the present invention.

First, the force prediction unit 230 generates a virtual image 521 representing a change in motion information over time based on time series motion information in a graph form and inputs it to the first convolutional layer 520 of the CNN. You can. For example, the virtual image 521 may be generated by converting a covariance matrix between vectorized time series motion information 522 and 523 into an image. In this case, the first convolution layer 520 may analyze the correlation between time-series operation information using the virtual image 521.

Next, the force prediction unit 230 may input the actual images 511 accumulated over time into a second convolutional layer 510 different from the first convolutional layer.

Next, the force prediction unit 230 uses a pooling layer 530 of the virtual images output from the first convolution layer 520 and the actual images output from the second convolution layer 510. Can be matched.

Lastly, the force prediction unit 230 classifies the matched information through the fully connected layer (FC layer) 540 to predict the interaction force of the robot arm and the physical properties of the object corresponding to the time series motion information and the actual image. have.

6 is a view showing the structure of a robot arm force prediction device included in an image-based force prediction system according to an embodiment of the present invention.

As shown in FIG. 6, the force prediction device 130 of the robot arm includes an image acquisition unit 610, a motion information acquisition unit 620, a force prediction unit 630, a haptic sensing information prediction unit 640, and feedback It may include a portion 640. In this case, the force prediction unit 630 and the haptic sensing information prediction unit 640 may be respective modules for performing different processes or programs included in one process.

The image acquisition unit 610 may acquire an actual image generated by the camera 601 linked to the robot arm 602 to predict the force by photographing the operation of the robot arm 602 associated with the object. In this case, the image acquiring unit 610 may be a communicator for receiving an actual image wirelessly from the camera 601, or a port through which a wire for receiving the actual image from the camera 601 is coupled. At this time, the actual image may include an object deformed by the operation of the robot arm 602. Specifically, the continuous real image may be included in a process in which an object is deformed by the operation of the robot arm 602.

The motion information acquisition unit 620 may acquire time series motion information related to the motion of the robot arm 602. At this time, the operation information acquisition unit 620 may be a communicator that receives operation information wirelessly from the robot arm 602 or a port through which a wire for receiving operation information from the robot arm 602 is coupled.

In addition, although the image acquisition unit 610 and the operation information acquisition unit 620 are described in FIG. 6 as separate configurations, the force prediction device 130 of the robot arm uses a single communicator to acquire actual image and operation information. It is also possible to perform all acquisitions.

The force predicting unit 630 may predict the interaction force of the robot arm based on real images and time series motion information. At this time, the force prediction unit 630 of the actual images, the time series motion information, the physical properties of the object and the learning result of the correlation between the interaction force of the robot arm is stored in the database 120 of the actual images, time series motion information and objects The interaction force of the robot arm can be predicted by applying material properties.

The haptic sensing information prediction unit 640 may predict haptic sensing information using the material property of the object and the predicted interaction force of the robot arm.

The feedback unit 640 may transmit a feedback signal to the robot arm controller 603 according to the predicted haptic sensing information to provide feedback to the user of the robot arm. At this time, the feedback unit 640 may be a communicator that transmits a feedback signal to the robot arm controller 603 wirelessly, or a port to which a wire for transmitting a feedback signal to the robot arm controller 603 is coupled.

The robot arm force prediction apparatus 130 stores actual images, time series motion information, and objects in a database 120 in which learning results of correlations between real images, time series motion information, object properties, and robot arm interaction forces are stored. By applying the material properties of the robot arm, the interaction force of the robot arm can be predicted without a sensor for measuring the interaction force of the robot arm.

7 is an example of a process for predicting an interaction force of a robot arm according to an embodiment of the present invention.

The force predicting device 130 for the robot arm may acquire actual images 710 generated by photographing the operation of the robot arm 602 from a camera linked to the robot arm 602. At this time, the continuous real images 710 are as shown in FIG. 7, from the image 711 where the robot arm 602 contacts the object to the image 714 that catches the object that the robot arm 602 contacts. The robot arm 602 may be an image sequentially photographing a process of interacting with an object. Also, the robot arm 602 may operate according to a control signal 706 transmitted from the robot arm controller 603 to the robot arm 602.

Then, the robot arm force prediction device 130 may obtain time-series motion information 720 related to the motion of the robot arm 602 from the robot arm 602.

Next, the robot arm force prediction device 130 may preprocess the actual images 710 and the time series motion information 720 using the CNN and the FC layer 701. In this case, the robot arm force prediction apparatus 130 may extract a region related to the robot's motion from the actual images 710 using the CNN and transmit the extracted region to the RNN 702. In addition, the robot arm force prediction device 130 may process the time series motion information of the robot to the FC layer and transmit it to the RNN 702.

Next, the robot arm force prediction device 130 may use the RNN 702 to check the relationship between the region that changes over time and time-series motion information. In addition, the robot arm force prediction apparatus 130 may search the interaction force of the robot arm and the physical properties of the object corresponding to the relationship between the region identified in the database 120 and time-series motion information.

Next, the robot arm force prediction apparatus 130 may output the retrieved interaction force of the robot arm and the physical properties of the object as a force prediction result 703.

In addition, the robot arm force prediction apparatus 130 may predict haptic sensing information to be fed back to the user according to the force prediction result 703. Then, the robot arm force prediction apparatus 130 may transmit a feedback signal 730 including haptic sensing information to the robot arm controller 603. At this time, the robot arm controller 603 may provide feedback to the user who controls the robot arm 602 according to the received feedback signal 730.

8 is another example of a process for predicting an interaction force of a robot arm according to an embodiment of the present invention.

FIG. 8 shows that when the robot arm 602 interacts with a breakable object (surgical tool, patient's organ), such as a surgical robot arm, the robot arm force predicting device 130 warns the risk of object damage. Yes.

The apparatus for predicting the force of the robot arm 130 may acquire actual images 810 generated by photographing an operation of the robot arm 602 from a camera linked to the robot arm 602. At this time, the actual images 810 may include the first image 811 to the current image 814 in the process of the object being deformed by the operation of the robot arm 602. Also, the robot arm 602 may operate according to the control signal 800 transmitted from the robot arm controller 603 to the robot arm 602.

Then, the robot arm force prediction device 130 may obtain time-series motion information 820 related to the motion of the robot arm 602 from the robot arm 602.

Next, the robot arm force prediction device 130 may preprocess the actual images 810 and the time series motion information 820 using the CNN and the FC layer 801. At this time, the robot arm force prediction device 130 may extract an area related to the robot's motion from the actual images 810 using the CNN and transmit it to the RNN 802. In addition, the robot arm force prediction device 130 may process the time series motion information of the robot to the FC layer and transmit it to the RNN 802.

Next, the robot arm force prediction device 130 may use the RNN 802 to check the relationship between the region that changes over time and time series motion information. In addition, the robot arm force prediction apparatus 130 may search the interaction force of the robot arm and the physical properties of the object corresponding to the relationship between the region identified in the database 120 and time-series motion information. Next, the robot arm force prediction apparatus 130 may output the retrieved interaction force of the robot arm and the physical properties of the object as a force prediction result 803.

At this time, the force predicting device 130 of the robot arm may determine the possibility of damage to the object by comparing the interaction force of the robot arm included in the force prediction result 803 and the physical properties of the object. In addition, when the probability of damage to the object is greater than or equal to a threshold, the force predicting device 130 of the robot arm may transmit a signal 840 warning the possibility of the object to be broken to the robot arm controller 603. At this time, the robot arm controller 603 may be set to operate the warning sound or warning lamp according to the received signal 840 or not to transmit the control signal 800 in the direction in which the object is damaged to the robot arm 602. have.

9 is a flowchart illustrating a method for learning an interaction force of a robot arm according to an embodiment of the present invention.

In operation 910, the image acquisition unit 210 may acquire an actual image generated by photographing an operation of the robot arm 202 associated with the object by the camera 201 linked to the learning robot arm 202. At this time, the motion information acquisition unit 220 may acquire time-series motion information related to the motion of the robot arm 202. In addition, the force learning device 110 of the robot arm may acquire both real image and time series motion information using a single communicator.

In operation 920, the force predicting unit 230 may input actual images and time series motion information into a deep learning algorithm to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object. In this case, the force predicting unit 230 may predict the interaction force of the robot arm using CNN and RNN, or the interaction force of the robot arm using only CNN.

In step 930, the correlation learning unit 240 measures the interaction force and object physical properties of the robot arm measured using the sensor, and the force predictor 230 predicts the interaction force of the robot arm and the physical property value of the object. By comparison, it is possible to learn a correlation between real images and time series motion information, physical properties of an object, and interaction force of a robot arm.

10 is a flowchart illustrating a method for predicting an interaction force of a robot arm according to an embodiment of the present invention.

In step 1010, the image acquisition unit 610 may acquire an actual image generated by photographing the operation of the robot arm 602 associated with the object by the camera 601 linked to the robot arm 602 to predict the force. . At this time, the motion information acquisition unit 620 may acquire time-series motion information related to the motion of the robot arm 602. Also, the force predicting device 130 of the robot arm may acquire both real image and time series motion information using a single communicator.

In step 1020, the force prediction unit 630 may predict the interaction force of the robot arm based on the actual images and time series motion information. At this time, the force predicting unit 630 of the actual images, time series motion information, the physical properties of the object and the learning result of the correlation between the interaction force of the robot arm is stored in the database 120 of the actual images, time series motion information and objects The interaction force of the robot arm can be predicted by applying material properties.

In step 1030, the haptic sensing information prediction unit 640 may predict the haptic sensing information using the physical properties of the object and the interaction force of the robot arm predicted in step 1020.

In step 1040, the feedback unit 640 may transmit a feedback signal to the robot arm controller 603 according to the haptic sensing information predicted in step 1030 to provide feedback to the user of the robot arm.

The present invention learns the correlation between the real image of the robot's motion and time series motion information, the object's physical properties and the interaction force of the robot's arm, and applies the real image and the time series motion information of the robot's motion to the learning results. By doing so, it is possible to predict the interaction force of the robot arm without a sensor for measuring the interaction force of the robot arm.

In addition, the present invention provides feedback to the user who controls the robot arm according to the predicted interaction force of the robot arm, thereby reproducing the physical sense to the user using only the camera without configuring a complex sensor network.

In addition, since the present invention can predict the interaction force of the robot arm using the actual image, when an error occurs in the sensor, the interaction force of the robot arm measured by the sensor and the robot arm predicted using the actual image By comparing the operating force, it is possible to determine whether the sensor is abnormal.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and constructed for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described by a limited drawing, a person skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

110: robot arm force learning device
120: database
130: robot arm force prediction device

Claims (20)

Obtaining real images generated by continuously photographing interactions between objects over time;
Obtaining time-series motion information of a robot arm related to a change in position of the objects or deformation of the objects over time; And
Predicting an interaction force between the objects based on a change in the position of the objects or a deformation of the objects included in the real image.
Including,
The predicting step,
The actual images, the time series motion information, the physical properties of the objects and the learning result of the correlation between the interaction force of the robot arm are stored in the database, and the physical images, the time series motion information, and the physical properties of the object are applied to the database. Interactive force prediction method for predicting the interaction force of a robot arm. delete delete delete According to claim 1,
The correlation is
The time series motion information and the actual images are input to a deep learning algorithm of a neural network structure, and the predicted interaction force of the robot arm and the physical force value of the object and the physical force value of the robot arm measured using the sensor and the physical property value of the object A method of predicting interaction forces that is learned by comparison. The method of claim 5,
The deep learning algorithm,
Extracting a region related to the motion of the robot from the real images by using a convolutional neural network (CNN) to which the real images are input;
Calculating class scores of the time-series motion information of the robot using a first fully-connected (FC) layer into which the time-series motion information of the robot is input;
Learning the relationship between the region and the time-series operation information that changes over time by inputting the extracted region and the class scores of the time-series operation information into a recurrent neural network (RNN); And
The process of predicting the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object by calculating class scores of the learning result to the second FC layer where the learning result of the RNN is input.
An interactive force prediction method for learning a correlation between the actual images and time series motion information of the robot and the interaction force of the robot arm by performing. The method of claim 5,
The deep learning algorithm,
Generating a virtual image representing a change of motion information over time based on the time-series motion information in a graph form and inputting it to a first convolutional layer of a convolutional neural network (CNN);
Inputting the actual images accumulated over time into a second convolutional layer different from the first convolutional layer;
Matching a virtual image output from the first convolutional layer and a real image output from the second convolutional layer using a pooling layer; And
The process of predicting the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object by classifying the matched information through a fully connected layer
An interactive force prediction method for learning a correlation between the actual images and time series motion information of the robot and the interaction force of the robot arm by performing. According to claim 1,
The time series operation information,
Interaction force prediction including at least one of a change in power used by the motor of the robot arm, a change in position of the robot arm, and a change in motion of the robot arm to perform the operation of the robot arm that changes over time. Way. According to claim 1,
Predicting haptic sensing information using a material property of the object and a predicted interaction force of the robot arm; And
And providing feedback to the user of the robot arm according to the haptic sensing information. Obtaining real images generated by photographing the motion of the robot arm associated with the object and time series motion information related to the motion of the robot arm;
Predicting the interaction force of the robot arm corresponding to the time series motion information and the real image and physical properties of the object by inputting the real images and the time series motion information into a deep learning algorithm;
Comparing the predicted interaction force of the robot arm and the physical force of the object with the sensor's interaction force and the physical force of the object, the actual images and the time series motion information, the physical property value of the object, and Learning a correlation between the interaction forces of the robot arm; And
Storing the correlation in a database
Robot arm force learning method comprising a. The method of claim 10,
The step of predicting the interaction force of the robot arm,
Extracting a region related to the motion of the robot from the real images using a CNN to which the real images are input;
Calculating class scores of the time-series motion information of the robot using a first fully-connected (FC) layer into which the time-series motion information of the robot is input;
Learning the relationship between the region changing over time and the time series operation information by inputting the extracted region and the class scores of the time series operation information to the RNN; And
Predicting the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical property value of the object by calculating class scores of the learning result to the second FC layer into which the learning result of the RNN is input.
Robot arm force learning method comprising a. The method of claim 10,
The step of predicting the interaction force of the robot arm,
Generating a virtual image representing a change in motion information over time based on the time series motion information in a graph form and inputting it to a first convolutional layer of a CNN;
Inputting the actual images accumulated over time into a second convolutional layer different from the first convolutional layer;
Matching the virtual image output from the first convolutional layer and the actual image output from the second convolutional layer using a pooling layer; And
Classifying the matched information through a fully connected layer to predict the time series motion information and the interaction force of the robot arm corresponding to the real image and the physical property value of the object
Robot arm force learning method comprising a. An image acquiring unit for acquiring an actual image generated by continuously photographing interactions between objects over time;
A motion information acquiring unit for acquiring time-series motion information of a robot arm related to a change in position of the objects or deformation of the objects over time; And
A force prediction unit predicting an interaction force between the objects based on a change in the position of the objects or a deformation of the objects included in the real image
Including,
The force prediction unit,
The actual images, the time series motion information, the physical properties of the object and the learning result of the correlation between the interaction force of the robot arm are stored in the database, and the physical images, the time series motion information, and the physical properties of the object are applied to the database. An interaction force prediction device that predicts the interaction force of a robot arm. delete delete The method of claim 13,
The time series operation information,
Interaction force prediction including at least one of a change in power used by the motor of the robot arm, a change in position of the robot arm, and a change in motion of the robot arm to perform the operation of the robot arm that changes over time. Device. The method of claim 13,
A haptic sensing information prediction unit predicting haptic sensing information using the predicted interaction force of the robot arm with the physical properties of the object; And
A feedback unit that provides feedback to the user of the robot arm according to the haptic sensing information
Interaction force prediction device further comprising. A communicator for acquiring real-time images generated by photographing the motion of the robot arm associated with the object and time-series motion information related to the motion of the robot arm;
A force predicting unit that inputs the real images and the time series motion information into a deep learning algorithm to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object; And
Comparing the predicted interaction force of the robot arm and the physical force of the object with the sensor's interaction force and the physical force of the object, the actual images and the time series motion information, the physical property value of the object, and Correlation learning unit for learning the correlation between the interaction forces of the robot arm
Robot arm force learning device comprising a. The method of claim 18,
The force prediction unit,
A class score of time-series motion information of the robot is extracted by using a first FC layer to which the time-series motion information of the robot is input, by extracting a region related to the robot motion from the real images using a CNN where the real images are input. Second, learning the relationship between the region changing over time and the time series operation information by inputting the extracted region and the class scores of the time series operation information to the RNN, and inputting the learning result of the RNN A force learning device for a robot arm that calculates class scores of the learning result by the FC layer and predicts the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object. The method of claim 18,
The force prediction unit,
Based on the time-series motion information, a virtual image representing a change in motion information over time in a graph form is generated and input to a first convolutional layer of CNN, and the accumulated real images over time are removed from the virtual image. Inputted to a second convolutional layer different from one convolutional layer, the virtual image output from the first convolutional layer and the actual images output from the second convolutional layer are matched and matched using an integrated layer. A robot arm force learning device that classifies information through an FC layer to predict the interaction force of the robot arm corresponding to the time series motion information and the actual image, and the physical property value of the object.

Images