CN111145257B - Article grabbing method and system and article grabbing robot - Google Patents
Article grabbing method and system and article grabbing robot Download PDFInfo
- Publication number
- CN111145257B CN111145257B CN201911383453.6A CN201911383453A CN111145257B CN 111145257 B CN111145257 B CN 111145257B CN 201911383453 A CN201911383453 A CN 201911383453A CN 111145257 B CN111145257 B CN 111145257B
- Authority
- CN
- China
- Prior art keywords
- article
- identification
- information
- target
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims description 26
- 230000011218 segmentation Effects 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 7
- 238000003709 image segmentation Methods 0.000 claims description 7
- 238000013135 deep learning Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 229940112822 chewing gum Drugs 0.000 description 4
- 235000015218 chewing gum Nutrition 0.000 description 4
- 230000033001 locomotion Effects 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 238000012015 optical character recognition Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 238000010146 3D printing Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000010147 laser engraving Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/06—Recognition of objects for industrial automation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Robotics (AREA)
- Manipulator (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an article grabbing method, which comprises the following steps: learning and training the articles to establish an identification model of the articles; receiving a user instruction and extracting identification information of a target object in the user instruction; comparing the identification information with the identification model to obtain a target object, and obtaining and outputting the position information of the target object; and grabbing the target object according to the position information. The invention also provides an article grabbing system and an article grabbing robot capable of realizing the method. The invention can improve the compatibility and accuracy of the grabbing of the articles.
Description
Technical Field
The invention relates to the technical field of industrial robots, in particular to an article grabbing method and system and an article grabbing robot.
Background
With the continuous development of robotics, robots have been widely used in various fields such as welding, assembly, handling, and painting. Robots are machine devices that automatically perform work, either by running pre-programmed programs to accept human command control or by performing actions according to guidelines established by artificial intelligence techniques.
During application, robots often need to perform object gripping tasks. However, the gripping of the articles by the existing robots is mostly realized by program control, for example, manually setting a program control mechanical arm to a specified position to grip the specified articles. In each grabbing process, the articles and the placing positions of the articles are fixed, so that the compatibility and the accuracy are not high.
Disclosure of Invention
The embodiment of the invention provides an article grabbing method, an article grabbing system and an article grabbing robot, which can solve the problems of low compatibility and accuracy in the prior art that the robot grabs an article.
In order to solve the above problems, the present invention provides an article gripping method, comprising:
learning and training the articles to establish an identification model of the articles;
receiving a user instruction and extracting identification information of a target object in the user instruction;
comparing the identification information with the identification model to obtain a target object, and obtaining and outputting the position information of the target object;
and grabbing the target object according to the position information.
Preferably, the learning and training the article to build the identification model of the article includes:
acquiring a plurality of images of an article;
performing feature analysis on at least one image of the article to obtain feature information of the article and setting an identification tag for the article;
and associating the characteristic information of the article with the identification tag to establish an identification model of the article.
Preferably, the comparing the identification information with the identification model to obtain the target object includes:
carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object;
extracting the characteristics of the image corresponding to each article to obtain the characteristic information of each article;
comparing the characteristic information of each article with the characteristic information in the identification model;
and when the comparison is successful, determining the object as the target object.
Preferably, the comparing the identification information with the identification model to obtain the target object includes:
matching the identification label of the target object with the identification label of each object in the identification model, and determining the matched object when the matching is successful;
carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object;
extracting the characteristics of the image corresponding to each article to obtain the characteristic information of each article;
and comparing the characteristic information of each article with the comparison characteristic information of the matched article, and determining the matched article as a target article when the comparison is successful.
The invention also provides an article grabbing system, which comprises the following key steps:
the model processing module is used for learning and training the articles so as to establish an identification model of the articles;
the user instruction processing module is used for receiving a user instruction and extracting identification information of a target object in the user instruction;
the article identification module is used for comparing the identification information with the identification model to obtain a target article, acquiring and outputting the position information of the target article;
and the grabbing module is used for grabbing the target object according to the position information.
Preferably, the model processing module includes:
an image acquisition unit configured to acquire a plurality of images of an article;
the image processing unit is used for carrying out feature analysis on at least one image of the article, obtaining feature information of the article and setting an identification tag for the article;
and the model building unit is used for associating the characteristic information of the article with the identification tag to build an identification model of the article.
Preferably, the article identification module includes:
the image segmentation unit is used for carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object;
the feature extraction unit is used for extracting features of the images corresponding to each article to obtain feature information of each article;
the feature comparison unit is used for comparing the feature information of each article with the feature information in the identification model;
and the target article confirming unit is used for determining the article as a target article when the comparison is successful.
Preferably, the article identification module includes:
the matching unit is used for matching the identification tag of the target object with the identification tag of each object in the identification model, and determining the matched object when the matching is successful;
the image segmentation unit is used for carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object;
the feature extraction unit is used for extracting features of the images corresponding to each article to obtain feature information of each article;
and the comparison unit is used for comparing the characteristic information of each article with the comparison characteristic information of the matched article, and determining the matched article as a target article when the comparison is successful.
The present invention also provides an article gripping robot comprising:
the tail end of the mechanical arm is provided with a grabbing or sucking device;
the image capturing device is connected with the mechanical arm and used for acquiring an object image;
the user instruction processing device is used for receiving a user instruction and extracting identification information of a target object in the user instruction;
and the article identification module is used for comparing the identification information with the identification model to obtain a target article, acquiring the position information of the target article and outputting the position information.
Preferably, the article identification module includes:
the image segmentation unit is used for carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object;
the feature extraction unit is used for extracting features of the images corresponding to each article to obtain feature information of each article;
the feature comparison unit is used for comparing the feature information of each article with the feature information in the identification model;
and the target article confirming unit is used for determining the article as a target article when the comparison is successful.
In the embodiment of the invention, the article identification model is trained in advance by associating the article characteristics with the association relation of the identification labels. When the object grabbing is carried out, the robot can divide and extract an object in a large range according to the label information in the user instruction and then compare the object with the identification model, so that the target object is determined. Compared with the prior art, the method and the device for grasping the object can grasp the appointed object through the user instruction, greatly improve the intelligent degree of the existing robot, be applicable to industrial production or daily life, expand the application range of the robot for grasping the object, improve compatibility, and improve the accuracy of the robot for grasping the object due to grasping according to the comparison result.
Drawings
Fig. 1 is a schematic flow chart of an article grabbing method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of step 101 in an article grabbing method according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of step 103 in an article grabbing method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a frame structure of an article gripping system according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a model processing module in an article grabbing system according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an article identification module in an article grabbing system according to an embodiment of the present invention;
FIG. 7 is a schematic view of another structure of an article identification module in an article grabbing system according to an embodiment of the present invention;
fig. 8 is a schematic view of a frame structure of an article gripping robot according to an embodiment of the present invention;
FIG. 9 is a schematic hardware configuration of an article gripping system according to an embodiment of the present invention;
FIG. 10 is a schematic view of an installation of an article gripping robot in an article gripping system gripping an article;
fig. 11 is a control flow chart of a computer controlled article gripping robot gripping a target article.
Detailed Description
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.
Embodiments of the present invention provide an article grabbing solution, and a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
First, an article gripping method provided by the embodiment of the present invention will be described. Referring to fig. 1, fig. 1 is a schematic flow chart of an article grabbing method provided by an embodiment of the present invention, where the article grabbing method of the present embodiment is applied to a robot, as shown in fig. 1, and includes the following steps:
step 101, learning and training an article to establish an identification model of the article; specifically, in an embodiment of the present invention, an image of an object to be grabbed may be acquired by a camera to learn and train to establish an object identification model. The camera can be connected with a robot or a computer, and can shoot at least one image for each article according to the concentration of each article to be grabbed in the learning and training stage. For example, nine images may be taken for each item and the images of each item may be sent to a robot or computer, which builds an item identification model based on the characteristics and names of the items. If the training process is processed by the robot, the camera sends the image of each object to the robot, and if the training process is processed by the computer, the camera sends the image of each object to the computer, and correspondingly, the robot or the computer receives the image sent by the camera. Of course, the learning and training process may also be processed by a camera, and the camera sends the recognition model obtained by training to a computer or a robot accordingly.
102, receiving a user instruction and extracting identification information of a target object in the user instruction; in the embodiment of the invention, the instruction can be acquired and converted through a computer, a robot or a communication interface connected with the robot so as to extract the identification information of the target object. The identification information may include: item name, quantity, mode of operation, etc. In one embodiment, the voice input device of the computer can receive the instruction input by the user, and after conversion, the article identification information is output to the camera so as to start the camera to work. In another embodiment, a microphone may be provided on the robot to acquire the instruction and process the instruction by providing an instruction processing device on the robot control board. In still another embodiment, a wireless communication module may be provided on the robot, and voice or text instructions may be input through a mobile phone APP or the like and transmitted to the robot. It should be noted that, in an alternative embodiment, the robot may include a microphone unit, and the robot receives the voice command based on the microphone unit, and in a third alternative embodiment and a fourth alternative embodiment, the computer may include a microphone unit, and the computer receives the voice command based on the microphone unit. The user instructions may also be text information, and correspondingly, in an application scenario, the robot may include an input interface, the robot receiving text information based on the input interface, and in a third application scenario, the computer may include an input interface, the computer receiving text information based on the input interface.
Step 103, comparing the identification information with the identification model to obtain a target object, and obtaining and outputting the position information of the target object; in an embodiment, what the target object is may be determined according to the image of the object, that is, the feature information of the object may be compared with the feature information of the object identification model. And under the condition of successful comparison, the target object can be identified. For example, the user inputs 'chewing gum' through voice, and the computer performs semantic conversion and outputs the semantic conversion to the camera. The camera starts working, photographs in the view finding range, and outputs the photographed photographs to the computer. The computer compares the input photos with the database, and as the chewing gum identification model is stored in advance, one of the articles can be identified as chewing gum when the comparison is successful. In another alternative embodiment, the identification may be performed directly according to the identification tag of the target object, so as to determine which object in the set of objects to be grabbed is the target object, for example, for the target object with a distinct identification tag (such as an object name), the identification tag in the object image may be directly identified by the computer, so as to determine which object is the target object. After the target object is determined, the camera outputs the coordinate information of the object to the computer or the robot.
And 104, grabbing the target object according to the position information. As before, if the camera outputs the coordinate information of the object to the computer, the computer converts the coordinate information into three-dimensional coordinates adapted to the robot and transmits to the robot, and then the robot controls the object gripping mechanism to reach the coordinate position to grip the chewing gum. Of course, the camera can also directly output the object coordinate information to the robot, and the robot directly executes corresponding grabbing actions.
In the embodiment of the invention, the related word key in the user instruction is extracted through the identification of the user instruction, and the capturing of the man-machine interaction is completed in combination with the deep learning, so that the man-machine interaction is more convenient and faster by adding the user instruction, and the requirements of various scenes of the user are met. Compared with the prior art, the embodiment of the invention can realize intelligent grabbing of the appointed articles, improve the accuracy of grabbing the articles and complete the intelligent and personalized grabbing function.
The following describes the method for capturing objects according to the embodiment of the present invention in detail based on a specific application scenario.
Referring to fig. 2, in a preferred embodiment of the present invention, the step 101 may include:
step 1011, acquiring a plurality of images of the object; as before, multiple images of an item may be acquired by a camera. For example, photographing at different angles is performed on each article through a camera, nine pictures are photographed on each article, labels of each article are given in sequence, picture training is performed through deep learning mobilent, and after training is completed, a model can find out corresponding articles according to the labels.
Step 1012, performing feature analysis on at least one image of the article to obtain feature information of the article and setting an identification tag for the article; adding an identification tag to each article, and performing classification training on the image of each article, for example, the classification training on the small sample image of the article can be realized through the mobility of deep learning to obtain an identification model, and specifically, the image of each article is subjected to feature analysis to obtain the comparison feature information of each article; and for each article, comparing the characteristic information of the associated article with the identification label of the article to obtain an identification model.
In step 1013, the feature information of the article is associated with the identification tag, and an identification model of the article is established. It should be noted that, in order to identify a target object from objects in the object set to be grasped by the recognition model, the object set to be grasped for training the recognition model must include the object set to be grasped, so that the recognition success rate of the recognition model can be improved.
In step 1013, in an alternative implementation manner, the method for gripping an object of the present embodiment is applied to a robot, where the robot may obtain an identification tag of a target object stored locally, where the identification tag of the target object is embedded in advance in a memory of the robot, and the purpose of the method is to specify the robot to grip the target object corresponding to the identification tag.
In another optional implementation manner, the method for grabbing an object in this embodiment is applied to a robot, where the robot may include a man-machine interaction interface, the man-machine interaction interface may receive an instruction input by a user, and after the robot parses the instruction, obtain an identification tag of a target object, so as to designate the robot to grab the target object corresponding to the identification tag.
In a third alternative embodiment, the method for grabbing an object in this embodiment is applied to a robot, where the robot integrates an interface that communicates with a computer, and the computer may receive an instruction input by a user and parse the instruction to obtain an identification tag of a target object, and send the identification tag of the target object to the robot through a communication interface connected to the robot, so as to designate the robot to grab the target object corresponding to the identification tag.
In a fourth alternative embodiment, the method for capturing an object of the present embodiment is applied to a computer, and the computer may receive an instruction input by a user other than a user instruction, and parse the instruction to obtain an identification tag of the target object.
Referring to fig. 3, in a preferred embodiment, the step 103 may include:
step 1031, performing region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object; in the embodiment of the invention, the camera can be used for photographing in the view finding range, and the photographed photo is output to the computer. Specifically, firstly, an image of an object set to be grabbed can be shot through a camera, the image comprises each object in the object set to be grabbed, the camera sends the image to a robot or a computer, and the robot or the computer performs region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object.
Step 1032, extracting the features of the images corresponding to each article to obtain the feature information of each article; the characteristic information here includes, but is not limited to, physical information (e.g., length, width, height, color, etc.) of the object that can be extracted by the camera.
Step 1033, comparing the characteristic information of each article with the characteristic information in the identification model; since each article has been previously learned and trained, physical information such as length, width, height, color, etc. of each article is known, and at this time, after an image of the article is obtained, the respective physical information of the image can be compared.
Step 1034, when the comparison is successful, determines the item as the target item.
In this embodiment of the present invention, there may be two embodiments, one of which is to first determine, based on each article in the set of articles to be grasped, an identification tag of each article based on the identification model, and then determine, based on the identification tag of the target article, which is the target article in a matching manner. In this embodiment, step 1033 specifically includes:
then, for each article, extracting the characteristics of the image corresponding to each article to obtain the characteristic information of each article; comparing the characteristic information of the article with the comparison characteristic information of each article in the identification model for each article; and under the condition that the comparison is successful, determining the identification label of the article corresponding to the comparison characteristic information which is successfully compared as the identification label of the article.
Finally, based on the identification tag of the target object, matching and determining which is the target object, and matching the identification tag of the target object with the identification tag of each object; and when the matching is successful, determining the object corresponding to the identification tag successfully matched with the identification tag of the target object as the target object.
In another embodiment, first, an identification tag of a target object is acquired, an object matched with the target object is determined based on an identification model, then, the object is compared with each object in a set of objects to be grasped, the set of the target object matched with the object is determined in the set of objects to be grasped, and the target object is positioned as which object in the set of objects to be grasped. In this embodiment, the step flow specifically includes:
matching the identification tag of the target object with the identification tag of each object in the identification model; under the condition that the matching is successful, determining a matched object; wherein the matched article is an article corresponding to the identification tag successfully matched with the identification tag of the target article; then under the condition that characteristic information of the articles to be grabbed and the comparison characteristic information of the matched articles are successfully compared, the matched articles are determined to be target articles; the matched articles are articles corresponding to the successfully matched characteristic information.
Specifically, firstly, matching an identification tag of a target object with an identification tag of each object in an identification model, and determining a matched object under the condition of successful matching; then, carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object, and carrying out feature extraction on the image corresponding to each object to obtain feature information of each object; the images corresponding to the object set to be grabbed are shot by the camera and sent.
Finally, comparing the characteristic information of each article with the comparison characteristic information of the matched articles, and determining the matched articles as target articles under the condition that the characteristic information of the articles to be grabbed is successfully compared with the comparison characteristic information of the matched articles; the matched article is an article corresponding to the characteristic information of the successfully matched comparison characteristic information of the matched article.
In step 1034, a previous step determines which item the target item is, i.e., locating the location of the item. Step 1034 specifically includes:
performing region segmentation on the image corresponding to the object set to be grabbed to obtain pixel coordinates of the identified target object;
position information determined based on pixel coordinates of the identified target item is acquired.
Specifically, in an alternative embodiment, the robot or the computer may perform region segmentation on the image corresponding to the object set to be grabbed to obtain the pixel coordinates of the identified target object, and then convert the pixel coordinates into three-dimensional coordinates, where the three-dimensional coordinates are the position information of the target object. In another embodiment, the camera may also perform region segmentation on the image corresponding to the object set to be grabbed to obtain the pixel coordinates of the identified target object, and then send the pixel coordinates to the robot or the computer, where the robot or the computer converts the pixel coordinates into three-dimensional coordinates.
In step 1034, in an alternative embodiment, the robot may grasp the target object of the location information directly based on the location information, and in another alternative embodiment, the computer may send a control instruction including the location information to the robot, so that the robot grasps the target object of the location information according to the control instruction.
The embodiment of the invention further provides an article grabbing system, which can realize the method. Referring to fig. 4, fig. 4 is a schematic frame structure of an article grabbing system according to an embodiment of the present invention. As shown in fig. 4, the article gripping system includes: a model processing module 41 for learning and training the article to build an identification model of the article; a user instruction processing module 42, configured to receive a user instruction and extract identification information of a target object in the user instruction; the article identifying module 43 is configured to compare the identifying information with the identifying model to obtain a target article, obtain position information of the target article, and output the position information; and the grabbing module 44 is used for grabbing the target object according to the position information.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a model processing module in the article grabbing system according to the embodiment of the present invention, and as shown in fig. 5, the model processing module 41 includes: an image acquisition unit 411 for acquiring a plurality of images of an article; an image processing unit 412, configured to perform feature analysis on at least one image of the article, obtain feature information of the article, and set an identification tag for the article; model creation section 413 associates the feature information of the article with the identification tag, and creates an identification model of the article.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an article identifying module in the article grabbing system according to the embodiment of the present invention, and as shown in fig. 6, the article identifying module 43 includes: an image segmentation unit 431, configured to segment an image corresponding to a set of objects to be grabbed in an area, and obtain an image corresponding to each object; a feature extraction unit 432, configured to perform feature extraction on an image corresponding to each article, to obtain feature information of each article; a feature comparison unit 433 for comparing the feature information of each article with the feature information in the identification model; a target item confirmation unit 434 for determining the item as a target item when the comparison is successful.
Referring to fig. 7, fig. 7 is another schematic structural diagram of an article identification module in the article grabbing system according to an embodiment of the present invention, and as shown in fig. 7, the article identification module 43 includes: a matching unit 435 that matches the identification tag of the target item with the identification tag of each item in the identification model, and determines a matching item when the matching is successful; an image segmentation unit 436, configured to segment an image corresponding to the object set to be grabbed in an area, so as to obtain an image corresponding to each object; a feature extraction unit 437, configured to perform feature extraction on an image corresponding to each item, to obtain feature information of each item; and a comparison unit 438, configured to compare the feature information of each item with the comparison feature information of the matching item, and determine the matching item as the target item when the comparison is successful.
According to the embodiment of the invention, the user instruction processing module 42 is used for identifying the user instruction, extracting the related word key in the user instruction, identifying the target object through the object identifying module 43 based on the identification model obtained by deep learning, and obtaining and outputting the position information of the target object, so that the user instruction processing module 42 and the deep learning are combined to jointly complete the grabbing of the human-computer interaction, the human-computer interaction is more convenient and faster due to the addition of the user instruction, and the various scene requirements of the user are met. Compared with the prior art, the embodiment of the invention can realize intelligent grabbing of the appointed articles, improve the accuracy of grabbing the articles and complete the intelligent and personalized grabbing function.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present invention.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
The embodiment of the invention also provides an article grabbing robot, as shown in fig. 8, which comprises:
the mechanical arm 81, wherein a grabbing or sucking device 82 is arranged at the tail end of the mechanical arm 81;
the image capturing device 83, the image capturing device 83 is connected with the mechanical arm 81 and is used for acquiring an object image;
user instruction processing means 84 for receiving the user instruction and extracting the identification information of the target object in the user instruction;
and the article identification module 85 is configured to compare the identification information with the identification model to obtain a target article, obtain location information of the target article, and output the location information.
In the embodiment of the present invention, the image capturing device 83 may be a camera set forth in the foregoing embodiment, and may be disposed above the mechanical arm 81. The user command processing means 84 may comprise the aforementioned user command processing module and a microprocessor integrated in the robot control circuit board for receiving and converting the user command. The article identification module 85 may include a memory device, a microprocessor, etc., and may also be integrated on a control circuit board for performing the functions of the article identification module in the article gripping device described above.
According to the embodiment of the invention, the image capturing device 83, the user instruction processing device 84 and the article identification module 85 are arranged on the article grabbing robot, so that the method flow can be realized through one robot, and the device is simple in structure, low in cost and convenient to operate.
The embodiment of the invention further provides an article grabbing system, which is used for grabbing the articles appointed by the user. The article grabbing system can comprise a plurality of electronic devices, and the electronic devices can grab articles appointed by users in a mutually matched mode. In a preferred embodiment, the article gripping system comprises: an article grabbing robot, a computer and an image capturing device; the article grabbing robot comprises a mechanical arm, wherein the tail end of the mechanical arm is provided with grabbing or sucking devices, the image capturing device is connected with a computer and used for acquiring an article image and outputting the article image to the computer, and the computer comprises:
the model processing module is used for learning and training the articles so as to establish an identification model of the articles;
the user instruction processing device is used for receiving a user instruction and extracting identification information of a target object in the user instruction;
and the article identification module is used for comparing the identification information with the identification model to obtain a target article, acquiring the position information of the target article and outputting the position information to the article grabbing robot.
Application scene: referring to fig. 9, fig. 9 is a schematic hardware configuration of an article gripping system according to an embodiment of the present invention. As shown in fig. 9, the article gripping system includes a microphone unit 91, a camera 92, a computer 93 and an article gripping robot 94, the camera 92 is connected to the computer 93, the computer 93 includes a display screen, the computer 93 obtains a user instruction of a user through the microphone unit 91, and the camera 92 may be a USB camera for capturing an image of a target article. Referring to fig. 10, fig. 10 is a schematic view illustrating an installation of an article gripping robot in an article gripping system, the article gripping robot having a suction cup kit installed at a distal end thereof to suction a target article.
The object gripping robot 94 of the embodiment of the present invention is a dobot magician robot, which is a desktop level intelligent robot arm that supports functions of teaching reproduction, script control, graphical programming, writing and drawing, laser engraving, 3D printing, visual recognition, and the like. The system also has rich I/O expansion interfaces for users to use in secondary development. Dobot magian consists of a base, a large arm, a small arm, and an end tool, etc., with four degrees of freedom, and is able to control movement of the end along the X, Y, Z axis and rotation of the Z axis. The mechanical arm movement modes include a jog mode, a point position mode (PTP), and an ARC movement mode (ARC). PTP and ARC may be collectively referred to as a deposit reproduction motion mode. The JUMP mode in PTP can be used to JUMP from the end position of each stroke to the start position of the next stroke when writing. The robot arm communicates with the computer 93 via a USB interface.
Referring to fig. 11, fig. 11 is a control flow chart of the object gripping robot gripping the object, which realizes gripping of the object by the object gripping robot controlled by the computer. As shown in fig. 11, the control flow is as follows:
firstly, the camera 92 shoots nine images of each article, tags are added to each article, and training is carried out through deep learning mobilet to obtain an identification model; transmitting the identification model to the computer 93;
then, the user inputs a voice command based on the microphone unit 91, the computer 93 obtains the voice command through the microphone unit 91, parses the voice command, extracts a keyword, and determines an identification tag of the target article based on the keyword;
next, the computer 93 identifies a target article corresponding to the identification tag among the articles to be grasped by the identification model;
then, the camera 92 shoots an image of the object to be grabbed, and the image is subjected to region segmentation to obtain pixel coordinates; sending the pixel coordinates to the computer 93;
then, the computer converts the pixel coordinates into three-dimensional coordinates;
next, the computer 93 transmits a control instruction including the three-dimensional coordinates to the article gripping robot 94;
finally, the article gripping robot 94 grips the target article based on the control instruction.
In practical application, first, the computer 93 accesses a front page of the system through a web browser, and after calling the microphone unit 91 for input, the microphone unit 91 records a voice command of the user through a RecordRTC, wherein the RecordRTC is a media record library developed by the browser.
The computer 93 then uploads the voice command to the Tencent server to invoke automatic speech recognition (Automatic Speech Recognition, ASR) for recognition. Meanwhile, the camera 92 captures an image of the object to be placed, the captured image is subjected to feature extraction and segmentation through the OpenCV visual library, and meanwhile, an identification tag label corresponding to the image is uploaded to a Tencent server to be identified through a visual feature identification (Optical Character Recognition, OCR) technology, and voice data converted into characters returned by the Tencent server are delivered to a language technology platform (Language Technology Platform, LTP) for processing, and neuro-linguistics (Natural Language Processing, NLP) are contained therein. The processed semantic data is then transferred to a Tensorflow framework for Machine Learning (ML), wherein the Tensorflow framework contains a series of technical knowledge such as NNs neural network, DL deep Learning, convolution algorithm and the like.
Then, the camera 92 generates a trained recognition model to achieve a human-like machine learning function.
Finally, the computer 93 determines the position information of the target object through hand-eye calibration based on the identification model sent by the camera 92, and outputs the position information to the object grabbing robot 94, and the object grabbing robot 94 notifies the mechanical arm to grab the target object which we want.
Therefore, the intelligent recognition of the object by the mechanical arm and the corresponding operation of grabbing the object are completed through the tensorsurface framework of google and the technologies of OCR, ASR, LTP and the like of vision and voice, and the aim of fusing and understanding the knowledge of the related fields of vision and voice technology and machine learning is achieved.
In addition, in another alternative embodiment, the article grabbing robot 94 in the article grabbing system is a robot with grabbing function, and an interface for communicating with other electronic devices in the article grabbing system is integrated on the article grabbing robot 94, so as to communicate with the other electronic devices, acquire information sent by the other electronic devices, and finally grab the article specified by the user under the cooperation of the other electronic devices. That is, the article grabbing robot 94 may include a processor, and may process information sent by other electronic devices, such as training a recognition model, analyzing a voice command, and so on, to finally grab an article specified by a user.
The object grabbing system provided in the embodiment of the present invention can also implement each process in the above method embodiment by independently setting the camera 92, the computer 93 and the object grabbing robot 94, and in order to avoid repetition, the description is omitted here.
The foregoing is merely illustrative embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the technical scope of the present invention, and the invention should be covered. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (3)
1. An article gripping method, characterized in that the article gripping method comprises:
learning and training the article to build an identification model of the article, comprising: acquiring a plurality of images of an article; performing feature analysis on at least one image of the article to obtain feature information of the article and setting an identification tag for the article; associating the characteristic information of the article with the identification tag, and establishing an identification model of the article;
receiving a user instruction and extracting identification information of a target object in the user instruction;
comparing the identification information with the identification model to obtain a target object, wherein the method comprises the steps of matching the identification label of the target object with the identification label of each object in the identification model, and determining a matched object when the matching is successful; carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object; extracting the characteristics of the image corresponding to each article to obtain the characteristic information of each article; comparing the characteristic information of each article with the comparison characteristic information of the matched article, and determining the matched article as a target article when the comparison is successful; acquiring and outputting the position information of the target object;
and grabbing the target object according to the position information.
2. An article gripping system, comprising:
the model processing module is used for learning and training the article to establish an identification model of the article, and comprises the following steps: an image acquisition unit configured to acquire a plurality of images of an article; the image processing unit is used for carrying out feature analysis on at least one image of the article, obtaining feature information of the article and setting an identification tag for the article; the model building unit is used for associating the characteristic information of the article with the identification tag to build an identification model of the article;
the user instruction processing module is used for receiving a user instruction and extracting identification information of a target object in the user instruction;
the article identification module is used for comparing the identification information with the identification model to obtain a target article, acquiring and outputting the position information of the target article, and comprises the following steps: the matching unit is used for matching the identification tag of the target object with the identification tag of each object in the identification model, and determining the matched object when the matching is successful; the image segmentation unit is used for carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object; the feature extraction unit is used for extracting features of the images corresponding to each article to obtain feature information of each article; the comparison unit is used for comparing the characteristic information of each article with the comparison characteristic information of the matched article, and determining the matched article as a target article when the comparison is successful;
and the grabbing module is used for grabbing the target object according to the position information.
3. An article gripping robot having an article gripping system according to claim 2, comprising:
the tail end of the mechanical arm is provided with a grabbing or sucking device;
the image capturing device is connected with the mechanical arm and used for acquiring an object image;
the user instruction processing device is used for receiving a user instruction and extracting identification information of a target object in the user instruction;
the article identification module is used for comparing the identification information with the identification model to obtain a target article, acquiring and outputting the position information of the target article, and comprises the following steps: the matching unit is used for matching the identification tag of the target object with the identification tag of each object in the identification model, and determining the matched object when the matching is successful; the image segmentation unit is used for carrying out region segmentation on the image corresponding to the object set to be grabbed to obtain an image corresponding to each object; the feature extraction unit is used for extracting features of the images corresponding to each article to obtain feature information of each article; and the comparison unit is used for comparing the characteristic information of each article with the comparison characteristic information of the matched article, and determining the matched article as a target article when the comparison is successful.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911383453.6A CN111145257B (en) | 2019-12-27 | 2019-12-27 | Article grabbing method and system and article grabbing robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911383453.6A CN111145257B (en) | 2019-12-27 | 2019-12-27 | Article grabbing method and system and article grabbing robot |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111145257A CN111145257A (en) | 2020-05-12 |
| CN111145257B true CN111145257B (en) | 2024-01-05 |
Family
ID=70521457
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911383453.6A Active CN111145257B (en) | 2019-12-27 | 2019-12-27 | Article grabbing method and system and article grabbing robot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111145257B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111724444B (en) * | 2020-06-16 | 2023-08-22 | 中国联合网络通信集团有限公司 | Method, device and system for determining grabbing point of target object |
| CN112068422B (en) * | 2020-08-04 | 2023-04-14 | 广州中国科学院先进技术研究所 | Grabbing learning method and device of intelligent robot based on small samples |
| CN112215132A (en) * | 2020-10-10 | 2021-01-12 | 北京如影智能科技有限公司 | Method and device for generating article information for adaptive smart kitchen system |
| CN112720474A (en) * | 2020-12-21 | 2021-04-30 | 深圳市越疆科技有限公司 | Pose correction method and device for robot, terminal device and storage medium |
| CN116184892B (en) * | 2023-01-19 | 2024-02-06 | 盐城工学院 | AI identification control method and system for robot object taking |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902271A (en) * | 2012-10-23 | 2013-01-30 | 上海大学 | Binocular vision-based robot target identifying and gripping system and method |
| CN107571260A (en) * | 2017-10-25 | 2018-01-12 | 南京阿凡达机器人科技有限公司 | The method and apparatus that control machine people captures object |
| CN108537841A (en) * | 2017-03-03 | 2018-09-14 | 株式会社理光 | A kind of implementation method, device and the electronic equipment of robot pickup |
| CN108876795A (en) * | 2018-06-07 | 2018-11-23 | 四川斐讯信息技术有限公司 | A kind of dividing method and system of objects in images |
| CN109102547A (en) * | 2018-07-20 | 2018-12-28 | 上海节卡机器人科技有限公司 | Robot based on object identification deep learning model grabs position and orientation estimation method |
| CN109410275A (en) * | 2018-10-15 | 2019-03-01 | 深圳步智造科技有限公司 | Position predicting method, device, equipment and readable storage medium storing program for executing |
| CN109784297A (en) * | 2019-01-26 | 2019-05-21 | 福州大学 | A kind of Three-dimensional target recognition based on deep learning and Optimal Grasp method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3458919B1 (en) * | 2016-05-19 | 2022-08-24 | Deep Learning Robotics Ltd. | Robot assisted object learning vision system |
| JP7136554B2 (en) * | 2017-12-18 | 2022-09-13 | 国立大学法人信州大学 | Grasping device, learning device, program, grasping system, and learning method |
-
2019
- 2019-12-27 CN CN201911383453.6A patent/CN111145257B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902271A (en) * | 2012-10-23 | 2013-01-30 | 上海大学 | Binocular vision-based robot target identifying and gripping system and method |
| CN108537841A (en) * | 2017-03-03 | 2018-09-14 | 株式会社理光 | A kind of implementation method, device and the electronic equipment of robot pickup |
| CN107571260A (en) * | 2017-10-25 | 2018-01-12 | 南京阿凡达机器人科技有限公司 | The method and apparatus that control machine people captures object |
| CN108876795A (en) * | 2018-06-07 | 2018-11-23 | 四川斐讯信息技术有限公司 | A kind of dividing method and system of objects in images |
| CN109102547A (en) * | 2018-07-20 | 2018-12-28 | 上海节卡机器人科技有限公司 | Robot based on object identification deep learning model grabs position and orientation estimation method |
| CN109410275A (en) * | 2018-10-15 | 2019-03-01 | 深圳步智造科技有限公司 | Position predicting method, device, equipment and readable storage medium storing program for executing |
| CN109784297A (en) * | 2019-01-26 | 2019-05-21 | 福州大学 | A kind of Three-dimensional target recognition based on deep learning and Optimal Grasp method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111145257A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111145257B (en) | Article grabbing method and system and article grabbing robot | |
| CN107223246B (en) | Image labeling method and device and electronic equipment | |
| CN107239728B (en) | Unmanned aerial vehicle interaction device and method based on deep learning attitude estimation | |
| KR101887637B1 (en) | Robot system | |
| CN111590611B (en) | Article classification and recovery method based on multi-mode active perception | |
| CN111723782A (en) | Deep learning-based visual robot grabbing method and system | |
| CN108427910A (en) | Deep-neural-network AR sign language interpreters learning method, client and server | |
| CN111319044B (en) | Article grabbing method and device, readable storage medium and grabbing robot | |
| Sudol et al. | Looktel—A comprehensive platform for computer-aided visual assistance | |
| CN108702458A (en) | Image pickup method and device | |
| CN109407709A (en) | A kind of meeting camera shooting automatic tracking system based on Kinect bone track algorithm | |
| CN113936340B (en) | AI model training method and device based on training data acquisition | |
| CN112947458B (en) | Robot accurate grabbing method based on multi-mode information and computer readable medium | |
| CN116434344A (en) | Pen holding gesture recognition model training method, recognition method and related devices | |
| CN212724028U (en) | Vision robot grasping system | |
| CN106997449A (en) | Robot and face identification method with face identification functions | |
| CN108960200A (en) | A kind of data processing method and electronic equipment based on intelligent interaction | |
| EP3646243B1 (en) | Learning template representation libraries | |
| CN113894779A (en) | Multi-mode data processing method applied to robot interaction | |
| US10708445B2 (en) | Information processing apparatus and non-transitory computer readable medium | |
| CN110021297A (en) | A kind of intelligent display method and its device based on audio-video identification | |
| Nashrom et al. | Assistive Shopping Tool for the Visually Impaired | |
| CN112017247A (en) | Method for realizing unmanned vehicle vision by using KINECT | |
| Xu | Gesture-based Human-robot Interaction for Field Programmable Autonomous Underwater Robots | |
| CN114550185B (en) | Document generation method, related device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210720 Address after: 518000 1003, building 2, Chongwen Park, Nanshan wisdom Park, 3370 Liuxian Avenue, Fuguang community, Taoyuan Street, Nanshan District, Shenzhen City, Guangdong Province Applicant after: SHENZHEN YUEJIANG TECHNOLOGY Co.,Ltd. Address before: 518000 1003, building 2, Chongwen Park, Nanshan wisdom Park, 3370 Liuxian Avenue, Fuguang community, Taoyuan Street, Nanshan District, Shenzhen City, Guangdong Province Applicant before: SHENZHEN YUEJIANG TECHNOLOGY Co.,Ltd. Applicant before: WUHAN YUEJIANG ZHIDAO TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |