CN113674341A - Robot visual identification and positioning method, intelligent terminal and storage medium - Google Patents
Robot visual identification and positioning method, intelligent terminal and storage medium Download PDFInfo
- Publication number
- CN113674341A CN113674341A CN202110962704.7A CN202110962704A CN113674341A CN 113674341 A CN113674341 A CN 113674341A CN 202110962704 A CN202110962704 A CN 202110962704A CN 113674341 A CN113674341 A CN 113674341A
- Authority
- CN
- China
- Prior art keywords
- image
- checkerboard
- robot
- disassembled
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000000007 visual effect Effects 0.000 title abstract description 7
- 239000011159 matrix material Substances 0.000 claims abstract description 34
- 238000006243 chemical reaction Methods 0.000 claims abstract description 20
- 238000012549 training Methods 0.000 claims description 57
- 230000009466 transformation Effects 0.000 claims description 22
- 230000004807 localization Effects 0.000 claims 3
- 239000002699 waste material Substances 0.000 abstract description 8
- 125000004122 cyclic group Chemical group 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 229910052799 carbon Inorganic materials 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 229910000838 Al alloy Inorganic materials 0.000 description 1
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 229910000831 Steel Inorganic materials 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000004033 plastic Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Manipulator (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a robot visual identification and positioning method, an intelligent terminal and a storage medium, wherein the method comprises the following steps: acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled; inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model; and determining target position information corresponding to each part to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot. According to the invention, the type information and the image position information of each part to be disassembled are output through the image recognition model, and the target position information is determined according to the image position information, so that the type information of the part to be disassembled can be accurately recognized, the position information of the part to be disassembled can be accurately positioned, the automatic classification disassembly of the shared bicycle and the cyclic utilization of the parts of the shared bicycle are realized, and the problem of resource waste caused by manual violent disassembly is solved.
Description
Technical Field
The invention relates to the technical field of machine identification, in particular to a robot visual identification and positioning method, an intelligent terminal and a storage medium.
Background
The sharing bicycle has the advantages of high degree of freedom, low price, low carbon, environmental protection and the like, and is favored by young people due to high use frequency. The shared bicycle brings great convenience to people, tens of millions of shared bicycles face to be scrapped every year due to the fact that a large number of shared bicycles are put in and various human factors are damaged, in order to solve the problem that the old shared bicycles are disorderly stopped and randomly put in, the recycled shared bicycles are treated as waste products after being violently disassembled in the conventional method, and the treatment mode causes great resource waste.
Thus, there is still a need for improvement and development of the prior art.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a robot vision identification and positioning method, an intelligent terminal and a storage medium, aiming at solving the problem of great resource waste caused by the existing manual violent disassembling and sharing bicycle mode.
The technical scheme adopted by the invention for solving the problems is as follows:
in a first aspect, an embodiment of the present invention provides a robot vision identification and positioning method, where the method is applied to an intelligent terminal connected to a camera and a robot, and the method includes:
acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled;
inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model;
and determining target positions corresponding to the parts to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot.
The robot vision identification and positioning method comprises the following steps of:
inputting training images in a training image set into a preset network model, and outputting prediction attribute labels corresponding to all parts in the training images through the preset network model; the training image set comprises training images and real attribute labels corresponding to all parts in the training images, the real attribute labels comprise real category information and real image position information, and the prediction attribute labels comprise prediction category information and predicted image position information;
updating the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continuing to execute the step of outputting the predicted attribute labels corresponding to all the components in the training image through the preset network model until the training condition of the preset network model meets the preset condition so as to obtain the image recognition model.
The robot vision identification and positioning method comprises the following steps of updating model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continuously executing the step of outputting the predicted attribute labels corresponding to all the parts in the training image through the preset network model until the training condition of the preset network model meets the preset condition:
determining a loss value according to the predicted attribute label and the real attribute label, and comparing the loss value with a preset threshold value;
and when the loss value is not less than the preset threshold value, updating the model parameters of the preset network model according to a preset parameter learning rate, and continuing to execute the step of outputting the prediction attribute labels corresponding to all the components in the training image through the preset network model until the loss value is less than the preset threshold value.
The robot vision identification and positioning method comprises the following steps of determining target positions corresponding to all parts to be disassembled according to the image position information and a predetermined conversion matrix from a camera to the tail end of the robot, wherein the step of determining the target positions corresponding to all the parts to be disassembled comprises the following steps:
determining the central position coordinates corresponding to each part to be disassembled according to the image position information;
and carrying out coordinate transformation on the coordinates of the central position according to a predetermined conversion matrix from the camera to the tail end of the robot, and determining the target position corresponding to each part to be disassembled.
The robot vision identification and positioning method comprises the following steps of determining center position coordinates corresponding to each part to be disassembled according to the image position information:
determining the minimum circumscribed rectangle corresponding to each part to be disassembled according to the image position information;
and acquiring the center point coordinate of the minimum external rectangle corresponding to each part to be disassembled, and determining the center point coordinate as the center position coordinate corresponding to each part to be disassembled.
The robot vision identification and positioning method comprises the following steps of:
acquiring a pre-designed checkerboard, and determining coordinates of each corner point in the checkerboard under a robot base coordinate system and coordinates of each corner point in the checkerboard under a camera coordinate system according to the checkerboard;
and determining a conversion matrix from the camera to the tail end of the robot according to the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system.
The robot vision identification and positioning method comprises the following steps of determining coordinates of each corner point in the checkerboard in a robot base coordinate system and coordinates of each corner point in the checkerboard in a camera coordinate system according to the checkerboard, wherein the steps comprise:
calibrating the camera according to the checkerboard, and determining internal and external parameters and distortion coefficients of the camera;
and acquiring the image coordinates of each corner point in the checkerboard, carrying out coordinate transformation on the image coordinates according to the internal and external parameters and the distortion coefficient of the camera, and determining the coordinates of each corner point in the checkerboard in a camera coordinate system.
The robot vision identification and positioning method comprises the following steps of determining coordinates of each corner point in the checkerboard in a robot base coordinate system and coordinates of each corner point in the checkerboard in a camera coordinate system according to the checkerboard, wherein the steps further comprise:
determining a transformation matrix from a checkerboard coordinate system to a robot tail end coordinate system and coordinates of each corner point in the checkerboard coordinate system according to the checkerboard;
and determining the coordinates of each angular point in the checkerboard under the coordinate system of the robot base according to a transformation matrix from the checkerboard coordinate system to the terminal coordinate system of the robot and the coordinates of each angular point in the checkerboard under the checkerboard coordinate system.
In a second aspect, an embodiment of the present invention further provides a robot vision identifying and positioning apparatus, where the apparatus includes:
the image acquisition module is used for acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled;
the image recognition module is used for inputting the image to be recognized into an image recognition model and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model;
and the target positioning module is used for determining the target position corresponding to each part to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot.
In a third aspect, an embodiment of the present invention provides an intelligent terminal, including: a processor, a storage medium communicatively coupled to the processor, the storage medium adapted to store a plurality of instructions; the processor is adapted to call instructions in the storage medium to execute the steps of implementing the robot visual recognition and positioning method.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a plurality of instructions are stored, wherein the instructions are adapted to be loaded and executed by a processor to perform the steps of implementing the robot visual identification and positioning method described above.
The invention has the beneficial effects that: the embodiment of the invention firstly obtains an image to be identified; the method comprises the steps of acquiring images to be recognized, inputting the images to be recognized into an image recognition model, outputting category information and image position information corresponding to each component to be recognized through the image recognition model, and finally determining target positions corresponding to the components to be recognized according to the image position information and a predetermined conversion matrix from a camera to the tail end of a robot.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a robot vision identification and positioning method according to an embodiment of the present invention;
FIG. 2 is a schematic block diagram of a robot vision identifying and positioning device provided by an embodiment of the invention;
fig. 3 is a schematic block diagram of an internal structure of an intelligent terminal according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
It should be noted that, if directional indications (such as up, down, left, right, front, and back … …) are involved in the embodiment of the present invention, the directional indications are only used to explain the relative positional relationship between the components, the movement situation, and the like in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indications are changed accordingly.
Compared with the traditional bicycle leasing, the shared bicycle has the following advantages: firstly, the shared bicycle does not need to be parked on a special parking pile, can be parked and moved in most areas, and has higher use freedom; secondly, compared with the price of a few yuan of the traditional lease and even dozens of yuan of the scenic spot, the shared bicycle has more advantages; in addition, the shared bicycle meets the current low-carbon, environment-friendly and green travel trend. The advantages enable the shared bicycle to hopefully meet daily high-frequency use requirements and become a new travel fashion for young people.
However, while the shared bicycle brings great convenience to people, tens of millions of shared bicycles are scrapped every year due to the large amount of thrown shared bicycles and the damage of various human factors, in order to solve the problem that the worn shared bicycles are disorderly parked and disorderly put, the existing method is to treat the recovered shared bicycles as waste products after manual violent disassembly, and the parts of aluminum alloy, plastic, steel and the like on the shared bicycles are available data, so that the treatment mode can cause great resource waste.
In order to solve the problems in the prior art, the embodiment provides a robot vision identification and positioning method, by which the category information of a part to be detached can be accurately identified and the position information of the part to be detached can be accurately positioned, the automatic classification detachment of a shared bicycle and the cyclic utilization of parts of the shared bicycle are realized, and the problem of resource waste caused by manual violent detachment is solved. In specific implementation, firstly, an image to be identified is obtained; the method comprises the steps of obtaining images to be recognized, inputting the images to be recognized into an image recognition model, outputting category information and image position information corresponding to each component to be recognized through the image recognition model, and finally determining target positions corresponding to the components to be recognized according to the image position information and a predetermined conversion matrix from a camera to the tail end of a robot.
Exemplary method
The embodiment provides a robot vision identification and positioning method which can be applied to an intelligent terminal connected with a camera and a robot. As shown in fig. 1 in particular, the method comprises:
s100, acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled.
Specifically, the image to be recognized is obtained by photographing the shared bicycle to be detached through the industrial camera, the shared bicycle to be detached comprises a plurality of parts to be detached, such as nuts, screws, bolts and the like, pixels of the industrial camera are as high as 200 thousands, the image to be recognized can be acquired at the speed of 15 frames per second, the industrial camera is connected with the intelligent terminal through a network cable, and then SOCKET communication is established by utilizing a UDP/IP protocol, so that data transmission from the application layer to the application layer is realized. When the object to be disassembled needs to be disassembled, if the scrapped shared bicycle needs to be disassembled, the object to be disassembled is photographed through the industrial camera, the image to be recognized is collected, then the collected image to be recognized is transmitted to the intelligent terminal through the industrial camera, so that in the subsequent step, the image to be recognized is input into the image recognition model, and the category information and the image position information corresponding to each part to be disassembled are obtained.
And S200, inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model.
Specifically, the category information is a category to which each component to be disassembled belongs, for example, the component to be disassembled belongs to a nut, a bolt, a screw, or the like, the image position information is a two-dimensional image coordinate of each component to be disassembled on the image to be recognized, the category information and the position information are obtained by recognizing and positioning the image to be recognized through an image recognition model, and correspondingly, the step of obtaining the category information and the image position information corresponding to each component to be disassembled may specifically be: and inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled in the shared bicycle to be disassembled through the image recognition model. The image recognition model comprises a basic convolutional neural network and a connected regression classification network, such as a convolutional layer, an activation layer, a pooling layer and other core trunk layers.
In a specific embodiment, the method for generating the image recognition model in step S200 includes:
step S210, inputting training images in a training image set into a preset network model, and outputting prediction attribute labels corresponding to all parts in the training images through the preset network model; the training image set comprises training images and real attribute labels corresponding to all parts in the training images, the real attribute labels comprise real category information and real image position information, and the prediction attribute labels comprise prediction category information and predicted image position information;
step S220, updating model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continuing to execute the step of outputting the predicted attribute labels corresponding to all the parts in the training image through the preset network model until the training condition of the preset network model meets a preset condition, so as to obtain an image recognition model.
Specifically, the training image set comprises training images and real attribute labels corresponding to all parts in the training images, the training images are obtained by photographing different types of parts through an industrial camera, and in order to improve the identification and positioning accuracy of the image identification model, when the training image set is collected, the robustness of the training image set is improved by changing the relative positions of the industrial camera and the parts and simulating different industrial background environments through different illumination, brightness, distance and image resolution. The real attribute label comprises real category information and real image position information corresponding to each component in the training image.
In this embodiment, a python compiling deep learning algorithm is used to construct a network model in advance, and the network model has the same structure as the image recognition model, and includes a basic convolutional neural network and a coupled regression classification network, such as a convolutional layer, an activation layer, a pooling layer, and some core trunk layers. After a training image set is obtained, training a network model in a python compiling deep learning algorithm, wherein the training process comprises setting of attributes such as training round, training batch and the like, and the training process of the network model specifically comprises the following steps: inputting training images in a training image set into a preset network model, and outputting prediction attribute labels corresponding to all parts in the training images through the preset network model, wherein the prediction attribute labels are similar to the real attribute labels and comprise prediction category information and predicted image position information; and then updating the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels until the training condition of the preset network model meets the preset condition so as to obtain an image recognition model.
In a specific embodiment, the step S220 of updating the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continuing to execute the step of outputting the predicted attribute labels corresponding to the components in the training image through the preset network model until the training condition of the preset network model meets a preset condition includes:
step S221, determining a loss value according to the predicted attribute label and the real attribute label, and comparing the loss value with a preset threshold value;
step S222, when the loss value is not less than the preset threshold, updating the model parameters of the preset network model according to a preset parameter learning rate, and continuing to execute the step of outputting the predicted attribute labels corresponding to the components in the training image through the preset network model until the loss value is less than the preset threshold.
Specifically, in this embodiment, a threshold value used for determining whether the training condition of the preset network model meets a preset condition is preset, and after the predicted attribute tag is obtained, the loss value is determined according to the predicted attribute tag and the real attribute tag. The smaller the general loss value is, the better the performance of the network model is, and after the loss value is obtained, whether the loss value is smaller than a preset threshold value is further judged; if so, indicating that the training condition of the preset network model meets the preset condition; if not, the training condition of the preset network model is indicated to be not satisfied with the preset condition, the model parameters of the preset network model are updated according to the preset parameter learning rate, the step of outputting the prediction attribute labels corresponding to all the parts in the training image through the preset network model is continuously executed, background real-time monitoring can be carried out through a tensisorbard in the training process of the network model, the network training condition is examined, and when the loss value tends to be stable and is smaller than the preset threshold value, the network model training is completed.
And step S300, determining target positions corresponding to the parts to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot.
Because the image position information output by the image recognition model and corresponding to each component to be disassembled is the image position coordinates corresponding to each component to be disassembled, the component to be disassembled can be conveniently and automatically disassembled by the robot, and the image position information needs to be converted into target position information under a terminal coordinate system of the robot. After the image position information corresponding to each component to be disassembled is acquired, coordinate transformation is further performed on the image position information through a predetermined conversion matrix from a camera to the tail end of the robot, and target position information corresponding to each component to be disassembled is obtained.
In one embodiment, step S300 specifically includes:
step S310, determining a central position coordinate corresponding to each part to be disassembled according to the image position information;
and step S320, carrying out coordinate transformation on the coordinates of the center position according to a predetermined conversion matrix from the camera to the tail end of the robot, and determining the target positions corresponding to the parts to be disassembled.
Specifically, in this embodiment, after image position information corresponding to the to-be-detached component is acquired, the center position coordinates corresponding to each to-be-detached component are determined according to the image position information, and then the center position coordinates are subjected to coordinate transformation according to a predetermined transformation matrix from the camera to the end of the robot, so as to determine the target position corresponding to each to-be-detached component.
In an embodiment, step S310 specifically includes:
step S311, determining the minimum circumscribed rectangle corresponding to each part to be disassembled according to the image position information;
step S312, obtaining the center point coordinate of the minimum circumscribed rectangle corresponding to each component to be disassembled, and determining the center point coordinate as the center position coordinate corresponding to each component to be disassembled.
In a specific embodiment, the center position coordinate corresponding to each to-be-detached component is the center point coordinate of the minimum circumscribed rectangle corresponding to each to-be-detached component, and after the image position information corresponding to each to-be-detached component is acquired in this embodiment, the maximum and minimum horizontal and vertical coordinates are selected from the image position information corresponding to each to-be-detached component as a bounding box, the minimum circumscribed rectangle corresponding to each to-be-detached component is determined, then, the center point coordinate of the minimum circumscribed rectangle corresponding to each to-be-detached component is acquired, and the center point coordinate is determined as the center position coordinate corresponding to each to-be-detached component.
In a specific embodiment, the method for determining the transformation matrix from the camera to the robot end in step S300 includes:
step M310, obtaining a pre-designed checkerboard, and determining coordinates of each corner point in the checkerboard under a robot base coordinate system and coordinates of each corner point under a camera coordinate system according to the checkerboard;
and step M320, determining a conversion matrix from the camera to the tail end of the robot according to the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system.
Specifically, camera calibration is a very important step in robot vision, and can help a robot to convert recognized visual information, so as to complete subsequent shared bicycle disassembly. Wherein, the calculation formula of the conversion matrix from the camera to the robot terminal isendTcamera＝(baseTend)-1baseP(cameraP)-1WhereinendTcamerais a transformation matrix of the camera to the end of the robot,baseTendis a transformation matrix of the robot tip to the base coordinate system,baseTendcan be obtained in real time according to positive kinematics of the robot,camerap is the coordinates of each corner point in the checkerboard under the camera coordinate system,baseand P is the coordinate of each angular point in the checkerboard under the coordinate system of the robot base.
In one embodiment, in step M310, the step of determining coordinates of each corner point in the checkerboard in the robot base coordinate system and the camera coordinate system according to the checkerboard includes:
step M311, calibrating the camera according to the checkerboard, and determining internal and external parameters and a distortion coefficient of the camera;
and step M312, obtaining the image coordinates of each corner point in the checkerboard, carrying out coordinate transformation on the image coordinates according to the internal and external parameters and the distortion coefficient of the camera, and determining the coordinates of each corner point in the checkerboard under a camera coordinate system.
Specifically, in the embodiment, when determining the coordinates of each corner point in the checkerboard in the camera coordinate system, the camera is calibrated according to the checkerboard, the internal and external parameters and the distortion coefficient of the camera are determined, then the image coordinates of each corner point in the checkerboard are obtained, the image coordinates are subjected to coordinate transformation according to the internal and external parameters and the distortion coefficient of the camera, and the coordinates of each corner point in the checkerboard in the camera coordinate system are determined. The calibration process of the camera comprises the following steps: the checkerboard is placed in a dark box, photos in different directions are taken for the checkerboard, a series of checkerboard pictures are obtained, coordinate values of all corner points on the checkerboard in a world coordinate system and coordinate values in the corresponding checkerboard pictures are obtained, a homography matrix is obtained through a least square method, and internal and external parameters and distortion coefficients of the camera are solved, so that the camera is calibrated.
In one embodiment, in step M310, the step of determining coordinates of each corner point in the checkerboard in the robot base coordinate system and the camera coordinate system according to the checkerboard further includes:
step M313, determining a transformation matrix from a checkerboard coordinate system to a robot tail end coordinate system and coordinates of each corner point in the checkerboard coordinate system according to the checkerboard;
and step M314, determining the coordinates of each corner point in the checkerboard under the robot base coordinate system according to the transformation matrix from the checkerboard coordinate system to the robot tail end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system.
Specifically, in this embodiment, when determining the coordinates of each corner point in the checkerboard under the robot base coordinate system, the coordinates of each corner point in the checkerboard under the checkerboard coordinate system and the translation coordinates of the checkerboard origin to the robot end coordinate origin may be determined by setting the upper left corner point of the checkerboard as the origin, then measuring or determining according to the design size the coordinates of each corner point in the checkerboard under the checkerboard coordinate system and the translation coordinates of the checkerboard origin to the robot end coordinate origin, then determining the coordinates of each corner point in the checkerboard under the robot base coordinate system according to the translation coordinates of the checkerboard origin to the robot end coordinate origin, and finally determining the coordinates of each corner point in the checkerboard under the robot base coordinate system according to the translation matrices of the checkerboard origin to the robot end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system. Wherein,the calculation formula of the coordinates of each angular point in the checkerboard under the robot base coordinate system is as follows:baseP＝baseTend endTboard boardp, wherein,baseTendis a transformation matrix of the robot tip to the base coordinate system,baseTendcan be obtained in real time according to positive kinematics of the robot,endTboardis a transformation matrix from the checkerboard coordinate system to the robot end coordinate system,boardand P is the coordinate of each corner point in the checkerboard in a checkerboard coordinate system.
Exemplary device
As shown in fig. 2, an embodiment of the present invention provides a robot vision identifying and positioning device, including: an image acquisition module 210, an image recognition module 220, and a target location module 230. Specifically, the image obtaining module 210 is configured to obtain an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled. The image recognition module 220 is configured to input the image to be recognized into an image recognition model, and output category information and image position information corresponding to each component to be disassembled through the image recognition model. The target positioning module 230 is configured to determine target positions corresponding to the components to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the end of the robot.
Based on the above embodiment, the present invention further provides an intelligent terminal, and a schematic block diagram thereof may be as shown in fig. 3. The intelligent terminal comprises a processor, a memory, a network interface, a display screen and a temperature sensor which are connected through a system bus. Wherein, the processor of the intelligent terminal is used for providing calculation and control capability. The memory of the intelligent terminal comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the intelligent terminal is used for being connected and communicated with an external terminal through a network. The computer program is executed by a processor to implement a robot vision recognition and positioning method. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the intelligent terminal is arranged inside the intelligent terminal in advance and used for detecting the operating temperature of internal equipment.
It will be understood by those skilled in the art that the block diagram shown in fig. 3 is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation to the intelligent terminal to which the solution of the present invention is applied, and a specific intelligent terminal may include more or less components than those shown in the figure, or combine some components, or have a different arrangement of components.
In one embodiment, an intelligent terminal is provided that includes a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled;
inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model;
and determining target positions corresponding to the parts to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, databases, or other media used in embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
In summary, the present invention discloses a robot vision identification and positioning method, an intelligent terminal and a storage medium, including: acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled; inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model; and determining target positions corresponding to the parts to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot. According to the invention, the type information and the image position information of each part to be disassembled are output through the image recognition model, and the target position information is determined according to the image position information, so that the type information of the part to be disassembled can be accurately recognized, the position information of the part to be disassembled can be accurately positioned, the automatic classification disassembly of the shared bicycle and the cyclic utilization of the parts of the shared bicycle are realized, and the problem of resource waste caused by manual violent disassembly is solved.
It is to be understood that the invention is not limited to the examples described above, but that modifications and variations may be effected thereto by those of ordinary skill in the art in light of the foregoing description, and that all such modifications and variations are intended to be within the scope of the invention as defined by the appended claims.
Claims (10)
1. A robot vision identification and positioning method is applied to an intelligent terminal connected with a camera and a robot, and comprises the following steps:
acquiring an image to be identified; the image to be recognized comprises a plurality of parts to be disassembled;
inputting the image to be recognized into an image recognition model, and outputting the category information and the image position information corresponding to each part to be disassembled through the image recognition model;
and determining target positions corresponding to the parts to be disassembled according to the image position information and a predetermined conversion matrix from the camera to the tail end of the robot.
2. The robot vision identifying and positioning method according to claim 1, wherein the method for generating the image identification model comprises:
inputting training images in a training image set into a preset network model, and outputting prediction attribute labels corresponding to all parts in the training images through the preset network model; the training image set comprises training images and real attribute labels corresponding to all parts in the training images, the real attribute labels comprise real category information and real image position information, and the prediction attribute labels comprise prediction category information and predicted image position information;
updating the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continuing to execute the step of outputting the predicted attribute labels corresponding to all the components in the training image through the preset network model until the training condition of the preset network model meets the preset condition so as to obtain the image recognition model.
3. The robot vision identifying and positioning method of claim 2, wherein the step of updating the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels and continuing to output the predicted attribute labels corresponding to each component in the training image through the preset network model until the training condition of the preset network model meets a preset condition comprises:
determining a loss value according to the predicted attribute label and the real attribute label, and comparing the loss value with a preset threshold value;
and when the loss value is not less than the preset threshold value, updating the model parameters of the preset network model according to a preset parameter learning rate, and continuing to execute the step of outputting the prediction attribute labels corresponding to all the components in the training image through the preset network model until the loss value is less than the preset threshold value.
4. The robot vision identifying and positioning method of claim 1, wherein the step of determining the target position corresponding to each component to be disassembled according to the image position information and a predetermined conversion matrix from a camera to a robot end comprises:
determining the central position coordinates corresponding to each part to be disassembled according to the image position information;
and carrying out coordinate transformation on the coordinates of the central position according to a predetermined conversion matrix from the camera to the tail end of the robot, and determining the target position corresponding to each part to be disassembled.
5. The robot vision identifying and positioning method of claim 4, wherein the step of determining the center position coordinates corresponding to each component to be disassembled according to the image position information comprises:
determining the minimum circumscribed rectangle corresponding to each part to be disassembled according to the image position information;
and acquiring the center point coordinate of the minimum external rectangle corresponding to each part to be disassembled, and determining the center point coordinate as the center position coordinate corresponding to each part to be disassembled.
6. The robot vision identifying and positioning method according to claim 1, wherein the method for determining the transformation matrix of the camera to the robot end comprises:
acquiring a pre-designed checkerboard, and determining coordinates of each corner point in the checkerboard under a robot base coordinate system and coordinates of each corner point in the checkerboard under a camera coordinate system according to the checkerboard;
and determining a conversion matrix from the camera to the tail end of the robot according to the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system.
7. The robot vision identification and localization method of claim 6, wherein the step of determining coordinates of each corner point in the checkerboard in a robot base coordinate system and in a camera coordinate system from the checkerboard comprises:
calibrating the camera according to the checkerboard, and determining internal and external parameters and distortion coefficients of the camera;
and acquiring the image coordinates of each corner point in the checkerboard, carrying out coordinate transformation on the image coordinates according to the internal and external parameters and the distortion coefficient of the camera, and determining the coordinates of each corner point in the checkerboard in a camera coordinate system.
8. The robot vision identifying and positioning method of claim 7, wherein the step of determining coordinates of each corner point in the checkerboard in a robot base coordinate system and in a camera coordinate system according to the checkerboard further comprises:
determining a transformation matrix from a checkerboard coordinate system to a robot tail end coordinate system and coordinates of each corner point in the checkerboard coordinate system according to the checkerboard;
and determining the coordinates of each angular point in the checkerboard under the coordinate system of the robot base according to a transformation matrix from the checkerboard coordinate system to the terminal coordinate system of the robot and the coordinates of each angular point in the checkerboard under the checkerboard coordinate system.
9. An intelligent terminal, comprising: a processor, a storage medium communicatively coupled to the processor, the storage medium adapted to store a plurality of instructions; the processor is adapted to invoke instructions in the storage medium to perform the steps of implementing the robot vision identification and localization method of any of the above claims 1-8.
10. A computer readable storage medium having stored thereon a plurality of instructions adapted to be loaded and executed by a processor to perform the steps of implementing the robot vision identification and localization method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110962704.7A CN113674341A (en) | 2021-08-20 | 2021-08-20 | Robot visual identification and positioning method, intelligent terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110962704.7A CN113674341A (en) | 2021-08-20 | 2021-08-20 | Robot visual identification and positioning method, intelligent terminal and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113674341A true CN113674341A (en) | 2021-11-19 |
Family
ID=78544569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110962704.7A Pending CN113674341A (en) | 2021-08-20 | 2021-08-20 | Robot visual identification and positioning method, intelligent terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113674341A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115546566A (en) * | 2022-11-24 | 2022-12-30 | 杭州心识宇宙科技有限公司 | Intelligent body interaction method, device, equipment and storage medium based on article identification |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108171748A (en) * | 2018-01-23 | 2018-06-15 | 哈工大机器人（合肥）国际创新研究院 | A kind of visual identity of object manipulator intelligent grabbing application and localization method |
| CN110660104A (en) * | 2019-09-29 | 2020-01-07 | 珠海格力电器股份有限公司 | Industrial robot visual identification positioning grabbing method, computer device and computer readable storage medium |
-
2021
- 2021-08-20 CN CN202110962704.7A patent/CN113674341A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108171748A (en) * | 2018-01-23 | 2018-06-15 | 哈工大机器人（合肥）国际创新研究院 | A kind of visual identity of object manipulator intelligent grabbing application and localization method |
| CN110660104A (en) * | 2019-09-29 | 2020-01-07 | 珠海格力电器股份有限公司 | Industrial robot visual identification positioning grabbing method, computer device and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 朱静: "面向再制造拆卸产品的工业机器人视觉识别与定位研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 07, pages 138 - 734 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115546566A (en) * | 2022-11-24 | 2022-12-30 | 杭州心识宇宙科技有限公司 | Intelligent body interaction method, device, equipment and storage medium based on article identification |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110706293B (en) | SURF feature matching-based electronic component positioning and detecting method | |
| CN111739020B (en) | Automatic labeling method, device, equipment and medium for periodic texture background defect label | |
| CN113838015B (en) | Electrical product appearance defect detection method based on network cooperation | |
| JP2023145545A5 (en) | ||
| CN110865077B (en) | Visual inspection system for appearance defects in RFID antenna production | |
| CN115330734A (en) | Automatic robot repair welding system based on three-dimensional target detection and point cloud defect completion | |
| Chen et al. | A comprehensive review of deep learning-based PCB defect detection | |
| CN112767354A (en) | Defect detection method, device and equipment based on image segmentation and storage medium | |
| CN113674341A (en) | Robot visual identification and positioning method, intelligent terminal and storage medium | |
| CN114677567B (en) | Model training method and device, storage medium and electronic equipment | |
| CN114445651A (en) | Training set construction method and device of semantic segmentation model and electronic equipment | |
| CN114445410A (en) | Circuit board detection method based on image recognition, computer and readable storage medium | |
| CN117173125A (en) | Panoramic view-based defect point location display method, device and storage medium | |
| CN112070762A (en) | Mura defect detection method and device for liquid crystal panel, storage medium and terminal | |
| CN116778458B (en) | Parking space detection model construction method, parking space detection method, equipment and storage medium | |
| CN113762286A (en) | Data model training method, device, equipment and medium | |
| CN117557565A (en) | Detection method and device for lithium battery pole piece | |
| CN113222025A (en) | Feasible region label generation method based on laser radar | |
| CN113688897B (en) | Full-automatic disassembly method for shared bicycle, intelligent terminal and storage medium | |
| CN117557538A (en) | PCB surface defect detection method, device, computer equipment and storage medium | |
| CN116994049A (en) | Full-automatic flat knitting machine and method thereof | |
| CN116664519A (en) | Welding defect detection method, equipment and storage medium based on weak supervision learning | |
| CN116385353A (en) | Camera module abnormality detection method | |
| CN115346270A (en) | Traffic police gesture recognition method and device, electronic equipment and storage medium | |
| CN114897797A (en) | Method, device and equipment for detecting defects of printed circuit board and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211119 |
|
| RJ01 | Rejection of invention patent application after publication |