CN114332985A - Portrait profile intelligent drawing method based on double mechanical arm cooperation - Google Patents
Portrait profile intelligent drawing method based on double mechanical arm cooperation Download PDFInfo
- Publication number
- CN114332985A CN114332985A CN202111477658.8A CN202111477658A CN114332985A CN 114332985 A CN114332985 A CN 114332985A CN 202111477658 A CN202111477658 A CN 202111477658A CN 114332985 A CN114332985 A CN 114332985A
- Authority
- CN
- China
- Prior art keywords
- track
- arm
- image
- coordinate system
- mechanical arm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 210000000697 sensory organ Anatomy 0.000 claims abstract description 17
- 230000033001 locomotion Effects 0.000 claims abstract description 16
- 238000007781 pre-processing Methods 0.000 claims abstract description 13
- 238000010845 search algorithm Methods 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 17
- 238000013527 convolutional neural network Methods 0.000 claims description 15
- 238000012163 sequencing technique Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 230000005540 biological transmission Effects 0.000 claims description 6
- 230000000877 morphologic effect Effects 0.000 claims description 6
- 230000003068 static effect Effects 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 4
- 230000003044 adaptive effect Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 3
- 238000010924 continuous production Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 6
- 238000013135 deep learning Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 abstract description 4
- 238000000605 extraction Methods 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010422 painting Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a portrait outline intelligent drawing method based on double mechanical arms cooperation, which comprises the steps of inputting an original RGB color image into a deep learning face recognition model to obtain pixel coordinate information of five sense organs; carrying out image preprocessing operation to obtain a binary image; obtaining a contour image through binary image contour extraction; then dividing the outline image into a left part and a right part; arranging pixel points of the left half outline image and the right half outline image to form a plurality of discontinuous track sections; converting the pixel coordinates into coordinates under a mechanical arm tool coordinate system; adding mechanical arm posture information to the track segment coordinates to form a series of tracks which can be recognized by the mechanical arm; planning the track by using non-interference double-arm drawing motion; and transmitting the non-interference track to the two mechanical arms through a TCP/IP protocol to draw the portrait picture by the cooperation of the two mechanical arms. The method realizes the function of double mechanical arms cooperative drawing, has high drawing efficiency and vivid effect, and is convenient to show in occasions such as education and teaching.
Description
Technical Field
The invention relates to a portrait outline intelligent drawing method based on double-mechanical-arm cooperation, and belongs to the technical field of robot drawing.
Background
With the progress of science and technology, the artificial intelligence technology is rapidly developed, and in the modern times, the high and new technology is also used in more and more occasions, especially in combination with robots, so that the application range of the intelligent robot is very wide. Painting through the mechanical arm also gradually goes into the public visual field.
At present, most of mechanical arm painting is performed by a single arm to print pixel points line by line, and the image-taking pictures can not be drawn by imitating the same drawing sequence and method of a human painter. The double-arm robot is used less at present in the aspect of intelligent drawing, and through the cooperation of two arms, can demonstrate gesture more elegant than single-arm robot and draw the skill, and draw efficiently, be convenient for demonstrate in multiple occasions such as education and teaching.
Disclosure of Invention
In order to solve the problems in the prior art, the invention aims to overcome the defects in the prior art and provide the portrait outline intelligent drawing method based on the cooperation of the two mechanical arms. Obtaining pixel coordinates and a binary image of five sense organs of the portrait image by utilizing a deep learning face recognition model and an image preprocessing technology; obtaining a contour image by adopting a contour extraction technology, and providing an inverse/clockwise search algorithm to sort a plurality of pixel points of each track segment in the contour; coordinate information under a mechanical arm tool coordinate system is obtained through a coordinate system conversion matrix, the coordinate information and mechanical arm posture information are combined to form a series of track points which can be used for mechanical arm motion control, and data transmission between an upper computer and the mechanical arm is achieved through a TCP/IP protocol so that portrait image contour drawing is achieved. The method realizes the function of drawing by the cooperation of the two mechanical arms, has high drawing efficiency and vivid effect, and is convenient to show in occasions such as education and teaching.
In order to achieve the purpose of the invention, the invention adopts the following technical scheme:
an intelligent portrait outline drawing method based on cooperation of two mechanical arms comprises the following operation steps:
(1) inputting the original RGB color image into a convolutional neural network CNN to obtain coordinate information of the five sense organs of the portrait image in a pixel coordinate system, hereinafter referred to as a coordinate system I;
(2) carrying out image preprocessing operation on the original RGB color image to obtain a binary image;
(3) extracting the contour of the binary image to obtain the coordinate information (x) of each pixel point in the contour in a coordinate system Ii，yi，zi) The coordinate system I is generally a planar coordinate system, default ziIs 0;
(4) dividing the contour image into a left part and a right part by utilizing a straight line which passes through the central positions of the two eyes and is parallel to the Y axis in the coordinate system I;
(5) for the left half contour image, sequencing pixel points in each track by adopting a counterclockwise search algorithm; for the right half outline image, adopting a clockwise search algorithm to sort the pixel points in each track; thus, a plurality of discontinuous track sections are formed;
(6) converting coordinate information of all track segment pixel points in a coordinate system I into coordinate information in a mechanical arm tool coordinate system, hereinafter referred to as a coordinate system T, for track drawing;
(7) adding mechanical arm posture information to the coordinate information converted into the coordinate system T, and combining the coordinate information and the posture information in the coordinate system T to obtain a series of track points which can be used for mechanical arm motion control;
(8) planning the non-interference double-arm drawing motion based on the left arm priority principle; taking the current track section of the left arm as a reference, finding a first track section which does not interfere with the left arm from all track sections of the right arm, and if the first track section is found, simultaneously drawing the current track sections by the left mechanical arm and the right mechanical arm; if the current track segment is not found, the right arm waits at a safe position until the left arm finishes drawing the current track segment; then, drawing the next track segment of the left arm according to the process until all track segments of the left arm are drawn; finally, drawing the left track segment of the right arm;
(9) and the upper computer obtains the static IP address of the mechanical arm through a TCP/IP protocol, realizes data transmission and transmits all the track segments to the controllers of the left mechanical arm and the right mechanical arm for drawing.
Preferably, in step 1, the inputting the original RGB color image into the convolutional neural network CNN to obtain the position information of the five sense organs of the portrait image includes:
the convolutional neural network consists of three cascade networks and is used for face recognition, so that the position information of five sense organs can be obtained;
position information of five sense organs is represented by (x)1，y1，z1)，(x2，y2，z2)，，(x3，y3，z3)，(x4，y4，z4)，(x5，y5，z5) Iso-coordinate representation in which (x)1，y1，z1) Represents the center coordinates of the left eye, (x)2，y2，z2) Represents the center coordinate of the right eye, (x)3，y3，z3) Represents the center coordinates of the nose, (x)4，y4，z4) Coordinates representing the left mouth corner, (x)5，y5，z5) Coordinates representing the right mouth angle.
Preferably, in step 2, the obtaining a binary image by performing an image preprocessing operation on the original RGB color image includes:
the image preprocessing comprises background removal, style conversion, image graying, adaptive threshold processing and morphological operation.
Preferably, in the step 4, the dividing the contour image into left and right parts by using a straight line passing through the center positions of the two eyes and parallel to the Y axis in the coordinate system I includes:
the left part and the right part comprise a left half outline image and a right half outline image which are respectively composed of a plurality of discontinuous track sections, each track section is composed of a plurality of adjacent pixel points, and in order to enable the robot to draw according to the track sections, the pixel points in the track need to be sequenced one by one to form a plurality of track sections composed of sequential consecutive pixel points.
Preferably, in the step 5, for the left half contour image, a counterclockwise search algorithm is adopted to sort the pixel points in each track; for the right half contour image, adopting a clockwise search algorithm to sort the pixel points in each track, comprising:
in the process of sequencing a plurality of pixel points in each track section, because the mechanical arm draws each track section in a continuous process, a pen is dropped at the initial position of the track section j, a pen is lifted at the tail end position of the track section j, and then the mechanical arm moves to the initial position of the (j + 1) th track section to drop, 1 and 0 marking pen dropping and pen lifting actions are set at the initial position and the tail end position of each track section;
the anticlockwise searching algorithm firstly traverses each line in the left half outline image, finds a first black pixel point, the pixel value is 0, the pixel point is taken as a central point, the next black pixel point is searched from the right top in an 8-neighborhood according to the anticlockwise direction, and if the pixel point is found, the pixel point is added into the current track segment queue; if not, continuing searching the pixel points in the next row until all the pixel points of the left half outline image are searched, wherein the pixel value of the searched pixel point is set to be 255, and the traversal is represented.
Preferably, in step 6, the converting the coordinate information of all track segment pixel points in the coordinate system I into the coordinate information in the coordinate system T for track drawing includes:
conversion formula is
Wherein
A transformation matrix representing the coordinate system I to the coordinate system T,TPirepresenting the coordinate information of the pixel point i in the coordinate system T, specifically being (TXi,TYi,TZi)，IPiRepresenting coordinate information of a pixel I in a coordinate system IIs specifically (IXi,IYi,IZi)。
Preferably, in step 7, adding pose information of the robot arm to the coordinate information converted into the coordinate system T, and combining the coordinate information and the pose information in the coordinate system T to obtain a series of coordinate points usable for the movement of the robot arm includes:
the method is characterized in that fourteen-degree-of-freedom redundant double mechanical arms are used for drawing, so each mechanical arm has seven degrees of freedom respectively, and the coordinate point data format is (X)i,Yi,Zi,q1,q2,q3,q4) The first three items are coordinate information of the tail end of the mechanical arm in a coordinate system T, and the last four items are expressed in a quaternion form and are attitude information of the mechanical arm;
the two mechanical arms always keep the posture that the pen point is vertical to the paper surface in the drawing process, namely only X is arranged in the drawing processi,Yi,ZiThree values will change, attitude information q1,q2,q3,q4Always kept unchanged; the drawing gesture of the mechanical arm is obtained by teaching, some potential obstacles need to be avoided in the teaching process, and the mechanical arm is kept to be stable and uniform in the process of moving from the initial gesture to the drawing gesture; after the drawing pose of the mechanical arm is taught for the first time, recording q1,q2,q3,q4As a fixed parameter by the program and Xi,Yi,ZiAnd combining the coordinate information to obtain a data format for controlling the motion of the mechanical arm.
Compared with the prior art, the invention has the following obvious and prominent substantive characteristics and remarkable advantages:
1. the redundant double-arm robot with fourteen degrees of freedom is adopted to draw the figure of the zodiac picture, so that singular points can be well avoided, and the robot is as flexible as the arms of a person;
2. according to the invention, an artificial intelligence deep learning technology is combined with the robot, so that intelligent efficient collaborative drawing of the two mechanical arms is realized, and the application of the intelligent robot is more flexible;
3. according to the method, the original RGB color portrait is subjected to background removal, so that the interference of background pixel points on foreground pixels in the subsequent preprocessing step is eliminated, the binary image is clear in outline, and the final outline drawing effect is improved;
4. the reverse/clockwise search algorithm provided by the invention has the advantages of high speed and high efficiency in pixel point sequencing.
Drawings
Fig. 1 is a general flowchart of a portrait outline intelligent drawing method based on two robot arms cooperation according to a preferred embodiment of the present invention.
Fig. 2 is a schematic diagram of a pixel coordinate system of a preferred embodiment of the present invention.
Fig. 3 is a flow chart of the reverse/clockwise search algorithm of the preferred embodiment of the present invention.
Fig. 4-1 and 4-2 are schematic diagrams of the outline point arranging method according to the preferred embodiment of the present invention.
Detailed Description
The above-described scheme is further illustrated below with reference to specific embodiments, which are detailed below:
the first embodiment is as follows:
in this embodiment, referring to fig. 1, an intelligent portrait contour drawing method based on cooperation of two robots includes the following operation steps:
(1) inputting the original RGB color image into a convolutional neural network CNN to obtain coordinate information of the five sense organs of the portrait image in a pixel coordinate system, hereinafter referred to as a coordinate system I;
(2) carrying out image preprocessing operation on the original RGB color image to obtain a binary image;
(3) extracting the contour of the binary image to obtain the coordinate information (x) of each pixel point in the contour in a coordinate system Ii，yi，zi) The coordinate system I is generally a planar coordinate system, default ziIs 0;
(4) dividing the contour image into a left part and a right part by utilizing a straight line which passes through the central positions of the two eyes and is parallel to the Y axis in the coordinate system I;
(5) for the left half contour image, sequencing pixel points in each track by adopting a counterclockwise search algorithm; for the right half outline image, adopting a clockwise search algorithm to sort the pixel points in each track; thus, a plurality of discontinuous track sections are formed;
(6) converting coordinate information of all track segment pixel points in a coordinate system I into coordinate information in a mechanical arm tool coordinate system, hereinafter referred to as a coordinate system T, for track drawing;
(7) adding mechanical arm posture information to the coordinate information converted into the coordinate system T, and combining the coordinate information and the posture information in the coordinate system T to obtain a series of track points which can be used for mechanical arm motion control;
(8) planning the non-interference double-arm drawing motion based on the left arm priority principle; taking the current track section of the left arm as a reference, finding a first track section which does not interfere with the left arm from all track sections of the right arm, and if the first track section is found, simultaneously drawing the current track sections by the left mechanical arm and the right mechanical arm; if the current track segment is not found, the right arm waits at a safe position until the left arm finishes drawing the current track segment; then, drawing the next track segment of the left arm according to the process until all track segments of the left arm are drawn; finally, drawing the left track segment of the right arm;
(9) and the upper computer obtains the static IP address of the mechanical arm through a TCP/IP protocol, realizes data transmission and transmits all the track segments to the controllers of the left mechanical arm and the right mechanical arm for drawing.
The method realizes the function of drawing by the cooperation of the two mechanical arms, has high drawing efficiency and vivid effect, and is convenient to show in occasions such as education and teaching.
Example two:
this embodiment is substantially the same as the first embodiment, and is characterized in that:
in this embodiment, referring to fig. 1 to 4, an intelligent portrait contour drawing method based on cooperation of two robots includes the following steps:
1. inputting an original RGB color image (520 pixels multiplied by 390 pixels) into a convolutional neural network CNN to obtain coordinate information of a portrait image five sense organs in a pixel coordinate system, hereinafter referred to as a coordinate system I, and the specific method is as follows:
1.1 the convolutional neural network is composed of three cascaded networks, which are respectively used for roughly positioning the face position, finely positioning the face position, and outputting the coordinate information of the five sense organs, in the actual operation, firstly, the size of the RGB color image is adjusted to 520 x 390, then, the image is input into a model trained by deep learning for prediction, and the obtained output is the coordinate information of the five sense organs in a pixel coordinate system.
1.2 location information of five sense organs (x)1，y1，z1)，(x2，y2，z2)，，(x3，y3，z3)，(x4，y4，z4)，(x5，y5，z5) Iso-coordinate representation in which (x)1，y1，z1) Represents the center coordinates of the left eye, (x)2，y2，z2) Represents the center coordinate of the right eye, (x)3，y3，z3) Represents the center coordinates of the nose, (x)4，y4，z4) Coordinates representing the left mouth corner, (x)5，y5，z5) The coordinate representing the right mouth corner is a default value of 0 in the coordinate since the pixel coordinate system is generally a two-dimensional coordinate system.
2. The method comprises the following steps of carrying out image preprocessing operation on the original RGB color image to obtain a binary image, and specifically comprises the following steps:
2.1 the image preprocessing comprises the operations of background removal, style conversion, image graying, adaptive threshold processing, morphological operation and the like, wherein the style conversion firstly converts an RGB image into a YUV color space, carries out edge detection on a Y channel independently, calculates a horizontal component and a vertical component for each pixel, and then adopts a line integral convolution algorithm to filter the image to finally obtain the converted style image.
The morphological operation mainly adopts morphological closed operation, and in OpenCV, the morphological operation is acted on a pixel point with a pixel value of 255, so that the black part line can be thinned by adopting the closed operation, and the image denoising effect is realized.
The steps 1 and 2 are divided into two branches for parallel processing.
4. Dividing the contour image into a left part and a right part by utilizing a straight line passing through the central positions of the two eyes and being parallel to the Y axis in the coordinate system I, wherein the specific method comprises the following steps:
4.1 coordinate information of five sense organs in coordinate System I obtained by step 1.1, (x)1，y1，z1) Represents the center coordinates of the left eye, (x)2，y2，z2) The center coordinates of the right eye are expressed, so the center positions of both eyes are expressed as (x)c，yc) Wherein x isc＝x1+(x2-x1)/2，yc＝y1+(y2-y1) 2, by passing through point (x)c，yc) And a straight line parallel to the Y-axis in the coordinate system I divides the contour image into two parts.
The left part and the right part comprise a left half outline image and a right half outline image which are respectively composed of a plurality of discontinuous track sections, each track section is composed of a plurality of adjacent pixel points, in order to draw the robot according to the track sections, the pixel points in the track need to be sequenced one by one, a plurality of track sections composed of sequential consecutive pixel points are formed, if one outline is composed of m sections of tracks, each section of track is composed of n pixel points, and the specific formats of the outline and the track sections are as follows:
outline format: [ [1 (x) ]1，y1，z1)，(x2，y2，z2)，……，(xn，yn，zn)，0]1，[1，(x1，y1，z1)，(x2，y2，z2)，……，(xn，yn，zn)，0]2，……，[1，(x1，y1，z1)，(x2，y2，z2)，……，(xn，yn，zn)，0]m]
Track segment format: [1, (x)1，y1，z1)，(x2，y2，z2)，……，(xn，yn，zn)，0]
5. For the left half contour image, sequencing pixel points in each track by adopting a counterclockwise search algorithm; and for the right half outline image, sequencing the pixel points in each track by adopting a clockwise search algorithm. Therefore, a plurality of discontinuous track sections are formed, and the specific method is as follows:
5.1 in the process of sequencing a plurality of pixel points in each track segment, because the mechanical arm draws each track segment in a continuous process, the pen is dropped at the initial position of the track segment j, the pen is lifted at the tail end position of the track segment j, and then the mechanical arm moves to the initial position of the j +1 th track segment to drop, 1 and 0 marks are set at the initial position and the tail end position of each track segment to mark the pen dropping and lifting actions, and the actions specifically refer to the above outline format and the track segment format.
5.2, referring to fig. 4-1, first traversing each line in the left half outline image, finding a first black pixel point, wherein the pixel value is 0, the pixel coordinate is (i, j), and the pixel point is taken as a central point, starting from the right top in an 8-neighborhood, namely the pixel point with the pixel coordinate being (i-1, j), searching for a next black pixel point in the counterclockwise direction, and if the pixel point is found, adding the pixel point into the current track segment queue; if not, continuing to search the pixel points in the next row until all the pixel points of the left half outline image are searched, wherein the pixel value of the searched pixel point is set to be 255, which represents that the image is traversed, and the clockwise search algorithm is similar to the counterclockwise search algorithm with reference to fig. 4-2.
6. Converting coordinate information of all track segment pixel points in a coordinate system I into coordinate information in a mechanical arm tool coordinate system (hereinafter referred to as a coordinate system T) for track drawing, wherein the specific method comprises the following steps:
6.1 conversion of coordinate System I to coordinate System T is given by
Wherein
A transformation matrix representing the coordinate system I to the coordinate system T,TPirepresenting the coordinate information of the pixel point i in the coordinate system T, specifically being (TXi,TYi,TZi)，IPiRepresenting the coordinate information of the pixel point I in the coordinate system I, specifically being (IXi,IYi,IZi)。
7. Adding mechanical arm attitude information to the coordinate information converted into the coordinate system T, and combining the coordinate information and the attitude information in the coordinate system T to obtain a series of track points which can be used for mechanical arm motion control, wherein the specific method comprises the following steps:
7.1 the method is to draw by fourteen-degree-of-freedom redundant double mechanical arms, so each mechanical arm has seven degrees of freedom respectively, namely the pose of the tail end of the mechanical arm is represented by seven parameters, and the data format is (X)i,Yi,Zi,q1,q2,q3,q4) The first three items are coordinate information of the tail end of the mechanical arm in a coordinate system T and are rectangular coordinate information, and the last four items are represented in a quaternion form and are attitude information of the mechanical arm.
7.2 the two mechanical arms always keep the pen point vertical to the paper surface in the drawing process, namely only X is in the drawing processi,Yi,ZiThree values will change, attitude information q1,q2,q3,q4Always kept unchanged. The drawing gesture of the mechanical arm is obtained through teaching, potential obstacles need to be avoided in the teaching process, and the mechanical arm is kept stable and uniform in the process of moving from the initial gesture to the drawing gesture. After the drawing pose of the mechanical arm is taught for the first time, recording q1,q2,q3,q4As a fixed parameter by the program and Xi,Yi,ZiAnd combining the coordinate information to obtain a data format for controlling the motion of the mechanical arm.
8. And planning the non-interference double-arm drawing motion based on the left arm priority principle. Taking the current track section of the left arm as a reference, finding a first track section which does not interfere with the left arm from all track sections of the right arm, and if the first track section is found, simultaneously drawing the current track sections by the left mechanical arm and the right mechanical arm; if the current track segment is not found, the right arm waits at a safe position until the left arm finishes drawing the current track segment; then, drawing the next track segment of the left arm according to the process until all track segments of the left arm are drawn; and finally, drawing the left track segment of the right arm.
9. The upper computer obtains the static IP address of the mechanical arm through a TCP/IP protocol to realize data transmission, and transmits all the track segments to controllers of the left mechanical arm and the right mechanical arm for drawing, and the specific method comprises the following steps:
the upper computer is connected with the mechanical arm controller through a network cable, the upper computer actively establishes a communication relation with the mechanical arm, after a static IP address of the mechanical arm is obtained, the IP address is input into a program, and the program is operated to carry out data transmission.
In this embodiment, the portrait outline intelligent drawing method based on the cooperation of two mechanical arms includes the following operation steps: inputting an original RGB color image into a deep learning face recognition model to obtain pixel coordinate information of five sense organs; carrying out image preprocessing operation on the original RGB color image to obtain a binary image; extracting the outline of the binary image to obtain an outline image; dividing the contour image into a left part and a right part by utilizing a straight line passing through the central positions of the two eyes and being parallel to the Y axis of the pixel coordinate system; under a pixel coordinate system, respectively utilizing a counterclockwise search algorithm and a clockwise search algorithm to arrange pixel points of the left half outline image and the right half outline image to form a plurality of discontinuous track sections; converting the pixel coordinates of the track segment into coordinates under a mechanical arm tool coordinate system; adding mechanical arm posture information to the track segment coordinates to form a series of tracks which can be recognized by the mechanical arm; performing non-interference double-arm drawing motion planning on the track based on a left arm priority principle to ensure that the two mechanical arms do not interfere in the drawing process; and transmitting the non-interference track to the two mechanical arms through a TCP/IP protocol to draw the portrait picture by the cooperation of the two mechanical arms. The method realizes the function of drawing by the cooperation of the two mechanical arms, has high drawing efficiency and vivid effect, and is convenient to show in occasions such as education and teaching.
The embodiments of the present invention have been described with reference to the accompanying drawings, but the present invention is not limited to the embodiments, and various changes and modifications can be made according to the purpose of the invention, and any changes, modifications, substitutions, combinations or simplifications made according to the spirit and principle of the technical solution of the present invention shall be equivalent substitutions, as long as the purpose of the present invention is met, and the present invention shall fall within the protection scope of the present invention without departing from the technical principle and inventive concept of the present invention.
Claims (7)
1. An intelligent portrait outline drawing method based on double mechanical arm cooperation is characterized by comprising the following operation steps:
(1) inputting the original RGB color image into a convolutional neural network CNN to obtain coordinate information of the five sense organs of the portrait image in a pixel coordinate system, hereinafter referred to as a coordinate system I;
(2) carrying out image preprocessing operation on the original RGB color image to obtain a binary image;
(3) extracting the contour of the binary image to obtain the coordinate information (x) of each pixel point in the contour in a coordinate system Ii，yi，zi) The coordinate system I is generally a planar coordinate system, default ziIs 0;
(4) dividing the contour image into a left part and a right part by utilizing a straight line which passes through the central positions of the two eyes and is parallel to the Y axis in the coordinate system I;
(5) for the left half contour image, sequencing pixel points in each track by adopting a counterclockwise search algorithm; for the right half outline image, adopting a clockwise search algorithm to sort the pixel points in each track; thus, a plurality of discontinuous track sections are formed;
(6) converting coordinate information of all track segment pixel points in a coordinate system I into coordinate information in a mechanical arm tool coordinate system, hereinafter referred to as a coordinate system T, for track drawing;
(7) adding mechanical arm posture information to the coordinate information converted into the coordinate system T, and combining the coordinate information and the posture information in the coordinate system T to obtain a series of track points which can be used for mechanical arm motion control;
(8) planning the non-interference double-arm drawing motion based on the left arm priority principle; taking the current track section of the left arm as a reference, finding a first track section which does not interfere with the left arm from all track sections of the right arm, and if the first track section is found, simultaneously drawing the current track sections by the left mechanical arm and the right mechanical arm; if the current track segment is not found, the right arm waits at a safe position until the left arm finishes drawing the current track segment; then, drawing the next track segment of the left arm according to the process until all track segments of the left arm are drawn; finally, drawing the left track segment of the right arm;
(9) and the upper computer obtains the static IP address of the mechanical arm through a TCP/IP protocol, realizes data transmission and transmits all the track segments to the controllers of the left mechanical arm and the right mechanical arm for drawing.
2. The portrait outline intelligent drawing method based on double-mechanical-arm cooperation of claim 1, wherein the step of inputting the original RGB color image into a Convolutional Neural Network (CNN) to obtain position information of five sense organs of the portrait image comprises the steps of:
the convolutional neural network consists of three cascade networks and is used for face recognition, so that the position information of five sense organs can be obtained;
position information of five sense organs is represented by (x)1，y1，z1)，(x2，y2，z2),，(x3，y3，z3)，(x4，y4，z4)，(x5，y5，z5) Iso-coordinate representation in which (x)1，y1，z1) Represents the center coordinates of the left eye, (x)2，y2，z2) Represents the center coordinate of the right eye, (x)3，y3，z3) Represents the center coordinates of the nose, (x)4，y4，z4) Indicating left mouth cornerCoordinate (x)5，y5，z5) Coordinates representing the right mouth angle.
3. The portrait outline intelligent drawing method based on double-mechanical-arm cooperation as claimed in claim 1, wherein the subjecting the original RGB color image to an image preprocessing operation to obtain a binary image comprises:
the image preprocessing comprises background removal, style conversion, image graying, adaptive threshold processing and morphological operation.
4. The portrait outline intelligent drawing method based on double-mechanical-arm cooperation of claim 1, wherein the dividing of the outline image into a left part and a right part by using a straight line passing through the center positions of the two eyes and being parallel to the Y axis in the coordinate system I comprises:
the left part and the right part comprise a left half outline image and a right half outline image which are respectively composed of a plurality of discontinuous track sections, each track section is composed of a plurality of adjacent pixel points, and in order to enable the robot to draw according to the track sections, the pixel points in the track need to be sequenced one by one to form a plurality of track sections composed of sequential consecutive pixel points.
5. The portrait outline intelligent drawing method based on double-mechanical-arm cooperation of claim 1, wherein for the left half outline image, a counterclockwise search algorithm is adopted to sort the pixel points in each track; for the right half contour image, adopting a clockwise search algorithm to sort the pixel points in each track, comprising:
in the process of sequencing a plurality of pixel points in each track section, because the mechanical arm draws each track section in a continuous process, a pen is dropped at the initial position of the track section j, a pen is lifted at the tail end position of the track section j, and then the mechanical arm moves to the initial position of the (j + 1) th track section to drop, 1 and 0 marking pen dropping and pen lifting actions are set at the initial position and the tail end position of each track section;
the anticlockwise searching algorithm firstly traverses each line in the left half outline image, finds a first black pixel point, the pixel value is 0, the pixel point is taken as a central point, the next black pixel point is searched from the right top in an 8-neighborhood according to the anticlockwise direction, and if the pixel point is found, the pixel point is added into the current track segment queue; if not, continuing searching the pixel points in the next row until all the pixel points of the left half outline image are searched, wherein the pixel value of the searched pixel point is set to be 255, and the traversal is represented.
6. The portrait profile intelligent drawing method based on double-mechanical-arm cooperation as claimed in claim 1, wherein the converting coordinate information of all track segment pixel points in a coordinate system I into coordinate information in a coordinate system T for track drawing comprises:
conversion formula is
Wherein
A transformation matrix representing the coordinate system I to the coordinate system T,TPirepresenting the coordinate information of the pixel point i in the coordinate system T, specifically being (TXi,TYi,TZi)，IPiRepresenting the coordinate information of the pixel point I in the coordinate system I, specifically being (IXi,IYi,IZi)。
7. The portrait profile intelligent drawing method based on two-robot-arm cooperation of claim 1, wherein the adding of robot arm posture information to the coordinate information converted into the coordinate system T, and the combining of the coordinate information and the posture information in the coordinate system T to obtain a series of coordinate points available for robot arm movement comprises:
the method is characterized by fourteen-degree-of-freedom redundancyThe two mechanical arms are used for drawing, so each mechanical arm has seven degrees of freedom respectively, and the coordinate point data format is (X)i,Yi,Zi,q1,q2,q3,q4) The first three items are coordinate information of the tail end of the mechanical arm in a coordinate system T, and the last four items are expressed in a quaternion form and are attitude information of the mechanical arm;
the two mechanical arms always keep the posture that the pen point is vertical to the paper surface in the drawing process, namely only X is arranged in the drawing processi,Yi,ZiThree values will change, attitude information q1,q2,q3,q4Always kept unchanged; the drawing gesture of the mechanical arm is obtained by teaching, some potential obstacles need to be avoided in the teaching process, and the mechanical arm is kept to be stable and uniform in the process of moving from the initial gesture to the drawing gesture; after the drawing pose of the mechanical arm is taught for the first time, recording q1,q2,q3,q4As a fixed parameter by the program and Xi,Yi,ZiAnd combining the coordinate information to obtain a data format for controlling the motion of the mechanical arm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111477658.8A CN114332985A (en) | 2021-12-06 | 2021-12-06 | Portrait profile intelligent drawing method based on double mechanical arm cooperation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111477658.8A CN114332985A (en) | 2021-12-06 | 2021-12-06 | Portrait profile intelligent drawing method based on double mechanical arm cooperation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114332985A true CN114332985A (en) | 2022-04-12 |
Family
ID=81049335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111477658.8A Pending CN114332985A (en) | 2021-12-06 | 2021-12-06 | Portrait profile intelligent drawing method based on double mechanical arm cooperation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114332985A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105701437A (en) * | 2014-11-11 | 2016-06-22 | 沈阳新松机器人自动化股份有限公司 | Portrait drawing system based robot |
| CN107107342A (en) * | 2014-12-26 | 2017-08-29 | 川崎重工业株式会社 | The teaching system of dual-arm robot and the teaching method of dual-arm robot |
| CN108460369A (en) * | 2018-04-04 | 2018-08-28 | 南京阿凡达机器人科技有限公司 | A kind of drawing practice and system based on machine vision |
| CN110405730A (en) * | 2019-06-06 | 2019-11-05 | 大连理工大学 | A kind of man-machine object interaction mechanical arm teaching system based on RGB-D image |
| CN110900581A (en) * | 2019-12-27 | 2020-03-24 | 福州大学 | Four-degree-of-freedom mechanical arm vision servo control method and device based on RealSense camera |
-
2021
- 2021-12-06 CN CN202111477658.8A patent/CN114332985A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105701437A (en) * | 2014-11-11 | 2016-06-22 | 沈阳新松机器人自动化股份有限公司 | Portrait drawing system based robot |
| CN107107342A (en) * | 2014-12-26 | 2017-08-29 | 川崎重工业株式会社 | The teaching system of dual-arm robot and the teaching method of dual-arm robot |
| CN108460369A (en) * | 2018-04-04 | 2018-08-28 | 南京阿凡达机器人科技有限公司 | A kind of drawing practice and system based on machine vision |
| CN110405730A (en) * | 2019-06-06 | 2019-11-05 | 大连理工大学 | A kind of man-machine object interaction mechanical arm teaching system based on RGB-D image |
| CN110900581A (en) * | 2019-12-27 | 2020-03-24 | 福州大学 | Four-degree-of-freedom mechanical arm vision servo control method and device based on RealSense camera |
Non-Patent Citations (3)
| Title |
|---|
| 李寿杰;王艳;刘东凯;李郑;兴雷;: "悬挂式绘图机器人系统设计与实现", 电子设计工程, no. 03, 5 February 2020 (2020-02-05) * |
| 邓宗乾;李静;沈南燕;何舟;李英杰: "基于近红外双目系统的目标定位软件设计", 《工业控制计算机》, 23 September 2021 (2021-09-23) * |
| 黄鑫: "敏捷型辅助搏击机器人研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, 30 June 2021 (2021-06-30), pages 140 - 283 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280856B (en) | Unknown object grabbing pose estimation method based on mixed information input network model | |
| CN108491880B (en) | Object classification and pose estimation method based on neural network | |
| CN107679491B (en) | 3D convolutional neural network sign language recognition method fusing multimodal data | |
| CN109108942A (en) | The mechanical arm motion control method and system of the real-time teaching of view-based access control model and adaptive DMPS | |
| CN103646249B (en) | A kind of greenhouse intelligent mobile robot vision navigation path identification method | |
| CN111968217A (en) | SMPL parameter prediction and human body model generation method based on picture | |
| CN106909216A (en) | A kind of Apery manipulator control method based on Kinect sensor | |
| CN109746916B (en) | Method and system for robot to write calligraphy | |
| CN111890357A (en) | Intelligent robot grabbing method based on action demonstration teaching | |
| CN114714365B (en) | Disordered workpiece grabbing method and system based on cloud platform | |
| CN110555404A (en) | Flying wing unmanned aerial vehicle ground station interaction device and method based on human body posture recognition | |
| Chen et al. | Combining reinforcement learning and rule-based method to manipulate objects in clutter | |
| WO2022228391A1 (en) | Terminal device positioning method and related device therefor | |
| CN115147488A (en) | Workpiece pose estimation method based on intensive prediction and grasping system | |
| CN116500901A (en) | Digital twin-driven man-machine cooperation task planning method under condition of unknown user intention | |
| CN111814823A (en) | Transfer learning method based on scene template generation | |
| CN108052901A (en) | A kind of gesture identification Intelligent unattended machine remote control method based on binocular | |
| CN113284184A (en) | Robot RGBD visual perception oriented 6D pose estimation method and system | |
| CN114332985A (en) | Portrait profile intelligent drawing method based on double mechanical arm cooperation | |
| CN115861780B (en) | Robot arm detection grabbing method based on YOLO-GGCNN | |
| CN112884799A (en) | Target tracking method in complex scene based on twin neural network | |
| CN116977506A (en) | Model action redirection method, device, electronic equipment and storage medium | |
| CN114998573A (en) | Grabbing pose detection method based on RGB-D feature depth fusion | |
| Ouyang et al. | Robot grasp with multi-object detection based on RGB-D image | |
| CN111950341B (en) | Real-time gesture recognition method and gesture recognition system based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |