CN108453739B - Stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting - Google Patents
Stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting Download PDFInfo
- Publication number
- CN108453739B CN108453739B CN201810296121.3A CN201810296121A CN108453739B CN 108453739 B CN108453739 B CN 108453739B CN 201810296121 A CN201810296121 A CN 201810296121A CN 108453739 B CN108453739 B CN 108453739B
- Authority
- CN
- China
- Prior art keywords
- target object
- hsv
- image
- mechanical arm
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Image Processing (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting, wherein the system comprises: the system comprises a binocular image acquisition module, an upper computer image processing module and a mechanical arm movement module; the binocular image acquisition module is used for acquiring images containing a target object in real time and transmitting the images containing the target object to the upper computer image processing module; the upper computer image processing module is used for processing images containing target objects; the mechanical arm motion module is used for receiving the space position coordinates of the target object calculated by the digital image processing module and realizing the grabbing action of the mechanical arm. The system improves a binocular stereoscopic vision positioning method using the surface centroid of the target object as a characteristic point through HSV automatic threshold segmentation and Hu-moment-based shape fitting, and has the characteristics of high measurement robustness, automatic thresholding parameter optimization, simple structure and low cost under different illumination conditions.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting.
Background
The mechanical arm senses the environment by means of a sensor and guides the mechanical arm to move by utilizing environment information so as to complete a certain work task. Wherein perceiving information from the complex environment is the basis for the achievement of the series of actions. Along with the reduction of hardware cost and the improvement of computer performance, a vision sensor represented by a camera can provide abundant and comprehensive real-time information for the motion of a mechanical arm, and becomes a commonly used sensing device. The machine vision is used as a branch of artificial intelligence, images collected by a camera can be processed through a digital technology, and information which has guiding significance on the movement of the mechanical arm is extracted and described by mathematical symbols.
The binocular stereo vision is a machine vision form, and three-dimensional space position information of an object is obtained by calculating position deviation of corresponding points of two images according to two-dimensional images of the same scene at the same time, which are acquired from two different positions, by utilizing a parallax principle of an ideal parallel binocular vision imaging model. The binocular stereo vision positioning method using the centroid of the surface of the target object as the corresponding characteristic point has the following defects:
(1) the extracted edge of the surface profile of the object is susceptible to change under the influence of illumination, so that the centroid positioning is inaccurate, and the spatial position coordinate precision obtained by the reprojection calculation is influenced;
(2) the positioning adaptability to objects with different shapes is poor, and parameters need to be adjusted repeatedly according to target objects with different shapes.
The Hu moment is a highly condensed image statistical feature with translation, rotation and scaling invariance, also known as Hu invariant moment. The Hu moment was originally proposed by Ming-Kuei Hu in 1962 as a typical image feature applied in digital image processing techniques.
Disclosure of Invention
Aiming at the technical problems in the related art, the invention provides a stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting, which improve a binocular stereoscopic vision positioning method using a surface centroid of a target object as a characteristic point through an HSV automatic threshold segmentation algorithm and a Hu moment-based shape fitting algorithm, and have the characteristics of high measurement robustness, automatic thresholding parameter optimization, simple structure and low cost under different illumination conditions.
In order to achieve the technical purpose, the technical scheme of the invention is as follows:
a stereoscopic vision positioning mechanical arm grabbing system based on automatic shape fitting comprises: the system comprises a binocular image acquisition module, an upper computer image processing module and a mechanical arm movement module;
the binocular image acquisition module is used for acquiring images containing a target object in real time and transmitting the images containing the target object to the upper computer image processing module;
the upper computer image processing module is used for carrying out image processing on the image containing the target object, and further comprises a digital image processing module and an HSV thresholding parameter adjusting module;
the HSV thresholding parameter adjusting module is used for setting HSV thresholding parameters, and the HSV thresholding parameters comprise an initial upper limit, an initial optimization step length and an optimal threshold searching range;
the digital image processing module is used for processing the image containing the target object and comprises an image preprocessing unit, an HSV automatic threshold segmentation unit, a morphological filtering unit, a Hu moment-based shape fitting unit, a geometric figure centroid calculation unit and a binocular stereo vision re-projection unit;
the Hu moment-based shape fitting unit is used for reducing the actual shape of the target object, and multiple Hu moments are used as the characteristics of geometric shapes to form multiple geometric shape templates; matching the actual shape of the target object with the plurality of geometric shape templates, calculating the Hu moment similarity between the actual shape of the target object and the plurality of geometric shape templates, and fitting the actual shape of the target object by using the geometric shape template with the most similar Hu moment characteristic;
the mechanical arm motion module is used for receiving the space position coordinates of the target object calculated by the digital image processing module and realizing the grabbing action of the mechanical arm.
The binocular image acquisition module further comprises a binocular camera and a UVC communication protocol;
the binocular camera adopts a binocular camera supporting a UVC communication protocol, acquires an image containing a target object at a frequency of 30Hz, and temporarily stores the image containing the target object in the binocular camera in a YUV color space format;
the UVC communication protocol is used for transmitting the image containing the target object to the upper computer image processing module in real time through the UVC communication protocol.
The mechanical arm motion module further comprises a Bluetooth serial port module, a steering engine control module and a mechanical arm execution mechanism;
the Bluetooth serial port module is used for realizing the communication between the digital image processing module and the steering engine control module through Bluetooth connection;
the steering engine control module is used for solving the angle that each joint of the mechanical arm should rotate on the premise of completing the grabbing task based on the kinematic model of the mechanical arm execution mechanism and the spatial position coordinate of the target object calculated by the upper computer image processing module; each joint of the mechanical arm is driven by the pulse width modulation waveform with variable duty ratio generated by the rudder control module.
The mechanical arm executing mechanism is used for realizing the grabbing action of the mechanical arm according to the angle which is output by the steering engine control module and is required to rotate by each joint of the mechanical arm on the premise of completing the grabbing task; the mechanical arm executing mechanism is a multi-degree-of-freedom serial mechanical arm executing mechanism, all connecting rods are connected through revolute pairs, and rotation of all joints is controlled by the steering engine control module.
Further, the image preprocessing unit is used for performing stereo matching on two images containing the target object, which are acquired by the binocular camera at the same time point, so that the imaging process of the two images containing the target object, which are acquired at the same time point, meets an ideal parallel binocular vision imaging model; converting the image containing the target object from a YUV color space to an HSV color space to obtain a corresponding HSV image; and performing Gaussian smoothing filtering on the HSV image.
Further, the HSV automatic threshold segmentation unit is configured to perform automatic threshold segmentation on the HSV image, and specifically includes: and respectively optimizing the initial upper limit and the initial lower limit of the threshold of the HSV image by a given step length according to the HSV thresholding parameter, so that the numerical value of the upper limit and the lower limit with the maximum segmentation result area of the HSV image is the optimal threshold, and performing threshold segmentation on the HSV image by using the optimal threshold.
Further, the binocular stereo vision re-projection unit is configured to calculate and obtain a spatial position coordinate of the target object, specifically: based on the parallax principle of an ideal parallel binocular vision imaging model, the surface geometric centroid of the target object is used as a feature point of binocular vision stereo matching, and the reprojection calculation is performed according to the position difference of the feature point in two images containing the target object, acquired by the binocular camera at the same time point, obtained by stereo matching, so as to obtain the spatial position coordinate of the target object.
The invention provides a stereoscopic vision positioning mechanical arm grabbing method based on automatic shape fitting, which comprises the following steps of:
s1: configuring HSV thresholding parameters, the HSV thresholding parameters including: initial upper and lower limits, optimizing step length and optimal threshold value searching range;
s2: capturing an image by using a binocular camera to obtain an image containing a target object;
s3: carrying out image preprocessing on an image containing a target object;
s4: performing HSV automatic threshold segmentation according to the HSV thresholding parameters to obtain a binary image;
s5: performing morphological filtering on the binary image by adopting a morphological opening operation method;
s6: obtaining a fitted graph by adopting a shape fitting algorithm based on the Hu moment;
s7: calculating the centroid of the fitted graph by adopting a centroid calculation method of a plane geometric graph to obtain the position coordinates of the centroid of the fitted graph in the binary image;
s8: calculating binocular stereoscopic vision reprojection to obtain a spatial position coordinate of the target object;
s9: and the mechanical arm finishes the grabbing action according to the space position coordinates of the target object.
Further, in S3, the image preprocessing step includes:
performing stereo matching on two images containing the target object, which are acquired by the binocular camera at the same time point, so that the imaging process of the two images containing the target object, which are acquired at the same time point, meets an ideal parallel binocular vision imaging model;
converting the image containing the target object from a YUV color space to an HSV color space to obtain a corresponding HSV image;
and performing Gaussian smoothing filtering on the HSV image.
Further, in S4, the HSV automatic threshold segmentation step includes:
determining HSV thresholds required by the HSV automatic threshold segmentation, wherein the HSV thresholds comprise an HSV threshold upper limit and an HSV threshold lower limit, and the HSV threshold upper limit and the HSV threshold lower limit further respectively comprise an H value, an S value and a V value, namely the HSV thresholds comprise an H value upper limit, an S value upper limit, a V value upper limit, an H value lower limit, an S value lower limit and a V value lower limit;
optimizing the HSV threshold values in sequence, carrying out threshold value segmentation, and taking the HSV threshold value with the largest graphic area of a segmentation result as the optimal HSV threshold value;
and performing threshold segmentation on the HSV image containing the target object by using the optimal HSV threshold to obtain a binary image.
Further, in S6, the Hu moment-based shape fitting algorithm includes:
extracting the maximum graph contour of the target object in the binary image subjected to morphological filtering to serve as a graph to be fitted;
calculating a plurality of Hu moment features of the graph to be fitted;
calculating Euclidean distances of multi-Hu moment feature vectors between the graph to be fitted and a plurality of geometric shape graph templates stored in advance;
and fitting the graph to be fitted by using a geometric graph template corresponding to the minimum Euclidean distance to obtain the fitted graph.
Further, in S8, the binocular stereo-vision re-projection calculation specifically includes: based on the parallax principle of an ideal parallel binocular vision imaging model, the surface geometric centroid of the target object is used as a feature point of binocular vision stereo matching, and the reprojection calculation is performed according to the position difference of the feature point in two images containing the target object, acquired by the binocular camera at the same time point, obtained by stereo matching, so as to obtain the spatial position coordinate of the target object.
The invention has the beneficial effects that:
(1) in the invention, the contour of a target image is identified by adopting a Hu moment-based shape fitting algorithm in a digital image processing module to obtain an optimal fitting geometric figure, and the centroid of the fitting geometric figure of a target object is extracted. Matching the geometric shape template with the actual shape of the target object through the Hu moments, taking the geometric shape template with the highest Hu moment feature similarity as the actual shape of the target object, and calculating to obtain the geometric centroid of the target object. The algorithm overcomes the defects of chromaticity change, color distortion at the edge of the outline and the like of a target object under the condition of uneven illumination, improves the accuracy of target object detection and the robustness in an illumination scene, and has higher application value in the fields of image recognition, vision measurement, security and the like.
(2) According to the method, an HSV automatic threshold segmentation algorithm is adopted in the aspect of image segmentation, threshold distribution of a target object in an HSV space is obtained through automatic HSV threshold optimization, binarization image segmentation is carried out, image noise reduction is carried out by Gaussian smoothing filtering, and a better preprocessing result is obtained. The algorithm overcomes the defects of large noise and large parameter adjusting workload of the existing image segmentation technology, has the advantages of strong practicability and high accuracy, and has certain guiding significance on engineering practices in the fields of image recognition, object detection, security protection and the like.
(3) The system has multiple functions, good expansibility, lower cost and wider application range, and has the cross-platform operation characteristic.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic structural diagram of a stereoscopic vision positioning mechanical arm grabbing system based on automatic shape fitting according to the invention;
FIG. 2 is a flow chart of a method for grabbing a mechanical arm for stereoscopic vision positioning based on automatic shape fitting according to the invention;
FIG. 3 is a flow chart of HSV threshold optimization according to the present invention;
FIG. 4 is a flow chart of a Hu moment based shape fitting algorithm according to the present invention;
FIG. 5 is two images of a target object captured by a binocular camera at the same point in time;
FIG. 6 is a calculation result of image pre-processing;
fig. 7 is the centroid calculation result of the fitted graph.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
As shown in fig. 1, the stereoscopic vision positioning mechanical arm grabbing system based on automatic shape fitting of the present invention includes: the system comprises a binocular image acquisition module, an upper computer image processing module and a mechanical arm movement module; solid arrows in fig. 1 indicate the composition and logical relationship of each module of the system, and broken line crop heads indicate the direction of information transfer between each module when the system is in operation.
The binocular image acquisition module is used for acquiring images containing a target object in real time and transmitting the images containing the target object to the upper computer image processing module;
the binocular image acquisition module further comprises a binocular camera and a UVC communication protocol;
the binocular camera adopts a CMOS (complementary metal oxide semiconductor) binocular camera supporting a UVC (universal video coding) communication protocol, acquires an image containing a target object at the frequency of 30Hz, the pixel of the image is 320 multiplied by 200, and temporarily stores the image containing the target object in the binocular camera in a YUV (YUV) color space format; the CMOS binocular camera is connected with a USB3.0 interface of an upper computer through a USB data line, and images shot in real time are transmitted to the upper computer through a UVC communication protocol and used as input of an image processing module of the upper computer;
the UVC communication protocol is used for transmitting the image containing the target object to the upper computer image processing module in real time through the UVC communication protocol.
The upper computer image processing module further comprises a digital image processing module and an HSV thresholding parameter adjusting module; in the upper computer image processing module, the upper computer runs a Windows10 operating system, and the Visual Studio 2013 integrated development environment is used for compiling, linking, debugging and running the algorithm programs of the digital image processing module and the HSV thresholding parameter adjusting module. The OpenCV cross-platform computer vision library is used as a basic tool for program development. Image data is captured from a binocular camera using the DirectShow development kit for streaming media on a Windows platform.
The HSV thresholding parameter adjusting module is used for setting HSV thresholding parameters which comprise initial upper and lower limits, optimizing step length and optimal threshold searching range.
The digital image processing module is used for processing an image containing a target object and comprises an image preprocessing unit, an HSV automatic threshold segmentation unit, a morphological filtering unit, a Hu moment-based shape fitting unit, a geometric figure centroid calculation unit and a binocular stereo vision re-projection unit.
The image preprocessing unit is used for carrying out stereo matching on two images containing the target object, which are acquired by the binocular camera at the same time point, so that the imaging process of the two images containing the target object, which are acquired at the same time point, meets an ideal parallel binocular vision imaging model; converting the image containing the target object from a YUV color space to an HSV color space to obtain a corresponding HSV image; and performing Gaussian smooth filtering on the HSV image.
The HSV automatic threshold segmentation unit is used for carrying out automatic threshold segmentation on the HSV image, and specifically comprises the following steps: and respectively optimizing the initial upper limit and the initial lower limit of the threshold of the HSV image by a given step length according to the HSV thresholding parameters, so that the numerical value of the upper limit and the lower limit with the maximum segmentation result area of the HSV image is the optimal threshold, and performing threshold segmentation on the HSV image by using the optimal threshold.
The morphological filtering unit is carried out by adopting a morphological opening operation method, namely, the same template set is used for carrying out corrosion operation on the object set, and then expansion operation is carried out, so that the effects of smoothing the outline of the object and eliminating the parts protruding out of surrounding points can be achieved.
The Hu moment-based shape fitting unit is used for reducing the actual shape of a target object, and a plurality of Hu moments are used as the characteristics of geometric shapes to form a plurality of geometric shape templates; matching the actual shape of the target object with the templates of various geometric shapes, calculating the similarity between the actual shape of the target object and the Hu moments of the templates of various geometric shapes, and fitting the actual shape of the target object by using the template of the geometric shape with the most similar Hu moment characteristics; the actual shape of the target object is restored to the maximum extent, and the influence of the deformation of the actual shape of the target object in the image on the geometric centroid position of the target object caused by factors such as light intensity, shadow and shielding is effectively reduced.
The geometric image centroid calculation unit calculates the position coordinates of the centroid of the fitted graph in the image by using a centroid calculation method of the plane geometric graph.
The binocular stereoscopic vision reprojection unit is used for calculating and obtaining the space position coordinates of a target object, and specifically comprises the following steps: based on the parallax principle of ideal parallel binocular vision, the surface geometric centroid of a target object is used as a characteristic point of binocular vision stereo matching, and the spatial position coordinate of the target object is obtained by carrying out re-projection and calculation according to the position difference of the characteristic point in two images containing the target object, which are acquired by a binocular camera at the same time point and obtained by stereo matching.
In the mechanical arm motion module, the Bluetooth serial port module is connected with an upper computer with a Bluetooth connection function by using an HC-06 wireless transparent transmission module, and is simulated to be a serial port device to communicate with the upper computer. The steering engine control module is designed by using an embedded open-source hardware Arduino platform, runs a mechanical arm control algorithm consisting of a space coordinate conversion algorithm, a mechanical arm kinematics model and a steering engine control algorithm, and is powered by using a 9V voltage-stabilizing switching power supply. Arduino uses a low-cost microprocessor (ATmega8 or ATmega128), integrates development environments similar to Java and C languages, and has short development period and low cost. The rudder control module is connected with the Bluetooth serial port module through a universal UART (universal asynchronous receiver/transmitter) serial data bus, serial port communication is carried out through a universal asynchronous serial port communication protocol, and spatial position information of a target object, which is obtained by the Bluetooth serial port module from the upper computer image processing module, is received. The steering engine control module generates a pulse width modulation waveform with a variable duty ratio to drive the joint steering engine to rotate by a specified angle. The mechanical arm executing mechanism is a multi-degree-of-freedom series mechanical arm, all connecting rods are connected through revolute pairs, and the rotation of an output shaft of each joint steering engine is controlled and driven by a steering engine control module.
The operation flow of the system is as follows:
(1) the device is started after being electrified, the CMOS binocular camera starts to shoot a scene containing a target object, and the acquired image is shown in figure 5; the mechanical arm movement module enters a grabbing initial position;
(2) manually setting HSV thresholding parameters of the HSV thresholding parameter adjusting module: initial upper and lower limits, optimizing step length and optimal step length searching range;
(3) after the setting is finished, the upper computer executes a digital image processing program, preprocesses the image, and automatically optimizes the required HSV threshold value according to the set HSV thresholding parameter, wherein the image preprocessing calculation result is shown in FIG. 6;
(4) carrying out binarization segmentation on the image according to the obtained optimal threshold value to obtain a binary image containing the graphic profile information of the target object;
(5) performing morphological filtering on the binary image, and smoothing the contour of the target object by using morphological opening operation;
(6) extracting multi-Hu moment characteristics of the maximum graph contour of the target object in the binary image, comparing the obtained value with Hu moment characteristics of a large number of geometric graph templates, and fitting the graph contour of the target object by using the geometric graph template with the most similar multi-Hu moment characteristics, wherein the fitting result is shown in FIG. 7;
(7) calculating the centroid position of the fitted graph, and performing binocular vision reprojection calculation by using the centroid position as a feature point to obtain a spatial position coordinate of the target object relative to a camera coordinate system;
(8) the steering engine control module receives the space position coordinates of the target object transmitted by the upper computer through the Bluetooth serial port, converts the space position coordinates into position coordinates of the target object relative to a mechanical arm coordinate system, and performs inverse kinematics solution to obtain the angle which each joint of the mechanical arm should rotate when finishing the target object grabbing task;
(9) and the steering engine control module drives the steering engine to move at a specified speed and a specified rotation angle, so that the mechanical arm executing mechanism moves to a target pose, a target object is grabbed and returns to the initial grabbing position to prepare for completing the next grabbing.
As shown in fig. 2, another aspect of the present invention provides a stereoscopic positioning mechanical arm grabbing method based on automatic shape fitting, including the following steps:
s1: configuring HSV thresholding parameters, wherein the HSV thresholding parameters comprise: initial upper and lower limits, optimizing step length and optimal threshold value searching range; opening the CMOS binocular camera and establishing Bluetooth connection;
s2: capturing an image by using a binocular camera to obtain an image containing a target object;
s3: carrying out image preprocessing on an image containing a target object;
in the image preprocessing, the parameters of a binocular camera obtained by experiments in advance are utilized to carry out stereo matching on a left image and a right image acquired by a CMOS binocular camera at the same time point, so that the imaging process of the left image and the right image which are acquired at the same time point and contain a target object meets an ideal parallel binocular vision imaging model; converting the image containing the target object from a YUV color space to an HSV color space to obtain a corresponding HSV image; and performing Gaussian smooth filtering on the HSV image to weaken noise which may be generated in the transmission process of the image.
S4: performing HSV automatic threshold segmentation according to the HSV thresholding parameters to obtain a binary image;
the HSV automatic threshold segmentation is that the initial upper limit and the initial lower limit of the HSV threshold are respectively explored by given step length on the premise of giving the initial upper limit and the initial lower limit of the threshold, optimizing step length and optimal threshold search range, so that the upper limit value and the lower limit value with the largest image segmentation result area are the optimal threshold. The HSV thresholds needed for segmenting the thresholds of the image include an HSV threshold upper limit and an HSV threshold lower limit, which further include an H value, an S value and a V value, respectively, that is, the HSV thresholds include an H value upper limit, an S value upper limit, a V value upper limit, an H value lower limit, an S value lower limit and a V value lower limit, so that the HSV thresholds include 6 integer values in total. As shown in fig. 3, the HSV automatic threshold segmentation algorithm optimizes the above 6 integer values in sequence, and when optimizing a certain value, fixes other values as the determined optimal value or the designated initial value, and within the optimal threshold search range of the value, increases the designated optimization step length from the minimum value satisfying the optimal threshold search range, and obtains a plurality of trial values until the trial value exceeds the optimal threshold search range. On the premise that other values are fixed, threshold value division is performed once by using each trial value, the graph area of the division result is calculated, and the trial value which enables the graph area value of the division result to be maximum is fixed as an optimal value. After all 6 integer values are optimized, the optimal HSV threshold required by image segmentation is obtained, the optimal HSV threshold is used for carrying out threshold segmentation on the image, and the binary image of the optimal threshold segmentation result under the current condition is obtained.
S5: the binary image is subjected to morphological filtering by adopting a morphological opening operation method, namely, the same template set is used for carrying out corrosion operation on the object set, and then expansion operation is carried out, so that the effects of smoothing the object contour and eliminating the parts protruding out of surrounding points can be achieved.
S6: obtaining a fitted graph by adopting a shape fitting algorithm based on the Hu moment;
as shown in fig. 4, the shape fitting algorithm based on the Hu moment first extracts the graph contour of the target object in the binary image after morphological filtering, retains the contour with the largest contour area as the graph to be fitted, and eliminates the irrelevant contours with too small remaining areas. And calculating a plurality of Hu moment features of the graph to be fitted, and storing the result in a multi-dimensional vector as a multi-Hu moment feature vector of the shape contour graph. And calculating Euclidean distances between the multi-Hu-moment feature vectors and the multi-Hu-moment feature vectors of a large number of pre-stored geometric shape graph templates, so that the geometric shape graph template with the minimum Euclidean distance between the vectors is used as an optimal fitting template to fit the graph to be fitted. And (3) using a least square method to connect the outline of the geometric figure template to the figure to be fitted in an external mode to obtain the fitted figure.
S7: calculating the centroid of the fitted graph by adopting a centroid calculation method of a plane geometric graph to obtain the position coordinate of the centroid of the fitted graph in the binary image;
s8: calculating binocular stereo vision reprojection to obtain a spatial position coordinate of the target object;
the binocular stereo vision reprojection calculation specifically comprises the following steps: based on the parallax principle of ideal parallel binocular vision, the surface geometric centroid of a target object is used as a characteristic point of binocular vision stereo matching, and the spatial position coordinate of the target object is obtained by carrying out re-projection and calculation according to the position difference of the characteristic point in two images containing the target object, which are acquired by a binocular camera at the same time point and obtained by stereo matching.
S9: and the mechanical arm finishes the grabbing action according to the space position coordinates of the target object.
Outputting the space position coordinate information of the target object to a steering engine control module in the mechanical arm movement module through a communication serial port established by the Bluetooth serial port module; the steering engine control module calculates the rotation angle of each joint of the mechanical arm required for enabling the tail end of the mechanical arm execution mechanism to move to the space coordinate where the target object is located; and the steering engine control module generates pulse width modulation waveforms according to the calculation result to drive the steering engines of all joints of the mechanical arm execution mechanism to rotate by an appointed angle, so that the grabbing action of the target object is completed.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (2)
1. The utility model provides a stereovision location arm grasping system based on automatic shape fitting which characterized in that includes: the system comprises a binocular image acquisition module, an upper computer image processing module and a mechanical arm movement module;
the binocular image acquisition module is used for acquiring images containing a target object in real time and transmitting the images containing the target object to the upper computer image processing module;
the upper computer image processing module is used for carrying out image processing on the image containing the target object, and further comprises a digital image processing module and an HSV thresholding parameter adjusting module;
in the upper computer image processing module, the upper computer runs a Windows10 operating system, and compiles, links, debugs and runs algorithm programs of the digital image processing module and the HSV thresholding parameter adjusting module by using a Visual Studio 2013 integrated development environment;
the HSV thresholding parameter adjusting module is used for setting HSV thresholding parameters, and the HSV thresholding parameters comprise an initial upper limit, an initial optimization step length and an optimal threshold searching range;
the digital image processing module is used for processing the image containing the target object and comprises an image preprocessing unit, an HSV automatic threshold segmentation unit, a morphological filtering unit, a Hu moment-based shape fitting unit, a geometric figure centroid calculation unit and a binocular stereo vision re-projection unit;
the HSV automatic threshold segmentation unit is used for carrying out automatic threshold segmentation on the HSV image, and specifically comprises the following steps: optimizing initial upper and lower limits of a threshold of the HSV image by a given step length according to the HSV thresholding parameters, so that the numerical values of the upper and lower limits with the maximum segmentation result area of the HSV image are the optimal threshold, and performing threshold segmentation on the HSV image by using the optimal threshold;
the Hu moment-based shape fitting unit is used for reducing the actual shape of the target object, and multiple Hu moments are used as the characteristics of geometric shapes to form multiple geometric shape templates; matching the actual shape of the target object with the plurality of geometric shape templates, calculating the Hu moment similarity between the actual shape of the target object and the plurality of geometric shape templates, and fitting the actual shape of the target object by using the geometric shape template with the most similar Hu moment characteristic;
the mechanical arm motion module is used for receiving the spatial position coordinates of the target object calculated by the digital image processing module to realize the grabbing action of the mechanical arm;
the binocular image acquisition module further comprises a binocular camera and a UVC communication protocol;
the binocular camera adopts a binocular camera supporting a UVC communication protocol, acquires an image containing a target object at a frequency of 30Hz, and temporarily stores the image containing the target object in the binocular camera in a YUV color space format; the binocular camera is connected with the upper computer image processing module through a USB data line;
the UVC communication protocol is used for transmitting the image containing the target object to the upper computer image processing module in real time through the UVC communication protocol;
the mechanical arm motion module further comprises a Bluetooth serial port module, a steering engine control module and a mechanical arm execution mechanism;
the Bluetooth serial port module is used for realizing the communication between the digital image processing module and the steering engine control module through Bluetooth connection; the Bluetooth serial port module is connected with an upper computer with a Bluetooth connection function by using an HC-06 wireless transparent transmission module, and is simulated into a serial port device to be communicated with the upper computer;
the steering engine control module is used for solving the angle that each joint of the mechanical arm should rotate on the premise of completing the grabbing task based on the kinematic model of the mechanical arm execution mechanism and the spatial position coordinate of the target object calculated by the upper computer image processing module; each joint of the mechanical arm is driven by a pulse width modulation waveform with variable duty ratio generated by the rudder control module; the steering engine control module is designed by using an embedded open-source hardware Arduino platform, runs a mechanical arm control algorithm consisting of a space coordinate conversion algorithm, a mechanical arm kinematics model and a steering engine control algorithm, and is powered by using a 9V voltage-stabilizing switching power supply; the rudder control module is connected with the Bluetooth serial port module by a UART universal serial data bus, carries out serial port communication by a universal asynchronous serial port communication protocol and receives the spatial position information of the target object obtained by the Bluetooth serial port module from the upper computer image processing module;
the mechanical arm executing mechanism is used for realizing the grabbing action of the mechanical arm according to the angle which is output by the steering engine control module and is required to rotate by each joint of the mechanical arm on the premise of completing the grabbing task; the mechanical arm executing mechanism is a multi-degree-of-freedom serial mechanical arm executing mechanism, all connecting rods are connected through revolute pairs, and the rotation of all joints is controlled by the steering engine control module;
the image preprocessing unit is used for carrying out stereo matching on two images containing the target object, which are acquired by the binocular camera at the same time point, so that the imaging process of the two images containing the target object, which are acquired at the same time point, meets an ideal parallel binocular vision imaging model; converting the image containing the target object from a YUV color space to an HSV color space to obtain a corresponding HSV image; performing Gaussian smoothing filtering on the HSV image;
the binocular stereoscopic vision re-projection unit is used for calculating and obtaining the space position coordinates of the target object, and specifically comprises the following steps: based on the parallax principle of ideal parallel binocular vision, the surface geometric centroid of the target object is used as a characteristic point of binocular vision stereo matching, and the reprojection calculation is carried out according to the position difference of the characteristic point in two images containing the target object, which are acquired by the binocular camera at the same time point and are obtained by stereo matching, so as to obtain the spatial position coordinate of the target object;
the Hu moment-based shape fitting unit is further configured to:
extracting the maximum graph contour of the target object in the binary image subjected to morphological filtering to serve as a graph to be fitted;
calculating a plurality of Hu moment features of the graph to be fitted;
calculating Euclidean distances of multi-Hu moment feature vectors between the graph to be fitted and a plurality of geometric shape graph templates stored in advance;
and fitting the graph to be fitted by using a geometric graph template corresponding to the minimum Euclidean distance to obtain the fitted graph.
2. A stereoscopic vision positioning mechanical arm grabbing method based on automatic shape fitting is characterized by comprising the following steps:
s1: configuring HSV thresholding parameters, the HSV thresholding parameters including: initial upper and lower limits, optimizing step length and optimal threshold value searching range;
s2: capturing an image by using a binocular camera to obtain an image containing a target object;
s3: carrying out image preprocessing on an image containing a target object;
s4: performing HSV automatic threshold segmentation according to the HSV thresholding parameters to obtain a binary image;
s5: performing morphological filtering on the binary image by adopting a morphological opening operation method;
s6: obtaining a fitted graph by adopting a shape fitting algorithm based on the Hu moment;
s7: calculating the centroid of the fitted graph by adopting a centroid calculation method of a plane geometric graph to obtain the position coordinates of the centroid of the fitted graph in the binary image;
s8: calculating binocular stereoscopic vision reprojection to obtain a spatial position coordinate of the target object;
s9: the mechanical arm finishes a grabbing action according to the space position coordinate of the target object;
in S3, the image preprocessing includes:
performing stereo matching on two images containing the target object, which are acquired by the binocular camera at the same time point, so that the imaging process of the two images containing the target object, which are acquired at the same time point, meets an ideal parallel binocular vision imaging model;
converting the image containing the target object from a YUV color space to an HSV color space to obtain a corresponding HSV image;
performing Gaussian smoothing filtering on the HSV image;
in S4, the HSV automatic threshold partitioning step includes:
determining HSV thresholds required by the HSV automatic threshold segmentation, wherein the HSV thresholds comprise an HSV threshold upper limit and an HSV threshold lower limit, and the HSV threshold upper limit and the HSV threshold lower limit further respectively comprise an H value, an S value and a V value, namely the HSV thresholds comprise an H value upper limit, an S value upper limit, a V value upper limit, an H value lower limit, an S value lower limit and a V value lower limit;
optimizing the HSV threshold values in sequence, carrying out threshold value segmentation, and taking the HSV threshold value with the largest graphic area of a segmentation result as the optimal HSV threshold value;
performing threshold segmentation on the HSV image containing the target object by using the optimal HSV threshold to obtain a binary image;
in S6, the Hu moment-based shape fitting algorithm includes:
extracting the maximum graph contour of the target object in the binary image subjected to morphological filtering to serve as a graph to be fitted;
calculating a plurality of Hu moment features of the graph to be fitted;
calculating Euclidean distances of multi-Hu moment feature vectors between the graph to be fitted and a plurality of geometric shape graph templates stored in advance;
fitting the graph to be fitted by using a geometric graph template corresponding to the minimum Euclidean distance to obtain a fitted graph;
in S8, the binocular stereo-vision reprojection calculation specifically includes: based on the parallax principle of ideal parallel binocular vision, the surface geometric centroid of the target object is used as a characteristic point of binocular vision stereo matching, and the reprojection calculation is performed according to the position difference of the characteristic point in two images containing the target object, acquired by the binocular camera at the same time point, obtained by stereo matching, so as to obtain the spatial position coordinate of the target object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810296121.3A CN108453739B (en) | 2018-04-04 | 2018-04-04 | Stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810296121.3A CN108453739B (en) | 2018-04-04 | 2018-04-04 | Stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108453739A CN108453739A (en) | 2018-08-28 |
| CN108453739B true CN108453739B (en) | 2021-03-12 |
Family
ID=63234555
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810296121.3A Active CN108453739B (en) | 2018-04-04 | 2018-04-04 | Stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108453739B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109767424B (en) * | 2018-12-13 | 2022-05-17 | 西安电子科技大学 | Binocular vision train water injection port detection and positioning method based on FPGA |
| TWI672206B (en) | 2018-12-19 | 2019-09-21 | 財團法人工業技術研究院 | Method and apparatus of non-contact tool center point calibration for a mechanical arm, and a mechanical arm system with said calibration function |
| CN110612813A (en) * | 2019-10-18 | 2019-12-27 | 江苏银宝农业科学研究院有限公司 | Leaf vegetable cutting system based on embedded visual platform |
| CN111325684B (en) * | 2020-02-01 | 2022-04-26 | 武汉大学 | Semi-automatic high-spatial-resolution remote sensing image extraction method for buildings with different shapes |
| CN116700724B (en) * | 2023-08-03 | 2023-10-20 | 江苏狄诺尼信息技术有限责任公司 | Interchange plane interactive graphic design system based on mode method |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902271A (en) * | 2012-10-23 | 2013-01-30 | 上海大学 | Binocular vision-based robot target identifying and gripping system and method |
| CN105729468B (en) * | 2016-01-27 | 2018-01-09 | 浙江大学 | A kind of robotic workstation based on the enhancing of more depth cameras |
| CN105894499B (en) * | 2016-03-25 | 2018-09-14 | 华南理工大学 | A kind of space object three-dimensional information rapid detection method based on binocular vision |
| CN106003036A (en) * | 2016-06-16 | 2016-10-12 | 哈尔滨工程大学 | Object grabbing and placing system based on binocular vision guidance |
| CN107192331A (en) * | 2017-06-20 | 2017-09-22 | 佛山市南海区广工大数控装备协同创新研究院 | A kind of workpiece grabbing method based on binocular vision |
-
2018
- 2018-04-04 CN CN201810296121.3A patent/CN108453739B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108453739A (en) | 2018-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108453739B (en) | Stereoscopic vision positioning mechanical arm grabbing system and method based on automatic shape fitting | |
| CN107767423B (en) | mechanical arm target positioning and grabbing method based on binocular vision | |
| CN111462154B (en) | Target positioning method and device based on depth vision sensor and automatic grabbing robot | |
| CN104156726B (en) | A kind of workpiece identification method and device based on geometric characteristic | |
| CN106909216B (en) | Kinect sensor-based humanoid manipulator control method | |
| CN102982557B (en) | Method for processing space hand signal gesture command based on depth camera | |
| Malima et al. | A fast algorithm for vision-based hand gesture recognition for robot control | |
| US8787663B2 (en) | Tracking body parts by combined color image and depth processing | |
| TWI394087B (en) | Method and apparatus for tracking target object | |
| CN111694428B (en) | Gesture and track remote control robot system based on Kinect | |
| CN103529855B (en) | A kind of object recognition and detection device of rotary adjustable binocular vision and the application at agriculture fruit picking mechanical thereof | |
| US11270461B2 (en) | System and method for posture sequence on video from mobile terminals | |
| Liu et al. | Combining background subtraction and three-frame difference to detect moving object from underwater video | |
| CN111085997A (en) | Capturing training method and system based on point cloud acquisition and processing | |
| CN103559491A (en) | Human body motion capture and posture analysis system | |
| CN112926503B (en) | Automatic generation method of grabbing data set based on rectangular fitting | |
| CN104268519A (en) | Image recognition terminal based on mode matching and recognition method of image recognition terminal | |
| Guo | Research of hand positioning and gesture recognition based on binocular vision | |
| CN203196911U (en) | Industrial CCD (Charge Coupled Device) based visual identification sorting system | |
| CN111784655A (en) | Underwater robot recovery positioning method | |
| CN112068705A (en) | Bionic robot fish interaction control method and system based on gesture recognition | |
| CN111241940B (en) | Remote control method of robot and human body boundary frame determination method and system | |
| CN114851209A (en) | Industrial robot working path planning optimization method and system based on vision | |
| CN113034526B (en) | Grabbing method, grabbing device and robot | |
| CN108983791A (en) | A kind of intelligence vehicle control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |