CN111582186A - Object edge identification method, device, system and medium based on vision and touch - Google Patents
Object edge identification method, device, system and medium based on vision and touch Download PDFInfo
- Publication number
- CN111582186A CN111582186A CN202010393054.4A CN202010393054A CN111582186A CN 111582186 A CN111582186 A CN 111582186A CN 202010393054 A CN202010393054 A CN 202010393054A CN 111582186 A CN111582186 A CN 111582186A
- Authority
- CN
- China
- Prior art keywords
- edge
- information
- algorithm
- target
- tactile
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 43
- 230000000007 visual effect Effects 0.000 claims abstract description 30
- 239000013598 vector Substances 0.000 claims description 28
- 230000009471 action Effects 0.000 claims description 23
- 238000005070 sampling Methods 0.000 claims description 22
- 230000008569 process Effects 0.000 claims description 19
- 238000000605 extraction Methods 0.000 claims description 17
- 230000001149 cognitive effect Effects 0.000 claims description 13
- 238000000513 principal component analysis Methods 0.000 claims description 13
- 230000004927 fusion Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 10
- 238000011217 control strategy Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 2
- 230000007797 corrosion Effects 0.000 claims description 2
- 238000005260 corrosion Methods 0.000 claims description 2
- 239000000126 substance Substances 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 9
- 230000033001 locomotion Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000019771 cognition Effects 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003628 erosive effect Effects 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses a method, a device, a system and a medium for identifying object edges based on vision and touch, wherein the method comprises the following steps: acquiring object global tactile information by using an active tactile exploration algorithm, and extracting an object edge contour 1 according to the object global tactile information; processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor; and fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain the final edge feature of the object. By implementing the embodiment of the invention, the vision and the touch are fused to realize the identification of the edge characteristics of the object, and the identified edge characteristics of the object can be used in the mechanical arm grabbing, so that the accuracy, the robustness and the reliability of the mechanical arm grabbing are improved.
Description
Technical Field
The invention relates to the technical field of robot cognition, in particular to a method, a device, a system and a medium for identifying object edges based on vision and touch.
Background
Today of the rapid development of robot technology, the application field of robot is more and more wide, and the market also is more and more high to the requirement of robot, especially the arm field of snatching. At present, the mechanical arm mainly depends on vision to recognize and grab objects, and only can grab objects with simple shapes and high hardness due to lack of tactile feedback, but cannot grab objects with complex geometric shapes, softness and fragility, so that the application range of the mechanical arm and the mechanical arm is severely limited.
The fusion of vision and touch can make up for the deficiency. Visual excellence provides geometric and positional information at far distances, while touch can provide a small range of geometric, texture and force information at close distances. Touch has better resolution and is not affected by occlusion. Furthermore, vision is effective for free space motion because it is able to locate and identify remotely inaccessible objects. The tactile sensation will be able to provide accurate information during contact with the operating object for making fine but vital adjustments. If the method is applied properly, the sensing fusion of vision and touch can improve the accuracy, robustness and reliability of mechanical arm grabbing.
Disclosure of Invention
In view of the foregoing technical deficiencies, it is an object of embodiments of the present invention to provide a method, an apparatus, a system, and a medium for recognizing an edge of an object based on vision and touch.
To achieve the above object, in a first aspect, an embodiment of the present invention provides a method for recognizing an edge of an object based on vision and touch, including:
acquiring object global tactile information by using an active tactile exploration algorithm, and extracting an object edge contour 1 according to the object global tactile information;
processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
and fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain the final edge feature of the object.
As a specific embodiment of the present application, the active haptic exploration algorithm includes:
and (3) data processing: obtaining the posterior probability of the t-1 th time in the sampling process of the touch sensor, calculating the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object according to the posterior probability, and determining the final cognitive target angle and target position according to the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object; wherein t is a natural number;
active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, wherein the touch sensor executes the current touch exploration action;
information encoding step: and acquiring current tactile information obtained by the tactile sensor executing the current tactile exploration action, coding the current tactile information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
As an embodiment of the present application, the specific process of extracting the object edge profile 2 includes:
calculating the surface curvature and normal vector of the object: acquiring object information acquired by a visual sensor, and extracting and processing the object information by adopting a principal component analysis algorithm to obtain object surface curvature and normal vectors;
extracting a target curved surface: generating a structural point cloud according to the object surface curvature, the normal vector and the three-dimensional point cloud information, performing curved surface segmentation on the structural point cloud by adopting a region growing algorithm, and extracting target curved surface information;
extracting the edge contour of the target plane: and extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm.
Further, extracting the object information by using a principal component analysis algorithm to obtain an object surface curvature and a normal vector, specifically comprising:
selecting a point for target calculation
Selecting the sum of the point clouds
K is a point of close proximity
Processing the three-dimensional point cloud information by using a principal component analysis algorithm, wherein the formula is as follows:
wherein the content of the first and second substances,
is the central point of the close neighbor,
the eigenvalue and eigenvector of the covariance matrix C;
and estimating the curvature of the surface of the object according to the characteristic value, and estimating the normal vector according to the characteristic vector.
Further, extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm specifically comprises the following steps:
performing expansion pixel point adding processing on the original image corresponding to the target curved surface to obtain a first image with the enlarged curved surface area;
carrying out corrosion edge pixel point removal processing on the original image corresponding to the target curved surface to obtain a second image with the reduced curved surface area;
and carrying out XOR processing on the first image and the second image to obtain the remaining points, namely the curved surface edge points.
As a specific embodiment of the present application, the object edge profile information 2 includes an edge angle and an edge position, and obtaining the final edge feature of the object specifically includes:
converting the edge angle and the edge position into probability distribution of a plurality of angle intervals and a plurality of position intervals by adopting a Gaussian distribution model;
and fusing the probability distribution with the object edge contour 1 to obtain the final edge feature of the object.
In a second aspect, an embodiment of the present invention provides a device for recognizing an edge of an object based on vision and touch, including an extraction module and a fusion module;
the extraction module is configured to:
acquiring object global tactile information by using an active tactile exploration algorithm, and extracting an object edge contour 1 according to the object global tactile information;
processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
the fusion module is configured to:
fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain a final edge feature of the object;
wherein the active haptic exploration algorithm comprises:
and (3) data processing: obtaining the posterior probability of the t-1 th time in the sampling process of the touch sensor, calculating the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object according to the posterior probability, and determining the final cognitive target angle and target position according to the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object;
active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, wherein the touch sensor executes the current touch exploration action;
information encoding step: and acquiring current tactile information obtained by the tactile sensor executing the current tactile exploration action, coding the current tactile information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
In a third aspect, the present invention provides an apparatus for recognizing an edge of an object based on vision and touch, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, where the memory is used for storing a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect.
In a fourth aspect, the present invention provides a readable storage medium, which stores a computer program, where the computer program includes program instructions, and the program instructions, when executed by a processor, implement the method of the first aspect.
In a fifth aspect, the embodiment of the present invention provides a vision and touch based object edge identification system, which includes a touch sensor, a vision sensor and an object edge identification device, which are in communication with each other, where the touch sensor and the time sensor are used for performing edge identification on a detected object. Wherein the object edge identification device is as described in the third aspect above.
By implementing the embodiment of the invention, the vision and the touch are fused to realize the identification of the edge characteristics of the object, and the identified edge characteristics of the object can be used in the mechanical arm grabbing, so that the accuracy, the robustness and the reliability of the mechanical arm grabbing are improved.
Drawings
In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings that are needed in the detailed description of the invention or the prior art will be briefly described below.
FIG. 1 is a schematic diagram of a robotic arm and sensor configuration environment;
FIG. 2 is a schematic flow chart of a method for recognizing an edge of an object based on vision and touch according to an embodiment of the present invention;
FIGS. 3a and 3b are schematic diagrams of angular and positional discretization;
FIG. 4 is a schematic diagram of a probability distribution;
FIG. 5 is a flow chart of a pure haptic detection algorithm;
FIGS. 6a and 6b are schematic diagrams of a tactile sensor movement exploration process;
FIG. 7 is a schematic diagram of edge extraction;
FIG. 8 is a schematic edge tangent;
FIG. 9 is a schematic representation of a Gaussian distribution;
FIG. 10 is a flow chart of an algorithm for merging visual and tactile detection, wherein the visual prior probability initialization process is added;
FIG. 11 is a schematic structural diagram of an apparatus for recognizing an edge of an object based on vision and touch according to a first embodiment of the present invention;
fig. 12 is a schematic structural diagram of an object edge recognition device based on vision and touch according to a second embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention has the following inventive concept: studies in biology and neuroscience have shown that animal brain cognition operates in a bayesian fashion. The invention is inspired by biology and neuroscience, and provides a novel visual and tactile multi-sensor fusion algorithm based on the Bayesian principle. The edge characteristics of the object can be collected by the algorithm, and the collected information can be used in mechanical arm grabbing to improve the grabbing accuracy and robustness, so that the grabbing effect of the mechanical arm is improved.
Based on the above inventive concept, the embodiment of the present invention provides an object edge identification method based on vision and touch. The robot arm and sensor environment configuration of the identification method is shown in fig. 1, and comprises a motion actuator provided with a touch sensor, the touch sensor arranged at the tail end of the motion actuator, a visual sensor and an object to be detected.
Based on the above environment configuration, referring to fig. 2, the method for recognizing the edge of an object based on vision and touch mainly includes:
s101, obtaining object global tactile information by using an active tactile exploration algorithm, and extracting an object edge outline 1 according to the object global tactile information.
And S102, processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2.
And obtaining the three-dimensional point cloud information by a visual sensor.
And S103, fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain the final edge feature of the object.
Regarding step S101: when the touch sensor is used to acquire the touch information, a point needs to be noticed that the touch sensor can only acquire local small-range information. If the global information is required to be acquired, the local information needs to be repeatedly acquired for many times, and the local information is integrated into the global information. Therefore, the invention provides an active exploration algorithm which can determine the position of the next tactile exploration according to the local information obtained by the tactile sensor, so that useful information is explored, and the tactile information acquisition efficiency is improved.
Regarding step S102: aiming at visual object edge extraction, the invention provides an object edge extraction algorithm framework based on principal component analysis and region segmentation algorithm. The algorithm uses point clouds, surface curvatures and surface normal vectors.
Regarding step S103: the vision and touch can complement each other, so the invention provides a vision and touch fusion algorithm based on the Bayes principle. In this algorithm, the visually captured edge profile of the object will be encoded as a probability distribution of the position and edge tangent angles, which will be fused with the probability distribution obtained by the haptic sense.
Active touch based object edge extraction
Specifically, in the present embodiment, the active haptic exploration algorithm includes:
and (3) data processing: obtaining the posterior probability of the t-1 th time in the sampling process of the touch sensor, calculating the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object according to the posterior probability, and determining the final cognitive target angle and target position according to the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object; wherein t is a positive integer greater than 1;
active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, wherein the touch sensor executes the current touch exploration action;
information encoding step: and acquiring current tactile information obtained by the tactile sensor executing the current tactile exploration action, coding the current tactile information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
Further, with respect to the data processing steps:
the formula involved in data processing is the most core part of the whole algorithm, and is responsible for deriving the angle and position of the edge. The edge feature is described by two variables of the edge tangent angle and the edge position in a coordinate system with the center of the touch sensor as an origin.
The part of data processing related to Bayesian formula uses probability, so that the angle and position are discretized in the application. The angle is divided into N in the interval of 0 to 360ornEach zone is divided into N parts at 0-10 mm positionsloc. For convenience of description, assume that Norn＝18，NlocAs a reference case, 20, each interval corresponds to a probability, as shown in fig. 3a and 3 b. The discrete probability distribution information may be applied to a bayesian algorithm to obtain an object edge profile.
The Bayesian formula involved in calculating the edge angle and the position characteristic information is as follows:
zt( t 1, 2, 3.) is a single tactile contact every t, z1:tFor all haptic information accumulated from 1 st to t th times, θiFor discretized angle information, rlIs discretized location information. P (z)t|rl，θi) Sampling for the tth hapticThe calculation process of the conditional probability of time is described in the previous step. P (r)l，θi|z1：t-1) The prior probabilities accumulated for the 1 st to t-1 st samples. P (z)t|zt-1) And the method plays a role of normalization in the formula for marginal probability. The marginal probability for normalization is obtained by the following equation:
it should be noted that the object edge feature extraction process is a process of repeatedly sampling and continuously accumulating probabilities. The posterior probability of the t-1 step is taken as the prior probability of the t step. The posterior probability may change during repeated sampling of the accumulated information. Therefore, the marginal probability of the edge angle can be calculated by the posterior probability.
Further, a marginal probability P (theta) of the edge angle of the object is calculatedi|z1:t) And marginal probability P (r) of edge positionl|z1:t) The formula of (1) is as follows:
selecting the angle and position interval with the highest corresponding marginal probability as the final cognitive angle thetadecAnd position rdecAs shown in fig. 4, in this example,
θdec＝argmax(P(θi|z1:t))
rdec＝argmax(P(rl|z1:t))
the obtained angle thetadecAnd position rdecAnd (4) displaying local coordinates of the touch sensor.
It should be noted that, in the process of repeated sampling, as the information is accumulated continuously, the marginal probability distribution is gradually narrowed, and the peak value of the probability distribution is higher and higher. Once edge angle of objectThe ratio P (theta)i|z1:t) Exceeds a set threshold value PdecThen the sampling is finished, and the sampling cycle is skipped, and the next sampling position is entered for sampling, as shown in fig. 5.
Further, regarding the active exploration step:
exploration is very important for the acquisition of tactile information. Unlike vision, a tactile sensor can collect a very small local range of information. In this case, it is necessary to search continuously and sample for many times to reconstruct relatively complete information.
In this embodiment, the current haptic exploration action is calculated by a predetermined control strategy according to the angle and position information learned in the previous step. The output exploration action consists of two execution actions: one is the exploration action along the tangential direction of the edge wheel buckle, and the other is the position deviation correction action along the normal vector of the edge. Please refer to fig. 6a and fig. 6 b.
The tactile sensor is moved along the tangent of the edge by a fixed amount Δ e and then moved along the orthogonal direction of the edge by Δ r (t). Δ e is a fixed value set manually, and Δ r (t) is a variable related to the edge position, which is obtained by the following equation:
Δr(t)＝g·(-rdec(t))
g is the gain, which is a value set to be greater than 0. r isdecThe position of the edge of the object after the data processing mentioned in the previous step with respect to the coordinates of the tactile sensor.
The tactile sensor changes its position and angle during its movement. In order to ensure the uniformity of coordinates, coordinate transformation is required to be performed all the time, and information acquired by the touch sensor is projected into a coordinate system of the touch sensor.
Further, with respect to the information encoding step:
the tactile information acquired by the tactile sensor cannot be directly applied to the Bayesian algorithm, and the information needs to be converted into probability distribution in the midway through an information encoding process so that the probability distribution can be processed and analyzed.
In the Bayesian equation mentioned in the data processing
Conditional probability P (z)t|rl，θi) Translated from raw tactile sensor information. The signal currently acquired by the touch sensor is ztA means is needed to convert the original signal ztConditional probability P (z) of transforming into angles and intervalst|rl，θi)。
The method trains the neural network phi to learn the transformation relation in the neural network by using a supervised learning method. The input of the neural network is the raw sensor signal, and the output is the corresponding dimension NlocNornProbability distribution of (2). The samples required for supervised learning can be derived by sampling on objects of known shape.
P(zt|rl,θi)＝Φ(zt)
Second, object edge information extraction based on vision
Specifically, in the present embodiment, the vision-based object edge extraction mainly includes:
calculating the surface curvature and normal vector of the object: acquiring object information acquired by a visual sensor, and extracting and processing the object information by adopting a principal component analysis algorithm to obtain object surface curvature and normal vectors;
extracting a target curved surface: generating a structural point cloud according to the object surface curvature, the normal vector and the three-dimensional point cloud information, performing curved surface segmentation on the structural point cloud by adopting a region growing algorithm, and extracting target curved surface information;
extracting the edge contour of the target plane: and extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm.
Further, the object surface curvature and normal vector calculation step:
the calculation of the normal vector and curvature of the object plane can be obtained by a Principal Component Analysis (PCA) algorithm. This process requires traversing the entire point cloud to obtain the curvatures and normal vectors of all points in the point cloud.
The specific method comprises the following steps: selecting a point for target calculation
Then selecting the sum in the point cloud
K adjacent points
After the neighboring points are selected, the point groups are processed by using a principal component analysis algorithm (PCA).
The specific formula is as follows:
wherein
Is the central point of the close neighbor,
are the eigenvalues and eigenvectors of the covariance matrix C. The eigenvalues can be used to estimate the curvature of the surface and the minimum eigenvector can be used to estimate the normal vector of the surface.
The direction of the normal vector calculated by PCA is not fixed, and the normal vector
Can be adjusted by the following formula
Wherein
To be considered the selected viewing angle.
Further, the target curved surface extraction step:
and generating a structural point cloud by post-processing the point cloud, the normal vector and the curvature obtained in the last step so as to ensure that each point corresponds to a pixel point of the 2D color image one by one. The processed point cloud can be used for segmenting the curved surface through a region growing algorithm and extracting target curved surface information. The region segmentation algorithm uses three information of distance, normal vector and curvature to segment the point cloud into different planes. The segmentation uses a region growing algorithm, and the region growing needs to meet three conditions of distance, normal vector and curvature.
The distance between the current point and the adjacent point is smaller than the preset value
The curvature of the current point is smaller than the preset value
The angle difference between the normal vector of the current point and the normal vector of the adjacent point is less than a preset value
The specific algorithm flow is as follows:
further, an edge contour extraction step of the target plane:
each point of the structural point cloud obtained in the step corresponds to a certain pixel in the two-dimensional image, so that the edge contour of the curved surface can be extracted by using a two-dimensional image processing algorithm. The extraction of the edge profile requires the use of dilation and erosion methods into the image processing.
Referring to fig. 7, the specific steps are as follows:
expanding the original image to add pixel points and enlarge the area of the curved surface;
corroding the original image to remove edge pixel points and reduce the area of the curved surface;
and performing XOR on the two points, wherein the rest points are plane edge points.
Third, sense fusion based on vision and touch
When the visual contact fusion is carried out, the edge outline of the object is firstly converted into probability distribution, and the specific steps are as follows.
The nearest point p on the edge to the touch sensor is selected, and then k adjacent points on the edge contour are selected. A linear regression algorithm is used for fitting a straight line by using the selected k adjacent points, and the fitted straight line is an edge tangent line. The tangent angle is the angle of the edge profile and the distance of the tangent from the sensor is the distance of the edge profile from the sensor, as shown in fig. 8.
The edge angle and edge position obtained by the above steps through vision are determined values, and need to be converted into probability distribution to be fused with the tactile information.
The visually acquired edge angles and positions are converted into a probability distribution of 18 angle intervals and 20 position intervals by a Gaussian distribution model. The formula of the gaussian distribution model is as follows:
the standard deviation σ affects the accuracy of the probability distribution, krAnd k isθIs a normalization parameter used to adjust the edge angle and the edge location range. The closer the angle interval and the position interval from the measurement angle θ to the measurement position r, the higher the assigned probability value, refer to fig. 9.
The edge angle and position features obtained by vision are converted into probability distribution P (r)l，θi|z0) This is then used as a priori probability to initialize the haptic exploration algorithm, as follows:
prior probability P (r) in the formulal，θi|z1：t-1) Will be provided withProbability distribution P (r) calculated by visionl，θi|z0) And (5) assignment initialization.
The object edge feature extraction process is a process of repeatedly sampling and continuously accumulating probabilities. The posterior probability of the t-1 step is taken as the prior probability of the t step. The algorithm will continually cycle through repeated sampling, accumulate information, and improve accuracy during the process of sample accumulation. The posterior probability may change during repeated sampling of the accumulated information. We can calculate the marginal probability of the edge angle feature by the posterior probability. Once the marginal probability of the edge angle of the object exceeds the set threshold, the sampling is finished, a sampling cycle is skipped, the next sampling position is entered, the process of continuously sampling the visual sense touch exploration algorithm which is fused with the vision can refer to the schematic diagram 10, and the process of the unfused visual sense touch exploration algorithm can refer to the diagram 5.
In the embodiment of the present application, the edge angle is divided into 18 sections, and the edge position offset is divided into 20 sections. Each interval is represented by a corresponding probability. Through the above algorithm, the angle and edge offset with the highest corresponding probability are selected as the angle and position of the final decision, referring to fig. 3a and 3 b.
Selecting the angle with the highest marginal probability as the edge angle characteristic of the final object:
selecting the position with the highest marginal rate as the edge position characteristic of the final object:
the edge angle and position obtained in the above steps are described in a coordinate system with the tactile sensor as an origin, and are local information with respect to the tactile sensor. Depending on the particular application, it may be desirable to obtain global information relative to a world coordinate system. The position and posture of the touch sensor are known, and if the characteristics of the edge contour of the object relative to the world coordinate system are to be acquired, the coordinate system is changed.
By implementing the object edge identification method provided by the embodiment of the invention, the vision and the touch are fused to realize the identification of the object edge characteristics, and the identified object edge characteristics can be used in the mechanical arm grabbing, so that the accuracy, robustness and reliability of the mechanical arm grabbing are improved.
Based on the same inventive concept, corresponding to the foregoing method embodiment, as shown in fig. 11, an embodiment of the present invention provides a visual and tactile-based object edge recognition apparatus, which includes an extraction module 100 and a fusion module 200.
Wherein the extraction module 100 is configured to:
acquiring object global tactile information by using an active tactile exploration algorithm, and extracting an object edge contour 1 according to the object global tactile information;
processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
the fusion module 200 is configured to:
fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain a final edge feature of the object;
wherein the active haptic exploration algorithm comprises:
and (3) data processing: obtaining the posterior probability of the t-1 th time in the sampling process of the touch sensor, calculating the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object according to the posterior probability, and determining the final cognitive target angle and target position according to the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object;
active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, wherein the touch sensor executes the current touch exploration action;
information encoding step: and acquiring current tactile information obtained by the tactile sensor executing the current tactile exploration action, coding the current tactile information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
Optionally, the embodiment of the invention also provides another object edge recognition device based on vision and touch. As shown in fig. 12, the edge identifying apparatus may include: one or more processors 101, one or more input devices 102, one or more output devices 103, and memory 104, the processors 101, input devices 102, output devices 103, and memory 104 being interconnected via a bus 105. The memory 104 is used for storing a computer program comprising program instructions, the processor 101 being configured for invoking the program instructions to perform the methods of the above-described visual and haptic based object edge identification method embodiment parts.
It should be understood that, in the embodiment of the present invention, the Processor 101 may be a Central Processing Unit (CPU), and the Processor may also be other general processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 102 may include a keyboard or the like, and the output device 103 may include a display (LCD or the like), a speaker, or the like.
The memory 104 may include read-only memory and random access memory, and provides instructions and data to the processor 101. A portion of the memory 104 may also include non-volatile random access memory. For example, the memory 104 may also store device type information.
In a specific implementation, the processor 101, the input device 102, and the output device 103 described in this embodiment of the present invention may execute the implementation manner described in the embodiment of the method for recognizing an object edge based on vision and touch provided in the first embodiment of the present invention, which is not described herein again.
It should be noted that, for a more specific description of the workflow of the two object edge recognition apparatuses, please refer to the foregoing method embodiment, which is not described herein again.
By implementing the object edge recognition device provided by the embodiment of the invention, the vision and the touch are fused to realize the recognition of the object edge characteristics, and the recognized object edge characteristics can be used in mechanical arm grabbing, so that the accuracy, robustness and reliability of mechanical arm grabbing are improved.
Further, corresponding to the visual and tactile-based object edge identification method and apparatus of the first embodiment, an embodiment of the present invention further provides a readable storage medium storing a computer program, the computer program comprising program instructions, which when executed by a processor, implement: the method for recognizing the edge of an object based on vision and touch according to the first embodiment is described above.
The computer readable storage medium may be an internal storage unit of the object edge identification device according to the foregoing embodiment, such as a hard disk or a memory of a system. The computer readable storage medium may also be an external storage device of the system, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) Card, Flash memory Card (Flash Card), etc. provided on the system. Further, the computer readable storage medium may also include both an internal storage unit and an external storage device of the system. The computer-readable storage medium is used for storing the computer program and other programs and data required by the system. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
Based on the same inventive concept, the embodiment of the invention provides a vision and touch based object edge identification system, which comprises a touch sensor, a vision sensor and an object edge identification device, wherein the touch sensor, the vision sensor and the object edge identification device are communicated with each other, and the touch sensor and the time sensor are used for carrying out edge identification on a detected object. The object edge recognition device is as described in the foregoing embodiments, and is not described herein again.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. An object edge identification method based on vision and touch is characterized by comprising the following steps:
acquiring object global tactile information by using an active tactile exploration algorithm, and extracting an object edge contour 1 according to the object global tactile information;
processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
and fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain the final edge feature of the object.
2. The object edge identification method of claim 1, wherein the active haptic exploration algorithm comprises:
and (3) data processing: obtaining the posterior probability of the t-1 th time in the sampling process of the touch sensor, calculating the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object according to the posterior probability, and determining the final cognitive target angle and target position according to the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object; wherein t is a positive integer greater than 1;
active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, wherein the touch sensor executes the current touch exploration action;
information encoding step: and acquiring current tactile information obtained by the tactile sensor executing the current tactile exploration action, coding the current tactile information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
3. The method for identifying the edge of the object according to claim 1, wherein the specific process for extracting the edge contour 2 of the object comprises:
calculating the surface curvature and normal vector of the object: acquiring object information acquired by a visual sensor, and extracting and processing the object information by adopting a principal component analysis algorithm to obtain object surface curvature and normal vectors;
extracting a target curved surface: generating a structural point cloud according to the object surface curvature, the normal vector and the three-dimensional point cloud information, performing curved surface segmentation on the structural point cloud by adopting a region growing algorithm, and extracting target curved surface information;
extracting the edge contour of the target plane: and extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm.
4. The method for identifying an edge of an object according to claim 3, wherein extracting the object information by using a principal component analysis algorithm to obtain the curvature of the surface of the object and a normal vector comprises:
selecting a point for target calculation
Selecting the sum of the point clouds
K is a point of close proximity
Processing the three-dimensional point cloud information by using a principal component analysis algorithm, wherein the formula is as follows:
wherein the content of the first and second substances,
is the central point of the close neighbor,
the eigenvalue and eigenvector of the covariance matrix C;
and estimating the curvature of the surface of the object according to the characteristic value, and estimating the normal vector according to the characteristic vector.
5. The object edge identification method according to claim 3, wherein extracting the edge contour of the target curved surface by using a two-dimensional image processing algorithm specifically comprises:
performing expansion pixel point adding processing on the original image corresponding to the target curved surface to obtain a first image with the enlarged curved surface area;
carrying out corrosion edge pixel point removal processing on the original image corresponding to the target curved surface to obtain a second image with the reduced curved surface area;
and carrying out XOR processing on the first image and the second image to obtain the remaining points, namely the curved surface edge points.
6. The object edge identification method according to claim 1, wherein the object edge profile information 2 includes an edge angle and an edge position, and obtaining the final edge feature of the object specifically includes:
converting the edge angle and the edge position into probability distribution of a plurality of angle intervals and a plurality of position intervals by adopting a Gaussian distribution model;
and fusing the probability distribution with the object edge contour 1 to obtain the final edge feature of the object.
7. An object edge recognition device based on vision and touch is characterized by comprising an extraction module and a fusion module;
the extraction module is configured to:
acquiring object global tactile information by using an active tactile exploration algorithm, and extracting an object edge contour 1 according to the object global tactile information;
processing the three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
the fusion module is configured to:
fusing the object edge contour 1 and the object edge contour 2 by using a Bayesian model to obtain a final edge feature of the object;
wherein the active haptic exploration algorithm comprises:
and (3) data processing: obtaining the posterior probability of the t-1 th time in the sampling process of the touch sensor, calculating the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object according to the posterior probability, and determining the final cognitive target angle and target position according to the marginal probability of the edge angle of the object and the marginal probability of the edge position of the object;
active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, wherein the touch sensor executes the current touch exploration action;
information encoding step: and acquiring current tactile information obtained by the tactile sensor executing the current tactile exploration action, coding the current tactile information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
8. An apparatus for visual and tactile based edge recognition of an object, comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, wherein the memory is configured to store a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method according to any one of claims 1-6.
9. A readable storage medium storing a computer program comprising program instructions, which when executed by a processor implement the method of any one of claims 1-6.
10. A vision and touch based object edge identification system comprising a touch sensor, a vision sensor and an object edge identification device in communication with each other, the touch sensor and the time sensor being used for edge identification of an object to be detected, wherein the object edge identification device is according to claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010393054.4A CN111582186B (en) | 2020-05-11 | 2020-05-11 | Object edge recognition method, device, system and medium based on vision and touch |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010393054.4A CN111582186B (en) | 2020-05-11 | 2020-05-11 | Object edge recognition method, device, system and medium based on vision and touch |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111582186A true CN111582186A (en) | 2020-08-25 |
| CN111582186B CN111582186B (en) | 2023-12-15 |
Family
ID=72112146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010393054.4A Active CN111582186B (en) | 2020-05-11 | 2020-05-11 | Object edge recognition method, device, system and medium based on vision and touch |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111582186B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420735A (en) * | 2021-08-23 | 2021-09-21 | 深圳市信润富联数字科技有限公司 | Contour extraction method, contour extraction device, contour extraction equipment, program product and storage medium |
| CN113714662A (en) * | 2021-09-06 | 2021-11-30 | 玖陆零信安创(盐城)科技有限公司 | Cutting size adjusting mechanism for intelligent household wood board cutting equipment |
| CN114851227A (en) * | 2022-06-22 | 2022-08-05 | 上海大学 | Device based on machine vision and sense of touch fuse perception |
| CN115760805A (en) * | 2022-11-24 | 2023-03-07 | 中山大学 | Positioning method for processing surface depression of element based on visual touch sense |
| CN116125840A (en) * | 2023-03-09 | 2023-05-16 | 四川长虹新网科技有限责任公司 | Intelligent feeding system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH05288543A (en) * | 1992-04-07 | 1993-11-02 | Fujitsu Ltd | Recognizing device integrated with visual information and tactile information |
| CN106272424A (en) * | 2016-09-07 | 2017-01-04 | 华中科技大学 | A kind of industrial robot grasping means based on monocular camera and three-dimensional force sensor |
| CN110091331A (en) * | 2019-05-06 | 2019-08-06 | 广东工业大学 | Grasping body method, apparatus, equipment and storage medium based on manipulator |
| CN111055279A (en) * | 2019-12-17 | 2020-04-24 | 清华大学深圳国际研究生院 | Multi-mode object grabbing method and system based on combination of touch sense and vision |
-
2020
- 2020-05-11 CN CN202010393054.4A patent/CN111582186B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH05288543A (en) * | 1992-04-07 | 1993-11-02 | Fujitsu Ltd | Recognizing device integrated with visual information and tactile information |
| CN106272424A (en) * | 2016-09-07 | 2017-01-04 | 华中科技大学 | A kind of industrial robot grasping means based on monocular camera and three-dimensional force sensor |
| CN110091331A (en) * | 2019-05-06 | 2019-08-06 | 广东工业大学 | Grasping body method, apparatus, equipment and storage medium based on manipulator |
| CN111055279A (en) * | 2019-12-17 | 2020-04-24 | 清华大学深圳国际研究生院 | Multi-mode object grabbing method and system based on combination of touch sense and vision |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420735A (en) * | 2021-08-23 | 2021-09-21 | 深圳市信润富联数字科技有限公司 | Contour extraction method, contour extraction device, contour extraction equipment, program product and storage medium |
| CN113420735B (en) * | 2021-08-23 | 2021-12-21 | 深圳市信润富联数字科技有限公司 | Contour extraction method, device, equipment and storage medium |
| CN113714662A (en) * | 2021-09-06 | 2021-11-30 | 玖陆零信安创(盐城)科技有限公司 | Cutting size adjusting mechanism for intelligent household wood board cutting equipment |
| CN113714662B (en) * | 2021-09-06 | 2022-06-24 | 玖陆零信安创(盐城)科技有限公司 | Cutting size adjusting mechanism for intelligent household wood board cutting equipment |
| CN114851227A (en) * | 2022-06-22 | 2022-08-05 | 上海大学 | Device based on machine vision and sense of touch fuse perception |
| CN114851227B (en) * | 2022-06-22 | 2024-02-27 | 上海大学 | Device based on machine vision and touch sense fusion perception |
| CN115760805A (en) * | 2022-11-24 | 2023-03-07 | 中山大学 | Positioning method for processing surface depression of element based on visual touch sense |
| CN115760805B (en) * | 2022-11-24 | 2024-02-09 | 中山大学 | Positioning method for processing element surface depression based on visual touch sense |
| CN116125840A (en) * | 2023-03-09 | 2023-05-16 | 四川长虹新网科技有限责任公司 | Intelligent feeding system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111582186B (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111582186B (en) | Object edge recognition method, device, system and medium based on vision and touch | |
| Khaloo et al. | Robust normal estimation and region growing segmentation of infrastructure 3D point cloud models | |
| EP2680228B1 (en) | Improvements in or relating to three dimensional close interactions. | |
| Ahn et al. | Real-time head orientation from a monocular camera using deep neural network | |
| US9098740B2 (en) | Apparatus, method, and medium detecting object pose | |
| Gökberk et al. | 3D shape-based face representation and feature extraction for face recognition | |
| JP5406705B2 (en) | Data correction apparatus and method | |
| Vemulapalli et al. | R3DG features: Relative 3D geometry-based skeletal representations for human action recognition | |
| CN107958458B (en) | Image segmentation method, image segmentation system and equipment comprising image segmentation system | |
| JP6349418B2 (en) | Object positioning by high-precision monocular movement | |
| Xiong et al. | Model-guided deformable hand shape recognition without positioning aids | |
| US10043279B1 (en) | Robust detection and classification of body parts in a depth map | |
| US20180174328A1 (en) | Turning radius-based corner detection algorithm | |
| KR20170024303A (en) | System and method for detecting feature points of face | |
| Mehryar et al. | Automatic landmark detection for 3d face image processing | |
| JP4921847B2 (en) | 3D position estimation device for an object | |
| Ye et al. | An improved algorithm for Harris corner detection | |
| JP2012243285A (en) | Feature point position decision device, feature point position determination method and program | |
| KR101528757B1 (en) | Texture-less object recognition using contour fragment-based features with bisected local regions | |
| Kröger et al. | Performance evaluation on contour extraction using Hough transform and RANSAC for multi-sensor data fusion applications in industrial food inspection | |
| Wang et al. | Robot grasping in dense clutter via view-based experience transfer | |
| Ghebreab et al. | Strings: variational deformable models of multivariate continuous boundary features | |
| Hsiao et al. | Confronting ambiguity in 6d object pose estimation via score-based diffusion on se (3) | |
| Zhang et al. | Corner detection based on tangent-to-point distance accumulation technique | |
| JP5814714B2 (en) | Image similarity determination device, image similarity determination method, and image similarity determination program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |