CN111582186B - Object edge recognition method, device, system and medium based on vision and touch - Google Patents
Object edge recognition method, device, system and medium based on vision and touch Download PDFInfo
- Publication number
- CN111582186B CN111582186B CN202010393054.4A CN202010393054A CN111582186B CN 111582186 B CN111582186 B CN 111582186B CN 202010393054 A CN202010393054 A CN 202010393054A CN 111582186 B CN111582186 B CN 111582186B
- Authority
- CN
- China
- Prior art keywords
- edge
- information
- touch
- angle
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 40
- 230000000007 visual effect Effects 0.000 claims abstract description 28
- 239000013598 vector Substances 0.000 claims description 25
- 238000009826 distribution Methods 0.000 claims description 23
- 238000000605 extraction Methods 0.000 claims description 20
- 238000005070 sampling Methods 0.000 claims description 18
- 238000000513 principal component analysis Methods 0.000 claims description 13
- 230000001149 cognitive effect Effects 0.000 claims description 12
- 230000004927 fusion Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 238000011217 control strategy Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 230000019771 cognition Effects 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 2
- 238000005260 corrosion Methods 0.000 claims description 2
- 230000007797 corrosion Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 6
- 238000010606 normalization Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000035807 sensation Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- CLOMYZFHNHFSIQ-UHFFFAOYSA-N clonixin Chemical compound CC1=C(Cl)C=CC=C1NC1=NC=CC=C1C(O)=O CLOMYZFHNHFSIQ-UHFFFAOYSA-N 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003628 erosive effect Effects 0.000 description 1
- 238000005530 etching Methods 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000003655 tactile properties Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application discloses an object edge identification method, device, system and medium based on vision and touch, wherein the method comprises the following steps: acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information; processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor; and fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object. By implementing the embodiment of the application, the vision and the touch are fused to realize the identification of the edge characteristics of the object, and the identified edge characteristics of the object can be used for grabbing by the mechanical arm, so that the accuracy, the robustness and the reliability of grabbing by the mechanical arm are improved.
Description
Technical Field
The application relates to the technical field of robot cognition, in particular to an object edge recognition method, device, system and medium based on vision and touch.
Background
Today, the robot technology rapidly develops, the application field of the robot is wider and wider, the requirements of the market on the robot are also higher and higher, and particularly the field of grabbing of the mechanical arm. At present, the mechanical arm is mainly used for grabbing objects by vision, and only objects with simple shapes and large hardness can be grabbed due to lack of tactile feedback, but objects with complex geometric shapes, softness and weakness can not be grabbed, so that the application range of the mechanical arm and the mechanical arm is severely limited.
The fusion of visual and tactile properties may complement each other. Visual tampering provides geometric and positional information at a long distance, while touch can provide a small range of geometric, texture and force information at close contact. The touch has better resolution and is not affected by occlusion. Moreover, vision is effective for free space movement because it is able to locate and identify objects that are far from reach. The haptic sensation will be able to provide accurate information during contact with the manipulation object to make fine but vital adjustments. If the method is properly applied, the accuracy, the robustness and the reliability of the grabbing of the mechanical arm can be improved through the visual and tactile sensing fusion.
Disclosure of Invention
In view of the above technical drawbacks, an object of an embodiment of the present application is to provide a method, apparatus, system and medium for identifying an edge of an object based on vision and touch.
To achieve the above object, in a first aspect, an embodiment of the present application provides a method for identifying an edge of an object based on vision and touch, including:
acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information;
processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
and fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object.
As a specific embodiment of the present application, the active haptic discovery algorithm includes:
and a data processing step: acquiring t-1 th posterior probability in the sampling process of the touch sensor, calculating marginal probability of an object edge angle and marginal probability of an object edge position according to the posterior probability, and determining a final cognitive target angle and target position according to the marginal probability of the object edge angle and the marginal probability of the object edge position; wherein t is a natural number;
an active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, and executing the current touch exploration action by the touch sensor;
and an information encoding step: and obtaining current haptic information obtained by the haptic sensor executing the current haptic exploration action, encoding the current haptic information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
As a specific embodiment of the present application, the specific process of extracting the object edge profile 2 includes:
calculating the curvature and the normal quantity of the object surface: acquiring object information acquired by a vision sensor, and extracting the object information by adopting a principal component analysis algorithm to obtain the curvature and normal vector of the object surface;
extracting a target curved surface: generating a structural point cloud according to the object surface curvature, the normal vector and the three-dimensional point cloud information, performing surface segmentation on the structural point cloud by adopting a region growing algorithm, and extracting target surface information;
edge contour extraction of the target plane: and extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm.
Further, the object information is extracted by adopting a principal component analysis algorithm to obtain the curvature and normal vector of the object surface, which specifically comprises the following steps:
selecting a point of target calculationSelecting and +.>Point ∈k nearest neighbor>
And processing the three-dimensional point cloud information by using a principal component analysis algorithm, wherein the formula is as follows:
wherein,is the center point of the neighbor->The eigenvalues and eigenvectors of the covariance matrix C;
and estimating the surface curvature of the object according to the characteristic value, and estimating the normal vector according to the characteristic vector.
Further, extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm, which specifically comprises the following steps:
performing expansion pixel point adding processing on an original image corresponding to the target curved surface to obtain a first image with the enlarged curved surface area;
performing corrosion edge pixel point removal treatment on the original image corresponding to the target curved surface to obtain a second image with a reduced curved surface area;
and performing exclusive OR processing on the first image and the second image to obtain the residual points which are the curved surface edge points.
As a specific embodiment of the present application, the object edge profile information 2 includes an edge angle and an edge position, and the obtaining the final edge feature of the object specifically includes:
converting the edge angles and the edge positions into probability distributions of a plurality of angle intervals and a plurality of position intervals by adopting a Gaussian distribution model;
and fusing the probability distribution with the object edge profile 1 to obtain the final edge characteristic of the object.
In a second aspect, an embodiment of the present application provides an object edge recognition device based on vision and touch, including an extraction module and a fusion module;
the extraction module is used for:
acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information;
processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
the fusion module is used for:
fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object;
wherein the active haptic exploration algorithm comprises:
and a data processing step: acquiring t-1 th posterior probability in the sampling process of the touch sensor, calculating marginal probability of an object edge angle and marginal probability of an object edge position according to the posterior probability, and determining a final cognitive target angle and target position according to the marginal probability of the object edge angle and the marginal probability of the object edge position;
an active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, and executing the current touch exploration action by the touch sensor;
and an information encoding step: and obtaining current haptic information obtained by the haptic sensor executing the current haptic exploration action, encoding the current haptic information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
In a third aspect, an embodiment of the present application provides a visual and tactile-based object edge recognition apparatus, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, and where the memory is configured to store a computer program, where the computer program includes program instructions, and where the processor is configured to invoke the program instructions to perform the method of the first aspect.
In a fourth aspect, an embodiment of the present application provides a readable storage medium storing a computer program comprising program instructions which, when executed by a processor, implement the method of the first aspect described above.
In a fifth aspect, an embodiment of the present application provides a vision and touch-based object edge recognition system, including a touch sensor, a vision sensor, and an object edge recognition device that are in communication with each other, where the touch sensor and the time sensor are used to perform edge recognition on a detected object. Wherein the object edge recognition device is as described in the third aspect above.
By implementing the embodiment of the application, the vision and the touch are fused to realize the identification of the edge characteristics of the object, and the identified edge characteristics of the object can be used for grabbing by the mechanical arm, so that the accuracy, the robustness and the reliability of grabbing by the mechanical arm are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a schematic diagram of a robot and sensor configuration environment;
FIG. 2 is a schematic flow chart of a vision and haptic based object edge recognition method provided by an embodiment of the present application;
FIGS. 3a and 3b are schematic views of angle and position discretization;
FIG. 4 is a schematic diagram of probability distribution;
FIG. 5 is a flow chart of a pure haptic detection algorithm;
FIGS. 6a and 6b are schematic diagrams of haptic sensor movement exploration processes;
FIG. 7 is an edge extraction schematic;
FIG. 8 is an edge tangent schematic;
FIG. 9 is a schematic diagram of a Gaussian distribution;
FIG. 10 is a flowchart of an algorithm for fusing visual and tactile detection, wherein the diagram includes a plurality of visual prior probability initialization processes;
FIG. 11 is a schematic view of an object edge recognition device based on vision and touch according to a first embodiment of the present application;
fig. 12 is a schematic structural view of an object edge recognition device based on vision and touch according to a second embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The application is characterized in that: biological and neuroscience studies have shown that animal brain cognition operates on the bayesian principle. The application provides a novel vision and touch multi-sensor fusion algorithm based on Bayesian principle, which is inspired by biology and neuroscience. The algorithm can collect the edge characteristics of the object, and the collected information can be used in the grabbing of the mechanical arm to improve the grabbing accuracy and robustness, so that the grabbing effect of the mechanical arm is improved.
Based on the above inventive concept, the embodiment of the application provides an object edge recognition method based on vision and touch. The mechanical arm and sensor environment configuration of the identification method is shown in fig. 1, and the identification method comprises a motion executing mechanism provided with a touch sensor, the touch sensor arranged at the tail end of the motion executing mechanism, a visual sensor and an object to be detected.
Based on the above-mentioned environment configuration, please refer to fig. 2, the object edge recognition method based on vision and touch mainly includes:
s101, acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information.
S102, processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2.
The three-dimensional point cloud information is obtained by a visual sensor.
And S103, fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object.
Regarding step S101: when using a tactile sensor to obtain tactile information, one needs to notice a point that the tactile sensor can only obtain local small-scale information. If the global information is to be acquired, the local information needs to be repeatedly taken for a plurality of times, and the local information is integrated into the global information. Therefore, the application provides an active exploration algorithm which can determine the position of the next tactile exploration according to the local information obtained by the tactile sensor, thereby exploring the useful information and improving the efficiency of obtaining the tactile information.
Regarding step S102: aiming at visual object edge extraction, the application provides an object edge extraction algorithm framework based on principal component analysis and a region segmentation algorithm. The algorithm uses point clouds, surface curvatures and surface normal vectors.
Regarding step S103: the vision can complement the touch, so the application provides a visual touch fusion algorithm which is based on the Bayesian principle. In this algorithm, the visually acquired object edge profile will be encoded as a probability distribution of position and edge tangent angle, which will be fused with the haptic derived probability distribution.
1. Active haptic based object edge extraction
Specifically, in this embodiment, the active haptic discovery algorithm includes:
and a data processing step: acquiring t-1 th posterior probability in the sampling process of the touch sensor, calculating marginal probability of an object edge angle and marginal probability of an object edge position according to the posterior probability, and determining a final cognitive target angle and target position according to the marginal probability of the object edge angle and the marginal probability of the object edge position; wherein t is a positive integer greater than 1;
an active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, and executing the current touch exploration action by the touch sensor;
and an information encoding step: and obtaining current haptic information obtained by the haptic sensor executing the current haptic exploration action, encoding the current haptic information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
Further, regarding the data processing step:
the formula involved in the data processing is the most core part of the whole algorithm, and is responsible for obtaining two characteristics of angle and position of the edge. Edge features are described in terms of two variables, edge tangent angle and edge position, in a coordinate system with the center of the tactile sensor as the origin.
The data processing involves the use of probabilities in part of the Bayesian formula, so the angle and position are discretized in the present application. The angle is divided into N in the interval of 0 to 360 orn The positions of the intervals are divided into N at 0 to 10mm loc . For convenience of description, assume N orn ＝18，N loc As a reference case, each interval corresponds to a probability, as shown in fig. 3a and 3b. Discrete probability distribution information may be applied to a bayesian algorithm to derive an object edge profile.
The Bayesian formula related to calculating the edge angle and the position characteristic information is as follows:
z t (t=1, 2, 3.) for every t single haptic contact, z 1:t θ for all accumulated haptic information 1 st to t th i R is discretized angle information l Is discretized location information. P (z) t |r l ，θ i ) For the conditional probability at the time of the tth tactile sample, the calculation process is described in the previous step. P (r) l ，θ i |z 1：t-1 ) The accumulated prior probabilities for samples 1 st through t-1 st. P (z) t |z t-1 ) The marginal probability plays a role in normalization in the formula. The marginal probability for normalization is obtained by the following formula:
it should be noted that, the object edge feature extraction process is a process of repeatedly sampling the continuous accumulation probability. The posterior probability of step t-1 will be referred to as the prior probability of step t. The posterior probability may change during repeated sampling of the accumulated information. Thus, the marginal probability of the edge angle can be calculated by posterior probability.
Further, a marginal probability P (θ i |z 1:t ) And marginal probability of edge position P (r l |z 1:t ) The formula of (2) is as follows:
selecting the angle and the position interval with the highest corresponding marginal probability as the final cognition angle theta dec And position r dec As shown in fig. 4, the present application,
θ dec ＝argmax(P(θ i |z 1:t ))
r dec ＝argmax(P(r l |z 1:t ))
the acquired angle theta dec And position r dec And (5) representing the partial coordinates of the touch sensor.
It should be noted that, in the process of repeated sampling, as information is accumulated, the marginal probability distribution gradually shrinks and narrows, and the peak value of the probability distribution becomes higher and higher. Once the object edge angle probability P (θ i |z 1:t ) Peak exceeding the set threshold P dec When the sampling is completed, the sampling cycle is skipped, and the next sampling position is entered for continuous sampling, and reference is made to fig. 5.
Further, regarding the active exploration step:
it is important to explore the collection of haptic information. Unlike vision, the tactile sensor can collect a very small local range of information. In this case, it is necessary to search continuously, and to sample multiple times to reconstruct relatively complete information.
In this embodiment, the current haptic sensation exploration is calculated according to the angle and position information learned in the previous step by a predetermined control strategy. The output exploration action consists of two execution actions: one is the exploring action along the tangential direction of the edge wheel button, and the other is the position deviation correcting action along the normal vector of the edge. Please refer to fig. 6a and 6b.
The tactile sensor moves along the edge tangent by a fixed amount deltae and then moves along the edge in the orthogonal direction by deltar (t). Δe is a artificially set constant value, Δr (t) is a variable related to the edge position, and is obtained by the following formula:
Δr(t)＝g·(-r dec(t) )
g is a gain, which is a artificially set value greater than 0. r is (r) dec The position of the edge of the object relative to the coordinates of the tactile sensor is the data processed as mentioned in the previous step.
The tactile sensor changes its position and angle as it moves. In order to ensure the unification of coordinates, the coordinates are required to be transformed at any time, and information acquired by the tactile sensor is projected into a coordinate system of the tactile sensor.
Further, regarding the information encoding step:
the haptic information collected by the haptic sensor cannot be directly applied to the Bayesian algorithm, and the haptic information can be processed and analyzed only by being converted into probability distribution through the information coding process.
In the Bayesian formula mentioned in the above data processing
Conditional probability P (z) t |r l ，θ i ) From primary haptic sensingThe device information is converted. The current signal acquired by the touch sensor is z t A means is required to convert the original signal z t The conditional probability of converting angles and intervals P (z t |r l ，θ i )。
The application trains the neural network phi to learn the conversion relation by using a supervised learning method. The neural network inputs the original sensor signal, and the output is the corresponding dimension N loc N orn Is a probability distribution of (c). The samples required for supervised learning can be derived by sampling on objects of known shape.
P(z t |r l ,θ i )＝Φ(z t )
2. Vision-based object edge information extraction
Specifically, in the present embodiment, vision-based object edge extraction mainly includes:
calculating the curvature and the normal quantity of the object surface: acquiring object information acquired by a vision sensor, and extracting the object information by adopting a principal component analysis algorithm to obtain the curvature and normal vector of the object surface;
extracting a target curved surface: generating a structural point cloud according to the object surface curvature, the normal vector and the three-dimensional point cloud information, performing surface segmentation on the structural point cloud by adopting a region growing algorithm, and extracting target surface information;
edge contour extraction of the target plane: and extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm.
Further, regarding the object surface curvature and normal amount calculation step:
the calculation of the normal vector and curvature of the object may be obtained by a principal component analysis algorithm (PCA). This process requires traversing the entire point cloud to obtain the curvature and normal vector for all points within the point cloud.
The specific method comprises the following steps: selecting a point of target calculationThen selecting the point cloud and ++>Point of k nearest neighbor->After the adjacent points are selected, the point group is processed by using a principal component analysis algorithm (PCA).
The specific formula is as follows:
wherein the method comprises the steps ofIs the center point of the neighbor->Is the eigenvalue and eigenvector of covariance matrix C. The eigenvalues may be used to estimate the curvature of the surface and the minimum eigenvector may be used to estimate the normal vector of the surface.
The direction of the normal vector calculated by PCA is not fixed, the normal vectorThe direction of (2) can be adjusted by the following formulaWherein->To consider the selected viewing angle.
Further, the target curved surface extraction step:
the point cloud, the normal vector and the curvature obtained in the last step are subjected to post-processing to generate a structural point cloud so as to ensure that each point corresponds to a pixel point of the 2D color image one by one. The processed point cloud can divide the curved surface through a region growing algorithm and extract target curved surface information. The region segmentation algorithm uses three information of distance, normal vector and curvature to segment the point cloud into different planes. The segmentation uses a region growing algorithm, and the region growing needs to meet three conditions of three distances, normal vectors and curvature.
The distance between the current point and the adjacent point is smaller than a preset value
The curvature of the current point is smaller than a preset value
The difference between the normal vector angle of the current point and the adjacent point is smaller than a preset value
The specific algorithm flow is as follows:
further, an edge contour extraction step of the target plane:
the structural point cloud obtained by the steps corresponds to a certain pixel in the two-dimensional image at each point, so that a two-dimensional image processing algorithm can be used for extracting the curved surface edge contour. Extraction of edge contours requires the use of dilation and erosion methods in image processing.
Referring to fig. 7, the specific steps are as follows:
expanding the original image, adding pixel points, and expanding the area of the curved surface;
etching the original image to remove edge phase pixel points, and reducing the area of a curved surface;
exclusive or is carried out on the two points, and the rest points are plane edge points.
3. Sense fusion based on vision and touch
When vision touch fusion is carried out, the edge contour of the object is firstly converted into probability distribution, and the method comprises the following specific steps.
The point p on the edge closest to the tactile sensor is selected first, then the point on the edge contour nearest k is selected. A linear regression algorithm is used for fitting a straight line by using the selected k adjacent points, and the fitted straight line is an edge tangent line. The tangential angle is the angle of the edge profile and the distance of the tangential distance sensor is the distance of the edge profile from the sensor, as shown in fig. 8.
The edge angle and the edge position obtained through vision are determined values, and the values need to be converted into probability distribution to be fused with the touch information.
The visually acquired edge angles and positions are converted into probability distributions for 18 angle intervals and 20 position intervals by a gaussian distribution model. The formula of the gaussian distribution model is as follows:
the standard deviation sigma affects the accuracy of the probability distribution, k r And k is equal to θ Is a normalization parameter used to adjust the edge angle and edge position range. The closer the distance measurement angle θ and the measurement position r are, the higher the probability value assigned to the angle section and the position section is, see fig. 9.
The visually obtained edge angle and position features are reconverted into a probability distribution P (r l ，θ i |z 0 ) Then, the haptic discovery algorithm is initialized by using the prior probability as follows:
the prior probability P (r) l ，θ i |z 1：t-1 ) The probability distribution P (r) that would be calculated visually l ，θ i |z 0 ) And initializing assignment.
The object edge feature extraction process is a process of repeatedly sampling the continuously accumulated probability. The posterior probability of step t-1 will be referred to as the prior probability of step t. The algorithm will continually cycle through repeated samples, accumulate information, and improve accuracy in the process of sample accumulation. The posterior probability may change during repeated sampling of the accumulated information. We can calculate the marginal probability of the edge angle feature by posterior probability. Once the marginal probability of the object edge angle exceeds the set threshold, the process of sampling the object edge angle at the next sampling position and continuing to sample the visual tactile exploration algorithm is referred to as a schematic diagram 10, and the process of unfused visual tactile exploration algorithm is referred to as fig. 5.
In an embodiment of the present application, the edge angle is divided into 18 bins and the edge position offset is divided into 20 bins. Each interval is represented by a corresponding probability. Through the above algorithm, the angle and the edge offset with the largest probability are selected as the angle and the position of the final decision, referring to fig. 3a and 3b.
The angle with the highest marginal probability is selected as the final object edge angle characteristic:
selecting the position with the highest rear marginal rate as the edge position characteristic of the final object:
the edge angle and the position obtained in the above steps are described by a coordinate system with the touch sensor as an origin, and are local information relative to the touch sensor. Depending on the particular application, it may be desirable to obtain global information relative to the world coordinate system. The position and posture of the touch sensor are known, and if the characteristics of the edge outline of the object relative to the world coordinate system are to be acquired, the coordinate system is changed.
By implementing the object edge recognition method provided by the embodiment of the application, the recognition of the object edge features is realized by combining the vision and the touch, and the recognized object edge features can be used for the mechanical arm grabbing, so that the accuracy, the robustness and the reliability of the mechanical arm grabbing are improved.
Based on the same inventive concept, corresponding to the foregoing method embodiment, as shown in fig. 11, an embodiment of the present application provides an object edge recognition device based on vision and touch, including an extraction module 100 and a fusion module 200.
Wherein, the extraction module 100 is used for:
acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information;
processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
the fusion module 200 is used for:
fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object;
wherein the active haptic exploration algorithm comprises:
and a data processing step: acquiring t-1 th posterior probability in the sampling process of the touch sensor, calculating marginal probability of an object edge angle and marginal probability of an object edge position according to the posterior probability, and determining a final cognitive target angle and target position according to the marginal probability of the object edge angle and the marginal probability of the object edge position;
an active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, and executing the current touch exploration action by the touch sensor;
and an information encoding step: and obtaining current haptic information obtained by the haptic sensor executing the current haptic exploration action, encoding the current haptic information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1.
Optionally, the embodiment of the application also provides another object edge recognition device based on vision and touch. As shown in fig. 12, the edge recognition apparatus may include: one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104, the processors 101, input devices 102, output devices 103, and memory 104 being interconnected by a bus 105. The memory 104 is used for storing a computer program comprising program instructions, the processor 101 being configured to invoke the program instructions for performing the method of the above-described embodiment of the vision-and haptic-based object edge recognition method.
It should be appreciated that in embodiments of the present application, the processor 101 may be a central processing unit (Central Processing Unit, CPU), which may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSPs), application specific integrated circuits (Application SpecificIntegrated Circuit, ASICs), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 102 may include a keyboard or the like, and the output device 103 may include a display (LCD or the like), a speaker or the like.
The memory 104 may include read only memory and random access memory and provides instructions and data to the processor 101. A portion of the memory 104 may also include non-volatile random access memory. For example, the memory 104 may also store information of device type.
In a specific implementation, the processor 101, the input device 102, and the output device 103 described in the embodiments of the present application may execute the implementation described in the embodiment of the visual and tactile-based object edge recognition method provided in the first embodiment of the present application, which is not described herein.
It should be noted that, for a more specific workflow description of the two object edge recognition devices, please refer to the foregoing method embodiment section, and a detailed description is omitted herein.
According to the object edge recognition device, vision and touch are combined to realize recognition of object edge features, and the recognized object edge features can be used for mechanical arm grabbing, so that the accuracy, the robustness and the reliability of mechanical arm grabbing are improved.
Further, corresponding to the method and apparatus for identifying an edge of an object based on vision and touch according to the first embodiment, the embodiment of the present application further provides a readable storage medium storing a computer program, the computer program comprising program instructions which when executed by a processor implement: the object edge recognition method based on vision and touch of the first embodiment described above.
The computer readable storage medium may be an internal storage unit of the object edge identification device according to the foregoing embodiment, for example, a hard disk or a memory of the system. The computer readable storage medium may also be an external storage device of the system, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the system. Further, the computer readable storage medium may also include both internal storage units and external storage devices of the system. The computer readable storage medium is used to store the computer program and other programs and data required by the system. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
Based on the same inventive concept, the embodiment of the application provides an object edge recognition system based on vision and touch, which comprises a touch sensor, a vision sensor and an object edge recognition device which are communicated with each other, wherein the touch sensor and the time sensor are used for carrying out edge recognition on a detected object. The object edge recognition device is described in the foregoing embodiments, and is not described herein.
While the application has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (9)
1. A method for identifying edges of an object based on vision and touch, comprising:
acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information;
processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object;
wherein the active haptic exploration algorithm comprises:
and a data processing step: acquiring t-1 th posterior probability in the sampling process of the touch sensor, calculating marginal probability of an object edge angle and marginal probability of an object edge position according to the posterior probability, and determining a final cognitive target angle and target position according to the marginal probability of the object edge angle and the marginal probability of the object edge position; wherein t is a positive integer greater than 1;
an active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, and executing the current touch exploration action by the touch sensor; the current visual exploration action consists of two execution actions: one is an exploring action along the tangential direction of the edge contour, and the other is a position offset correcting action along the normal vector of the edge;
and an information encoding step: acquiring current haptic information obtained by the haptic sensor executing a current haptic exploration action, encoding the current haptic information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1;
wherein the data processing step is responsible for calculating edge angle and position characteristics, and the related formulas are as follows:
z t (t=1, 2, 3.) for every t word tactile contacts, z 1：t θ for all accumulated haptic information 1 st to t th i R is discretized angle information l For discretized position information, P (z t |r l ，θ i ) For conditional probability at the t-th haptic sample, P (r l ，θ i |z 1：t-1 ) The accumulated prior probabilities, P (z), for samples 1 st through t-1 st t |z t-1 ) The marginal probability can be obtained by the following formula:
calculating the marginal probability P (theta) of the object edge angle i |z 1：t ) And marginal probability of edge position P (r l |z 1：t ) The formula of (2) is as follows:
selecting the angle and the position interval with the highest corresponding marginal probability as the final cognition angle theta dec And position r dec ，
θ dec ＝argmax(P(θ i |z 1：t ))
r dec ＝argmax(P(r l |z 1：t ))
Acquired angle theta dec And position r dec And (5) representing the partial coordinates of the touch sensor.
2. The object edge identification method as claimed in claim 1, wherein the specific process of extracting the object edge profile 2 comprises:
calculating the curvature and the normal quantity of the object surface: acquiring object information acquired by a vision sensor, and extracting the object information by adopting a principal component analysis algorithm to obtain the curvature and normal vector of the object surface;
extracting a target curved surface: generating a structural point cloud according to the object surface curvature, the normal vector and the three-dimensional point cloud information, performing surface segmentation on the structural point cloud by adopting a region growing algorithm, and extracting target surface information;
edge contour extraction of the target plane: and extracting the edge contour of the target curved surface by adopting a two-dimensional image processing algorithm.
3. The method for recognizing an object edge according to claim 2, wherein the object information is extracted by a principal component analysis algorithm to obtain an object surface curvature and a normal vector, comprising:
selecting a point of target calculationSelecting a point +.>
And processing the three-dimensional point cloud information by using a principal component analysis algorithm, wherein the formula is as follows:
wherein,is the center point of the neighbor->The eigenvalues and eigenvectors of the covariance matrix C;
and estimating the surface curvature of the object according to the characteristic value, and estimating the normal vector according to the characteristic vector.
4. The method for recognizing an edge of an object according to claim 2, wherein the extracting the edge contour of the target curved surface by using a two-dimensional image processing algorithm comprises:
performing expansion pixel point adding processing on an original image corresponding to the target curved surface to obtain a first image with the enlarged curved surface area;
performing corrosion edge pixel point removal treatment on the original image corresponding to the target curved surface to obtain a second image with a reduced curved surface area;
and performing exclusive OR processing on the first image and the second image to obtain the residual points which are the curved surface edge points.
5. The method for identifying an edge of an object according to claim 1, wherein the object edge profile information 2 includes an edge angle and an edge position, and the obtaining the final edge feature of the object specifically includes:
converting the edge angles and the edge positions into probability distributions of a plurality of angle intervals and a plurality of position intervals by adopting a Gaussian distribution model;
and fusing the probability distribution with the object edge profile 1 to obtain the final edge characteristic of the object.
6. The object edge recognition device based on vision and touch is characterized by comprising an extraction module and a fusion module; the extraction module is used for:
acquiring global object touch information by using an active touch exploration algorithm, and extracting an object edge contour 1 according to the global object touch information;
processing three-dimensional point cloud information by using a visual algorithm, and extracting object edge contour information 2; the three-dimensional point cloud information is obtained by a visual sensor;
the fusion module is used for:
fusing the object edge profile 1 and the object edge profile 2 by using a Bayesian model to obtain the final edge characteristics of the object;
wherein the active haptic exploration algorithm comprises:
and a data processing step: acquiring t-1 th posterior probability in the sampling process of the touch sensor, calculating marginal probability of an object edge angle and marginal probability of an object edge position according to the posterior probability, and determining a final cognitive target angle and target position according to the marginal probability of the object edge angle and the marginal probability of the object edge position;
an active exploration step: calculating a current touch exploration action according to a preset control strategy, a final cognitive target angle and a target position, and executing the current touch exploration action by the touch sensor; the current visual exploration action consists of two execution actions: one is an exploring action along the tangential direction of the edge contour, and the other is a position offset correcting action along the normal vector of the edge;
and an information encoding step: acquiring current haptic information obtained by the haptic sensor executing a current haptic exploration action, encoding the current haptic information to obtain target data, and processing the target data by adopting a Bayesian algorithm to obtain the object edge profile 1;
wherein the data processing step is responsible for calculating edge angle and position characteristics, and the related formulas are as follows:
z t (t＝1，2，3...) For every t word tactile contacts, z 1：t θ for all accumulated haptic information 1 st to t th i R is discretized angle information l For discretized position information, P (z t |r l ，θ i ) For conditional probability at the t-th haptic sample, P (r l ，θ i |z 1：t-1 ) The accumulated prior probabilities, P (z), for samples 1 st through t-1 st t |z t-1 ) The marginal probability can be obtained by the following formula:
calculating the marginal probability P (theta) of the object edge angle i |z 1：t ) And marginal probability of edge position P (r l |z 1：t ) The formula of (2) is as follows:
selecting the angle and the position interval with the highest corresponding marginal probability as the final cognition angle theta dec And position r dec ，
θ dec ＝argmax(P(θ i |z 1：t ))
r dec ＝argmax(P(r l |z 1：t ))
Acquired angle theta dec And position r dec And (5) representing the partial coordinates of the touch sensor.
7. An object edge recognition apparatus based on vision and touch, characterized in that it comprises a processor, an input device, an output device and a memory, said processor, input device, output device and memory being interconnected, wherein said memory is adapted to store a computer program comprising program instructions, said processor being configured to invoke said program instructions to perform the method according to any of claims 1-5.
8. A readable storage medium storing a computer program comprising program instructions, which when executed by a processor implement the method of any one of claims 1-5.
9. An object edge recognition system based on vision and touch, comprising a touch sensor, a vision sensor and an object edge recognition device in communication with each other, the touch sensor and the vision sensor being used for edge recognition of a detected object, characterized in that the object edge recognition device is as claimed in claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010393054.4A CN111582186B (en) | 2020-05-11 | 2020-05-11 | Object edge recognition method, device, system and medium based on vision and touch |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010393054.4A CN111582186B (en) | 2020-05-11 | 2020-05-11 | Object edge recognition method, device, system and medium based on vision and touch |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111582186A CN111582186A (en) | 2020-08-25 |
| CN111582186B true CN111582186B (en) | 2023-12-15 |
Family
ID=72112146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010393054.4A Active CN111582186B (en) | 2020-05-11 | 2020-05-11 | Object edge recognition method, device, system and medium based on vision and touch |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111582186B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420735B (en) * | 2021-08-23 | 2021-12-21 | 深圳市信润富联数字科技有限公司 | Contour extraction method, device, equipment and storage medium |
| CN113714662B (en) * | 2021-09-06 | 2022-06-24 | 玖陆零信安创(盐城)科技有限公司 | Cutting size adjusting mechanism for intelligent household wood board cutting equipment |
| CN114851227B (en) * | 2022-06-22 | 2024-02-27 | 上海大学 | Device based on machine vision and touch sense fusion perception |
| CN115760805B (en) * | 2022-11-24 | 2024-02-09 | 中山大学 | Positioning method for processing element surface depression based on visual touch sense |
| CN116125840A (en) * | 2023-03-09 | 2023-05-16 | 四川长虹新网科技有限责任公司 | Intelligent feeding system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH05288543A (en) * | 1992-04-07 | 1993-11-02 | Fujitsu Ltd | Recognizing device integrated with visual information and tactile information |
| CN106272424A (en) * | 2016-09-07 | 2017-01-04 | 华中科技大学 | A kind of industrial robot grasping means based on monocular camera and three-dimensional force sensor |
| CN110091331A (en) * | 2019-05-06 | 2019-08-06 | 广东工业大学 | Grasping body method, apparatus, equipment and storage medium based on manipulator |
| CN111055279A (en) * | 2019-12-17 | 2020-04-24 | 清华大学深圳国际研究生院 | Multi-mode object grabbing method and system based on combination of touch sense and vision |
-
2020
- 2020-05-11 CN CN202010393054.4A patent/CN111582186B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH05288543A (en) * | 1992-04-07 | 1993-11-02 | Fujitsu Ltd | Recognizing device integrated with visual information and tactile information |
| CN106272424A (en) * | 2016-09-07 | 2017-01-04 | 华中科技大学 | A kind of industrial robot grasping means based on monocular camera and three-dimensional force sensor |
| CN110091331A (en) * | 2019-05-06 | 2019-08-06 | 广东工业大学 | Grasping body method, apparatus, equipment and storage medium based on manipulator |
| CN111055279A (en) * | 2019-12-17 | 2020-04-24 | 清华大学深圳国际研究生院 | Multi-mode object grabbing method and system based on combination of touch sense and vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111582186A (en) | 2020-08-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111582186B (en) | Object edge recognition method, device, system and medium based on vision and touch | |
| JP5406705B2 (en) | Data correction apparatus and method | |
| Vemulapalli et al. | R3DG features: Relative 3D geometry-based skeletal representations for human action recognition | |
| US9098740B2 (en) | Apparatus, method, and medium detecting object pose | |
| EP2680228B1 (en) | Improvements in or relating to three dimensional close interactions. | |
| EP3537375B1 (en) | Image segmentation methods, image segmentation system and device comprising same, and storage medium | |
| Sonkusare et al. | A review on hand gesture recognition system | |
| Nalepa et al. | Wrist localization in color images for hand gesture recognition | |
| US10147015B2 (en) | Image processing device, image processing method, and computer-readable recording medium | |
| JP2008191816A (en) | Image processor, image processing method, and computer program | |
| Al-Osaimi et al. | Integration of local and global geometrical cues for 3D face recognition | |
| Zhang et al. | Symbol recognition with kernel density matching | |
| KR20170024303A (en) | System and method for detecting feature points of face | |
| Guo et al. | A hybrid framework based on warped hierarchical tree for pose estimation of texture-less objects | |
| Thalmann et al. | First-person palm pose tracking and gesture recognition in augmented reality | |
| CN112541092B (en) | Three-dimensional image contour retrieval method and system based on tangential domain and storage medium | |
| KR20150043697A (en) | Texture-less object recognition using contour fragment-based features with bisected local regions | |
| KR101783629B1 (en) | System and method for detecting feature points of face | |
| JP5814714B2 (en) | Image similarity determination device, image similarity determination method, and image similarity determination program | |
| Chong et al. | Modeling video-based anomaly detection using deep architectures: Challenges and possibilities | |
| JP7364959B2 (en) | Feature point separation device for each subject, feature point separation method for each subject, and computer program | |
| JP2019039864A (en) | Hand recognition method, hand recognition program and information processing device | |
| Deutsch et al. | Finding structure in point cloud data with the robust isoperimetric loss | |
| Langs et al. | Automatic quantification of destructive changes caused by rheumatoid arthritis | |
| Świtoński et al. | Recognition of human gestures represented by depth camera motion sequences |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |