CN113902910B - Vision measurement method and system - Google Patents
Vision measurement method and system Download PDFInfo
- Publication number
- CN113902910B CN113902910B CN202111504115.0A CN202111504115A CN113902910B CN 113902910 B CN113902910 B CN 113902910B CN 202111504115 A CN202111504115 A CN 202111504115A CN 113902910 B CN113902910 B CN 113902910B
- Authority
- CN
- China
- Prior art keywords
- measurement
- geometric
- image
- primitive
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000691 measurement method Methods 0.000 title claims description 16
- 238000005259 measurement Methods 0.000 claims abstract description 127
- 238000000034 method Methods 0.000 claims abstract description 51
- 238000000605 extraction Methods 0.000 claims abstract description 41
- 238000012549 training Methods 0.000 claims abstract description 33
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 18
- 230000006399 behavior Effects 0.000 claims description 83
- 238000004364 calculation method Methods 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 31
- 230000000007 visual effect Effects 0.000 claims description 20
- 230000008569 process Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 12
- 238000003860 storage Methods 0.000 claims description 8
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- XAGFODPZIPBFFR-UHFFFAOYSA-N aluminium Chemical compound [Al] XAGFODPZIPBFFR-UHFFFAOYSA-N 0.000 description 15
- 229910052782 aluminium Inorganic materials 0.000 description 15
- 238000002372 labelling Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- AZDRQVAHHNSJOQ-UHFFFAOYSA-N alumane Chemical group [AlH3] AZDRQVAHHNSJOQ-UHFFFAOYSA-N 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000008672 reprogramming Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a vision measuring method and a system, wherein the method comprises the following steps: acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image; inputting the real-time image and the template image into a trained interesting outline primitive extraction model to obtain an outline primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives; based on the measurement behavior tree, geometric operation is carried out according to the outline element confidence map to obtain the measurement result of the object to be measured.
Description
Technical Field
The invention relates to the technical field of machine vision, in particular to a vision measurement method and a vision measurement system.
Background
The vision measurement is one of typical applications of machine vision, and aims to accurately acquire information such as a spatial pose, a size and a structure of a target object based on a camera model and an image. Along with the increasingly diversified and rapid update of tasks in the fields of industrial production, robot operation and the like, the flexible vision measurement system is an important link for realizing multifunctional robot and flexible intelligent manufacturing, has a comprehensive and practical prospect, and further needs to improve the flexibility degree in order to quickly adapt to various different measurement task requirements.
At present, a vision measurement system based on images is mostly developed for specific tasks and can only be used for object types and measurement requirements specified in the specific tasks, when the object types or the measurement requirements are changed, an expert is often required to reprogram, develop and debug, and quick function switching cannot be realized, so that the application of the vision measurement system is not flexible enough, particularly when facing to small-batch diversified objects to be measured, the flexibility and the intelligence degree of vision measurement are low, and the vision measurement of the objects to be measured cannot be realized accurately.
Therefore, a vision measuring method and system are needed to solve the above problems.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a vision measuring method and a system.
The invention provides a vision measuring method, which comprises the following steps:
acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image;
inputting the real-time image and the template image into a trained interesting outline primitive extraction model to obtain an outline primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives;
and performing geometric operation according to the profile element confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured.
According to the visual measurement method provided by the invention, the trained interesting outline primitive extraction model is obtained by the following steps:
acquiring an object image set containing a plurality of different style types;
marking interested contour primitives on sample object images in the object image set to obtain a training sample set; wherein the contour elements of interest comprise line segment type contour elements and/or arc type contour elements;
and training the weight parameters of the convolutional neural network based on the training sample set to obtain a trained interesting outline primitive extraction model.
According to the vision measuring method provided by the invention, before the geometric operation is performed according to the contour primitive confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured, the method further comprises the following steps:
constructing a geometric operation function library according to the geometric operation functions corresponding to the image space and the camera model;
and determining condition nodes and geometric calculation nodes based on preset geometric variable conditions and the geometric operation function library, and constructing a measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric calculation nodes.
According to the vision measuring method provided by the invention, the geometric operation function of the image space at least comprises the following steps: linear fitting, circle fitting, ellipse fitting, intersection point of two straight lines, included angle of the two straight lines, point-to-straight line distance, point-to-point distance and function of central lines of the two straight lines are solved;
the geometric operation function corresponding to the camera model at least comprises: affine transformation, PnP pose estimation, and binocular position measurement.
According to the visual measurement method provided by the invention, the building of the measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric calculation nodes comprises the following steps:
constructing corresponding measuring behavior subtrees through different geometric operation processes according to the condition nodes and the geometric calculation nodes;
and constructing the measurement behavior tree according to the plurality of measurement behavior subtrees.
According to the visual measurement method provided by the invention, after the real-time image and the template image are input into a trained extraction model of the interesting profile primitives to obtain a profile primitive confidence map corresponding to the real-time image, the method further comprises the following steps:
storing the profile primitive confidence map into a data blackboard, wherein the data blackboard comprises geometric variables required by the measuring behavior tree;
the obtaining of the measurement result of the object to be measured by performing geometric operation according to the profile element confidence map based on the measurement behavior tree includes:
and reading corresponding data in the data blackboard through the measuring behavior tree to perform geometric operation, so as to obtain a measuring result of the object to be measured.
The present invention also provides a vision measuring system comprising:
the image acquisition module is used for acquiring a real-time image of an object to be measured and determining a template image corresponding to the real-time image, wherein the template image is marked with an interesting contour element;
the contour primitive extraction module is used for inputting the real-time image and the template image into a trained interesting contour primitive extraction model to obtain a contour primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives;
and the measuring module is used for carrying out geometric operation according to the profile element confidence map based on the measuring behavior tree to obtain the measuring result of the object to be measured.
The present invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the vision measuring method as described in any one of the above when executing the program.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the vision measurement method as described in any of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, carries out the steps of the vision measuring method as described in any one of the above.
According to the visual measurement method and system provided by the invention, the model is obtained through convolutional neural network training, the confidence map of the interesting profile primitive on the object to be measured is extracted, the obtained measurement behavior tree is constructed based on the geometric calculation process, and geometric operation is carried out according to the confidence map of the interesting profile primitive, so that a user can efficiently and conveniently apply the visual measurement system to different tasks and objects only by configuring a template image, labeling the profile primitive and constructing a corresponding measurement behavior tree, the flexibility and the intelligence degree of the visual measurement system are obviously improved, and an accurate visual measurement result is obtained.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a vision measuring method provided by the present invention;
FIG. 2 is a schematic diagram illustrating labeling of interesting outline primitives of a template image provided by the present invention;
FIG. 3 is a schematic view of a measurement behavior tree constructed based on aluminum profiled parts according to the present invention;
FIG. 4 is a schematic structural diagram of a vision measuring system provided by the present invention;
fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When facing small-batch diversified objects to be measured, various measurement requirements of workpieces can be met on line through flexible visual measurement, the workpieces can simultaneously correspond to various different workpieces through reprogramming development and debugging, and the measurement data is fed back to processing equipment in real time by being combined with automatic transmission equipment on a production site. However, the application of the existing flexible vision measurement is still not flexible enough, when the object type or the measurement requirement changes, the measurement scheme needs to be reprogrammed again, and the accuracy of the vision measurement needs to be further improved.
In order to solve the problems of low flexibility degree, dependence on expert programming and insufficient application flexibility of the conventional vision measuring system, the invention provides a flexible intelligent vision measuring method based on interested contour primitives and a measuring behavior tree, which divides the vision measurement of an image into two stages of image feature extraction and geometric calculation, so that the image feature extraction process and the geometric calculation process are flexible and can be universally used for various different object types, thereby being applicable to different objects and tasks through simple and intuitive configuration and ensuring that the system deployment does not rely on complicated computer programming debugging.
Fig. 1 is a schematic flow chart of a vision measuring method provided by the present invention, and as shown in fig. 1, the present invention provides a vision measuring method, including:
step 101, acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image.
In the invention, an industrial camera is used for shooting an object to be measured to obtain a real-time image I of the object; then, determining a template image T according to the real-time image I, wherein the type of the object in the template image T is the same as that of the object to be measured, and the object in the template image T is marked with N interesting contour primitives
These interesting outline primitives mainly include line segment type outline primitives and circular arc type outline primitives, and through these interesting outline primitives, basic outline information of an object can be embodied in a template image.
in the invention, the real-time image I and the template image T obtained in the step 101 are input into a trained interesting outline primitive extraction model F to obtain a confidence map of N outline primitives in the real-time image I
，
. It should be noted that, in the present invention, the interesting outline primitive extraction model is obtained by labeling corresponding outline primitives for training based on sample images of various objects of different types during the training process, and during the actual vision measurement process, the real-time image I and the template image T contain objects of the same type, and the two images are allowed to have a certain difference in view angle, illumination and background.
Further, extracting the multi-scale characteristic graphs of the real-time image I and the template image T of the object by the model; then, based on the manual labeling
And using masked average pooling to obtain description vectors of N interesting contour primitives from the multi-scale feature map of the template image T
(ii) a Finally, based on cosine similarity measurement, comparing description vector pixel by pixel
And multi-scale characteristic map of real-time object image ISimilarity, obtaining N confidence maps
。
And 103, performing geometric operation according to the profile element confidence map based on the measurement behavior tree to obtain a measurement result of the object to be measured.
According to the invention, a behavior tree node and a topological structure for performing geometric calculation on the object to be measured can be designed according to actual measurement requirements. Specifically, a topological structure of the behavior tree is designed by using a graphical interface or a structured text, each node is defined, and the definition content comprises a node type, a node name, a node target, an input data number, an output data number and the like, so that the measurement behavior tree required by the invention is constructed.
And further, running a measuring behavior tree, and accessing each condition node and action node under the regulation and control of the flow nodes according to the contour primitive confidence map obtained in the step, thereby executing a series of geometric operation steps. Preferably, in the present invention, the acquired profile element confidence map and the geometric variables required by the measurement task are stored in the data blackboard, so that each node of the measurement behavior tree can directly read corresponding data (i.e. the profile element confidence map and the geometric variables) from the data blackboard, and after the measurement behavior tree performs a series of geometric operation steps, the geometric variables in the data blackboard are updated. Further, the measuring behavior tree reads a return value of the root node, judges whether the measurement is successful, and outputs a numerical value of a geometric variable of the object to be measured if the measurement is successful; and if the measurement fails, outputting internal error reporting information corresponding to the node returning to the failure state. It should be noted that the interesting outline primitive extraction model and the measurement behavior tree framework provided by the present invention are flexible and universal, and when aiming at a specific measurement task and an object to be measured, an instantiated measurement behavior tree needs to be formed by specific configuration according to a corresponding template image.
According to the visual measurement method provided by the invention, a model is obtained through convolutional neural network training, the confidence map of the interesting outline primitive on the object to be measured is extracted, the obtained measurement behavior tree is constructed based on the geometric calculation process, and geometric operation is carried out according to the confidence map of the interesting outline primitive, so that a user can efficiently and conveniently apply the visual measurement system to different tasks and objects only by configuring a template image, labeling the outline primitive and constructing a corresponding measurement behavior tree, the flexibility and the intelligence degree of the visual measurement system are obviously improved, and an accurate visual measurement result is obtained.
On the basis of the above embodiment, the trained contour primitive extraction model of interest is obtained by the following steps:
acquiring an object image set containing a plurality of different style types;
marking interested contour primitives on sample object images in the object image set to obtain a training sample set; wherein the contour elements of interest comprise line segment type contour elements and/or arc type contour elements.
And training the weight parameters of the convolutional neural network based on the training sample set to obtain a trained interesting outline primitive extraction model.
In the invention, N line segment type contour elements or arc type contour elements on the sample object image are regarded as interested contour elements, and corresponding labeling is carried out on the sample object image, so as to obtain the artificial labeling of N contour elements
And further obtaining a training sample set. In the invention, the sample object image marked with the interesting outline primitive can be used in the training process of the model, and also can provide the prior knowledge for the model when the model is actually used, so that the model can quickly determine the type of the object to be measured according to the prior knowledge, and extract the type of the object to be measured from the real-time object image ITaking out N interesting contour primitives and respectively using N confidence maps
Is represented in a confidence map
Where the confidence of the background pixel is 0, the pixels on the contour primitive of interest should have a confidence of approximately 1.
Further, in the invention, the trained extraction model F of the contour primitive of interest is obtained by training with a convolutional neural network, training is performed by inputting the training sample set into the convolutional neural network, and after a preset training condition is met (for example, a preset training number is reached), the training is stopped, so as to obtain the trained model. Preferably, the invention selects a ResNet-50 backbone network for model training.
On the basis of the above embodiment, before the geometric operation is performed according to the contour primitive confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured, the method further includes:
constructing a geometric operation function library according to the geometric operation functions corresponding to the image space and the camera model;
and determining condition nodes and geometric calculation nodes based on preset geometric variable conditions and the geometric operation function library, and constructing a measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric calculation nodes.
In the invention, before constructing the measuring behavior tree, a geometric operation function library is constructed firstly, and the function library is used for realizing a geometric calculation process required by visual measurement. Because the straight line fitting function and the ellipse fitting function have the functions of binaryzation and shape fitting, foreground pixels are collected from a binarized confidence map of the contour primitive of interest, and then the accurate geometric equation of a line segment or a circular arc is calculated by using a least square method or M-estimation.
Specifically, the geometric operation function of the image space at least includes: solving the intersection point of the two straight lines, solving the included angle of the two straight lines, solving the distance between points and the straight lines, solving the distance between the points and solving the function of the central lines of the two straight lines; the geometric operation function corresponding to the camera model at least comprises: linear fitting, circular fitting, ellipse fitting, affine transformation, PnP pose estimation, and binocular position measurement.
Further, in the geometric operation function library, a function library required by geometric calculation of an image space is calculated based on geometric parameters such as points, lines, ellipses and the like in the image space, and comprises straight line fitting, circle fitting, ellipse fitting, intersection point calculation of two straight lines, included angle calculation of the two straight lines, distance calculation of the points to the points, central lines calculation of the two straight lines and the like; geometric operation based on the camera model is to calculate geometric variables in a three-dimensional Cartesian space by using the calibrated camera model and geometric variables in an image space, and comprises affine transformation, PnP pose estimation, binocular position measurement and the like.
After the construction of the geometric operation function library is completed, a measurement behavior tree framework is further constructed for realizing flexible and easy-to-use geometric calculation process modeling. In the invention, the measuring behavior tree is a directed tree formed by four types of nodes including a root node, a flow node, a condition node and a geometric calculation node. The method comprises the steps of accessing in a depth-first mode and constructing nodes of a measurement behavior tree, wherein a root node is an initial node of the access measurement behavior tree, leaf nodes of the measurement behavior tree comprise condition nodes and geometric calculation nodes, and a flow node is used for determining the execution sequence of each leaf node. After each node except the root node is accessed and executed, a result of successful or failed execution is returned to the father node, and specifically, the basis of returning the execution result by each type of nodes is as follows:
1. condition nodes: when the specified geometric variable meets the specified condition, the node returns success; otherwise, the node returns failure;
2. geometric calculation node: when the input data format of the geometric computation function is correct and no error is reported in the intermediate process, the node returns success; otherwise, the node returns failure;
3. selecting a flow node: when one child node returns success, the node returns success and does not access the rest child nodes any more; if all child nodes return failures, the node returns failures;
4. and (3) sequential flow nodes: when all child nodes return success, the node returns success; otherwise, the node returns failure;
5. parallel flow nodes: when at least n child nodes in all the child nodes return success, the node returns success; otherwise the node returns a failure.
On the basis of the above embodiment, the constructing a measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric computation nodes includes:
constructing corresponding measuring behavior subtrees through different geometric operation processes according to the condition nodes and the geometric calculation nodes;
and constructing the measurement behavior tree according to the plurality of measurement behavior subtrees.
In the invention, subtrees of the behavior tree can be constructed based on different geometric operation processes, so that the measurement behavior tree has the characteristic of modularization, and the geometric computation stage is flexible by taking the constructed behavior tree as the subtree of the measurement behavior tree to be constructed according to different measurement task requirements.
On the basis of the above embodiment, after the real-time image and the template image are input to the trained extraction model of the contour primitive of interest to obtain the confidence map of the contour primitive corresponding to the real-time image, the method further includes:
storing the profile primitive confidence map into a data blackboard, wherein the data blackboard comprises geometric variables required by the measuring behavior tree;
the obtaining of the measurement result of the object to be measured by performing geometric operation according to the profile element confidence map based on the measurement behavior tree includes:
and reading corresponding data in the data blackboard through the measuring behavior tree to perform geometric operation, so as to obtain a measuring result of the object to be measured.
In the invention, the data blackboardIs used for declaring and storing the confidence map of the interesting outline primitive and all the geometric variables involved in the measurement task, and performs data initialization. The data in the data blackboard can be read and written by each node of the measuring behavior tree, the geometric variable corresponding to each data has a number #, and each node of the measuring behavior tree can access the specified geometric variable according to the number. Specifically, the line segment quadruples
It is shown that, among others,
is the parameter of the linear equation where the line segment is located, v represents whether the parameter of the linear equation is acquired or not,
a confidence map representing the # th contour primitive of interest. Circular arcs for quadruplets
It is shown that, among others,
the parameters of the equation representing the ellipse in which the arc lies,
indicates whether the parameters of the elliptical equation have been acquired,
is the confidence map for the # th contour primitive of interest.
Respectively representing the geometric variables of pose, point, angle and distance. It should be noted that the numbers of the input data and the output data of each node in the measurement behavior tree should correspond to the data in the data blackboard, so that the measurement behavior tree can read and write the corresponding data in the data blackboard during execution.
In the invention, N extracted confidence maps of the profile elements of interest are loaded into a data blackboard; then, operating the measuring behavior tree under the regulation and control of the flow nodes, accessing each condition node and each action node, thereby executing a series of geometric operation steps, updating geometric variables in the data blackboard, judging whether the measurement is successful or not by reading the return value of the root node, and outputting the geometric variable value of the object to be measured if the measurement is successful; and if the measurement fails, outputting internal error reporting information returned by each failure state node.
In an embodiment, an aluminum special-shaped part is selected as an object to be measured for explanation, fig. 2 is a schematic diagram of labeling of interesting profile primitives of a template image provided by the present invention, and referring to fig. 2, 7 interesting profile primitives are labeled on an object in the template image corresponding to the aluminum special-shaped part, and labeled data is stored in the format of an image or a parameter file, where the object in the template image is similar to the structure type of the object to be measured.
Furthermore, the measuring task of the invention is to acquire the six-degree-of-freedom pose of the aluminum special-shaped part relative to the camera coordinate system, and the geometric calculation process of the measurement is also related to the shape of the aluminum special-shaped part. Referring to fig. 2, the interface of the aluminum profile component is in a shape of a Chinese character 'tu', and a circular hole is formed in the middle of the aluminum profile component, the length of the aluminum profile component is 52mm, and the thickness of the aluminum profile component is 1 cm. It should be noted that, in the present invention, the geometric structure and the size of the object to be measured are known, and in order to obtain the pose of the object to be measured with 6 degrees of freedom with respect to the camera coordinate system, a model and a template image may be extracted based on the trained contour primitive of interest, 7 contour primitives of interest may be extracted from the aluminum special-shaped part, then, the measurement behavior tree is subjected to a series of geometric calculations to obtain 7 key points on the aluminum special-shaped part, and finally, the pose of the aluminum special-shaped part with six degrees of freedom may be calculated based on the PnP algorithm and the camera model.
Specifically, fig. 3 is a schematic diagram of a measurement behavior tree constructed based on an aluminum-made special-shaped part, as shown in fig. 3, two subtrees are located below a root node of the measurement behavior tree, the left subtree is used for extracting a key point from an image, the right subtree is used for executing a PnP algorithm, and if the left subtree returns to a success state, the PnP algorithm is used for calculating a pose with 6 degrees of freedom; if the left subtree returns a failure status, the PnP algorithm is not executed and the root node returns a failure. The measuring behavior tree (located in the left sub-tree) has two parallel nodes, and the parallel node returns success as long as at least 2 sub-nodes return success from the 3 sub-nodes of the first parallel node; and if at least 2 child nodes in 4 child nodes of the second parallel node return success, the parallel node returns success. Therefore, when partial key points are missing due to the fact that partial interesting contour primitives are extracted unsuccessfully, as long as at least 4 key points in 7 key points in the image are acquired successfully, the root node of the measurement behavior tree returns to a success state, and therefore robustness of visual measurement to the loss condition of the interesting contour primitives is improved.
Further, in the present invention, there are 15 data variables in the data blackboard, the first 7 variables store 7 interesting contour primitives (including straight line 1, straight line 2, straight line 3, straight line 4, straight line 5, straight line 6 and circular arc 7), and in addition, the image coordinates of 7 key points to be solved in the object to be measured (i.e. key point 8 to key point 14 referred to in fig. 3) and the pose of 6 degrees of freedom of the object to be measured are updated by the automatic execution of the measurement behavior tree. After extracting the interesting contour primitive of the aluminum special-shaped part, geometric operation processing is carried out through a measuring behavior tree, wherein a key point 8 is obtained by solving the intersection point of straight lines 1 and 2, a key point 9 is obtained by solving the intersection point of the straight lines 2 and 3, a key point 10 is obtained by solving the center of a circular arc 7, a key point 11 is obtained by solving the intersection point of the straight lines 1 and 5, a key point 12 is obtained by solving the intersection point of the straight lines 3 and 5, a key point 13 is obtained by solving the intersection point of the straight lines 4 and 5, and a key point 14 is obtained by solving the intersection point of the straight lines 5 and 6.
According to the vision measurement method provided by the invention, a plurality of interesting contour elements are extracted from the image of the object to be measured according to the prior knowledge provided by the template image and the manual label, and the calculation process from the interesting contour elements to the measured value is realized based on the measurement behavior tree. Compared with the existing specialized vision measuring method, the vision measuring method is more flexible and flexible. After the interesting outline extraction model construction and measurement behavior tree framework is constructed, a user can enable the vision measurement system to be efficiently and conveniently applied to different tasks and objects only by configuring template images, labeling outline primitives and designing a measurement behavior tree, and the flexibility and the intelligence degree of the vision measurement system are obviously improved.
The following describes the vision measuring system provided by the present invention, and the vision measuring system described below and the vision measuring method described above may be referred to in correspondence with each other.
Fig. 4 is a schematic structural diagram of a vision measuring system provided by the present invention, and as shown in fig. 4, the present invention provides a vision measuring system, which includes an image obtaining module 401, a contour primitive extracting module 402 and a measuring module 403, where the image obtaining module 401 is configured to obtain a real-time image of an object to be measured, and determine a template image corresponding to the real-time image, and the template image is marked with a contour primitive of interest; the contour primitive extraction module 402 is configured to input the real-time image and the template image into a trained extraction model of a contour primitive of interest, so as to obtain a confidence map of the contour primitive corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives; the measurement module 403 is configured to perform geometric operation according to the profile primitive confidence map based on the measurement behavior tree to obtain a measurement result of the object to be measured.
In the present invention, the image acquisition module 401 captures an object to be measured by an industrial camera, and acquires a real-time image of the object. In the invention, the model of the industrial camera is BaslercA 2440-35uc, the focal length of the lens is 25 mm, and the internal parameters of the industrial camera are calibrated. The object to be measured is a profiled aluminium part of known geometry, placed next to a checkerboard calibration plate and left still. Preferably, in the invention, the vision measuring system is provided with a mechanical arm (model abbibrb 1200) for changing the position and the posture of the industrial camera, so that the industrial camera shoots the object to be measured from different angles, distances and relative positions, and the background and the illumination near the object to be measured can be randomly changed in the shooting process.
Further, after the image obtaining module 401 obtains an image of the part, the image is input to the contour primitive extracting module 402, an interesting contour primitive of the image of the part is extracted, and the pose of the aluminum special-shaped part with six degrees of freedom relative to the camera coordinate system is calculated according to the interesting contour primitive through the measuring behavior tree in the measuring module 403.
Specifically, the contour primitive extraction module 402 extracts 7 contour primitives of interest from the current image (i.e. the image of the aluminum profile part) based on the contour primitive extraction model of interest, and then the measurement module 403 calculates 7 key points on the object to be measured and 6-degree-of-freedom pose of the object to be measured with respect to the camera coordinate system according to the 7 contour primitive confidence maps based on the designed measurement behavior treeCTO。
In order to verify the vision measurement precision, the invention also measures the pose of the checkerboard relative to the camera coordinate system based on the checkerboard calibration board and the PnP algorithmWTODue to the precise checkerboard structure and the large number of characteristic points, thereforeWTOThe measurement precision of (2) is very high, the relative pose between the object to be measured and the checkerboard can pass
And (4) calculating. Since the object to be measured and the grid are relatively stationary,WTOtheoretically a constant. By moving the camera to 50 different poses, measurements can be taken each timeWTOAndCTOis calculated to obtainWTOThe measurement success rate is 90%; then, according to 50WTOStatistical three-dimensional position coordinates (x, y, z) and three-dimensional attitude euler angles (f)x,fy,fz) The standard deviations of (a) are 1.5 mm, 0.2 mm, 6.2 mm, 7.7 degrees, 3.4 degrees and 0.6 degrees, respectively.
According to the visual measurement system provided by the invention, the model is obtained through convolutional neural network training, the confidence map of the interesting outline primitive on the object to be measured is extracted, the obtained measurement behavior tree is constructed based on the geometric calculation process, and geometric operation is carried out according to the confidence map of the interesting outline primitive, so that a user can efficiently and conveniently apply the visual measurement system to different tasks and objects only by configuring a template image, labeling the outline primitive and constructing the corresponding measurement behavior tree, the flexibility and the intelligence degree of the visual measurement system are obviously improved, and the accurate visual measurement result is obtained.
The system provided by the present invention is used for executing the above method embodiments, and for the specific processes and details, reference is made to the above embodiments, which are not described herein again.
Fig. 5 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 5, the electronic device may include: a processor (processor) 501, a communication interface (communication interface) 502, a memory (memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 are communicated with each other through the communication bus 504. The processor 501 may invoke logic instructions in the memory 503 to perform a vision measurement method comprising: acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image; inputting the real-time image and the template image into a trained interesting outline primitive extraction model to obtain an outline primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives; and performing geometric operation according to the profile element confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured.
In addition, the logic instructions in the memory 503 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform a vision measurement method provided by the above methods, the method comprising: acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image; inputting the real-time image and the template image into a trained interesting outline primitive extraction model to obtain an outline primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives; and performing geometric operation according to the profile element confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the vision measuring method provided by the above embodiments, the method comprising: acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image; inputting the real-time image and the template image into a trained interesting outline primitive extraction model to obtain an outline primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives; and performing geometric operation according to the profile element confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (7)
1. A vision measuring method, comprising:
acquiring a real-time image of an object to be measured, and determining a template image corresponding to the real-time image, wherein an interesting outline element is marked on the template image;
inputting the real-time image and the template image into a trained interesting outline primitive extraction model to obtain an outline primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives;
based on the measurement behavior tree, performing geometric operation according to the profile element confidence map to obtain a measurement result of the object to be measured;
before the geometric operation is performed according to the contour primitive confidence map based on the measurement behavior tree to obtain the measurement result of the object to be measured, the method further comprises:
constructing a geometric operation function library according to the geometric operation functions corresponding to the image space and the camera model;
determining condition nodes and geometric calculation nodes based on preset geometric variable conditions and the geometric operation function library, and constructing a measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric calculation nodes;
after the real-time image and the template image are input into the trained extraction model of the contour primitive of interest to obtain the confidence map of the contour primitive corresponding to the real-time image, the method further comprises the following steps:
storing the profile primitive confidence map into a data blackboard, wherein the data blackboard comprises geometric variables required by the measuring behavior tree;
the obtaining of the measurement result of the object to be measured by performing geometric operation according to the profile element confidence map based on the measurement behavior tree includes:
and reading corresponding data in the data blackboard through the measuring behavior tree to perform geometric operation, so as to obtain a measuring result of the object to be measured.
2. A visual measurement method according to claim 1, wherein the trained contour primitive extraction model of interest is obtained by:
acquiring an object image set containing a plurality of different style types;
marking interested contour primitives on sample object images in the object image set to obtain a training sample set; wherein the contour elements of interest comprise line segment type contour elements and/or arc type contour elements;
and training the weight parameters of the convolutional neural network based on the training sample set to obtain a trained interesting outline primitive extraction model.
3. The vision measuring method of claim 1, wherein the geometric operation function of the image space at least includes: linear fitting, circle fitting, ellipse fitting, intersection point of two straight lines, included angle of the two straight lines, point-to-straight line distance, point-to-point distance and function of central lines of the two straight lines are solved;
the geometric operation function corresponding to the camera model at least comprises: affine transformation, PnP pose estimation, and binocular position measurement.
4. The visual measurement method of claim 1, wherein the constructing a measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric computation nodes comprises:
constructing corresponding measuring behavior subtrees through different geometric operation processes according to the condition nodes and the geometric calculation nodes;
and constructing the measurement behavior tree according to the plurality of measurement behavior subtrees.
5. A vision measuring system, comprising:
the image acquisition module is used for acquiring a real-time image of an object to be measured and determining a template image corresponding to the real-time image, wherein the template image is marked with an interesting contour element;
the contour primitive extraction module is used for inputting the real-time image and the template image into a trained interesting contour primitive extraction model to obtain a contour primitive confidence map corresponding to the real-time image; the trained interesting outline primitive extraction model is obtained by training a convolutional neural network by using an object image set marked with sample outline primitives;
the measuring module is used for carrying out geometric operation according to the profile element confidence map based on the measuring behavior tree to obtain a measuring result of the object to be measured;
the system is further configured to:
constructing a geometric operation function library according to the geometric operation functions corresponding to the image space and the camera model;
determining condition nodes and geometric calculation nodes based on preset geometric variable conditions and the geometric operation function library, and constructing a measurement behavior tree corresponding to the object to be measured according to the condition nodes and the geometric calculation nodes;
the system is further configured to:
storing the profile primitive confidence map into a data blackboard, wherein the data blackboard comprises geometric variables required by the measuring behavior tree;
the measurement module is specifically configured to:
and reading corresponding data in the data blackboard through the measuring behavior tree to perform geometric operation, so as to obtain a measuring result of the object to be measured.
6. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the vision measurement method according to any of claims 1 to 4 when executing the computer program.
7. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the vision measurement method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111504115.0A CN113902910B (en) | 2021-12-10 | 2021-12-10 | Vision measurement method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111504115.0A CN113902910B (en) | 2021-12-10 | 2021-12-10 | Vision measurement method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113902910A CN113902910A (en) | 2022-01-07 |
| CN113902910B true CN113902910B (en) | 2022-04-08 |
Family
ID=79025526
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111504115.0A Active CN113902910B (en) | 2021-12-10 | 2021-12-10 | Vision measurement method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113902910B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108615229A (en) * | 2018-03-09 | 2018-10-02 | 南京信息工程大学 | Collision detection optimization method based on curvature points cluster and decision tree |
| CN111161229A (en) * | 2019-12-23 | 2020-05-15 | 大连理工大学 | Change detection method based on geometric active contour model and sparse self-coding |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2168815B1 (en) * | 2008-09-24 | 2014-06-04 | Robert Bosch GmbH | Method and device for detecting possibly colliding objects in a blind spot area |
| US10510154B2 (en) * | 2017-12-21 | 2019-12-17 | Intel Corporation | Adaptive processing of spatial imaging data |
| CN110891044B (en) * | 2018-09-11 | 2021-04-27 | 中国科学院信息工程研究所 | NPC generation and depiction method in network test scene |
| EP3734507A1 (en) * | 2019-05-03 | 2020-11-04 | Essilor International | Apparatus for machine learning-based visual equipment selection |
| CN111062955A (en) * | 2020-03-18 | 2020-04-24 | 天津精诊医疗科技有限公司 | Lung CT image data segmentation method and system |
| CN111340800B (en) * | 2020-03-18 | 2024-02-27 | 联影智能医疗科技(北京)有限公司 | Image detection method, computer device, and storage medium |
| CN111753692B (en) * | 2020-06-15 | 2024-05-28 | 珠海格力电器股份有限公司 | Target object extraction method, product detection method, device, computer and medium |
| CN112837805B (en) * | 2021-01-12 | 2024-03-29 | 浙江大学 | Eyelid topological morphology feature extraction method based on deep learning |
| CN112985293B (en) * | 2021-01-14 | 2023-05-30 | 北京航空航天大学 | Binocular vision measurement system and measurement method for single-camera double-spherical mirror image |
| CN113192105B (en) * | 2021-04-16 | 2023-10-17 | 嘉联支付有限公司 | Method and device for indoor multi-person tracking and attitude measurement |
| CN113435442B (en) * | 2021-05-07 | 2022-11-25 | 三峡大学 | Water level measuring method and device, water gauge and electronic equipment |
| CN113310433A (en) * | 2021-05-27 | 2021-08-27 | 伏燕军 | Virtual binocular stereo vision measuring method based on line structured light |
| CN113298837B (en) * | 2021-07-27 | 2021-11-26 | 南昌工程学院 | Image edge extraction method and device, storage medium and equipment |
-
2021
- 2021-12-10 CN CN202111504115.0A patent/CN113902910B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108615229A (en) * | 2018-03-09 | 2018-10-02 | 南京信息工程大学 | Collision detection optimization method based on curvature points cluster and decision tree |
| CN111161229A (en) * | 2019-12-23 | 2020-05-15 | 大连理工大学 | Change detection method based on geometric active contour model and sparse self-coding |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113902910A (en) | 2022-01-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108656107B (en) | Mechanical arm grabbing system and method based on image processing | |
| CN109816049B (en) | Assembly monitoring method and device based on deep learning and readable storage medium | |
| CN110827383B (en) | Attitude simulation method and device of three-dimensional model, storage medium and electronic equipment | |
| CN109472828B (en) | Positioning method, positioning device, electronic equipment and computer readable storage medium | |
| CN109410316B (en) | Method for three-dimensional reconstruction of object, tracking method, related device and storage medium | |
| Labbé et al. | Single-view robot pose and joint angle estimation via render & compare | |
| CN105139007B (en) | Man face characteristic point positioning method and device | |
| US20160098615A1 (en) | Apparatus and method for producing image processing filter | |
| WO2021009062A1 (en) | Methods of estimating a bare body shape from a concealed scan of the body | |
| CN104919492A (en) | Device for detecting feature-point position, method for detecting feature-point position, and program for detecting feature-point position | |
| CN113393524A (en) | Target pose estimation method combining deep learning and contour point cloud reconstruction | |
| Sun et al. | TriVoC: Efficient voting-based consensus maximization for robust point cloud registration with extreme outlier ratios | |
| CN113902910B (en) | Vision measurement method and system | |
| CN113160401A (en) | Object-oriented visual SLAM lightweight semantic map creation method | |
| Solichah et al. | Marker-less motion capture based on openpose model using triangulation | |
| Hajder et al. | Automatic estimation of sphere centers from images of calibrated cameras | |
| US20230071291A1 (en) | System and method for a precise semantic segmentation | |
| CN113947716A (en) | Closed loop detection method, closed loop detection device, robot and storage medium | |
| Tushev et al. | Parallel algorithms for effective correspondence problem solution in computer vision | |
| CN112926550A (en) | Human-computer interaction method and device based on three-dimensional image human body posture matching | |
| CN112906432A (en) | Error detection and correction method applied to human face key point positioning task | |
| Woo et al. | An efficient method for camera calibration using multilayer perceptron type neural network | |
| CN115222799B (en) | Method and device for acquiring image gravity direction, electronic equipment and storage medium | |
| CN112465898B (en) | Object 3D pose tag acquisition method based on checkerboard calibration plate | |
| CN117274525B (en) | Virtual tape measure measurement simulation method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |