CN113255702B - Target detection method and target detection device based on graph matching - Google Patents

Target detection method and target detection device based on graph matching Download PDF

Info

Publication number
CN113255702B
CN113255702B CN202110739846.7A CN202110739846A CN113255702B CN 113255702 B CN113255702 B CN 113255702B CN 202110739846 A CN202110739846 A CN 202110739846A CN 113255702 B CN113255702 B CN 113255702B
Authority
CN
China
Prior art keywords
vertex
vertexes
detected
matching result
standard template
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110739846.7A
Other languages
Chinese (zh)
Other versions
CN113255702A (en
Inventor
李杰明
杨洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Huahan Weiye Technology Co ltd
Original Assignee
Shenzhen Huahan Weiye Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Huahan Weiye Technology Co ltd filed Critical Shenzhen Huahan Weiye Technology Co ltd
Priority to CN202110739846.7A priority Critical patent/CN113255702B/en
Publication of CN113255702A publication Critical patent/CN113255702A/en
Application granted granted Critical
Publication of CN113255702B publication Critical patent/CN113255702B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

The application relates to a target detection method and a target detection device based on graph matching, wherein the target detection method is used for detecting local characteristics of an image to be detected according to a preset vertex detection model to obtain a plurality of vertexes on a target object; matching a plurality of vertexes on a target object with a standard template according to a preset image matching algorithm to obtain a first matching result; calculating matching scores and forming each vertex in the first matching result into a second matching result according to the matching scores; deducing undetected vertexes on the target object through the variation relation between each vertex in the second matching result and the standard template, and forming a third matching result by using the detected vertexes and the undetected vertexes; and determining the position and the angle of the target object in the image to be detected according to the third matching result, and forming a detection result of the target object. The technical scheme can improve the accuracy of target object detection, thereby obtaining a better detection result.

Description

Target detection method and target detection device based on graph matching
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a target detection method and a target detection apparatus based on graph matching.
Background
In recent years, artificial intelligence and big data have become the focus of attention in various fields at home and abroad. In the field of computer vision, the image algorithm based on deep learning has wide application. The convolutional neural network is trained by using the one-to-one correspondence of the images and the labeling information, and the work of classification, target detection, semantic segmentation and the like can be completed. The target detection convolutional neural network has a great number of applications in industry, such as identification and counting of pipeline products.
The target detection convolutional neural network (hereinafter referred to as target detection network) still faces many difficulties, such as identification of an occlusion object, identification of an extreme dimension and a shape object; furthermore, accurate recognition of the direction of an object is also a great demand for industrial machine vision. The existing target detection algorithms based on deep learning include YOLO, SSD, RCNN, etc., which are trained by building a CNN (convolutional neural network) and using labeled data, after training is completed, an image is input to the convolutional neural network and a feature map is output, and detection results, such as the type of an object, the central coordinate and the length and width of an external rectangle, are obtained by calculating the feature map.
In existing applications, identifying occluding objects is one of the difficulties in target detection. One technique for optimizing the recognition of occluding objects is: enhancing the training set image, such as randomly covering a portion of the object to be detected in the image with noise, or randomly setting pixels of the portion of the object to be detected in the image to a fixed pixel value (e.g., 0); the disadvantage of this method is that too much noise may be introduced into the training data, making convergence of the target detection model more difficult, and the training data is not necessarily processed in the same way as the real occlusion, so that erroneous detection results may still be obtained. In addition, the method for optimizing the multi-scale detection comprises the following steps: the characteristic pyramid is used for obtaining a plurality of characteristic graphs with different receptive fields by fusing the characteristic graphs with different scales, and then classifying and performing square regression on the characteristic graphs respectively to obtain results; the disadvantage of this method is that the amount of calculation is increased, and the aspect ratio of the shape of the object to be identified needs to be in a moderate interval (typically between 1:3 and 3: 1), and the detection of an object with an extreme aspect ratio is difficult to achieve. The three methods for identifying the object direction comprise the following steps: on the basis of an original target detection network, one or more angle regressions are added, fitting is carried out through training data, and the direction of an object can be identified; the method has the disadvantages that angle labeling needs to be additionally added during labeling, so that the workload of labeling is increased, and the training difficulty is increased due to the addition of angle regression.
Disclosure of Invention
The technical problem that this application mainly solved is: how to accurately detect and identify a target object in an image to be detected. In order to solve the above technical problem, the present application provides a target detection method and a target detection apparatus based on graph matching.
According to a first aspect, an embodiment provides a target detection method based on graph matching, including: acquiring an image to be detected of a target object; detecting local characteristics of the image to be detected according to a preset vertex detection model to obtain a plurality of vertexes of the target object; the vertex is used for representing a local feature of the surface of the object; matching a plurality of vertexes on the target object with a standard template according to a preset image matching algorithm to obtain matched vertexes and form a first matching result; the standard template is an information set of all vertexes on a standard object corresponding to the target object; calculating a matching score of the first matching result relative to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold; deducing undetected vertexes on a target object in the image to be detected according to the variation relation between each vertex in the second matching result and the standard template, and forming a third matching result by using each vertex in the second matching result and the undetected vertexes; and determining the position and the angle of the target object in the image to be detected according to the third matching result, and forming a detection result of the target object.
The vertex detection model is obtained by training a convolutional neural network based on deep learning, and the training process includes: obtaining at least one sample image of a target object, wherein the sample image comprises position information, angle information and category information of a plurality of vertexes on the target object; inputting the sample image into the convolutional neural network, iteratively updating network parameters until a loss function corresponding to the convolutional neural network is converged, and taking the trained convolutional neural network as the vertex detection model.
The matching of the plurality of vertexes on the target object with a standard template according to a preset graph matching algorithm to obtain matched vertexes and form a first matching result includes: acquiring a standard template of a standard object corresponding to the target object and an information search range of the standard template; the standard template comprises position information, angle information and category information of all vertexes on the standard object, and the information search range is used for setting detection ranges of angles and positions; forming a set of vertexes to be detected by using a plurality of vertexes detected in the image to be detected, calculating the position change relation between any two vertexes in the set of vertexes to be detected and any two vertexes in the standard template, and constructing the connection relation between the vertexes in the set of vertexes to be detected; comparing the connection relation between each vertex and other vertexes in the vertex set to be detected, and adding the vertex and other vertexes of which the connection relation accords with a preset screening condition into the vertex set X h And determining a change relationship
Figure 100002_DEST_PATH_IMAGE001
(ii) a Set the vertex X h The change relation corresponding to each vertex and the determined change relation
Figure 494518DEST_PATH_IMAGE001
Comparing, and determining the vertex set X by voting h A plurality of vertexes matched with the standard template; and obtaining the first matching result by utilizing the corresponding relation between the matched vertexes and the corresponding vertexes in the standard template.
The calculating the position change relationship between any two vertexes in the vertex set to be detected and any two vertexes in the standard template, and constructing the connection relationship between the vertexes in the vertex set to be detected, includes: for any two vertexes in the vertex set to be detectedk、lAnd any two vertices in the standard templatei、jComputing verticeskAnd vertexlIs expressed asβ kl Computing verticesiAnd vertexjIs expressed asβ ij (ii) a If the vertex is judgediAnd vertexkAre the same, and the vertexjAnd vertexlIf the category information of the two is the same, the position relation is calculatedβ ij Relative to the position relationβ kl Is expressed asδ ij-kl (ii) a The variation relation is used for representing the angle offset and the distance scaling amount of the relative conversion; judging the relationship of changesδ ij-kl Whether the vertex is in the information searching range of the standard template, if so, constructing the vertex in the set of the vertices to be detectedkAnd vertexlIn connection withγ kl Represents; angles are set in the information search rangeαCoordinates of the samexCoordinates of the sameyAnd a detection range of the distance scaling scale.
Comparing the connection relation between each vertex and other vertexes in the vertex set to be detected, and adding the vertex and other vertexes of which the connection relation accords with a preset screening condition into the vertex set X h And determining a change relationship
Figure 67451DEST_PATH_IMAGE002
The method comprises the following steps: judging that the vertex set to be detected is constructed with the connection relation between the vertexes, and storing all the vertexes in the vertex set to be detected into a stackS t (ii) a From the stackS t Sequentially popping each vertex and recording as the vertexhSequentially establishing a vertex set X h And make the vertexhAdding vertex set X h (ii) a Traversing the vertices in the set of vertices to be detectedhOther vertexes with connection relation therebetween, voting to determine one of the connection relations and obtaining a corresponding change relation
Figure 378347DEST_PATH_IMAGE002
And using the sum vertexhOther vertexes having a connection relation therebetween form a candidate point setP(ii) a For the candidate point setPEach vertex in (1)oObtaining verticesoThe connection relation and the corresponding change relation with each vertex in the set of vertices to be detected, if the vertex is judgedoAnd the vertex set X h The variation relation corresponding to the connection relation of any one vertex in the set is equal to the variation relation
Figure 180081DEST_PATH_IMAGE002
Then the vertex is putoTemporarily joining the vertex set X h Performing the following steps; then obtaining the vertex set X h Newly added internal connection relation among internal vertexesl in And each internal vertex and the vertex set X h Newly added external connection relation between external vertexesl out Judging the newly added internal connection relationshipl in Whether the number of the external connection relation is less than the newly added external connection relationl out If so, the verticesoMoving out of the vertex set X h If not, the new external connection relation is addedl out Upper and top pointoThe associated vertices are also added to the set of candidate pointsP(ii) a Traverse the set of candidate pointsPAnd updating the set of vertices X h Outputting the vertex set X finally formed h And the determined change relation
Figure 525611DEST_PATH_IMAGE001
。
The vertex is set to X h The change relation corresponding to each vertex and the determined change relation
Figure 3866DEST_PATH_IMAGE001
Comparing, and determining the vertex set X by voting h The plurality of vertexes matched with the standard template comprise: obtaining the vertex set X h Middle vertexhConnections to the remaining verticesRelationships and corresponding variation relationships; if the vertex ishThe variation relation corresponding to the connection relation between any other vertex is equal to the variation relation
Figure 802058DEST_PATH_IMAGE001
If so, marking the connection relation as a first value; and counting and voting the vertexes marked as the first values in the connection relation to obtain voting results of all the vertexes in the vertex set, and determining a plurality of vertexes matched with the standard template according to the voting results.
The calculating a matching score of the first matching result relative to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold value includes: calculating an affine transformation matrix of the standard template to the first matching result and representing asG(ii) a Transforming the standard template to a coordinate system which is the same as the first matching result, and calculating a back projection error; the back projection error is formulated as
Figure 204220DEST_PATH_IMAGE003
(ii) a Wherein the content of the first and second substances,p c representing any vertex in the standard template, satisfyingp c ϵ M 1And isM 1Representing the standard template;q c representing the AND in the first matching resultp c Corresponding vertex, satisfyq c ϵ F 1And isF 1Representing the first matching result; calculating a matching score of the first matching result relative to the standard template according to the back projection error, wherein the matching score is expressed as
Figure DEST_PATH_IMAGE004
(ii) a Wherein the content of the first and second substances,edenotes the natural logarithmF 1| is the number of vertices in the first matching result, countM 1I is the number of the top points in the standard template; judgment ofμ>λThen, the first matching result is used to formA second matching result; wherein the content of the first and second substances,λis a preset score threshold.
Deducing undetected vertexes on a target object in the image to be detected through the variation relation between each vertex in the second matching result and the standard template, and forming a third matching result by using each vertex in the second matching result and the undetected vertexes, wherein the third matching result comprises the following steps: constructing the inferred undetected vertexes on the target object in the image to be detected into a first setTComparing the standard template with the first setTThe corresponding vertexes are formed into a second setR(ii) a The first setTAnd the second setRCan satisfyt=Gr(ii) a Wherein the content of the first and second substances,rrepresenting the second setRAnd satisfy any vertex inrϵ R，tRepresenting the first setTNeutralization ofrCorresponding vertex and satisfytϵ T(ii) a If it is usedF 2If the second matching result is represented, the third matching result is represented as
Figure 60050DEST_PATH_IMAGE005
。
According to a second aspect, there is provided in an embodiment an object detection apparatus comprising: the camera is used for obtaining an image to be detected of the target object by image capture; a processor connected to the camera, configured to process the image to be detected by the target detection method in the first aspect, so as to obtain a detection result of the target object; and the display is connected with the processor and used for displaying the image to be detected and/or the detection result of the target object.
The processor comprises an acquisition module, a detection module and a processing module, wherein the acquisition module is used for acquiring an image to be detected of a target object from the camera; the first detection module is used for detecting local characteristics of the image to be detected according to a preset vertex detection model to obtain a plurality of vertexes of the target object; the vertex is used for representing a local feature of the surface of the object; the second detection module is used for matching a plurality of vertexes on the target object with a standard template according to a preset image matching algorithm to obtain matched vertexes and form a first matching result; the standard template is an information set of all vertexes on a standard object corresponding to the target object; the first processing module is used for calculating a matching score of the first matching result relative to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold; the second processing module deduces the undetected vertexes on the target object in the image to be detected through the variation relation between the vertexes in the second matching result and the standard template, and forms a third matching result by using the vertexes in the second matching result and the undetected vertexes; and the third processing module is used for determining the position and the angle of the target object in the image to be detected according to the third matching result and forming the detection result of the target object.
According to a third aspect, an embodiment provides a computer-readable storage medium having a program stored thereon, the program being executable by a processor to implement the object detection method described in the first aspect above.
The beneficial effect of this application is:
according to the target detection method and the target detection device based on the graph matching, the target detection method is that local features of an image to be detected are detected according to a preset vertex detection model, and a plurality of vertexes of a target object are obtained; matching a plurality of vertexes on a target object with a standard template according to a preset image matching algorithm to obtain matched vertexes and form a first matching result; calculating a matching score and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold; deducing undetected vertexes on a target object in the image to be detected through the variation relation between each vertex in the second matching result and the standard template, and forming a third matching result by using each vertex in the second matching result and the undetected vertexes; and determining the position and the angle of the target object in the image to be detected according to the third matching result, and forming a detection result of the target object. On the first hand, as the local characteristic detection processing is firstly carried out on the image to be detected of the target object through the vertex detection model, individual vertexes with obvious local characteristics on the target object can be conveniently obtained, and technical conditions are provided for the matching of the vertexes and the standard template; in the second aspect, because the detected partial vertexes are matched with the standard template through the graph matching algorithm, not only the vertex matching problem is converted into the graph matching algorithm to be solved so as to improve the accuracy and stability of subsequent image registration, but also the proper vertexes are screened by calculating the position relation among the vertexes, so that the calculation complexity is reduced and the operation efficiency of the algorithm is improved; in the third aspect, since the first matching result is filtered by using the matching score, a second matching result with higher accuracy can be obtained; in the fourth aspect, the technical scheme infers the undetected vertexes through the change relation between the detected vertexes and the standard template, so that the rest undetected vertexes can be inferred only by a small number of detected vertexes, and the processing overhead of the previous vertex detection is greatly reduced; in the fifth aspect, since the third matching result includes information of almost all vertices on the target object, when the position and the angle of the target object in the image to be detected are determined according to the third matching result, the accuracy of detecting the target object can be improved, and thus a better detection result can be obtained.
In order to obtain a first matching result of a plurality of detected vertexes in an image to be detected, a graph matching algorithm is provided for matching the detected vertexes within the information search range of a standard template, and converting a vertex matching problem into a graph matching problem for solving and calculating, so that the corresponding relation between the plurality of vertexes and the vertexes in the standard template is realized; then, under the condition of obtaining the corresponding relation of the vertex part, the transformation matrix between the corresponding parts is convenient to calculate according to the corresponding relation, thereby deducing and identifying the rest undetected vertexes on the target object through the change relation, and finally completing the matching and detection of each vertex on the target object.
The technical scheme only needs to detect the local characteristics of the target object in the image to be detected, and the whole target object can be further detected according to the detection result of part of the vertexes. For a target object with an extreme length-width ratio, the target object can be completely detected only by selecting a local feature with a moderate length-width ratio on the target object; for the shielded target object, the inference and the detection of the whole can be finished only by identifying at least two vertexes; in addition, for objects with different scales needing to be detected at the same time, the target objects with different scales can be detected at the same time without adopting a feature pyramid by only selecting local features with the same scale size. That is to say, the target detection algorithm proposed by the technical scheme has a good detection effect for detecting objects with extreme length-width ratio and partially shielded objects and simultaneously detecting objects with multiple scales.
Drawings
FIG. 1 is a flow chart of a graph matching based target detection method of the present application;
FIG. 2 is a flow chart of obtaining a first matching result;
FIG. 3 is a flow chart of obtaining a second matching result;
fig. 4 is a schematic view of setting a vertex on a dial to be detected;
FIG. 5 is a schematic diagram of building a standard template for a watch face;
FIG. 6 is a diagram illustrating vertices and connections of a standard template;
FIG. 7 is a set of vertices X h The vertex and the connection relation of (2);
FIG. 8 is a schematic diagram of counting votes;
FIG. 9 is a flow chart for building a vertex detection model;
FIG. 10 is a schematic diagram of a vertex detection model;
FIG. 11 is a flowchart of matching an image to be detected and obtaining a detection result of a target object;
FIG. 12 is a schematic diagram of the structure of the target detection device of the present application;
FIG. 13 is a schematic diagram of a processor;
fig. 14 is a schematic structural diagram of an object detection device in another embodiment.
Detailed Description
The present application will be described in further detail below with reference to the accompanying drawings by way of specific embodiments. Wherein like elements in different embodiments are numbered with like associated elements. In the following description, numerous details are set forth in order to provide a better understanding of the present application. However, those skilled in the art will readily recognize that some of the features may be omitted or replaced with other elements, materials, methods in different instances. In some instances, certain operations related to the present application have not been shown or described in detail in order to avoid obscuring the core of the present application from excessive description, and it is not necessary for those skilled in the art to describe these operations in detail, so that they may be fully understood from the description in the specification and the general knowledge in the art.
Furthermore, the features, operations, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. Also, the various steps or actions in the method descriptions may be transposed or transposed in order, as will be apparent to one of ordinary skill in the art. Thus, the various sequences in the specification and drawings are for the purpose of describing certain embodiments only and are not intended to imply a required sequence unless otherwise indicated where such sequence must be followed.
The numbering of the components as such, e.g., "first", "second", etc., is used herein only to distinguish the objects as described, and does not have any sequential or technical meaning. The term "connected" and "coupled" when used in this application, unless otherwise indicated, includes both direct and indirect connections (couplings).
In many cases of industrial production, the shape of the target object to be detected is relatively fixed, and these shape changes can be regarded as affine transformation approximately at this time, which often occurs in the detection of standardized products such as metal parts and plastic housings. A complete target object may be divided into a plurality of feature vertices, hereinafter referred to as vertices (these vertices need to be set as local regions that better reflect the features of the object to be detected). A set formed by a plurality of vertexes is called as a template, and the overall detection of the template can be completed by detecting each vertex; that is, the detection of the whole target object can be completed by detecting the local target object.
The technical solution of the present application will be specifically described with reference to the following examples.
The first embodiment,
Referring to fig. 1, the present application discloses a target detection method based on graph matching, which includes steps 110 and 160, which are described below.
Step 110, acquiring an image to be detected of the target object. The target object can be a product on an industrial production line, a mechanical part in an article box or a tool on an operation table, and the like, and the target object may have partial occlusion, surface damage and abnormal shape in some scenes, which brings some difficulties to the detection of the surface characteristics of the target object.
In some practical applications, a camera or a video camera may be used to capture images of a target object in some scenes, so as to obtain an image to be detected of the target object, and a processor acquires the image to be detected from the camera or the video camera to perform further image processing.
And 120, detecting local characteristics of the image to be detected according to a preset vertex detection model to obtain a plurality of vertexes of the target object. The vertex here is used to characterize a local feature of the surface of the object.
It should be noted that the preset vertex detection model may be obtained by training a convolutional neural network based on deep learning, and the training process includes: (1) obtaining at least one sample image of a target object, wherein each sample image comprises position information, angle information and category information of a plurality of vertexes on the target object; (2) and inputting the sample image into a convolutional neural network, iteratively updating network parameters until a loss function corresponding to the convolutional neural network is converged, and taking the trained convolutional neural network as a vertex detection model. The construction process of the vertex detection model will be described in detail below.
And step 130, matching a plurality of vertexes on the target object with a standard template according to a preset graph matching algorithm to obtain matched vertexes and form a first matching result. The standard template is a set of information of all vertices on the standard object corresponding to the target object.
It should be noted that the standard object corresponding to the target object is a target object with no occlusion, no surface defect, no deformation, and normal posture, and the standard object has complete object surface features, and if these object surface features are represented by feature vertices, the standard template of the standard object is the set of these vertex information.
And 140, calculating a matching score of the first matching result relative to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold value.
Here, the first matching result is filtered by using the matching score, and when a condition of a preset score threshold is satisfied, a second matching result is formed by using each vertex in the first matching result, so that the second matching result with higher accuracy can be obtained.
And 150, deducing undetected vertexes on the target object in the image to be detected through the variation relation between each vertex in the second matching result and the standard template, and forming a third matching result by using each vertex in the second matching result and the undetected vertexes.
It should be noted that, here, the undetected vertex is actually inferred by using the variation relationship between the detected vertex in the second matching result and the standard template, and then the remaining undetected vertex can be inferred by only using a small number of detected vertices, so that the processing overhead of the previous vertex detection is greatly reduced.
And 160, determining the position and the angle of the target object in the image to be detected according to the third matching result, and forming a detection result of the target object.
It should be noted that, since the third matching result includes information of almost all vertices on the target object, when the position and the angle of the target object in the image to be detected are determined according to the third matching result, the accuracy of detecting the target object can be improved, so as to obtain a better detection result.
In the present embodiment, the step 120 mentioned above relates to the vertex detection model, and if the vertex detection model is to be used to perform the detection processing of the local features on the image to be detected, the vertex detection model must be constructed in advance, and the construction process of the model will be described in detail below.
In one embodiment, referring to FIG. 9, the vertex detection model constructing step may specifically include steps 121 and 122, which are described below.
And 121, constructing a convolutional neural network based on deep learning, and training the convolutional neural network by using the sample image with each vertex labeled until a loss function corresponding to the convolutional neural network is converged.
In one embodiment, the convolutional neural network based on deep learning can adopt a network type such as YOLO, Retina-Net, SSD and the like; since these types of networks are common, it is easy to configure the corresponding loss function of the convolutional neural network. The training process of the sample image participating in the convolutional neural network is the updating process of the network weight coefficient, and the corresponding loss function gradually tends to converge along with the increase of the updating times; generally, the convolutional neural network training can be considered to be completed when the loss function converges.
And step 122, using the trained convolutional neural network as a vertex detection model, wherein the vertex detection model is used for detecting local features of the image to be detected of the target object, so as to obtain a plurality of vertices on the target object.
It should be noted that after the vertex detection model is obtained, the image to be detected of the target object may be input, and a plurality of vertices on the target object may be output by detecting the local features on the target object in the image to be detected. It will be appreciated that the information for each vertex output may include: vertex position coordinates, vertex categories, vertex external rectangles and the like; what information can be specifically output is related to the self structure of the convolutional neural network and the vertex information of the sample image involved in training.
With respect to the structure of the convolutional neural network, referring to fig. 10, a backbone network for image feature extraction, a classification detection network for classification, and a block regression network for regression are specifically configured in the network. The backbone network may be configured with operations such as convolution, activation function, pooling, and the like, and may extract image features from an input image (e.g., the image of the dial 1 in fig. 4), so as to obtain a corresponding feature map. The classification detection network can set operations such as convolution, activation function and pooling, and can focus on the processing of feature classification due to the difference of network parameters, so that the obtained feature map is further classified, the feature map of the vertex on the target object about the classification information is obtained, and the classification information of the vertex can be obtained. The square regression network can also be provided with operations such as convolution, activation function and pooling, and the obtained feature graph can be subjected to further regression processing according to the difference of network parameters, so that the feature graph of the vertex on the target object regressing relative to the square frame is obtained, and the position of the vertex and the circumscribed rectangle/outsourcing rectangle can be obtained.
In this embodiment, referring to fig. 2, the step 130 mainly relates to a process of forming the first matching result by the matching process, and specifically may include steps 131 and 135, which are respectively described as follows.
Step 131, a standard template of a standard object corresponding to the target object and an information search range of the standard template are obtained.
Here, the standard template may include position information, angle information, and class information of all vertices on the standard object, where a vertex is a local feature for characterizing the surface of the object. In addition, the information search range of the standard template is used for setting the detection range of the angle, the position and the distance scaling scale. It should be noted that, for any target object or corresponding standard object, the local features of the surface thereof include, but are not limited to: the object can represent the convex-concave shape of the core characteristic of the object, the printing pattern of the core characteristic of the object, and the like; if a certain kind of local features is selected once, all the vertexes belonging to the local features need to be labeled completely. It is understood that each vertex includes at least the following 2 pieces of information: the abscissa and ordinate of the vertex (i.e., the coordinate of the vertex), the category of the characterized local feature (i.e., the category of the vertex); of course, in addition to the 2 pieces of information, other information may be optionally added to accelerate the subsequent matching speed, and generally, the optional information includes: the minimum bounding rectangle of the local feature (i.e. the vertex bounding rectangle, the information of which may include the rectangle center point, the rectangle length and width, and may refer to a standard rectangle with a rotation angle), the bounding rectangle of the local feature (i.e. the vertex bounding rectangle, the information of which may include the rectangle center point, the rectangle length and width, and the direction, and may refer to a standard rectangle without a rotation angle). It will be appreciated that since each vertex on an object is a representation of a local feature that has a well-defined class, exact location and relative angle of rotation, and size of the region in the overall object image, the vertices can be described by specific values of angle, location, class, rectangle, etc.
In one embodiment, the standard template and the information search range of the standard object may be obtained by:
(1) the method comprises the steps of obtaining a standard image of a standard object corresponding to a target object from a camera device, obtaining labeling information of all vertexes on the standard object in the standard image, and generating a standard template according to the labeling information of all vertexes on the standard object. The labeling information of all the vertexes on the standard object can be generated through a manual labeling mode, and the labeling information comprises position information, angle information and category information of each vertex on the standard object; of course, the label information may also include values such as a vertex bounding rectangle and a vertex bounding rectangle.
(2) Acquiring a reference direction and a rotation central point configured for a standard template, and maximum variation of an angle, a position and a distance scaling configured for the standard template; and setting the detection ranges of the angle, the position and the distance scaling scale according to the configured reference direction, the rotation center point and the maximum variation of the angle, the position and the distance scaling scale, thereby forming the information search range of the standard template.
For example, as shown in fig. 4 and 5, the dial to be detected is used as a standard object corresponding to the target object, some significant local features on the dial 1 are labeled, such as a vertex a1 at the center of the large dial, vertices a2 and a4 at the center of the small dial, and a vertex A3 at the edge number of the small dial, so that a simple standard template can be generated by using labeling information of these vertices, specifically see a1-a2-A3-a4 in fig. 5; in fig. 5, the reference direction of the standard template is L1, and the rotation center point is L0. Since the same vertex shows different shapes in images with different sizes and rotation directions, some adjustments are needed to match the shapes of the vertices, and the adjustment content includes not only the angular offset and the position offset of the vertex itself, but also the distance scaling amount between the vertex and another vertex. The information search range here means that a standard template transformed within the range can be detected, and vice versa.
And 132, forming a set of vertexes to be detected by using a plurality of vertexes detected in the image to be detected, calculating the position change relationship between any two vertexes in the set of vertexes to be detected and any two vertexes in the standard template, and constructing the connection relationship between the vertexes in the set of vertexes to be detected.
Since the standard template is constructed in advance, the vertex of each vertex in the standard template can be obtainedv i }(i=1,2,…,N 1) And the connection relationship between the vertexes, then a great distance is formed between the vertex sets to be detectedp i }(i=1,2,…,N 2) In the case of (2), the problem to be solved is the correspondence between the two. In order to quickly solve the correspondence between the two, the matching problem of the vertex is converted into the matching problem of the graph to be solved and calculated.
In one embodiment, step 132 specifically includes the following processes:
(1) for detection to be detectedVertex setP={p i Any two vertices ink、lAnd a standard templateV={v i Any two vertices ini、jComputing verticeskAnd vertexlIs expressed asβ kl Computing verticesiAnd vertexjIs expressed asβ ij 。
(2) Obtaining an angleαCoordinates of the samexCoordinates of the sameyAnd a detection range (which can be set by a user) from the scale, thereby setting an information search range of the standard template, which can be expressed as Ω.
(3) Judging the type of the vertex, if the vertex is judgediAnd vertexkAre the same, and the vertexjAnd vertexlIf the category information of the two is the same, the position relation is calculatedβ ij Relative to the position relationβ kl Is expressed asδ ij-kl (ii) a Relationship of variation hereδ ij-kl Angular offset and distance scaling, i.e. positional relationship, for characterizing relative transformationβ ij Change to a positional relationshipβ kl Desired angular offset, and vertexv i —v j Distance between to vertexv k —v l The amount of distance scaling required for the distance between.
(4) Judging the relationship of changesδ ij-kl Whether the information is in the information searching range omega of the standard template or not is judged, if so, the information is shownδ ij-kl ϵ omega, constructing the vertex in the vertex set to be detectedkAnd vertexlIn connection withγ kl And (4) showing. It can be understood that the angle is set in the information search range ΩαCoordinates of the samexCoordinates of the sameyAnd the detection range of the distance scaling scale, so that the relationship is transformedδ ij-kl When all the information in the vertex set meets the corresponding detection range, the vertex in the vertex set to be detected is constructedkAnd vertexlThe connection relationship of (1); this time indicating the vertexkAnd vertexlMay be a vertexiAnd vertexjTransformed by a certain transformation method, in this case called vertexkAnd vertexlOne edge of the graph can be constructed and there is a connection relationship.
Step 133, comparing the connection relationship between each vertex and other vertices in the vertex set to be detected, and adding the vertex and other vertices whose connection relationship meets the preset screening condition into the vertex set X h And determining a change relationship
Figure 459938DEST_PATH_IMAGE001
。
In a specific embodiment, step 133 specifically includes the following processes:
(1) judging that the vertex set to be detected is constructed with the connection relation between the vertexes, and storing all the vertexes in the vertex set to be detected into a stackS t That is, all the vertices to be detected are stacked, and thus can be recorded asS t 。
(2) From the stackS t Sequentially popping each vertex and recording as the vertexhSequentially establishing a vertex set X h And make the vertexhAdding vertex set X h (ii) a Because a vertex set X is newly built h Then the vertex can be matchedhAdding vertex set X h ；
In the vertex set X, the vertex set X is defined as a vertex h In (2), set the vertex to X h The connection between internal vertices is called an internal connection relationshipl in Set the vertices to X h The connection line between the internal vertex and any vertex outside the set is marked as the external connection relationl out (ii) a Of course, if there are multiple interior points connected to the same point outside the vertex set, the exterior connections are counted only once.
(3) Traversing the vertices in the vertex set to be detectedhOther vertexes having connection relation therebetween, voting determinationOne of the connection relations is obtained and the corresponding change relation is obtained
Figure DEST_PATH_IMAGE006
And using the sum vertexhOther vertexes having a connection relation therebetween form a candidate point setPI.e. all and the vertexhThe vertices where the connecting line exists are added to the candidate point set.
It is understood that due to the combination ofhEach vertex with a connection relation is stored with a change relation (including an angle offset and a distance scaling amount), and then the mode of the transformation relation is calculated, so that voting can be carried out to determine the change relation
Figure 126850DEST_PATH_IMAGE002
The process of (1). Furthermore, for the initial set of vertices X h Where there is only one vertex, the inner edge is now empty.
(4) For candidate point setPEach vertex in (1)oObtaining verticesoThe connection relation and the corresponding change relation with each vertex in the vertex set to be detected, if the vertex is judgedoAnd vertex set X h The variation relation corresponding to the connection relation of any one vertex in the set is equal to the variation relation
Figure 395020DEST_PATH_IMAGE002
Then the vertex is putoTemporarily joining vertex set X h In (1). Next, vertex set X is obtained h Newly added internal connection relation among internal vertexesl in And internal vertices and vertex set X h Newly added external connection relation between external vertexesl out Then, the newly added internal connection relationship can be judgedl in Whether the number of the external connection relation is less than the newly added external connection relationl out If the number of vertices is greater than the number of verticesoMoving out vertex set X h If not, the new external connection relation is addedl out Upper and top pointoThe associated vertices are also added to the candidate point setP。
It can be understood that the vertex pointsoAdding vertex set X h Later, the new internal connection relation can be checkedl in And newly added external connection relationl out To make a number judgment if a new internal connection relationship is addedl in External connection relation more than newly addedl out Then candidate vertexoAdding the newly added vertex into the vertex set X according with the requirement h (ii) a If the new internal connection relation is addedl in Less than the newly added external connection relationl out If the candidate vertex is not qualified, the candidate vertex should be selectedoFrom vertex set X h And deleted.
(5) Traversing a set of candidate pointsPAll vertices in and update vertex set X h That is, repeating step (4) until the candidate point setPIf the vertex set becomes empty, the vertex set X finally formed is output h And the determined change relation
Figure 324930DEST_PATH_IMAGE002
。
It will be appreciated that the above is from the stackS t Pop one vertexhAnd form a vertex set X h And the determined change relation
Figure 285933DEST_PATH_IMAGE001
Then the slave stack is needed nextS t Pop the next vertex and then perform a similar calculation until the stackS t Becomes empty.
Step 134, set the vertices X h The change relation corresponding to each vertex and the determined change relation
Figure 183351DEST_PATH_IMAGE001
Comparing, and determining vertex set X by voting h A plurality of vertexes matched with the standard template;
in one embodiment, step 134 specifically includes the following processes:
(1) for one vertexhObtaining vertex set X h Middle vertexhAnd the connection relation and the corresponding change relation between the other vertexes.
(2) If the vertex ishThe variation relation corresponding to the connection relation between any other vertex is equal to the variation relation
Figure 661736DEST_PATH_IMAGE001
The connection is marked as a first value, such as set to 1.
(3) And counting and voting the vertexes marked with the first values in the connection relation to obtain a voting result of each vertex in the vertex set, and determining a plurality of vertexes matched with the standard template according to the voting result.
Referring to fig. 6 and 7, the standard template is provided with vertexes U-V-W, and connecting lines among the three vertexes form a connection relation; set of vertices X h The four vertexes are connected with each other through connecting lines. Then, the vertex set X in Table 1 below can be obtained h And the corresponding relation and the transformation relation of the standard template.
TABLE 1 set of vertices X h Corresponding relation and change relation with standard template
Figure 836366DEST_PATH_IMAGE007
For the variation relationships in Table 1, if the variation relationships are equal to
Figure DEST_PATH_IMAGE008
Set to 1, unsatisfied set to 0, there are the flagged results in table 2.
TABLE 2 numerical labeling of transform relationships
Figure 358483DEST_PATH_IMAGE009
Then, a count vote may be performed for each vertex according to table 2, with the votes for the different classes of vertices set to-1, and then the count vote results of table 3.
TABLE 3 tally voting results
Figure 493929DEST_PATH_IMAGE010
It is more intuitive to show the tally voting results of table 3 with fig. 8, with the number of votes between vertex E and vertex U being 1, the number of votes between vertex F and vertex V also being 1, and the number of votes between vertex H and vertex W being 2. Then, vertex E may be matched to vertex U, vertex F may be matched to vertex V, and vertex H may be matched to vertex W.
And 135, obtaining a first matching result by using the matched vertexes and the corresponding relation between the vertexes and the corresponding vertexes in the standard template.
It will be appreciated that it can be certain that vertex set X is h There is some correspondence with the standard template, but the vertex set X is not clear h The corresponding relation exists between the vertex in the template and the vertex in the standard template; the solution is implemented by using a counting and voting algorithm, so as to obtain a vertex set X h The corresponding relation between the vertex of (1) and the vertex of the standard template, called the first matching result, can be expressed asF 1。
It can be understood that, since some detected vertexes in the image to be detected are matched in the information search range of the standard template through the graph matching algorithm, not only the vertex matching problem is converted into the graph matching algorithm to be solved to improve the accuracy and stability of subsequent image registration, but also the appropriate vertexes are screened through calculating the position relationship among the vertexes, so that the calculation complexity is reduced and the operation efficiency of the algorithm is improved.
In order to obtain a first matching result of a plurality of detected vertexes in an image to be detected, the technical scheme provides a graph matching algorithm for matching the detected vertexes in an information searching range of a standard template, and converts a vertex matching problem into a graph matching problem for solving and calculating, so that the corresponding relation between the plurality of vertexes and the vertexes in the standard template is realized; then, under the condition of obtaining the corresponding relation of the vertex part, the transformation matrix between the corresponding parts is convenient to calculate according to the corresponding relation, thereby deducing and identifying the rest undetected vertexes on the target object through the change relation, and finally completing the matching and detection of each vertex on the target object.
In this embodiment, referring to fig. 3, the step 140 mainly involves the processes of calculating the matching score and forming the second matching result, and specifically may include steps 141 and 144, which are respectively described as follows.
Step 141, an affine transformation matrix of the standard template transformed to the first matching result is calculated, and for convenience of the following calculation description, the affine transformation matrix can be expressed asG。
Step 142, transforming the standard template to the coordinate system same as the first matching result, and calculating the back projection error, wherein the back projection error is expressed as formula
Figure DEST_PATH_IMAGE011
；
Wherein the content of the first and second substances,p c represents any vertex in the standard template, satisfiesp c ϵ M 1And isM 1Representing a standard template;q c represents the AND in the first matching resultp c Corresponding vertex, satisfyq c ϵ F 1And isF 1Indicating a first match result.
Step 143, calculating a matching score of the first matching result with respect to the standard template according to the back projection error, and then the matching score is expressed as
Figure 494115DEST_PATH_IMAGE012
；
Wherein the content of the first and second substances,edenotes the natural logarithmF 1L is the firstA number of vertices in the matching result, the luminanceM 1And | is the number of vertices in the standard template.
Step 144, obtaining the matching scoreμIn the case of (3), the preset score threshold value may be set toλ(the magnitude of the value can be customized by the user), then the match score can be determinedμAnd a preset score thresholdλThe magnitude relationship of (1). When judgingμ>λThen, the first matching result is used to form the second matching result, and the second matching result can be recorded asF 2(ii) a When judgingμ<=λAnd if so, discarding the current first matching result.
It should be noted that, for the affine transformation matrixGCan be generally expressed as
Figure 523251DEST_PATH_IMAGE013
。
If the rotation angle is set asθThe translation amount of the abscissa is
Figure DEST_PATH_IMAGE014
The translation amount of the ordinate is
Figure 156882DEST_PATH_IMAGE015
The abscissa scaling factor is
Figure DEST_PATH_IMAGE016
The ordinate scaling factor is
Figure 779625DEST_PATH_IMAGE017
Then affine transformation matrixGCan be expressed as
Figure 193288DEST_PATH_IMAGE018
。
Can be obtained by simultaneous equations
Figure 201564DEST_PATH_IMAGE019
。
Since the rotation angle of the second matching result relative to the standard template can be calculatedθAnd combining the direction of the standard template, namely calculating the direction of the second matching result.
In this embodiment, the foregoing step 150 mainly relates to a process of inferring undetected vertices and forming a third matching result, and specifically may include the following processes:
(1) constructing the inferred undetected vertices on the target object in the image to be detected as a first setTComparing the standard template with the first setTThe corresponding vertexes are formed into a second setR。
(2) According to the first setTAnd a second setRVertex correspondences between, then first setTAnd a second setRCan satisfyt=Gr. Wherein the content of the first and second substances,rrepresenting a second setRAnd satisfy any vertex inrϵ R，tRepresenting a first setTNeutralization ofrCorresponding vertex and satisfytϵ T。
(3) If it is usedF 2Representing the second match result, the third match result can be represented as
Figure DEST_PATH_IMAGE020
。
Note that, since the vertex set X is obtained h The vertex of the standard template and the vertex of the standard template, the angular offset and the distance scaling required when the standard object in the standard template is transformed to the target object in the image to be detected can be determined, so that the transformation relation of the target object in the image to be detected relative to the standard object in the standard template is formed, and the change relation between each vertex in the second matching result and the standard template can also be formed. Since some of the detected vertices in the image to be detected can be matched with some of the vertices in the standard template, the undetected vertices in the image to be detected can also be matched with the remaining vertices in the standard template, so that it can be inferred that the target object in the image to be detected is undetectedThereby obtaining position information, angle information, and category information of the undetected vertex.
In this embodiment, the disclosed target detection method based on graph matching may also specifically refer to fig. 11. In order to construct a vertex detection model, some sample images about a target object can be acquired from some image data set, information labeling is carried out on each vertex on the target object in the sample images, and then the sample images are used for training a convolutional neural network until a loss function corresponding to the convolutional neural network converges to obtain the vertex detection model. And for the image to be detected of the target object, carrying out detection processing on local characteristics of the image to be detected by using the trained vertex detection model so as to obtain a plurality of vertexes on the target object.
In order to match a plurality of vertexes on a target object by using a graph matching algorithm, a standard template and an information search range of a standard object need to be acquired, so that a standard image of the standard object corresponding to the target object and annotation information of all vertexes on the standard object in the standard image can be acquired from an image pickup device, the standard template is generated according to the annotation information of all vertexes on the standard object, and the information search range of the standard template is automatically set or manually set.
In the execution process of the graph matching algorithm, firstly, a set of vertexes to be detected and a connection relation are formed by utilizing partial detected vertexes (namely a plurality of vertexes on a target object obtained through detection) in an image to be detected, the position change relation between any two vertexes in the set of vertexes to be detected and any two vertexes in a standard template is calculated, and the connection relation between the vertexes in the set of vertexes to be detected is constructed; then, comparing the connection relation between each vertex and other vertexes in the vertex set to be detected, and adding the vertex and other vertexes with the connection relation meeting the preset screening condition into the vertex set X h And determining a change relationship
Figure 878533DEST_PATH_IMAGE001
(ii) a Then, set the vertex X h The change relation corresponding to each vertex and the determined change relation
Figure 706681DEST_PATH_IMAGE001
Comparing, and determining vertex set X by voting h A plurality of vertexes matched with the standard template; and finally, obtaining a first matching result by utilizing the corresponding relation between the matched vertexes and the corresponding vertexes in the standard template.
After the first matching result is obtained, an affine transformation matrix for transforming the standard template to the first matching result can be calculated, the standard template is transformed to the same coordinate system as the first matching result by means of the affine transformation matrix, and a back projection error is calculated, so that a matching score of the first matching result relative to the standard template is calculated according to the back projection error. Then, the relation between the matching score and the preset score threshold can be judged, and when the matching score is judged to be larger than the preset score threshold, a second matching result is formed by using the first matching result. Next, the undetected vertex on the target object in the image to be detected is inferred from the variation relationship between each vertex in the second matching result and the standard template, and a third matching result can be formed by using each vertex in the second matching result and the undetected vertex. And finally, determining the position and the angle of the target object in the image to be detected according to the third matching result, thereby forming a detection result of the target object.
It can be understood that the technical solution related to the target detection method only needs to detect the local features of the target object in the image to be detected, and the whole target object can be further detected according to the detection result of part of the vertices. For a target object with an extreme length-width ratio, the target object can be completely detected only by selecting a local feature with a moderate length-width ratio on the target object; for the shielded target object, the inference and the detection of the whole can be finished only by identifying at least two vertexes; in addition, for objects with different scales needing to be detected at the same time, the target objects with different scales can be detected at the same time without adopting a feature pyramid by only selecting local features with the same scale size. That is to say, the target detection algorithm proposed by the technical scheme has a good detection effect for detecting objects with extreme length-width ratio and partially shielded objects and simultaneously detecting objects with multiple scales.
Example II,
On the basis of the graph matching-based target detection method disclosed in the first embodiment, the first embodiment discloses a target detection device.
Referring to fig. 12, the object detection device in the present embodiment mainly includes a camera 31, a processor 32 and a display 33, which are respectively described below.
The camera 31 is used for obtaining an image to be detected of the target object by image capture. The target object can be a product on an industrial production line, a mechanical part in an article box or a tool on an operation table, and the like, and the target object may have partial occlusion, surface damage and abnormal shape in some scenes, which brings some difficulties to the detection of the surface characteristics of the target object.
The processor 32 is connected to the camera 31, and configured to detect the image to be detected by using the target detection method disclosed in the first embodiment, so as to obtain a detection result of the target object. It is understood that the processor 32 may be a CPU, a GPU, an FPGA, a microcontroller or a digital integrated circuit with data processing function, as long as the target detection method implemented in steps 110 and 160 in the first embodiment above can be implemented according to its own logic instructions.
The display 33 is connected to the processor 32 and is used for displaying the image to be detected and/or the detection result of the target object. It is understood that the display 23 may be a screen with an image display function, and may be capable of displaying the image to be detected and the detection result separately or together, and the specific screen type and display layout are not limited.
In this embodiment, referring to fig. 13, the processor 32 includes an obtaining module 32-1, a first detecting module 32-2, a second detecting module 32-3, a first processing module 32-4, a second processing module 32-5, and a third processing module 32-6.
The acquisition module 32-1 may communicate with the camera 31 to acquire an image to be detected of the target object from the camera 31.
The first detection module 32-2 is configured to perform local feature detection processing on an image to be detected according to a preset vertex detection model to obtain a plurality of vertices on a target object; the vertex here is used to characterize a local feature of the surface of the object. For the functions of the first detection module 32-2, reference may be made to step 120 in the first embodiment, which is not described herein again.
It should be noted that the vertex detection model referred to herein is obtained by training a convolutional neural network based on deep learning, and the training process includes: obtaining at least one sample image of a target object, wherein each sample image comprises position information, angle information and category information of a plurality of vertexes on the target object; and inputting the sample image into a convolutional neural network, iteratively updating network parameters until a loss function corresponding to the convolutional neural network is converged, and taking the trained convolutional neural network as a vertex detection model.
The second detection module 32-3 is configured to match a plurality of vertices on the target object with a standard template according to a preset graph matching algorithm, obtain matched vertices, and form a first matching result; the standard template is an information set of all vertexes on the standard object corresponding to the target object. For example, a standard template of a standard object corresponding to the target object and an information search range of the standard template are obtained; forming a set of vertexes to be detected by using a plurality of vertexes detected in an image to be detected, calculating a position change relation between any two vertexes in the set of vertexes to be detected and any two vertexes in a standard template, and constructing a connection relation between the vertexes in the set of vertexes to be detected; comparing the connection relation between each vertex and other vertexes in the vertex set to be detected, and adding the vertex and other vertexes of which the connection relation accords with the preset screening condition into the vertex set X h And determining a change relationship
Figure 658457DEST_PATH_IMAGE008
(ii) a Set the vertex to X h The change relation of each vertex in the table isDetermined change relationship
Figure 334289DEST_PATH_IMAGE008
Comparing, and determining the vertex set X by voting h A plurality of vertexes matched with the standard template; and obtaining a first matching result by utilizing the corresponding relation between the matched vertexes and the corresponding vertexes in the standard template. The functions of the second detection module 32-3 can be referred to in step 131 and step 135 of the first embodiment, which are not described herein again.
The first processing module 32-4 is configured to calculate a matching score of the first matching result with respect to the standard template, and form each vertex in the first matching result as a second matching result when the matching score exceeds a preset score threshold. For example, an affine transformation matrix of the standard template transformed to the first matching result is calculated, the standard template is transformed to a coordinate system identical to the first matching result, and a back projection error is calculated; and calculating the matching score of the first matching result relative to the standard template according to the back projection error, and forming a second matching result by using the first matching result when the matching score is judged to be larger than a preset score threshold value. The functions of the first processing module 32-4 can be referred to the step 141 and the step 144 in the first embodiment, which are not described herein again.
The second processing module 32-5 infers the undetected vertex on the target object in the image to be detected through the variation relationship between each vertex in the second matching result and the standard template, and forms a third matching result by using each vertex in the second matching result and the undetected vertex. For example, the inferred undetected vertices on the target object in the image to be detected are formed as a first setTMatching the standard template with the first setTThe corresponding vertexes are formed into a second setR(ii) a First setTAnd a second setRCan satisfyt=GrWhereinrRepresenting a second setRAnd satisfy any vertex inrϵ R，tRepresenting a first setTNeutralization ofrCorresponding vertex and satisfytϵ TIf it is usedF 2The second matching result is represented, and the third matching result is represented as
Figure 103530DEST_PATH_IMAGE020
. For the functions of the second processing module 32-5, reference may be made to step 150 in the first embodiment, which is not described herein again.
The third processing module 32-6 is configured to determine a position and an angle of the target object in the image to be detected according to the third matching result, and form a detection result of the target object. For the functions of the third processing module 32-6, reference may be made to step 160 in the first embodiment, which is not described herein again.
Example III,
Referring to fig. 14, the present embodiment discloses an object detection apparatus, and the object detection apparatus 4 mainly includes a memory 41 and a processor 42.
The main components of the image processing apparatus 4 are a memory 41 and a processor 42. The memory 41 is used as a computer-readable storage medium and is mainly used for storing a program, and the program may be a program code corresponding to the object detection method in the first embodiment.
Wherein the processor 42 is connected to the memory 41 for executing the program stored in the memory 31 to implement the object detection method. The functions implemented by the processor 42 can refer to the processor 32 in the second embodiment, and will not be described in detail here.
Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments may be implemented by hardware, or may be implemented by computer programs. When all or part of the functions of the above embodiments are implemented by a computer program, the program may be stored in a computer-readable storage medium, and the storage medium may include: a read only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, etc., and the program is executed by a computer to realize the above functions. For example, the program may be stored in a memory of the device, and when the program in the memory is executed by the processor, all or part of the functions described above may be implemented. In addition, when all or part of the functions in the above embodiments are implemented by a computer program, the program may be stored in a storage medium such as a server, another computer, a magnetic disk, an optical disk, a flash disk, or a removable hard disk, and may be downloaded or copied to a memory of a local device, or may be version-updated in a system of the local device, and when the program in the memory is executed by a processor, all or part of the functions in the above embodiments may be implemented.
The present invention has been described in terms of specific examples, which are provided to aid understanding of the invention and are not intended to be limiting. For a person skilled in the art to which the invention pertains, several simple deductions, modifications or substitutions may be made according to the idea of the invention.

Claims (10)

1. A target detection method based on graph matching is characterized by comprising the following steps:
acquiring an image to be detected of a target object;
detecting local characteristics of the image to be detected according to a preset vertex detection model to obtain a plurality of vertexes of the target object; the vertex is used for representing a local feature of the surface of the object;
matching a plurality of vertexes on the target object with a standard template according to a preset image matching algorithm to obtain matched vertexes and form a first matching result; the standard template is an information set of all vertexes on a standard object corresponding to the target object;
calculating a matching score of the first matching result relative to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold;
deducing undetected vertexes on a target object in the image to be detected according to the variation relation between each vertex in the second matching result and the standard template, and forming a third matching result by using each vertex in the second matching result and the undetected vertexes;
determining the position and the angle of a target object in the image to be detected according to the third matching result, and forming a detection result of the target object;
the matching of the plurality of vertexes on the target object with a standard template according to a preset graph matching algorithm to obtain matched vertexes and form a first matching result comprises the following steps: acquiring a standard template of a standard object corresponding to the target object and an information search range of the standard template; the standard template comprises position information, angle information and category information of all vertexes on the standard object, and the information search range is used for setting detection ranges of angles and positions; forming a set of vertexes to be detected by using a plurality of vertexes detected in the image to be detected, calculating the position change relation between any two vertexes in the set of vertexes to be detected and any two vertexes in the standard template, and constructing the connection relation between the vertexes in the set of vertexes to be detected; comparing the connection relation between each vertex and other vertexes in the vertex set to be detected, and adding the vertex and other vertexes of which the connection relation accords with a preset screening condition into the vertex set X h And determining a change relationship
Figure DEST_PATH_IMAGE001
(ii) a Set the vertex X h The change relation corresponding to each vertex and the determined change relation
Figure 647793DEST_PATH_IMAGE001
Comparing, and determining the vertex set X by voting h A plurality of vertexes matched with the standard template; and obtaining the first matching result by utilizing the corresponding relation between the matched vertexes and the corresponding vertexes in the standard template.
2. The object detection method of claim 1, wherein the vertex detection model is obtained by training a deep learning-based convolutional neural network, and the training process comprises:
obtaining at least one sample image of a target object, wherein the sample image comprises position information, angle information and category information of a plurality of vertexes on the target object;
inputting the sample image into the convolutional neural network, iteratively updating network parameters until a loss function corresponding to the convolutional neural network is converged, and taking the trained convolutional neural network as the vertex detection model.
3. The target detection method of claim 1, wherein the calculating the variation relationship of the positions between any two vertexes in the set of vertexes to be detected and any two vertexes in the standard template, and the constructing the connection relationship between the vertexes in the set of vertexes to be detected comprises:
for any two vertexes in the vertex set to be detectedk、lAnd any two vertices in the standard templatei、jComputing verticeskAnd vertexlIs expressed asβ kl Computing verticesiAnd vertexjIs expressed asβ ij ；
If the vertex is judgediAnd vertexkAre the same, and the vertexjAnd vertexlIf the category information of the two is the same, the position relation is calculatedβ ij Relative to the position relationβ kl Is expressed asδ ij-kl (ii) a The variation relation is used for representing the angle offset and the distance scaling amount of the relative conversion;
judging the relationship of changesδ ij-kl Whether the vertex is in the information searching range of the standard template, if so, constructing the vertex in the set of the vertices to be detectedkAnd vertexlIn connection withγ kl Represents; angles are set in the information search rangeαCoordinates of the samexCoordinates of the sameyAnd a detection range of the distance scaling scale.
4. The method of claim 3, wherein said comparing each of said set of vertices to be detectedThe connection relation between each vertex and other vertexes, and adding the vertex and other vertexes whose connection relation accords with preset screening condition into vertex set X h And determining a change relationship
Figure 33775DEST_PATH_IMAGE001
The method comprises the following steps:
judging that the vertex set to be detected is constructed with the connection relation between the vertexes, and storing all the vertexes in the vertex set to be detected into a stackS t (ii) a From the stackS t Sequentially popping each vertex and recording as the vertexhSequentially establishing a vertex set X h And make the vertexhAdding vertex set X h ；
Traversing the vertices in the set of vertices to be detectedhOther vertexes with connection relation therebetween, voting to determine one of the connection relations and obtaining a corresponding change relation
Figure 141408DEST_PATH_IMAGE001
And using the sum vertexhOther vertexes having a connection relation therebetween form a candidate point setP；
For the candidate point setPEach vertex in (1)oObtaining verticesoThe connection relation and the corresponding change relation with each vertex in the set of vertices to be detected, if the vertex is judgedoAnd the vertex set X h The variation relation corresponding to the connection relation of any one vertex in the set is equal to the variation relation
Figure 5459DEST_PATH_IMAGE001
Then the vertex is putoTemporarily joining the vertex set X h Performing the following steps; then obtaining the vertex set X h Newly added internal connection relation among internal vertexesl in And each internal vertex and the vertex set X h Newly added external connection relation between external vertexesl out To judge newly addedInterconnection of componentsl in Whether the number of the external connection relation is less than the newly added external connection relationl out If so, the verticesoMoving out of the vertex set X h If not, the new external connection relation is addedl out Upper and top pointoThe associated vertices are also added to the set of candidate pointsP；
Traverse the set of candidate pointsPAnd updating the set of vertices X h Outputting the vertex set X finally formed h And the determined change relation
Figure 895530DEST_PATH_IMAGE001
。
5. The object detection method of claim 4, wherein said grouping said vertices into X h The change relation corresponding to each vertex and the determined change relation
Figure 311468DEST_PATH_IMAGE001
Comparing, and determining the vertex set X by voting h The plurality of vertexes matched with the standard template comprise:
obtaining the vertex set X h Middle vertexhThe connection relation and the corresponding change relation with other vertexes;
if the vertex ishThe variation relation corresponding to the connection relation between any other vertex is equal to the variation relation
Figure 781764DEST_PATH_IMAGE001
If so, marking the connection relation as a first value;
and counting and voting the vertexes marked as the first values in the connection relation to obtain voting results of all the vertexes in the vertex set, and determining a plurality of vertexes matched with the standard template according to the voting results.
6. The object detection method of claim 1, wherein the calculating a matching score of the first matching result with respect to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold comprises:
calculating an affine transformation matrix of the standard template to the first matching result and representing asG；
Transforming the standard template to a coordinate system which is the same as the first matching result, and calculating a back projection error; the back projection error is formulated as
Figure 59292DEST_PATH_IMAGE002
；
Wherein the content of the first and second substances,p c representing any vertex in the standard template, satisfyingp c ϵ M 1And isM 1Representing the standard template;q c representing the AND in the first matching resultp c Corresponding vertex, satisfyq c ϵ F 1And isF 1Representing the first matching result;
calculating a matching score of the first matching result relative to the standard template according to the back projection error, wherein the matching score is expressed as
Figure 56067DEST_PATH_IMAGE003
；
Wherein the content of the first and second substances,edenotes the natural logarithmF 1| is the number of vertices in the first matching result, countM 1I is the number of the top points in the standard template;
judgment ofμ>λIf so, forming a second matching result by using the first matching result; wherein the content of the first and second substances,λis a preset score threshold.
7. The object detection method of claim 6, wherein the deducing undetected vertexes on the object in the image to be detected through the variation relationship between the vertexes in the second matching result and the standard template, and forming a third matching result by using the vertexes in the second matching result and the undetected vertexes, comprises:
constructing the inferred undetected vertexes on the target object in the image to be detected into a first setTComparing the standard template with the first setTThe corresponding vertexes are formed into a second setR；
The first setTAnd the second setRCan satisfyt=Gr(ii) a Wherein the content of the first and second substances,rrepresenting the second setRAnd satisfy any vertex inrϵ R，tRepresenting the first setTNeutralization ofrCorresponding vertex and satisfytϵ T；
If it is usedF 2If the second matching result is represented, the third matching result is represented asF 3=T∪F 2。
8. An object detection device, comprising:
the camera is used for obtaining an image to be detected of the target object by image capture;
a processor connected with the camera and used for processing the image to be detected by the target detection method of any one of claims 1 to 7 to obtain the detection result of the target object;
and the display is connected with the processor and used for displaying the image to be detected and/or the detection result of the target object.
9. The object detection device of claim 8, wherein the processor comprises:
the acquisition module is used for acquiring an image to be detected of the target object from the camera;
the first detection module is used for detecting local characteristics of the image to be detected according to a preset vertex detection model to obtain a plurality of vertexes of the target object; the vertex is used for representing a local feature of the surface of the object;
the second detection module is used for matching a plurality of vertexes on the target object with a standard template according to a preset image matching algorithm to obtain matched vertexes and form a first matching result; the standard template is an information set of all vertexes on a standard object corresponding to the target object; the second detection module matches a plurality of vertexes on the target object with a standard template according to a preset graph matching algorithm, obtains the matched vertexes and forms a first matching result, and the second detection module comprises: acquiring a standard template of a standard object corresponding to the target object and an information search range of the standard template; the standard template comprises position information, angle information and category information of all vertexes on the standard object, and the information search range is used for setting detection ranges of angles and positions; forming a set of vertexes to be detected by using a plurality of vertexes detected in the image to be detected, calculating the position change relation between any two vertexes in the set of vertexes to be detected and any two vertexes in the standard template, and constructing the connection relation between the vertexes in the set of vertexes to be detected; comparing the connection relation between each vertex and other vertexes in the vertex set to be detected, and adding the vertex and other vertexes of which the connection relation accords with a preset screening condition into the vertex set X h And determining a change relationship
Figure 128060DEST_PATH_IMAGE001
(ii) a Set the vertex X h The change relation corresponding to each vertex and the determined change relation
Figure 351230DEST_PATH_IMAGE001
Comparing, and determining the vertex set X by voting h In match with said standard templateA plurality of vertices; obtaining the first matching result by utilizing the corresponding relation between the matched plurality of vertexes and the corresponding vertexes in the standard template;
the first processing module is used for calculating a matching score of the first matching result relative to the standard template, and forming each vertex in the first matching result into a second matching result when the matching score exceeds a preset score threshold;
the second processing module deduces the undetected vertexes on the target object in the image to be detected through the variation relation between the vertexes in the second matching result and the standard template, and forms a third matching result by using the vertexes in the second matching result and the undetected vertexes;
and the third processing module is used for determining the position and the angle of the target object in the image to be detected according to the third matching result and forming the detection result of the target object.
10. A computer-readable storage medium, characterized in that the medium has stored thereon a program executable by a processor to implement the object detection method according to any one of claims 1-7.
CN202110739846.7A 2021-07-01 2021-07-01 Target detection method and target detection device based on graph matching Active CN113255702B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110739846.7A CN113255702B (en) 2021-07-01 2021-07-01 Target detection method and target detection device based on graph matching

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110739846.7A CN113255702B (en) 2021-07-01 2021-07-01 Target detection method and target detection device based on graph matching

Publications (2)

Publication Number Publication Date
CN113255702A CN113255702A (en) 2021-08-13
CN113255702B true CN113255702B (en) 2021-11-09

Family

ID=77190519

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110739846.7A Active CN113255702B (en) 2021-07-01 2021-07-01 Target detection method and target detection device based on graph matching

Country Status (1)

Country Link
CN (1) CN113255702B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115526881B (en) * 2022-10-18 2023-07-18 深圳市安仕新能源科技有限公司 Battery cell polarity detection method and device based on image modeling

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110263795A (en) * 2019-06-04 2019-09-20 华东师范大学 One kind is based on implicit shape and schemes matched object detection method
CN111914874A (en) * 2020-06-09 2020-11-10 上海欣巴自动化科技股份有限公司 Target detection method and system
CN112232420A (en) * 2020-10-19 2021-01-15 深圳市华汉伟业科技有限公司 Image labeling method, target detection method and device and storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5375823B2 (en) * 2008-04-24 2013-12-25 日本電気株式会社 Image collation apparatus, image collation method, and program for image collation
US8204340B2 (en) * 2008-09-29 2012-06-19 Two Pic Mc Llc Methods and apparatus for dot marker matching
CN110415339B (en) * 2019-07-19 2021-07-13 清华大学 Method and device for calculating matching relation between input three-dimensional shapes

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110263795A (en) * 2019-06-04 2019-09-20 华东师范大学 One kind is based on implicit shape and schemes matched object detection method
CN111914874A (en) * 2020-06-09 2020-11-10 上海欣巴自动化科技股份有限公司 Target detection method and system
CN112232420A (en) * 2020-10-19 2021-01-15 深圳市华汉伟业科技有限公司 Image labeling method, target detection method and device and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
获取目标姿态的图像匹配方法研究;方志强 等;《制造业自动化》;20190430;第41卷(第4期);第51-55页 *
采用方向场配准和图匹配的指纹匹配算法;魏鸿磊 等;《大连理工大学学报》;20070930;第47卷(第5期);第683-688页 *

Also Published As

Publication number Publication date
CN113255702A (en) 2021-08-13

Similar Documents

Publication Publication Date Title
CN106599830B (en) Face key point positioning method and device
EP3506160B1 (en) Semantic segmentation of 2d floor plans with a pixel-wise classifier
WO2020037937A1 (en) Facial recognition method and apparatus, terminal, and computer readable storage medium
JP5677798B2 (en) 3D object recognition and position and orientation determination method in 3D scene
JP6433149B2 (en) Posture estimation apparatus, posture estimation method and program
JP4924165B2 (en) Shape comparison device and shape comparison method
CN113378976B (en) Target detection method based on characteristic vertex combination and readable storage medium
CN108550166B (en) Spatial target image matching method
CN108875903B (en) Image detection method, device, system and computer storage medium
CN113221956B (en) Target identification method and device based on improved multi-scale depth model
CN115690708A (en) Method and device for training three-dimensional target detection model based on cross-modal knowledge distillation
CN114581456B (en) Multi-image segmentation model construction method, image detection method and device
CN109409388B (en) Dual-mode deep learning descriptor construction method based on graphic primitives
CN114743259A (en) Pose estimation method, pose estimation system, terminal, storage medium and application
US20200005078A1 (en) Content aware forensic detection of image manipulations
CN113255702B (en) Target detection method and target detection device based on graph matching
Hsiao et al. Flat2layout: Flat representation for estimating layout of general room types
Seib et al. Object recognition using hough-transform clustering of surf features
CN112861785A (en) Shielded pedestrian re-identification method based on example segmentation and image restoration
JP2017151797A (en) Geometry verification device, program and method
JP6558803B2 (en) Geometric verification apparatus and program
JP2001143073A (en) Method for deciding position and attitude of object
JP2019211981A (en) Information processor, information processor controlling method and program
CN113344996B (en) Vertex registration method and device based on graph matching and storage medium
CN114820755A (en) Depth map estimation method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant