CN117576590A - Multi-machine multi-target matching method based on graph matching - Google Patents

Multi-machine multi-target matching method based on graph matching Download PDF

Info

Publication number
CN117576590A
CN117576590A CN202311551293.8A CN202311551293A CN117576590A CN 117576590 A CN117576590 A CN 117576590A CN 202311551293 A CN202311551293 A CN 202311551293A CN 117576590 A CN117576590 A CN 117576590A
Authority
CN
China
Prior art keywords
target
matching
features
feature
unmanned aerial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311551293.8A
Other languages
Chinese (zh)
Inventor
王亮
闻思凯
孙仁武
石晓然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202311551293.8A priority Critical patent/CN117576590A/en
Publication of CN117576590A publication Critical patent/CN117576590A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/17—Terrestrial scenes taken from planes or by drones
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a multi-machine multi-target matching method based on graph matching, which comprises the following steps: respectively acquiring at least one image of a detection area at the same moment by using an unmanned aerial vehicle A and an unmanned aerial vehicle B, wherein the detection area comprises a plurality of targets; inputting each image to a DLA34 network to obtain a feature map, and inputting the feature map to a feature extraction head so that the feature extraction head performs target detection based on the feature map and extracts relevant features of a target; the related features comprise target features and features of edges between targets; constructing an affinity matrix according to the target detection result and the related characteristics corresponding to the unmanned aerial vehicle A and the target detection result and the related characteristics corresponding to the unmanned aerial vehicle B; based on the affinity matrix, a matching result between the targets is calculated. The method improves the characteristic utilization capability, efficiency and robustness of the algorithm, introduces more usable characteristics particularly under the condition that the characteristics of the target are not clear, and can still obtain a better matching result.

Description

Multi-machine multi-target matching method based on graph matching
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a multi-machine multi-target matching method based on graph matching.
Background
Object Re-Identification (Re-ID) is an important research task in the field of computer vision, aimed at accurately identifying and associating the same Object or person with images/videos in different cameras or time periods. Unlike object detection, object re-recognition is more focused on the ability to track the same object across cameras, multiple angles, and for long periods of time. The target re-identification technology is widely applied to the fields of video monitoring, pedestrian tracking, intelligent traffic and the like, the appearance characteristics of targets are described by extracting the characteristic representation of the targets, then the characteristics of target objects are compared and matched, finally the similarity between different targets is measured through distance calculation, and whether the targets are the same target is judged based on the similarity.
The development of the target re-identification technology brings great convenience and benefit to the fields of real-time monitoring, crowd management and security, and the problem of multi-machine cognition uniformity refers to the problem that uniform cognition on the same target is difficult in the process of cooperatively carrying out tasks by multiple unmanned planes, and cognition uniformity is the basis of all clustered intelligent tasks. However, with the progressive complexity of unmanned aerial vehicle cluster system architecture and application environment, the effect of completing the problem of multi-machine cognition uniformity by using a pure target re-recognition method is worse. If the aim of improving the cognition unifying effect under the multi-computer cluster system is to increase the acquired image size, the method uses equipment with higher calculation power and a model to extract and compare the characteristics, but the method has higher requirements on the load and is difficult to improve the effect essentially.
The problem is solved by a multi-machine cognition unification method based on graph matching. At present, solutions to the problem of multimachine cognitive uniformity can be divided into three main categories. The first type of method is an image matching method based on the traditional image technology, the characteristic extraction of the identity is carried out on the target through a SIFT operator or a corner point, then the global search and the matching are carried out on the previously extracted identity characteristic in another photo, the matching success rate of the method is low, and the algorithm time is long. The second method is a method based on target re-identification, which comprises the steps of firstly acquiring all targets and positions thereof in a visual angle range by using a target detection algorithm, then extracting features of each detected target by using a target re-identification algorithm, and judging whether the targets are the same target or not by comparing corresponding features among image targets acquired by different unmanned aerial vehicles. The third type of method is a method based on affine transformation, which comprises the steps of firstly respectively obtaining all targets and positions thereof in a visual angle range by using a target detection algorithm on a plurality of unmanned aerial vehicles, then endowing each target with a globally uniform identity ID by using a tracking algorithm, then solving affine transformation among photos, projecting one photo onto the other photo through affine transformation, and obtaining a matching relation between the targets based on the relation between affine transformation and IOU (object oriented unit) among the targets.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a multi-machine multi-objective matching method based on graph matching. The technical problems to be solved by the invention are realized by the following technical scheme:
the invention provides a multi-machine multi-target matching method based on graph matching, which comprises the following steps:
respectively acquiring at least one image of a detection area at the same moment by using an unmanned aerial vehicle A and an unmanned aerial vehicle B, wherein the detection area comprises a plurality of targets;
inputting each image to a DLA34 network to obtain a feature map, and inputting the feature map to a feature extraction head so that the feature extraction head performs target detection based on the feature map and extracts relevant features of a target; the related features comprise target features and features of edges between targets;
constructing an affinity matrix according to the target detection result and the related characteristics corresponding to the unmanned aerial vehicle A and the target detection result and the related characteristics corresponding to the unmanned aerial vehicle B;
and calculating a matching result between targets based on the affinity matrix.
In one embodiment of the invention, the feature extraction head comprises: a target detection head, a target feature head, and an inter-target feature head, the target detection head comprising: the device comprises a center detection unit, a center correction unit and an edge frame prediction unit, wherein the center detection unit, the center correction unit, the edge frame prediction unit, a target detection head, a target feature head and an inter-target feature head all comprise a convolution layer with a convolution kernel of 3, a ReLU layer and a convolution layer with a convolution kernel of 1, which are sequentially connected.
In an embodiment of the invention, the center detection unit is configured to detect a target center point (X cen ,Y cen )，X cen 、Y cen Respectively representing coordinate values of the detected target center point at X, Y axes;
the center correction unit is used for predicting the correction amount (X reg ,Y reg )；
The frame prediction unit is used for predicting the edge frame (l, r, u, d) of the target, l, r, u, d respectively represents the corrected target center point (X cen +X reg ，Y cen +Y reg ) Distances to the left, right, above and below the target;
the target feature head is used for extracting target features;
the inter-target feature head is used for extracting inter-target edge features of edges formed by connecting any two target corrected center points.
In one embodiment of the present invention, the step of constructing the affinity matrix according to the target detection result and the relevant feature corresponding to the unmanned aerial vehicle a and the target detection result and the relevant feature corresponding to the unmanned aerial vehicle B includes:
obtaining a target detection result corresponding to the unmanned aerial vehicle AAnd the target detection result corresponding to the unmanned plane B> Respectively represent nth obtained by image detection acquired by unmanned plane A 1 ,n 2 ,...,n A Target(s)>Respectively represent nth obtained by image detection acquired by unmanned plane B 1 ,n 2 ,...,n B A target;
according to the objectIs->Related features, objects->Is->Is located at the first position of the calculationLine, th->Column elements, constructed toSize n aff *n aff Is a matrix of affinity matrices; wherein n is aff ＝n A *n B ，n 1 <n i <n A ，n 1 <n p <n A ，n 1 <n j <n B ，n 1 <n q <n B 。
In one embodiment of the invention, when n C ＝n E 、n D ＝n F When according to the targetIs->Related features, objects->Is->Is located at +.>Line, th->The elements of the columns being structured to give a size n aff *n aff Comprises the steps of:
acquisition targetTarget characteristics and target->And obtaining the +.sup.th in said affinity matrix by calculating the similarity of the two>Line, th->Column elements.
In one embodiment of the invention, when n C ≠n E 、n D ≠n F When according to the targetIs->Related features, objects->Is->Is located at +.>Line, th->The elements of the columns being structured to give a size n aff *n aff Comprises the steps of:
acquisition targetCorrected center point and target->Target inter-edge feature of edge to which the corrected center point is connected, and target +.>Is->Connected toAnd obtaining elements positioned on the diagonal of the affinity matrix by calculating the similarity of the edge characteristics between the targets of the connected edges.
In one embodiment of the present invention, the step of calculating a matching result between targets based on the affinity matrix includes:
calculating to obtain a size of the matrix by using a re-weighted random walk matching algorithm according to the affinity matrixIs a probability matrix of (2); the elements in the probability matrix represent the probability that the target corresponding to the row where the element is located matches the target corresponding to the column where the element is located;
and solving and obtaining a matching result between targets by using a Hungary matching algorithm according to the probability matrix.
In one embodiment of the present invention, after the step of obtaining the matching result between the targets by using the hungarian matching algorithm according to the probability matrix, the method further includes:
aiming at the matched target pairs in the matching result, judging whether the similarity of target features of the matched target pairs is larger than a preset threshold value; if yes, the processing is not performed; if not, deleting the target pair from the matching result to obtain a final matching result.
In one embodiment of the invention, for the target pairs in the final matching result, an edge box is used for display.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a multi-machine multi-target matching method based on graph matching, which solves the problem of poor matching effect in the prior art, improves the characteristic utilization capacity and efficiency of an algorithm, and can improve the robustness of the algorithm to achieve higher matching precision; particularly, under the condition that the characteristics of the target are not clear, more usable characteristics are introduced, and a better matching result can be obtained.
The present invention will be described in further detail with reference to the accompanying drawings and examples.
Drawings
FIG. 1 is a flowchart of a multi-machine multi-objective matching method based on graph matching according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a multi-machine multi-objective matching method based on graph matching according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a DLA34 network according to an embodiment of the present invention;
FIG. 4 is a schematic structural view of a feature extraction head according to an embodiment of the invention;
FIG. 5 is a schematic diagram of a target detection result and related features according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an affinity matrix provided by an embodiment of the present invention;
FIG. 7 is a schematic diagram of a matching result provided by an embodiment of the present invention;
FIG. 8a is a schematic diagram of a matching result using only target features provided by an embodiment of the present invention;
FIG. 8b is a schematic diagram of a matching result using only edge features between objects according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to specific examples, but embodiments of the present invention are not limited thereto.
Fig. 1 is a flowchart of a multi-machine multi-objective matching method based on graph matching according to an embodiment of the present invention, and fig. 2 is a schematic diagram of a multi-machine multi-objective matching method based on graph matching according to an embodiment of the present invention. As shown in fig. 1-2, an embodiment of the present invention provides a multi-machine multi-objective matching method based on graph matching, including:
s1, respectively acquiring at least one image of a detection area at the same moment by using an unmanned aerial vehicle A and an unmanned aerial vehicle B, wherein the detection area comprises a plurality of targets;
s2, inputting each image into a DLA34 network to obtain a feature image, and inputting the feature image into a feature extraction head to enable the feature extraction head to perform target detection based on the feature image and extract relevant features of a target; the related features comprise target features and features of edges between targets;
s3, constructing an affinity matrix according to the target detection result and the related characteristics corresponding to the unmanned aerial vehicle A and the target detection result and the related characteristics corresponding to the unmanned aerial vehicle B;
s4, calculating a matching result between targets based on the affinity matrix.
It should be noted that, the detection areas at the same moment can be shot by the cameras of the unmanned aerial vehicle a and the unmanned aerial vehicle B, and the shooting angles of the unmanned aerial vehicle a and the unmanned aerial vehicle B are different, so that one or more continuous images containing the target are respectively acquired.
In step S2, operations such as clipping, scaling, preprocessing and the like are performed on images obtained by two unmanned aerial vehicles, each image is scaled to a size of 3×768×1024, and then input to a DLA34 network, fig. 3 is a schematic structural diagram of the DLA34 network provided by the embodiment of the present invention, as shown in fig. 3, the network structure of each of the images is composed of six levels, i.e., levels 0 to 6 shown in the figure, each level is composed of a base layer or a base layer and a aggregation layer, specifically, the base layer is composed of a convolution layer with a convolution kernel size of 3, a BN layer and a RELU layer, level0 is composed of a convolution layer with a convolution kernel size of 3, a BN layer and a RELU layer, level1 is composed of a convolution layer with a convolution kernel size of 3, a stride of 2, each level is in a release net structure, a residual error structure is internally adopted, and finally, outputs from a release net and a residual error are combined by using Root. level3 is deeper and wider than level2, level4 is the same, and finally level5 is used for feature integration and output, and the size of the feature map extracted by the DLA34 network is 1×64×152×272.
In this embodiment, the output end of the DLA34 network is connected to a feature extraction head, which is used to learn multiple required targets, so as to obtain multiple target information at the same time. Fig. 4 is a schematic structural diagram of a feature extraction head according to an embodiment of the invention. Alternatively, as shown in fig. 4, the feature extraction head includes: target detection head, target feature head and target inter-feature head, the target detection head includes: the device comprises a center detection unit, a center correction unit and an edge frame prediction unit, wherein the center detection unit, the center correction unit, the edge frame prediction unit, a target detection head, a target feature head and an inter-target feature head all comprise a convolution layer with a convolution kernel of 3, a ReLU layer and a convolution layer with a convolution kernel of 1, which are sequentially connected.
Optionally, the center detecting unit is configured to detect a target center point (X cen ,Y cen )，X cen 、Y cen Respectively representing coordinate values of the detected target center point at X, Y axes;
the center correction unit predicts a correction amount (X reg ,Y reg )；
The frame prediction unit is used for predicting edge frames (l, r, u, d) of the target, and l, r, u, d respectively represents the corrected target center point (X cen +X reg ，Y cen +Y reg ) Distances to the left, right, above and below the target;
the target feature head is used for extracting target features;
the inter-target feature head is used for extracting inter-target edge features of edges formed by connecting any two target corrected center points.
Specifically, in this embodiment, by controlling the number of convolution kernels with a size of 1 in the feature extraction header, a feature representation feature map of a detection target center point feature map, a target center point offset error feature map, and a distance from a target center point to a target edge frame for a target detection task are obtained, where the sizes are 1×1×152×272, 1×2×152×272, and 1×4×152×272, respectively, and simultaneously, a target feature representation feature map, a size of 1×128×152×272, and a feature representation feature map of a target inter-edge feature are obtained for the feature extraction task, and the size of 1×64×152×272.
Fig. 5 is a schematic diagram of a target detection result and related features according to an embodiment of the present invention. Referring to fig. 5, in the present embodiment, the target center point detected based on the above-described detection target center point feature map is denoted as (X cen ,Y cen )，X cen 、Y cen Coordinate values of the target center point on the X, Y axis are respectively expressed, and a target center point correction amount predicted based on the target center point offset error is denoted as (X) reg ,Y reg )，X reg 、Y reg Respectively representing the correction amount of the target center point at X, Y axis, and marking the target edge frame obtained by detecting the characteristic representation characteristic map based on the distance from the target center point to the target edge frame as (l, r, u, d), wherein l, r, u, d respectively represent the distances between the corrected target center point and the left, right, upper and lower sides of the target, and the corrected target center point coordinate is (X cen +X reg ,Y cen +Y reg ) The whole of the object is denoted as (X min ,Y min ,X max ,Y max )，X min ＝X-l，Y min ＝Y-u，X max ＝X+r，Y max =y+d. Further, the coordinates (X) are taken on the target feature representation feature map cen +X reg ,Y cen +Y reg ) The 128-dimensional vector of the points of (a) is taken as the characteristic representation of the object, and the characteristics of the edges between the objects are represented by using the characteristic set of the points, namely, for one edge, the corresponding characteristics on the characteristic representation characteristic graph of the edge between the objects by using the two object center points of the end points and the characteristic representation of the edge between the two center points are taken as the characteristic representation of the edge.
In step S3, the step of constructing an affinity matrix according to the target detection result and the relevant feature corresponding to the unmanned aerial vehicle a and the target detection result and the relevant feature corresponding to the unmanned aerial vehicle B includes:
s301, acquiring a target detection result corresponding to the unmanned aerial vehicle AAnd the target detection result corresponding to the unmanned plane B> Respectively represent nth obtained by image detection acquired by unmanned plane A 1 ,n 2 ,...,n A Target(s)>Respectively represent nth obtained by image detection acquired by unmanned plane B 1 ,n 2 ,...,n B A target;
s302, according to the targetIs->Related features, objects->Is->Is located at +.>Line, th->The elements of the columns being structured to give a size n aff *n aff Affinity matrix of (a); wherein n is aff ＝n A *n B ，n 1 <n i <n A ，n 1 <n p <n A ，n 1 <n j <n B ，n 1 <n q <n B 。
Specifically, when n C ＝n E 、n D ＝n F When according to the targetIs->Related features, objects->With the objectIs located at +.>Line, th->The elements of the columns being structured to give a size n aff *n aff Comprises the steps of:
acquisition targetTarget characteristics and target->And obtaining the +.sup.th in the affinity matrix by calculating the similarity of the two>Line, th->Column elements.
On the other hand, when n C ≠n E 、n D ≠n F When according to the targetIs->Related features, objects->Is->Is located at +.>Line, th->The elements of the columns being structured to give a size n aff *n aff Comprises the steps of:
acquisition targetCorrected center point and target->Target inter-edge feature of edge to which the corrected center point is connected, and target +.>Is->And obtaining elements positioned on the diagonal of the affinity matrix by calculating the similarity of the edge characteristics between the targets of the connected edges.
To facilitate an understanding of the present embodiment, the affinity matrix will be described herein.
The affinity matrix is used to measure the distance or similarity between two points in a space. In computer vision tasks, the affinity matrix is typically represented as a weighted graph that treats each pixel point as a node and connects each pair of pixels by an edge.
In this embodiment, all detected targets are regarded as points in the affinity matrix, and the corrected center points of the two targets are connected to form edges, so as to construct the affinity matrix, and solve the problem of cognitive uniformity among unmanned aerial vehicles by solving the matching of the points in the affinity matrix.
Optionally, the construction of the affinity matrix is performed using the target detection result, the target feature representation, and the inter-target edge feature representation extracted by the DLA34 network and the feature extraction head. The construction of the affinity matrix firstly needs to determine a measurement mode, so that the result is closer to 1 when two variables are similar, and the measurement result is closer to 0 when the two variables are different, and the measurement mode is used for representing the correlation between two characteristics.
Aiming at images acquired by each unmanned aerial vehicle, the target detection result is expressed as a matrix N det The size n×4, n represents the number of detected targets, each of which can be described as (X min ,Y min ,X max ,Y max )，X min 、Y min 、X max 、Y max Respectively expressed as the X-axis coordinate and the Y-axis coordinate of the left top vertex and the X-axis coordinate and the Y-axis coordinate of the right bottom vertex of the target edge frame. The further extracted target features are represented as a matrix N NF Size n x 128, i.e., the representation dimension of each target feature is 128; the inter-object edge features are represented as a matrix N EF Size n 2 X 320, where n 2 For the number of edges, 320 is a representation of the characteristics of each edge. Illustratively, cosine similarity is used as a metric to construct an affinity matrix and to encode both the target features and the inter-target edge features.
FIG. 6 is a schematic diagram of an affinity matrix provided by an embodiment of the present invention. Specifically, the target detection result for the unmanned plane a is expressed asThe number of detected targets is n A The target detection result corresponding to unmanned plane B is expressed as +.>The number of detected targets is n B The constructed affinity matrix is denoted as N aff Size n aff *n aff Wherein n is aff ＝n a *n b . As shown in FIG. 6, in the construction process, N is used in turn first detA The elements in the tree are used as cores to traverse N one by one detB The elements of (2) are located in the affinity matrix at +.>Line, th->Elements of a row, representing the target->Target inter-edge feature and target +.>Similarity of edge characteristics between targets connected by the corrected center points; in particular, when n C ＝n E 、n D ＝n F At the time of->Line, th->The column element is the diagonal element of the affinity matrix, which element can be calculated as target +.>Target feature and target->Is obtained.
Obviously, in order to complete the cognitive unification between unmanned aerial vehicles, the embodiment performs mapping processing on the acquired related features, that is, regards the object itself as one vertex in the graph, uses the extracted object feature representation for the feature representation of the point, regards the relationship between the object and the object as the edge between two vertices in the graph, and uses the extracted edge feature representation between the objects for the feature representation of the edge, thereby constructing and obtaining the affinity matrix.
In step S4, a step of calculating a matching result between targets based on the affinity matrix includes:
s401, calculating to obtain a size of the matrix by using a re-weighted random walk matching algorithm according to the affinity matrixIs a probability matrix of (2); the elements in the probability matrix represent the probability that the target corresponding to the row where the element is located matches the target corresponding to the column where the element is located;
s402, solving and obtaining a matching result between targets by using a Hungary matching algorithm according to the probability matrix.
The present embodiment selectively uses the RRWM (Reweighted Random Walk Matching, re-weighted random walk matching) algorithm to solve for correspondence between targets. The RRWM performs matching by calculating similarity based on a random walk algorithm, and the basic idea is to perform random walk on a constructed graph (affinity matrix) using the random walk theory in graph theory, thereby calculating the similarity between them.
Specifically, in step S401, the affinity matrix is normalized first, so as to ensure that the sum of elements in each row is 1 and the value of each element is greater than 0; given the affinity matrix W and the re-weighting factor α, the transfer matrix P is initialized using the affinity matrix, and the probability matrix x, x is initialized with averaged values. The desired probability matrix y is calculated by performing an affinity preserving edge-random walk calculation under the constraint of bi-directional constraint re-weighting using the probability matrix x and the transition matrix P repeatedly. The probability matrix x= (1- α) y+αx is updated with the re-weighting factor α and the desired probability matrix y.
And repeating the process until the x converges, and solving the probability matrix x by using a Hungary matching algorithm to obtain a final matching result.
Of course, in some other embodiments of the present application, the matching result between the targets may be solved in step S401 by using other graph matching algorithms, which is not limited in this embodiment.
Further, after the step of obtaining the matching result between the targets by utilizing the hungarian matching algorithm according to the probability matrix, the method further comprises the following steps:
aiming at matched target pairs in the matching result, judging whether the similarity of target features of the matched target pairs is larger than a preset threshold value; if yes, the processing is not performed; if not, deleting the target pair from the matching result to obtain a final matching result.
Specifically, after solving the probability matrix P by using the hungarian algorithm, the obtained matching result is expressed as a similarity matrix, in order to avoid the situation that different targets are successfully matched, similarity calculation can be performed on target features of matched target pairs, and if the similarity of the two targets is smaller than a preset threshold value, such as 0.3, the similarity is deleted from the matching result.
Aiming at the target pair in the final matching result, the edge frame can be used for displaying, so that a user can intuitively acquire the matching result.
The multi-machine multi-objective matching method based on graph matching provided by the invention is further described below through simulation experiments.
Specifically, two images of the MDMT dataset, which are taken simultaneously but have a certain viewing angle difference, are input simultaneously, and at least one identical object exists in the two images simultaneously.
Fig. 7 is a schematic diagram of a matching result provided by the embodiment of the present invention. Taking two unmanned aerial vehicles as an example, the corresponding relation of targets appearing in two images can be calculated, as shown in fig. 7, the matching result is accurate through analysis, the information in the pictures is fully utilized, and even if most of targets are separated from the field of view, the relation between the targets can be used for connection.
FIG. 8a is a schematic diagram of a matching result using only target features provided by an embodiment of the present invention, FIG. 8b
The embodiment of the invention provides a schematic diagram of a matching result only using the edge characteristics of the target. Further, simulation conditions are the same as above, but target features are not input when an affinity matrix is constructed, the target features and the target inter-edge features are respectively and independently used for matching, and matching results are shown in fig. 8a and 8b, so that partial problems can be solved under both conditions, but the effect is good when the target features and the target inter-edge features are not used at the same time, thereby proving the superior effect of the invention, and better solving the problem of cognition uniformity.
According to the above embodiments, the beneficial effects of the invention are as follows:
the invention provides a multi-machine multi-target matching method based on graph matching, which solves the problem of poor matching effect in the prior art, improves the characteristic utilization capacity and efficiency of an algorithm, and can improve the robustness of the algorithm to achieve higher matching precision; particularly, under the condition that the characteristics of the target are not clear, more usable characteristics are introduced, and a better matching result can be obtained.
In the description of the present invention, a description of the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Further, one skilled in the art can engage and combine the different embodiments or examples described in this specification.
The foregoing is a further detailed description of the invention in connection with the preferred embodiments, and it is not intended that the invention be limited to the specific embodiments described. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the invention, and these should be considered to be within the scope of the invention.

Claims (9)

1. A multi-machine multi-target matching method based on graph matching is characterized by comprising the following steps:
respectively acquiring at least one image of a detection area at the same moment by using an unmanned aerial vehicle A and an unmanned aerial vehicle B, wherein the detection area comprises a plurality of targets;
inputting each image to a DLA34 network to obtain a feature map, and inputting the feature map to a feature extraction head so that the feature extraction head performs target detection based on the feature map and extracts relevant features of a target; the related features comprise target features and features of edges between targets;
constructing an affinity matrix according to the target detection result and the related characteristics corresponding to the unmanned aerial vehicle A and the target detection result and the related characteristics corresponding to the unmanned aerial vehicle B;
and calculating a matching result between targets based on the affinity matrix.
2. The graph-based multi-machine multi-objective matching method according to claim 1, wherein the feature extraction head comprises: a target detection head, a target feature head, and an inter-target feature head, the target detection head comprising: the device comprises a center detection unit, a center correction unit and an edge frame prediction unit, wherein the center detection unit, the center correction unit, the edge frame prediction unit, a target detection head, a target feature head and an inter-target feature head all comprise a convolution layer with a convolution kernel of 3, a ReLU layer and a convolution layer with a convolution kernel of 1, which are sequentially connected.
3. The graph-matching-based multi-machine multi-objective matching method according to claim 2, wherein,
the center detecting unit is used for detecting a target center point (X cen ,Y cen )，X cen 、Y cen Respectively representing coordinate values of the detected target center point at X, Y axes;
the center correction unit is used for predicting the correction amount (X reg ,Y reg )；
The frame prediction unit is used for predicting the edge frame (l, r, u, d) of the target, and l, r, u, d respectively represents correctionPost target center point (X cen +X reg ，Y cen +Y reg ) Distances to the left, right, above and below the target;
the target feature head is used for extracting target features;
the inter-target feature head is used for extracting inter-target edge features of edges formed by connecting any two target corrected center points.
4. The graph-matching-based multi-machine multi-objective matching method according to claim 3, wherein the step of constructing the affinity matrix according to the objective detection result and the relevant features corresponding to the unmanned aerial vehicle a and the objective detection result and the relevant features corresponding to the unmanned aerial vehicle B comprises the steps of:
obtaining a target detection result corresponding to the unmanned aerial vehicle AAnd the target detection result corresponding to the unmanned plane B> Respectively represent nth obtained by image detection acquired by unmanned plane A 1 ,n 2 ,...,n A Target(s)>Respectively represent nth obtained by image detection acquired by unmanned plane B 1 ,n 2 ,...,n B A target;
according to the objectIs->Related features, objects->Is->Is located at the first position of the calculationLine, th->The elements of the columns being structured to give a size n aff *n aff Is a matrix of affinity matrices; wherein n is aff ＝n A *n B ，n 1 <n i <n A ，n 1 <n p <n A ，n 1 <n j <n B ，n 1 <n q <n B 。
5. The graph-matching-based multi-machine multi-objective matching method of claim 4, wherein when n C ＝n E 、n D ＝n F When according to the targetIs->Related features, objects->Is->Is located at the first position of the calculationLine, th->The elements of the columns being structured to give a size n aff *n aff Comprises the steps of:
acquisition targetTarget characteristics and target->And obtaining the +.sup.th in said affinity matrix by calculating the similarity of the two>Line, th->Column elements.
6. The graph-matching-based multi-machine multi-objective matching method of claim 4, wherein when n C ≠n E 、n D ≠n F When according to the targetIs->Related features, objects->Is->Is located at the first position of the calculationLine, th->The elements of the columns being structured to give a size n aff *n aff Comprises the steps of:
acquisition targetCorrected center point and target->Target inter-edge feature of edge to which the corrected center point is connected, and target +.>Is->And obtaining elements positioned on the diagonal of the affinity matrix by calculating the similarity of the edge characteristics between the targets of the connected edges.
7. The graph-based multi-machine multi-objective matching method according to claim 1, wherein the step of calculating a matching result between objects based on the affinity matrix comprises:
calculating to obtain a size of the matrix by using a re-weighted random walk matching algorithm according to the affinity matrixIs a probability matrix of (2); the elements in the probability matrix represent the probability that the target corresponding to the row where the element is located matches the target corresponding to the column where the element is located;
and solving and obtaining a matching result between targets by using a Hungary matching algorithm according to the probability matrix.
8. The graph-matching-based multi-machine multi-objective matching method according to claim 7, further comprising, after the step of solving the matching result between the objects by using a hungarian matching algorithm according to the probability matrix:
aiming at the matched target pairs in the matching result, judging whether the similarity of target features of the matched target pairs is larger than a preset threshold value; if yes, the processing is not performed; if not, deleting the target pair from the matching result to obtain a final matching result.
9. The graph-based multi-machine multi-objective matching method of claim 8, wherein for objective pairs in the final matching result, an edge box is utilized for display.
CN202311551293.8A 2023-11-20 2023-11-20 Multi-machine multi-target matching method based on graph matching Pending CN117576590A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311551293.8A CN117576590A (en) 2023-11-20 2023-11-20 Multi-machine multi-target matching method based on graph matching

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311551293.8A CN117576590A (en) 2023-11-20 2023-11-20 Multi-machine multi-target matching method based on graph matching

Publications (1)

Publication Number Publication Date
CN117576590A true CN117576590A (en) 2024-02-20

Family

ID=89891323

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311551293.8A Pending CN117576590A (en) 2023-11-20 2023-11-20 Multi-machine multi-target matching method based on graph matching

Country Status (1)

Country Link
CN (1) CN117576590A (en)

Similar Documents

Publication Publication Date Title
CN111274916B (en) Face recognition method and face recognition device
WO2018133666A1 (en) Method and apparatus for tracking video target
CN113065558A (en) Lightweight small target detection method combined with attention mechanism
CN109711416B (en) Target identification method and device, computer equipment and storage medium
US8406535B2 (en) Invariant visual scene and object recognition
Murillo et al. From omnidirectional images to hierarchical localization
CN108596010B (en) Implementation method of pedestrian re-identification system
CN112084869A (en) Compact quadrilateral representation-based building target detection method
CN107545256B (en) Camera network pedestrian re-identification method combining space-time and network consistency
CN104850857B (en) Across the video camera pedestrian target matching process of view-based access control model spatial saliency constraint
CN111047626A (en) Target tracking method and device, electronic equipment and storage medium
Budvytis et al. Large scale joint semantic re-localisation and scene understanding via globally unique instance coordinate regression
Li et al. Image Matching Algorithm based on Feature-point and DAISY Descriptor.
CN112287802A (en) Face image detection method, system, storage medium and equipment
Zhang et al. Indoor space recognition using deep convolutional neural network: a case study at MIT campus
CN110827304A (en) Traditional Chinese medicine tongue image positioning method and system based on deep convolutional network and level set method
Shalnov et al. Convolutional neural network for camera pose estimation from object detections
CN111652314A (en) Temperature detection method and device, computer equipment and storage medium
CA2787856A1 (en) Systems and methods for estimating the geographic location at which image data was captured
CN112508863B (en) Target detection method based on RGB image and MSR image double channels
Fan et al. VLSG-SANet: A feature matching algorithm for remote sensing image registration
CN113011359A (en) Method for simultaneously detecting plane structure and generating plane description based on image and application
CN110516559B (en) Target tracking method and device suitable for accurate monitoring and computer equipment
CN117576590A (en) Multi-machine multi-target matching method based on graph matching
CN112926487B (en) Pedestrian re-identification method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination