CN110969657B

CN110969657B - Gun ball coordinate association method and device, electronic equipment and storage medium

Info

Publication number: CN110969657B
Application number: CN201811146931.7A
Authority: CN
Inventors: 许剑华
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-09-29
Filing date: 2018-09-29
Publication date: 2023-11-03
Anticipated expiration: 2038-09-29
Also published as: CN110969657A

Abstract

The embodiment of the invention provides a method and a device for associating gun ball coordinates, electronic equipment and a storage medium. The method comprises the following steps: respectively inputting a gun camera image and a dome camera image to be associated with coordinates into a pre-trained neural network to obtain a first output result and a second output result; determining a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result; any group of matching points comprise one coordinate point in a gun camera image and one coordinate point in a dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions belonging to the same target and in the image areas corresponding to the same target have correspondence; based on the determined sets of matching points, a coordinate mapping matrix is calculated for the bolt face image and the ball face image. By applying the embodiment of the invention, the aim of effectively determining the matching points and then effectively realizing the correlation of the gun ball coordinates can be fulfilled.

Description

Gun ball coordinate association method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of video monitoring, and in particular, to a method and apparatus for associating gun ball coordinates, an electronic device, and a storage medium.

Background

In the field of video monitoring, common video monitoring devices include a rifle bolt, a ball machine and the like. The gun camera is a gun type camera, the ball camera is an intelligent ball camera, and both the gun camera and the ball camera can be used for shooting images.

In some scenarios, such as tracking the same target in the camera image and the dome camera image, it is necessary to establish a mapping relationship between the coordinates of the camera image and the coordinates of the dome camera image, that is, to achieve the correlation between the coordinates of the dome camera. In the prior art, the process of the gun ball coordinate association method is as follows: extracting a plurality of feature points in a gun camera image and a dome camera image respectively by using traditional feature extraction filters such as SIFT (Scale-invariant feature transform, scale invariant feature transform algorithm), ORB (Oriented FAST and Rotated BRIEF, rapid feature point extraction and description algorithm) or SURF (Speeded-Up Robust Features, rapid robust feature algorithm) and the like; then, determining a plurality of groups of matching points in the extracted characteristic points of the gun camera and the extracted characteristic points of the ball camera, wherein one group of matching points comprises one characteristic point of a gun camera image and one characteristic point of the ball camera image, which are matched by pixel characteristics; and finally, calculating a coordinate mapping matrix by utilizing the determined multiple groups of matching points, wherein the coordinate mapping matrix is used for representing the mapping relation between the coordinates of the gun camera image and the coordinates of the dome camera image.

The feature extraction filter used in the gun ball coordinate correlation method needs to be designed manually. However, because the types of image features considered when the feature extraction filter is manually designed are limited, the problems of insufficient feature points or excessive noise points and the like may exist when the image is subjected to coordinate extraction, which definitely leads to difficulty in effectively determining the matching points, and finally, the gun-ball coordinate correlation cannot be effectively realized.

Disclosure of Invention

The embodiment of the invention aims to provide a method, a device, electronic equipment and a storage medium for associating gun ball coordinates, so as to effectively determine matching points and further effectively realize the purpose of gun ball coordinate association. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides a method for associating gun ball coordinates, where the method includes:

respectively inputting a gun camera image and a dome camera image to be associated with coordinates into a pre-trained neural network to obtain a first output result and a second output result; the neural network is used for identifying the position information of the targets in the image, the first output result comprises the position information of each target in the gun camera image, and the second output result comprises the position information of each target in the dome camera image;

Determining a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result; any group of matching points comprise one coordinate point in the gun camera image and one coordinate point in the dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions belonging to the same target and in the image areas corresponding to the same target have correspondence;

based on the determined sets of matching points, a coordinate mapping matrix is calculated for the bolt face image and the ball face image.

Optionally, the determining a plurality of sets of matching points based on the position information included in the first output result and the position information included in the second output result includes:

determining at least one identical object in the gun camera image and the dome camera image based on the position information included in the first output result and the position information included in the second output result;

and determining a plurality of groups of matching points from coordinate points corresponding to at least one identical target in the gun camera image and the dome camera image.

Optionally, the determining at least one identical object of the gun camera image and the ball camera image based on the position information included in the first output result and the position information included in the second output result includes:

Determining each first image block in the bolt face image and each second image block in the dome camera image; the first image block is an image area corresponding to the position information included in the first output result, and the second image block is an image area corresponding to the position information included in the second output result;

forming a plurality of image block groups by the determined first image block and the second image block, and calculating the similarity of the first image block and the second image block in each image block group; wherein, any one image block group consists of a first image block and a second image block;

determining at least one similarity meeting preset conditions among the similarities corresponding to the image block groups;

and determining targets corresponding to the first image block and the second image block in the image block group as the same target aiming at the image block group corresponding to each similarity in the at least one similarity.

Optionally, the first output result further includes a category of each target in the camera image, and the second output result further includes a category of each target in the camera image;

The grouping the determined first image block and second image block into a plurality of image block groups includes:

for each category, the first image block and the second image block of the corresponding target belonging to the category are combined into a plurality of image block groups.

Optionally, for each image block group, calculating the similarity between the first image block and the second image block in the image block group includes:

inputting each image block group into a pre-trained measurement network to obtain the similarity of a first image block and a second image block in the image block group;

wherein the metric network is a neural network for calculating image similarity.

Optionally, the determining, among the similarities corresponding to the plurality of image block groups, at least one similarity satisfying a preset condition includes:

determining the similarity larger than a preset similarity threshold value in the similarity corresponding to the image block groups to obtain a plurality of first similarities;

for any two first similarities in the plurality of first similarities, when judging that the image block group corresponding to the any two first similarities exists: deleting the smaller similarity of any two first similarities when the first image blocks are identical or the second image blocks are identical;

And determining the remaining first similarity as the similarity meeting the preset condition.

Optionally, the at least one identical object is one identical object;

and determining multiple groups of matching points from coordinate points corresponding to at least one identical target in the gun camera image and the dome camera image, wherein the determining comprises the following steps:

determining coordinate points at a plurality of preset positions in coordinate points corresponding to the same target in the gun camera image, and determining coordinate points at the plurality of preset positions in coordinate points corresponding to the same target in the dome camera image; wherein the number of the plurality of preset positions is not less than four;

and aiming at each preset position, taking a coordinate point at the preset position in the gun camera image and a coordinate point at the preset position in the dome camera image as a group of matching points.

Optionally, the at least one identical object is at least four identical objects;

for each same target, determining a coordinate point at a preset position in the coordinate points corresponding to the same target in the gun camera image, determining the coordinate point at the preset position in the coordinate points corresponding to the same target in the ball camera image, and taking the two determined coordinate points as a group of matching points.

In a second aspect, an embodiment of the present invention provides a device for associating gun ball coordinates, the device including:

the obtaining module is used for respectively inputting the gun camera image and the dome camera image to be associated with the coordinates into a pre-trained neural network to obtain a first output result and a second output result; the neural network is used for identifying the position information of the targets in the image, the first output result comprises the position information of each target in the gun camera image, and the second output result comprises the position information of each target in the dome camera image;

a determining module, configured to determine a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result; any group of matching points comprise one coordinate point in the gun camera image and one coordinate point in the dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions belonging to the same target and in the image areas corresponding to the same target have correspondence;

a calculation module for calculating a coordinate mapping matrix for the bolt face image and the dome camera image based on the determined sets of matching points.

Optionally, the determining module includes:

a first determining sub-module configured to determine at least one identical object of the bolt face image and the ball face image based on position information included in the first output result and position information included in the second output result;

and the second determining submodule is used for determining a plurality of groups of matching points from coordinate points corresponding to at least one identical target in the gun camera image and the ball camera image.

Optionally, the first determining sub-module includes:

a first determining unit configured to determine each first image block in the bolt face image and each second image block in the dome camera image; the first image block is an image area corresponding to the position information included in the first output result, and the second image block is an image area corresponding to the position information included in the second output result;

a calculation unit configured to group the determined first image block and second image block into a plurality of image block groups, and calculate, for each image block group, a similarity of the first image block and second image block in the image block group; wherein, any one image block group consists of a first image block and a second image block;

A second determining unit, configured to determine at least one similarity satisfying a preset condition among the similarities corresponding to the plurality of image block groups;

and a third determining unit, configured to determine, for each image block group corresponding to each of the at least one similarity, the targets corresponding to the first image block and the second image block in the image block group as the same target.

the computing unit is specifically configured to:

Optionally, the computing unit is specifically configured to:

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, where,

the memory is used for storing a computer program;

The processor is used for realizing the steps of the gun ball coordinate association method provided by the embodiment of the invention when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements the steps of the method for associating gun-ball coordinates provided by the embodiment of the present invention.

In the scheme provided by the embodiment of the invention, first, a first output result of a gun camera image and a second output result of a dome camera image are obtained by utilizing a pre-trained neural network; secondly, determining a plurality of groups of matching points in coordinate points with positions corresponding to the same targets in the gun camera image and the ball camera image based on the position information of each target included in the first output result and the position information of each target included in the second output result; and finally, calculating a coordinate mapping matrix about the gun camera image and the dome camera image by using the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the ball camera image, and as the same targets in the gun camera image and the ball camera image have the same characteristics, the number of coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the accuracy of the position information is ensured, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the effective determination of the matching points is realized, and the aim of the gun-ball coordinate association is further effectively realized.

Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow chart of a method for associating gun ball coordinates according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a detection result of a gun camera image or a dome camera image according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart of another method for associating gun ball coordinates according to an embodiment of the present invention;

FIG. 4 (a) is a schematic diagram of the detection results for an exemplary bolt face image; fig. 4 (b) is a schematic diagram of the detection result of the dome camera image for example;

fig. 5 is a schematic structural diagram of a gun ball coordinate correlation device according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In order to effectively determine matching points and then effectively achieve the aim of gun ball coordinate association, the embodiment of the invention provides a gun ball coordinate association method, a device, electronic equipment and a storage medium.

The embodiment of the invention provides a gun ball coordinate correlation device, which may be operated in an electronic apparatus. The electronic device may be a server or a terminal device, but is not limited thereto.

Next, first, a method for associating gun ball coordinates provided by the embodiment of the invention is described.

As shown in fig. 1, the method for associating gun ball coordinates provided by the embodiment of the invention may include the following steps:

S101, respectively inputting a gun camera image and a dome camera image to be associated with coordinates into a pre-trained neural network to obtain a first output result and a second output result;

the camera image and the ball camera image to be associated with the coordinates have the common feature, specifically, the camera image and the ball camera image to be associated with the coordinates can be images of the same scene, or images of the same target in different scenes, etc.

In the embodiment of the invention, firstly, a gun camera image and a ball camera image to be associated with coordinates can be acquired, and then the acquired gun camera image and ball camera image to be associated with the coordinates are respectively input into a pre-trained neural network. The mode of acquiring the camera image and the dome camera image to be associated with the coordinates can be to acquire the camera image from the camera in real time and correspondingly acquire the dome camera image from the dome camera; it is reasonable to acquire the gun camera image and the ball camera image to be associated with the coordinates from a preset storage position.

The neural network is a detection network based on deep learning and is used for identifying the position information of the target in the image. Wherein the object includes, but is not limited to, a person, a car, a building, a plant, an animal, and the like. Because the neural network is used for identifying the position information of the targets in the image, the first output result comprises the position information of each target in the gun camera image, and the second output result comprises the position information of each target in the dome camera image.

Specifically, first, the neural network may detect each target in the bolt face image and the dome camera image. As can be seen from fig. 2, the neural network can identify the detected target on the original image by using detection frames, and each detection frame covers all image areas of the corresponding target. As shown in fig. 2, there are two objects in the image, one being a person and the other being a vehicle. Then, the neural network may output an output result of the image, wherein the output result includes positional information of each target. In fig. 2, optionally, the position information of the target may be a coordinate point of a rectangular detection frame corresponding to the target, that is, a coordinate point on a border of the rectangular detection frame.

Of course, the shape of the detection frame is not limited to the rectangular shape shown in fig. 2, and may be a circular shape, a trapezoid shape, or the like.

It can be understood that the location information of the target is not limited to the coordinate point of the detection frame corresponding to the target, and any information that can be located to the target can be used as the location information of the target, for example: the location information of the target may also be: azimuth information such as the center, the upper left and the like, or contour points of the target, or coordinate points included in a detection frame corresponding to the target, and the like.

The neural network is obtained by training according to the sample camera image, the sample ball camera image, the position information of each target in the sample camera image and the position information of each target in the sample ball camera image, and the training process of the neural network is described later for the sake of layout clarity.

S102, determining a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result;

and considering that the image areas corresponding to the same target in the gun camera image and the dome camera image have the same image characteristics. Therefore, in the embodiment of the invention, the matching point can be determined in the coordinate points of the same target of the gun camera image and the ball camera image, so that the searching range of the matching point is reduced, the determining speed of the matching point is improved, and the effective matching point is obtained.

Any group of matching points comprise one coordinate point in the gun camera image and one coordinate point in the dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions belonging to the same object and in the image area corresponding to the same object have correspondence. It should be emphasized that, when the position information of the object is a coordinate point of a rectangular frame corresponding to the object, the image area corresponding to any object is an area surrounded by the rectangular frame.

For ease of understanding, a set of matching points is illustrated, for example, the two coordinate points included in the set of matching points include coordinate point a in the camera image and coordinate point B in the dome camera image, where a and B belong to the same target C, and where a is a center in the image area corresponding to the target C in the camera image and B is a center in the image area corresponding to the target C in the dome camera image.

It should be noted that, according to the requirement of the calculation method of the coordinate mapping matrix, the number of the sets of the plurality of sets of matching points is not less than four.

It should be noted that, based on the position information included in the first output result and the position information included in the second output result, specific implementation manners of determining multiple sets of matching points exist in multiple ways. For the sake of clear layout and clear solution, a specific implementation manner of determining multiple sets of matching points based on the location information included in the first output result and the location information included in the second output result is described in connection with a specific embodiment.

And S103, calculating a coordinate mapping matrix about the gun camera image and the dome camera image based on the determined multiple groups of matching points.

There may be a variety of specific implementations of calculating the coordinate mapping matrix for the bolt face image and the dome camera image based on the determined sets of matching points. Optionally, in a specific implementation, the calculation process may include the following steps:

1) And determining a coordinate matrix of the gun camera image and a coordinate matrix of the dome camera image based on the two-dimensional coordinates of each coordinate point in the N groups of matching points, wherein N is a natural number greater than or equal to 4.

The coordinate matrix of the gun camera image is as follows:

coordinate moment of ball machine imageThe array is as follows:

wherein, (x, y) is the two-dimensional coordinates of the coordinate point corresponding to the bolt face image in the matching point. And (u, v) is the two-dimensional coordinates of the coordinate point corresponding to the dome camera image in the matching point. The last of S and D acts as an array to facilitate matrix computation.

2) Let the coordinate mapping matrix be H, which is a 3x3 matrix, i.e.:

assume that:

h＝(H ₁₁ ,H ₁₂ ,H ₁₃ ,H ₂₁ ,H ₂₂ ,H ₂₃ ,H ₃₁ ,H ₃₂ ,H ₃₃ ) ^T ；

a _x,u ＝(-x,-y,-1,0,0,0,ux,uy,u) ^T ；

a _y,v ＝(0,0,0,-x,-y,-1,vx,vy,v) ^T ；

wherein h, a _x,u 、a _y,v And a are variables that are constructed for ease of computation.

Then, solving equation ah=0 can result in a solution for H, which in turn results in the coordinate mapping matrix H.

3) SVD decomposition of A can be achieved:

[U,Σ,V]＝svd(A)

the right singular vector and the left singular vector of A are obtained through SVD decomposition, the sigma and V are ordered from large to small according to the value in the sigma in a corresponding relation, and the right singular vector corresponding to the minimum value in the sigma is the approximate solution of h:

h＝V[[min(∑)],:]

where min (Σ) represents the minimum value of Σ.

Since the process of calculating the approximate solution of h is prior art, it will not be described in detail here.

After the approximate solution of H is obtained, the coordinate mapping matrix H can be determined.

After the coordinate mapping matrix is obtained, the coordinates (u, v) of the dome camera image can be solved through mapping of the coordinates (x, y) of the gun camera image, namely:

of course, the coordinates (x, y) of the gun camera image can also be solved by using the coordinate mapping matrix through the mapping of the coordinates (u, v) of the dome camera image, and the specific calculation method is not described herein.

The training process of the neural network is described in the following, and may include the following steps:

the method comprises the steps of firstly, acquiring a sample gun camera image, a sample ball camera image, and position information of each target in the sample gun camera image;

in this step, multiple sets of training sets may be acquired, where any set of training sets includes a sample camera image, a sample dome camera image, location information of each target in the sample camera image, and location information of each target in the sample dome camera image.

The position information of each target can be calibrated manually, and of course, calibration can be automatically completed by using other tools.

And secondly, training a pre-built initial neural network by using the sample camera image, the sample ball camera image, the position information of each target in the sample camera image and the position information of each target in the sample ball camera image to obtain the neural network.

Wherein, the initial neural network can be an existing neural network.

The training process of the neural network about this step may specifically be:

1) And inputting a plurality of groups of training sets into the initial neural network, and taking the position information of each target in the sample rifle bolt image and the position information of each target in the sample ball machine image in one group of training sets as true values of the initial neural network corresponding to the training sets.

2) Parameters in the initial neural network including connection weights of neurons, etc. are randomly initialized within the (0, 1) range.

3) And training each training set through the initial neural network to obtain a corresponding training result.

4) Comparing the training result with the corresponding true value to obtain an output result;

5) Calculating the value of a Loss function Loss of the initial neural network according to the output result;

6) And (3) adjusting parameters of the initial neural network according to the value of the Loss, and repeating the steps 3) -6) until the value of the Loss reaches a certain convergence condition, namely the value of the Loss reaches the minimum, at the moment, determining the parameters of the initial neural network, and completing training of the initial neural network to obtain the trained neural network.

In the embodiment of the invention, the position information of the target can be obtained quickly by utilizing the advantages of the neural network in the aspect of image recognition, the cost of data processing is saved, and the accuracy of determining the position information of the target is improved.

The following describes a method for associating gun ball coordinates provided by the embodiment of the invention with reference to specific embodiments.

As shown in fig. 3, the method for associating gun ball coordinates provided by the embodiment of the invention may include the following steps:

s301, respectively inputting a gun camera image and a dome camera image to be associated with coordinates into a pre-trained neural network to obtain a first output result and a second output result;

s301 is the same as S101, and will not be described here.

S302, determining at least one identical target in the gun camera image and the dome camera image based on the position information included in the first output result and the position information included in the second output result;

in this embodiment, S302-S303 are an alternative implementation of S102 in the above embodiment.

Optionally, in an embodiment of the present invention, step S302 may include the following steps a-d:

step a, determining each first image block in the gun camera image and each second image block in the dome camera image; the first image block is an image area corresponding to the position information included in the first output result, and the second image block is an image area corresponding to the position information included in the second output result;

Step b, the determined first image block and the determined second image block form a plurality of image block groups, and the similarity of the first image block and the second image block in each image block group is calculated; wherein, any one image block group consists of a first image block and a second image block;

step c, determining at least one similarity meeting a preset condition among the similarities corresponding to the plurality of image block groups;

and d, determining targets corresponding to the first image block and the second image block in the image block group as the same target aiming at the image block group corresponding to each similarity in the at least one similarity.

Specifically, in step a, two-dimensional coordinates of each coordinate point on a border of a detection frame including a target in the first output result may be utilized, in the gun camera image, image areas corresponding to the coordinate points are located, then an image of the image area is intercepted, and the intercepted image is taken as a first image block corresponding to the target; similarly, in the second output result, two-dimensional coordinates of each coordinate point on the border of the detection frame including the target may be utilized, in the dome camera image, image areas corresponding to the coordinate points are located, then an image of the image area is intercepted, and the intercepted image is used as the second image block corresponding to the target. Whereby a plurality of first image tiles in the bolt face image and a plurality of second image tiles in the dome camera image can be obtained.

In step b, the similarity is used to characterize the probability that the two images are similar, and the form of the similarity may be a percentage, such as 70%, or the like, and the form of the similarity may also be a value between 0 and 1, such as 0.7, or the like, and of course, the form of the similarity is not limited to the above.

Alternatively, as an implementation manner, the step b may include the following steps b1 to b2:

step b1, arranging and combining a plurality of first image blocks in a gun camera image and a plurality of second image blocks in a dome camera image to form a plurality of image block groups;

specifically, a probability method may be adopted to arrange and combine all the first image blocks in the camera image and all the second image blocks in the dome camera image to form a plurality of image block groups.

It will be appreciated that the determined plurality of image block groups encompasses all permutations and combinations of the first image block and the second image block.

And b2, calculating the similarity of the first image block and the second image block in each image block group.

The similarity can be calculated by adopting a traditional manually designed measurement algorithm.

As a preferred way of calculating the similarity, each image block group may be input into a pre-trained metric network, resulting in the similarity of the first image block and the second image block in the image block group. Wherein the metric network is a neural network for calculating image similarity. The similarity is calculated by using the measurement network, so that the advantage of the neural network can be utilized, and a calculation result of the similarity can be obtained rapidly and accurately. It should be noted that the metric network is trained according to two sample images and the similarity of the two sample images. For the training process of the metric network, reference may be made to the training process of the neural network described above, which is not described herein.

In addition, since the neural network in deep learning can be used for identifying the category of the target, in the embodiment of the invention, the neural network can detect not only the position information of the target, but also the category of the target, wherein the category includes but is not limited to people, vehicles, buildings, plants, animals and the like. Therefore, in order to improve the processing efficiency, the first output result further includes a category of each target in the camera image, and the second output result further includes a category of each target in the camera image. Accordingly, alternatively, as another implementation manner, the step b may include the following steps b3 to b4:

step b3, aiming at each category, forming a plurality of image block groups by the first image block and the second image block of which the corresponding targets belong to the category;

it can be understood that the same target belongs to the same category, and then the first image block and the second image block corresponding to the target in the same category are used as an image block group, so that the searching range of the same target can be narrowed, and the subsequent rapid and accurate determination of the same target in the first image block and the second image block is facilitated.

This step is illustrated for ease of understanding: assume that three categories of targets are in the first output result, namely, a person, a vehicle and a building; the first image block corresponding to the object of the person is D, the first image block corresponding to the object of the vehicle is E, and the first image block corresponding to the object of the building is F; two kinds of targets are respectively a person and a car in the second output result; the second image block corresponding to the object of the person in the category is G, and the second image block corresponding to the object of the vehicle in the category is H;

Then, the first image block D and the second image block G corresponding to the category person can be combined into an image block group; and forming a first image block E and a second image block H which are corresponding to the category vehicle into an image block group, and obtaining two image block groups.

It will be appreciated that the number of image block sets is reduced in this implementation compared to the previous implementation in which the determined image block set does not contain the image block set associated with the first image block F.

Step b4, for each image block group, calculating the similarity of the first image block and the second image block in the image block group.

Step b4 is the same as step b2, and will not be described here again.

It will be appreciated that in this implementation, the computation time of the similarity is reduced as the number of image block sets is reduced compared to the previous implementation. And since the first image block and the second image block for which the similarity is calculated belong to the same category, it is more effective to determine the same object with the obtained similarity.

It should be noted that, in this implementation manner, the neural network is obtained by training according to the sample camera image, the sample ball camera image, the position information of each target in the sample ball camera image, the type of each target in the sample camera image, and the type of each target in the sample ball camera image, and the training process of the neural network is not repeated.

In addition, for step c:

optionally, in an embodiment of the present invention, the step c may include the following steps c1 to c3:

step c1, determining the similarity larger than a preset similarity threshold value in the similarity corresponding to the image block groups to obtain a plurality of first similarities;

the preset similarity threshold may be set according to an empirical value, for example, may be 60%.

Step c2, for any two first similarities among the plurality of first similarities, when judging that the image block group corresponding to the any two first similarities exists: deleting the smaller similarity of any two first similarities when the first image blocks are identical or the second image blocks are identical;

this step is illustrated for ease of understanding: assuming that the two first similarities are 70% (the similarity between the first image block i and the second image block j) and 80% (the similarity between the first image block i and the second image block k), respectively, the first image block i is judged to have the same two similarities, and then, the smaller 70% of the two similarities is deleted.

As can be seen from the above-mentioned value of the first similarity, the probability that the targets corresponding to the first image block i and the second image block j are the same target is smaller than the probability that the targets corresponding to the first image block i and the second image block k are the same target. Since there is no possibility that i and j are the same target, while i and k are the same target, the similarity of the first image block i and the second image block j may be deleted to further narrow the search range of the same target.

In the implementation process of this step, the deleting operation may be performed on the one or more judged groups of similarities meeting the requirement, where the same first image block or the same second image block exists in the two image block groups corresponding to the one judged group of similarities meeting the requirement.

And traversing all the first similarities, and performing the deleting operation on all the judged groups of similarities meeting the requirements until a group of similarities meeting the requirements does not exist in the plurality of first similarities.

And c3, determining the remaining first similarity as the similarity meeting the preset condition.

It can be understood that the remaining first similarities are obtained by deleting the first similarities satisfying the requirement from the original first similarities.

In the above example, the target corresponding to the first image block i and the target corresponding to the second image block k may be determined to be the same target for the image block group (the image block group composed of the first image block i and the second image block k) corresponding to the similarity of 80%.

To facilitate understanding of step S302, one specific implementation of this step is illustrated:

Referring to fig. 4, fig. 4 (a) is a schematic diagram of the detection result of the camera image for an example, and fig. 4 (b) is a schematic diagram of the detection result of the dome camera image for an example. In fig. 4 (a), there are three targets, X1, X2 and Y1, respectively, and in fig. 4 (b), there are two targets, X3 and Y2, respectively, wherein the categories of X1, X2 and X3 are human, and the categories of Y1 and Y2 are vehicles.

A first step of determining a plurality of first image blocks and a plurality of second image blocks;

a first image block corresponding to X1, a first image block corresponding to X2 and a first image block corresponding to Y1 can be obtained; and a second image block corresponding to X3 and a second image block corresponding to Y2.

And secondly, judging that targets with the same category as X3 exist in the gun camera image aiming at X3 in the ball camera image. Then determining a target of the same category as X3 as X1 in the gun camera image, calculating the similarity of the second image block corresponding to X3 and the first image block corresponding to X1 to obtain a similarity ρ _X3,X1 =80%; judging and knowing ρ _X3,X1 If the similarity is larger than 60% of the preset similarity threshold, and if the similarity is judged to be calculated by X3 and other targets in the gun camera image, storing ρ _X3,X1 。

And for X3 in the ball machine image, continuously judging whether targets with the same category as the X3 exist in the gun machine image, and determining that another target with the same category as the X3 in the gun machine image is X2. Then, calculating the similarity between the second image block corresponding to X3 and the first image block corresponding to X2 to obtain a similarity ρ _X3,X2 =85%; judging and knowing ρ _X3,X2 Is larger than the preset similarity threshold value by 60 percent, and the similarity calculated by X1 in the images of the gun camera before X3 is judged and known, then ρ is compared _X3,X2 And ρ _X3,X1， Finding ρ _X3,X2 >ρ _X3,X1 Then delete ρ _X3,X1 Save ρ _X3,X1 。

And judging that targets with the same category as X3 no longer exist in the gun camera image, and selecting the next target in the dome camera image to repeatedly execute the step.

Namely: for Y2 in the dome camera image, determining that the target of the same category is Y1 in the gun camera image, and calculating the similarity of a second image block corresponding to Y2 and a first image block corresponding to Y1 to obtain a similarity rho _Y2,Y1 =70%; judging ρ _Y2,Y1 If the similarity is larger than 60% of the preset similarity threshold, continuing to judge that Y2 is not similar to the target in other gun camera images, and storing rho _Y2,Y1 。

Finally, according to the saved ρ _X3,X1 And ρ _Y2,Y1 And determining that the targets corresponding to X3 and X1 are the same target, and the targets corresponding to Y2 and Y1 are the same target.

If it is determined that there is no target of the same category as the target in the camera image for one target in the camera image, the next target in the camera image is replaced. And deleting the similarity if the similarity of the first image block and the second image block is not greater than a preset similarity threshold value.

Since the shooting range of the rifle bolt is larger than that of the ball bolt, the number of targets in the rifle bolt image is larger than that in the ball bolt image. Therefore, in the above implementation, in order to reduce the calculation amount, a manner of selecting one object in the camera image, determining whether the same object exists among the plurality of objects in the camera image, is not adopted, but a manner of selecting one object in the camera image, and determining whether the same object exists among the plurality of objects in the camera image is adopted.

S303, determining a plurality of groups of matching points from coordinate points corresponding to at least one identical target in the gun camera image and the dome camera image;

a specific implementation of this step is described below for different numbers of identical objects.

1) When the at least one identical object is one identical object;

the determining a plurality of groups of matching points from coordinate points corresponding to at least one identical object in the gun camera image and the ball camera image may include the following steps:

the method comprises the steps of firstly, determining coordinate points at a plurality of preset positions in coordinate points corresponding to the same target in a gun camera image, and determining coordinate points at the plurality of preset positions in the coordinate points corresponding to the same target in the dome camera image; wherein the number of the plurality of preset positions is not less than four;

This step is illustrated for ease of understanding: the coordinate points of the four vertices of the rectangular detection frame including the same target may be determined for the same target among the coordinate points corresponding to the same target in the bolt face image, and similarly, the coordinate points of the four vertices of the rectangular detection frame including the same target may be determined for the same target among the coordinate points corresponding to the same target in the bolt face image.

Of course, it is also possible to determine coordinate points of center points of four side lines of the rectangular detection frame containing the same object; alternatively, a coordinate point of the center point of the three side lines of the rectangular detection frame containing the same object, a coordinate point of the center point of the rectangular detection frame, and the like are determined. Here, coordinate points at a plurality of preset positions in the embodiment of the present invention are not limited.

And secondly, aiming at each preset position, taking a coordinate point at the preset position in the gun camera image and a coordinate point at the preset position in the dome camera image as a group of matching points.

Taking a plurality of preset positions as four vertices of a rectangular detection frame containing the same object as an example, the following description will be given: for each vertex, a coordinate point at the vertex in the bolt face image and a coordinate point at the vertex in the dome camera image are used as a set of matching points. Thus, four sets of matching points can be obtained.

2) When the at least one identical object is two identical objects or three identical objects;

and determining a plurality of groups of matching points from coordinate points corresponding to at least one same target in the gun camera image and the ball camera image, wherein the determining comprises the following steps: and determining four groups of matching points from the coordinate points corresponding to the two identical targets or the three identical targets.

Specifically, in the camera image, the coordinate points at four preset positions may be determined from the coordinate points corresponding to the two identical targets or the three identical targets, and in the ball camera image, the coordinate points at the four preset positions may be determined from the coordinate points corresponding to the two identical targets or the three identical targets; and aiming at each preset position, taking a coordinate point at the preset position in the gun camera image and a coordinate point at the preset position in the dome camera image as a group of matching points.

Taking two identical targets as an example, a process of determining coordinate points at four preset positions is described: the coordinate points at two preset positions can be determined in the coordinate points corresponding to the first same target, and the coordinate points at the other two preset positions are determined in the coordinate points corresponding to the second same target;

Or, it is reasonable to determine the coordinate point at one preset position among the coordinate points corresponding to one identical object, determine the coordinate points at three preset positions among the coordinate points corresponding to another identical object, and the like.

In the same manner, for three identical targets, the coordinate points at two preset positions may be determined among the coordinate points corresponding to one identical target, the coordinate points at one preset position may be determined among the coordinate points corresponding to the other two identical targets, and the like, and of course, the manner of determining the coordinate points at four preset positions for three identical targets is not limited thereto.

It should be noted that, the four preset positions may include: any four points on or in the border of a rectangular detection frame containing the same object, for example, the preset positions may be: vertices of a rectangular detection frame containing the same object, center points of edges, center points of rectangular detection frames, and so forth.

3) When the at least one identical object is at least four identical objects;

the determining a plurality of sets of matching points from coordinate points corresponding to at least one identical object in the gun camera image and the ball camera image may include:

It will be appreciated that at least four sets of matching points may be obtained.

Also, the preset position may be one of four vertices of a rectangular detection frame containing the same object, one of the center points of four edges, the center point of the rectangular detection frame, any point on the edge of the rectangular detection frame other than the vertex, or any point inside the rectangular detection frame other than the center point, or the like.

For the convenience of calculation, the center point of the rectangular detection frame may be selected as the preset position.

S304, calculating a coordinate mapping matrix about the gun camera image and the dome camera image based on the determined multiple groups of matching points.

Step S304 is the same as step S103, and will not be described here.

In the scheme provided by the embodiment of the invention, first, a first output result of a gun camera image and a second output result of a dome camera image are obtained by utilizing a pre-trained neural network; secondly, determining a plurality of image block groups based on the position information included in the first output result and the position information included in the second output result, and determining at least one identical object in the gun camera image and the dome camera image by calculating the similarity of a first image block and a second image block in each image block group; next, determining a plurality of groups of matching points by utilizing position correspondence from coordinate points corresponding to at least one identical target in the gun camera image and the dome camera image; and finally, calculating a coordinate mapping matrix about the gun camera image and the dome camera image by using the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the ball camera image, and as the same targets in the gun camera image and the ball camera image have the same characteristics, the number of coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the accuracy of the position information is ensured, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the effective determination of the matching points is realized, and the aim of the gun-ball coordinate association is further effectively realized.

Meanwhile, for the gun camera and the ball camera which already comprise the detection function, the neural network provided by the embodiment of the invention can multiplex the deep learning network of the original gun camera or the ball camera without adding an additional network, and the memory and the power consumption requirements of the gun camera or the ball camera are not increased; according to the embodiment of the invention, the similarity of the targets is calculated by using the measurement network, so that the accuracy can be improved; in the embodiment of the invention, the time-consuming calculation process can be carried out by a GPU or a deep learning/neural network acceleration module so as to improve the calculation speed; and the advantage of architecture can be fully utilized in the deep learning camera, so that the deep learning camera can be carried with an acceleration module of a deep learning network to quickly execute network calculation, and coordinate association can be quickly and efficiently completed.

Corresponding to the above method embodiment, the embodiment of the present invention further provides a device for associating the coordinates of the gun ball, as shown in fig. 5, where the device includes:

the obtaining module 501 is configured to input a gun camera image and a dome camera image to be associated with coordinates into a neural network trained in advance, so as to obtain a first output result and a second output result; the neural network is used for identifying the position information of the targets in the image, the first output result comprises the position information of each target in the gun camera image, and the second output result comprises the position information of each target in the dome camera image;

A determining module 502, configured to determine a plurality of groups of matching points based on the location information included in the first output result and the location information included in the second output result; any group of matching points comprise one coordinate point in the gun camera image and one coordinate point in the dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions belonging to the same target and in the image areas corresponding to the same target have correspondence;

a calculation module 503, configured to calculate a coordinate mapping matrix for the bolt face image and the ball machine image based on the determined plurality of sets of matching points.

Optionally, in an embodiment of the present invention, the determining module 502 includes:

Optionally, in an embodiment of the present invention, the first determining sub-module includes:

Optionally, in an embodiment of the present invention, the first output result further includes a category of each target in the camera image, and the second output result further includes a category of each target in the camera image;

The computing unit is specifically configured to:

Optionally, in an embodiment of the present invention, the computing unit is specifically configured to:

Optionally, in an embodiment of the present invention, the second determining unit is specifically configured to:

Optionally, in an embodiment of the present invention, the at least one identical object is one identical object;

The second determining sub-module is specifically configured to:

optionally, in an embodiment of the present invention, the at least one identical object is at least four identical objects;

the second determining sub-module is specifically configured to:

Corresponding to the above-described method embodiments, the present invention also provides an electronic device, as shown in fig. 6, which may include a processor 601 and a memory 602, where,

the memory 602 is used for storing a computer program;

the processor 601 is configured to implement the steps of the gun ball coordinate association method according to the embodiment of the present invention when executing the program stored in the memory 602.

The Memory may include RAM (Random Access Memory ) or NVM (Non-Volatile Memory), such as at least one magnetic disk Memory. Optionally, the memory may be at least one memory device located remotely from the processor.

The processor may be a general-purpose processor, including a CPU (Central Processing Unit ), NP (Network Processor, network processor), etc.; but also DSP (Digital Signal Processor ), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), FPGA (Field-Programmable Gate Array, field programmable gate array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

Through the electronic equipment, the following steps can be realized: firstly, obtaining a first output result of a gun camera image and a second output result of a dome camera image by utilizing a pre-trained neural network; secondly, determining a plurality of groups of matching points in coordinate points with positions corresponding to the same targets in the gun camera image and the ball camera image based on the position information of each target included in the first output result and the position information of each target included in the second output result; and finally, calculating a coordinate mapping matrix about the gun camera image and the dome camera image by using the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the ball camera image, and as the same targets in the gun camera image and the ball camera image have the same characteristics, the number of coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the accuracy of the position information is ensured, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the effective determination of the matching points is realized, and the aim of the gun-ball coordinate association is further effectively realized.

In addition, corresponding to the gun ball coordinate correlation method provided in the above embodiment, the embodiment of the present invention provides a computer readable storage medium, in which a computer program is stored, and the steps of the gun ball coordinate correlation method provided in the embodiment of the present invention are implemented when the computer program is executed by a processor.

The computer readable storage medium stores an application program for executing the gun ball coordinate association method provided by the embodiment of the invention when running, so that the method can be realized: firstly, obtaining a first output result of a gun camera image and a second output result of a dome camera image by utilizing a pre-trained neural network; secondly, determining a plurality of groups of matching points in coordinate points with positions corresponding to the same targets in the gun camera image and the ball camera image based on the position information of each target included in the first output result and the position information of each target included in the second output result; and finally, calculating a coordinate mapping matrix about the gun camera image and the dome camera image by using the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the ball camera image, and as the same targets in the gun camera image and the ball camera image have the same characteristics, the number of coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the accuracy of the position information is ensured, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the effective determination of the matching points is realized, and the aim of the gun-ball coordinate association is further effectively realized.

For the electronic device and the computer-readable storage medium embodiments, since the method content involved is substantially similar to the method embodiments described above, the description is relatively simple, and references to the relevant portions of the description of the method embodiments are only needed.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for associating gun ball coordinates, comprising:

Determining a plurality of groups of matching points in an image area corresponding to the same target of the gun camera image and the dome camera image based on the position information included in the first output result and the position information included in the second output result; any group of matching points comprise one coordinate point in the gun camera image and one coordinate point in the dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions in the image areas corresponding to the same targets are the same;

calculating a coordinate mapping matrix for the bolt face image and the dome camera image based on the determined plurality of sets of matching points;

wherein determining at least one identical object of the bolt face image and the ball face image based on the position information included in the first output result and the position information included in the second output result includes:

2. The method of claim 1, wherein the determining a plurality of sets of matching points in an image area corresponding to the same target of the bolt face image and the ball face image based on the position information included in the first output result and the position information included in the second output result comprises:

3. The method of claim 1, wherein the first output result further comprises a category of each object in the bolt face image, and the second output result further comprises a category of each object in the bolt face image;

4. A method according to claim 1 or 3, wherein for each group of image blocks, calculating the similarity of a first image block and a second image block in the group of image blocks comprises:

5. A method according to claim 1 or 3, wherein the determining at least one similarity satisfying a preset condition among the similarities corresponding to the plurality of image block groups includes:

6. The method of claim 2, wherein the at least one identical object is one identical object;

7. The method of claim 2, wherein the at least one identical target is at least four identical targets;

8. A gun ball coordinate correlation device, comprising:

the determining module is used for determining a plurality of groups of matching points in the image areas corresponding to the same targets of the gun camera image and the dome camera image based on the position information included in the first output result and the position information included in the second output result; any group of matching points comprise one coordinate point in the gun camera image and one coordinate point in the dome camera image, and the two coordinate points included in any group of matching points meet the following conditions: the positions in the image areas corresponding to the same targets are the same;

A calculation module for calculating a coordinate mapping matrix for the bolt face image and the dome camera image based on the determined plurality of sets of matching points;

wherein the determination module comprises a first determination sub-module comprising:

9. The apparatus of claim 8, wherein the determining module comprises:

10. The apparatus of claim 8, wherein the first output further comprises a category of each object in the bolt face image and the second output further comprises a category of each object in the bolt face image;

the computing unit is specifically configured to:

11. The apparatus according to claim 8 or 10, characterized in that the computing unit is specifically configured to:

12. An electronic device comprising a processor and a memory, wherein,

the memory is used for storing a computer program;

the processor is configured to implement the method steps of any of claims 1-7 when executing a program stored on the memory.

13. A computer-readable storage medium comprising,

the computer readable storage medium has stored therein a computer program which, when executed by a processor, carries out the method steps of any of claims 1-7.