CN110969657A

CN110969657A - Gun and ball coordinate association method and device, electronic equipment and storage medium

Info

Publication number: CN110969657A
Application number: CN201811146931.7A
Authority: CN
Inventors: 许剑华
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-09-29
Filing date: 2018-09-29
Publication date: 2020-04-07
Anticipated expiration: 2038-09-29
Also published as: CN110969657B

Abstract

The embodiment of the invention provides a gun and sphere coordinate association method and device, electronic equipment and a storage medium. The method comprises the following steps: respectively inputting a gunlock image and a dome camera image to be associated with the coordinates into a pre-trained neural network to obtain a first output result and a second output result; determining a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result; wherein, arbitrary group matching point includes a coordinate point in rifle bolt image and a coordinate point in the ball machine image, and two coordinate points that arbitrary group matching point includes satisfy: the positions which belong to the same target and are in the image area corresponding to the same target have the correspondence; based on the determined sets of matching points, a coordinate mapping matrix is calculated for the image of the gun camera and the image of the dome camera. By applying the embodiment of the invention, the matching points can be effectively determined, and then the aim of associating gun and sphere coordinates is effectively fulfilled.

Description

Gun and ball coordinate association method and device, electronic equipment and storage medium

Technical Field

The invention relates to the field of video monitoring, in particular to a gun and ball coordinate association method and device, electronic equipment and a storage medium.

Background

In the field of video monitoring, commonly used video monitoring devices include gunlocks, ball machines, and the like. The rifle bolt is a gun type camera, the ball machine is an intelligent ball camera, and the rifle bolt and the ball machine can be used for shooting images.

In some scenes, for example, tracking the same target in a gun camera image and a dome camera image, etc., a mapping relationship needs to be established between the coordinates of the gun camera image and the coordinates of the dome camera image, that is, the gun-ball coordinate association is realized. In the prior art, a gun and sphere coordinate association method comprises the following processes: extracting a plurality of feature points in a gunlock image and a dome camera image respectively by using a traditional feature extraction filter such as SIFT (Scale-invariant feature transform), ORB (ordered FAST and Robust BRIEF (FAST feature point extraction and description algorithm), SURF (Speeded-Up Robust Features algorithm) and the like; then, determining a plurality of groups of matching points in the extracted plurality of characteristic points of the rifle bolt and the plurality of characteristic points of the ball machine, wherein one group of matching points comprises one characteristic point of the image of the rifle bolt and one characteristic point of the image of the ball machine, which are matched with the pixel characteristics; and finally, calculating a coordinate mapping matrix by utilizing the determined multiple groups of matching points, wherein the coordinate mapping matrix is used for representing the mapping relation between the coordinates of the gun camera image and the coordinates of the dome camera image.

The feature extraction filter utilized by the gun-sphere coordinate correlation method described above needs to be designed manually. However, the types of image features considered when the feature extraction filter is designed manually are limited, so that problems of insufficient feature points or excessive noise points and the like may exist when the image is subjected to coordinate extraction, which undoubtedly results in difficulty in effectively determining matching points and finally fails to effectively realize gun and ball coordinate association.

Disclosure of Invention

The embodiment of the invention aims to provide a gun and sphere coordinate correlation method, a gun and sphere coordinate correlation device, electronic equipment and a storage medium, so as to effectively determine a matching point and further effectively achieve the purpose of gun and sphere coordinate correlation. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides a gun and sphere coordinate association method, where the method includes:

respectively inputting a gunlock image and a dome camera image to be associated with the coordinates into a pre-trained neural network to obtain a first output result and a second output result; the neural network is used for identifying position information of targets in an image, the first output result comprises position information of each target in the gunlock image, and the second output result comprises position information of each target in the dome camera image;

determining a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result; wherein, arbitrary group matching point includes a coordinate point in the rifle bolt image with a coordinate point in the ball machine image, and two coordinate points that arbitrary group matching point includes satisfy: the positions which belong to the same target and correspond to the same target in the image area have the correspondence;

based on the determined sets of matching points, a coordinate mapping matrix is calculated for the gun image and the dome image.

Optionally, the determining multiple sets of matching points based on the position information included in the first output result and the position information included in the second output result includes:

determining at least one identical target in the image of the gun camera and the image of the ball machine based on the position information included in the first output result and the position information included in the second output result;

and determining multiple groups of matching points from coordinate points corresponding to at least one same target in the gun camera image and the dome camera image.

Optionally, the determining at least one same target in the image of the bolt face and the image of the ball machine based on the position information included in the first output result and the position information included in the second output result includes:

determining each first image block in the gunlock image and each second image block in the dome camera image; the first image block is an image area corresponding to the position information included in the first output result, and the second image block is an image area corresponding to the position information included in the second output result;

the determined first image block and the second image block form a plurality of image block groups, and for each image block group, the similarity of the first image block and the second image block in the image block group is calculated; any image block group consists of a first image block and a second image block;

determining at least one similarity meeting a preset condition in the similarities corresponding to the image block groups;

and determining the targets corresponding to the first image block and the second image block in the image block group as the same target for the image block group corresponding to each similarity in the at least one similarity.

Optionally, the first output result further includes a category of each target in the gun image, and the second output result further includes a category of each target in the dome camera image;

the step of combining the determined first image block and the second image block into a plurality of image block groups includes:

and aiming at each category, a first image block and a second image block of which the corresponding targets belong to the category are combined into a plurality of image block groups.

Optionally, the calculating, for each image block group, a similarity between a first image block and a second image block in the image block group includes:

inputting each image block group into a pre-trained measurement network to obtain the similarity of a first image block and a second image block in the image block group;

wherein the measurement network is a neural network for calculating image similarity.

Optionally, the determining at least one similarity meeting a preset condition among the similarities corresponding to the plurality of image block groups includes:

determining the similarity greater than a preset similarity threshold in the similarities corresponding to the image block groups to obtain a plurality of first similarities;

for any two first similarities in the plurality of first similarities, when it is determined that the image block groups corresponding to the any two first similarities exist: deleting the smaller similarity of any two first similarities when the same first image block or the same second image block;

and determining the remaining first similarity as the similarity meeting the preset condition.

Optionally, the at least one same target is one same target;

determining multiple groups of matching points from coordinate points corresponding to at least one same target in the gun camera image and the dome camera image, wherein the determining comprises the following steps:

determining coordinate points at a plurality of preset positions in coordinate points corresponding to the same target in the gun camera image, and determining coordinate points at the plurality of preset positions in coordinate points corresponding to the same target in the dome camera image; wherein the number of the plurality of preset positions is not less than four;

and aiming at each preset position, taking the coordinate point at the preset position in the gun camera image and the coordinate point at the preset position in the dome camera image as a group of matching points.

Optionally, the at least one identical target is at least four identical targets;

and for each same target, determining a coordinate point at a preset position in coordinate points corresponding to the same target in the gun camera image, determining the coordinate point at the preset position in the coordinate points corresponding to the same target in the dome camera image, and taking the two determined coordinate points as a group of matching points.

In a second aspect, an embodiment of the present invention provides a gun and sphere coordinate correlation apparatus, including:

the acquisition module is used for respectively inputting the images of the gunlock and the dome camera to be associated with the coordinates into a pre-trained neural network to obtain a first output result and a second output result; the neural network is used for identifying position information of targets in an image, the first output result comprises position information of each target in the gunlock image, and the second output result comprises position information of each target in the dome camera image;

a determining module, configured to determine multiple sets of matching points based on the location information included in the first output result and the location information included in the second output result; wherein, arbitrary group matching point includes a coordinate point in the rifle bolt image with a coordinate point in the ball machine image, and two coordinate points that arbitrary group matching point includes satisfy: the positions which belong to the same target and correspond to the same target in the image area have the correspondence;

a calculation module to calculate a coordinate mapping matrix for the bolt face image and the ball machine image based on the determined sets of matching points.

Optionally, the determining module includes:

a first determining submodule configured to determine at least one same target in the image of the gun camera and the image of the dome camera based on position information included in the first output result and position information included in the second output result;

and the second determining submodule is used for determining a plurality of groups of matching points from coordinate points corresponding to at least one same target in the gun camera image and the dome camera image.

Optionally, the first determining sub-module includes:

the first determining unit is used for determining each first image block in the gun camera image and each second image block in the dome camera image; the first image block is an image area corresponding to the position information included in the first output result, and the second image block is an image area corresponding to the position information included in the second output result;

the calculating unit is used for forming the determined first image block and the second image block into a plurality of image block groups, and calculating the similarity of the first image block and the second image block in each image block group; any image block group consists of a first image block and a second image block;

a second determining unit, configured to determine at least one similarity that meets a preset condition among the similarities corresponding to the multiple image block groups;

a third determining unit, configured to determine, for each image block group corresponding to each similarity in the at least one similarity, targets corresponding to the first image block and the second image block in the image block group as the same target.

the computing unit is specifically configured to:

Optionally, the computing unit is specifically configured to:

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein,

the memory is used for storing a computer program;

the processor is used for realizing the steps of the gun and sphere coordinate association method provided by the embodiment of the invention when executing the program stored in the memory.

In a fourth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the gun and sphere coordinate association method provided by the embodiment of the present invention.

In the scheme provided by the embodiment of the invention, firstly, a pre-trained neural network is utilized to obtain a first output result of a gunlock image and a second output result of a dome camera image; secondly, determining a plurality of groups of matching points in coordinate points with corresponding positions in image areas corresponding to the same targets of the gunlock image and the dome camera image based on the position information of each target included in the first output result and the position information of each target included in the second output result; and finally, calculating to obtain a coordinate mapping matrix related to the images of the gunlock and the dome camera by utilizing the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the dome camera image, and the same targets in the gun camera image and the dome camera image have the same characteristics, so that the number of the coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the position information is ensured to be more accurate, therefore, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the matching points are effectively determined, and the purpose of the gun and sphere coordinate association is effectively realized.

Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a gun-sphere coordinate association method according to an embodiment of the present invention;

fig. 2 is a schematic diagram illustrating a detection result of a gunlock image or a dome camera image according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart illustrating a gun-sphere coordinate correlation method according to an embodiment of the present invention;

fig. 4(a) is a schematic diagram of the detection results for an exemplary bolt face image; FIG. 4(b) is a schematic diagram of the detection results for an exemplary dome camera image;

FIG. 5 is a schematic structural diagram of a gun-sphere coordinate correlation apparatus according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In order to achieve the purpose of effectively determining a matching point and then effectively achieving gun-sphere coordinate association, embodiments of the present invention provide a gun-sphere coordinate association method, an apparatus, an electronic device, and a storage medium.

It should be noted that an implementation subject of the gun-sphere coordinate association method provided by the embodiment of the present invention may be a gun-sphere coordinate association apparatus, and the apparatus may be implemented in an electronic device. The electronic device may be a server or a terminal device, but is not limited thereto.

First, a gun-sphere coordinate association method provided by an embodiment of the present invention is described below.

As shown in fig. 1, a gun-sphere coordinate association method provided by an embodiment of the present invention may include the following steps:

s101, inputting a gunlock image and a dome camera image to be associated with coordinates into a pre-trained neural network respectively to obtain a first output result and a second output result;

the image of the gun camera and the image of the ball camera to be coordinate-related have a common feature, and specifically, the image of the gun camera and the image of the ball camera to be coordinate-related can be images of the same scene, or images of the same target in different scenes.

In the embodiment of the invention, the image of the gun camera and the image of the ball camera to be associated with the coordinates can be obtained firstly, and then the obtained image of the gun camera and the image of the ball camera to be associated with the coordinates are respectively input into the pre-trained neural network. The method for acquiring the image of the gun camera and the image of the dome camera to be associated with the coordinates can be that the image of the gun camera is acquired from the gun camera in real time, and the image of the dome camera is correspondingly acquired from the dome camera; it is also reasonable to acquire the images of the gun camera, the ball camera and the like to be associated with the coordinates from a preset storage position.

The neural network is a detection network based on deep learning, and the neural network is used for identifying the position information of the target in the image. Wherein, the target includes but is not limited to human, vehicle, building, plant and animal, etc. Since the neural network is used for identifying the position information of the targets in the image, the first output result comprises the position information of each target in the image of the gunlock, and the second output result comprises the position information of each target in the image of the dome camera.

Specifically, first, the neural network may detect each target in the gun camera image and the dome camera image. Referring to the schematic diagram of the detection result of the image of the gun camera or the image of the ball camera shown in fig. 2, it can be seen from fig. 2 that the neural network can identify the detected target by using detection frames on the original image, and each detection frame covers the whole image area of the corresponding target. As shown in fig. 2, two objects exist in the image, one object being a person and the other object being a vehicle. The neural network may then output an output result of the image, wherein the output result includes location information of each target. In fig. 2, optionally, the position information of the target may be a coordinate point of a rectangular detection frame corresponding to the target, that is, a coordinate point on an edge of the rectangular detection frame.

Of course, the shape of the detection frame is not limited to the rectangular shape shown in fig. 2, and may be a circular shape, a trapezoidal shape, or the like.

It is understood that the position information of the target is not limited to the coordinate point of the detection frame corresponding to the target, and any information that can be located to the target may be used as the position information of the target, for example: the position information of the target may also be: the center, the upper left, and the like, or contour points of the target, or coordinate points included in a detection frame corresponding to the target, and the like.

The neural network is trained according to the sample gun camera image, the sample dome camera image, the position information of each target in the sample gun camera image, and the position information of each target in the sample dome camera image, and for clarity of layout, the training process of the neural network will be described later.

S102, determining a plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result;

the image areas corresponding to the same target in the gun camera image and the dome camera image have the same image characteristics. Therefore, in the embodiment of the invention, the matching point can be determined in the coordinate points of the same target of the gun camera image and the dome camera image, so that the search range of the matching point is reduced, the determination speed of the matching point is improved, and the effective matching point is obtained.

Wherein, arbitrary group matching point includes a coordinate point in the rifle bolt image with a coordinate point in the ball machine image, and two coordinate points that arbitrary group matching point includes satisfy: the positions belonging to the same object and in the image area corresponding to the same object have a correspondence. It should be emphasized that, when the position information of the target is the coordinate point of the rectangular frame corresponding to the target, the image area corresponding to any target is the area surrounded by the rectangular frame.

For ease of understanding, a set of matching points is illustrated, for example, a set of matching points includes two coordinate points including a coordinate point a in the gun image and a coordinate point B in the dome image, a and B belong to the same target C, and a is centered in an image area corresponding to the target C in the gun image and B is also centered in an image area corresponding to the target C in the dome image.

It should be noted that, according to the requirement of the calculation method of the coordinate mapping matrix, the number of the sets of the multiple sets of matching points is not less than four sets.

It should be noted that there are various specific implementation manners for determining the multiple sets of matching points based on the position information included in the first output result and the position information included in the second output result. For clarity of layout and a clear scheme, a specific implementation manner for determining the plurality of groups of matching points based on the position information included in the first output result and the position information included in the second output result is described later with reference to a specific embodiment.

S103, calculating a coordinate mapping matrix about the image of the gun camera and the image of the ball machine based on the determined multiple groups of matching points.

There may be a variety of specific implementations of computing the coordinate mapping matrix for the gun image and the dome image based on the determined sets of matching points. Optionally, in a specific implementation, the calculation process may include the following steps:

1) and determining a coordinate matrix of the gun camera image and a coordinate matrix of the dome camera image based on the two-dimensional coordinates of each coordinate point in the N groups of matching points, wherein N is a natural number greater than or equal to 4.

The coordinate matrix of the bolt face image is:

the coordinate matrix of the dome camera image is:

wherein, (x, y) is the two-dimensional coordinate of the coordinate point corresponding to the image of the gun camera in the matching point. And (u, v) are two-dimensional coordinates of coordinate points corresponding to the dome camera image in the matching points. The last row in S and D is an array added to facilitate matrix computation.

2) Let the coordinate mapping matrix be H, which is a 3x3 matrix, i.e.:

suppose that:

h＝(H₁₁,H₁₂,H₁₃,H₂₁,H₂₂,H₂₃,H₃₁,H₃₂,H₃₃)^T；

a_x,u＝(-x,-y,-1,0,0,0,ux,uy,u)^T；

a_y,v＝(0,0,0,-x,-y,-1,vx,vy,v)^T；

wherein h, a_x,u、a_y,vAnd A are variables that are constructed for ease of calculation.

Then solving the equation Ah equal to 0 can result in a solution for H, which in turn results in the coordinate mapping matrix H.

3) SVD decomposition of A can obtain:

[U,Σ,V]＝svd(A)

sorting sigma and V according to the value in sigma from large to small according to the corresponding relation of the right singular vector and the left singular vector of A obtained by SVD, wherein the right singular vector corresponding to the minimum value in sigma is the approximate solution of h:

h＝V[[min(∑)],:]

where min (Σ) represents taking the minimum value of Σ.

Since the process of calculating the approximate solution of h is prior art, it will not be described in detail here.

After an approximate solution for H is obtained, a coordinate mapping matrix H may be determined.

After obtaining the coordinate mapping matrix, the coordinates (u, v) of the dome image can be obtained by mapping the coordinates (x, y) of the gun image, that is:

of course, the coordinates (x, y) of the image of the gun camera can also be mapped and solved through the coordinates (u, v) of the image of the ball machine by using a coordinate mapping matrix, and the specific calculation method is not described herein again.

The following supplementary description describes the training process of the neural network, which may include the following steps:

the method comprises the steps of firstly, obtaining a sample gun camera image, a sample ball machine image, position information of each target in the sample gun camera image and position information of each target in the sample ball machine image;

in this step, a plurality of sets of training sets may be obtained, where any set of training set includes a sample gunlock image, a sample dome camera image, position information of each target in the sample gunlock image, and position information of each target in the sample dome camera image.

The position information of each target may be calibrated manually, or may be calibrated automatically by using other tools.

And secondly, training a pre-constructed initial neural network by using the sample gun camera image, the sample dome camera image, the position information of each target in the sample gun camera image and the position information of each target in the sample dome camera image to obtain the neural network.

Wherein the initial neural network may be an existing neural network.

The training process of the neural network related to this step may specifically be:

1) and inputting a plurality of groups of training sets into the initial neural network, and taking the position information of each target in the sample gunlock image and the position information of each target in the sample ball machine image in one group of training sets as a true value of the initial neural network corresponding to the group of training sets.

2) Parameters in the initial neural network are initialized randomly in the range of (0,1), and the parameters comprise connection weights of neurons and the like.

3) And training each training set through the initial neural network to obtain a corresponding training result.

4) Comparing the training result with the corresponding true value to obtain an output result;

5) calculating the Loss function Loss value of the initial neural network according to the output result;

6) and adjusting parameters of the initial neural network according to the Loss value, and repeating the steps 3) -6) until the Loss value reaches a certain convergence condition, namely the Loss value reaches the minimum value, at the moment, determining the parameters of the initial neural network, finishing the training of the initial neural network, and obtaining the trained neural network.

In the embodiment of the invention, the position information of the target can be quickly obtained by utilizing the advantages of the neural network in the aspect of image identification, the cost of data processing is saved, and the accuracy of determining the position information of the target is improved.

The following describes a gun-sphere coordinate association method provided by an embodiment of the present invention with reference to specific embodiments.

As shown in fig. 3, a gun-sphere coordinate association method provided by an embodiment of the present invention may include the following steps:

s301, inputting a gunlock image and a dome camera image to be associated with the coordinates into a pre-trained neural network respectively to obtain a first output result and a second output result;

s301 is the same as S101, and will not be described herein.

S302, determining at least one same target in the image of the gunlock and the image of the ball machine based on the position information included in the first output result and the position information included in the second output result;

in this embodiment, S302-S303 are an optional implementation manner of S102 in the above embodiment.

Optionally, in this embodiment of the present invention, step S302 may include the following steps a to d:

step a, determining each first image block in the gun camera image and each second image block in the dome camera image; the first image block is an image area corresponding to the position information included in the first output result, and the second image block is an image area corresponding to the position information included in the second output result;

b, the determined first image block and the second image block form a plurality of image block groups, and the similarity of the first image block and the second image block in each image block group is calculated; any image block group consists of a first image block and a second image block;

step c, determining at least one similarity meeting a preset condition in the similarities corresponding to the plurality of image block groups;

and d, aiming at the image block group corresponding to each similarity in the at least one similarity, determining the targets corresponding to the first image block and the second image block in the image block group as the same target.

Specifically, in step a, two-dimensional coordinates of each coordinate point on an edge line of a detection frame including a target in the first output result may be used to locate an image area corresponding to the coordinate points in the bolt image, then intercept an image of the image area, and use the intercepted image as a first image block corresponding to the target; similarly, the two-dimensional coordinates of each coordinate point on the edge line of the detection frame including the target in the second output result may be used to locate an image area corresponding to the coordinate points in the dome camera image, and then the image of the image area may be captured, and the captured image may be used as the second image block corresponding to the target. Thus, a plurality of first image blocks in the gun camera image and a plurality of second image blocks in the dome camera image can be obtained.

In step b, the similarity is used to characterize the probability that two images are similar, and the similarity may be in the form of a percentage, such as 70%, and the similarity may also be in the form of a value between 0 and 1, such as 0.7, and the similarity is not limited to the above.

Optionally, as an implementation manner, the step b may include the following steps b 1-b 2:

b1, arranging and combining a plurality of first image blocks in the gun camera image and a plurality of second image blocks in the dome camera image to form a plurality of image block groups;

specifically, all the first image blocks in the gun camera image and all the second image blocks in the dome camera image may be arranged and combined by a probabilistic method to form a plurality of image block groups.

It is understood that the determined plurality of image block groups covers all permutation combinations of the first image block and the second image block.

Step b2, for each image block group, calculating the similarity of the first image block and the second image block in the image block group.

The method for calculating the similarity can adopt a traditional manually designed measurement algorithm.

As a preferred way to calculate the similarity, each image block group may be input into a pre-trained metric network to obtain the similarity between the first image block and the second image block in the image block group. Wherein the measurement network is a neural network for calculating image similarity. The similarity is calculated by using the measurement network, and the calculation result of the similarity can be quickly and accurately obtained by using the advantages of the neural network. It should be noted that the metric network is trained according to the similarity between the two sample images and the two sample images. For the training process of the metric network, reference may be made to the training process of the neural network described above, and details are not repeated here.

In addition, since the neural network in deep learning can be used to identify the category of the target, in the embodiment of the present invention, the neural network can detect not only the location information of the target, but also the category of the target, wherein the category includes, but is not limited to, people, cars, buildings, plants, animals, and the like. Therefore, in order to improve processing efficiency, the first output result further includes the category of each target in the gun camera image, and the second output result further includes the category of each target in the dome camera image. Accordingly, optionally, as another implementation manner, the step b may include the following steps b 3-b 4:

step b3, aiming at each category, combining the first image block and the second image block of the corresponding target belonging to the category into a plurality of image block groups;

it can be understood that the same object belongs to the same category, and then, the first image block and the second image block corresponding to the object of the same category are taken as one image block group, so that the search range of the same object can be reduced, and the subsequent quick and accurate determination of the same object in the first image block and the second image block is facilitated.

For ease of understanding, this step is illustrated: assuming that there are three categories of objects in the first output result, which are people, cars and buildings, respectively; the first image block corresponding to the object with the type being a person is D, the first image block corresponding to the object with the type being a vehicle is E, and the first image block corresponding to the object with the type being a building is F; the second output result has two types of targets which are respectively a person and a vehicle; the second image block corresponding to the object with the type being the person is G, and the second image block corresponding to the object with the type being the vehicle is H;

then, the first image block D and the second image block G corresponding to the person in the category may be combined into one image block group; and forming the first image block E and the second image block H corresponding to the vehicle in the type into one image block group, and obtaining two image block groups.

It will be appreciated that the number of groups of blocks is reduced compared to the previous implementation in which the determined group of blocks does not contain a group of blocks associated with the first image block F.

Step b4, for each image block group, calculating the similarity of the first image block and the second image block in the image block group.

Wherein, step b4 is the same as step b2, and is not repeated here.

It will be appreciated that in this implementation, the computation time of the similarity is reduced due to the reduced number of image block groups, compared to the previous implementation. And since the first image block and the second image block for calculating the similarity belong to the same category, it is more effective to determine the same object using the obtained similarity.

It should be added that, in this implementation manner, the neural network is obtained by training according to the sample gun camera image, the sample dome camera image, the position information of each target in the sample gun camera image, the position information of each target in the sample dome camera image, the category of each target in the sample gun camera image, and the category of each target in the sample dome camera image, and the training process about the neural network is not repeated.

In addition, for step c:

optionally, in this embodiment of the present invention, the step c may include the following steps c1 to c 3:

step c1, determining the similarity greater than the preset similarity threshold in the similarity corresponding to the plurality of image block groups to obtain a plurality of first similarities;

the preset similarity threshold may be set according to an empirical value, and may be, for example, 60%.

Step c2, for any two first similarities in the multiple first similarities, when it is determined that the image block groups corresponding to the any two first similarities exist: deleting the smaller similarity of any two first similarities when the same first image block or the same second image block;

for ease of understanding, this step is illustrated: assuming that there are two first similarities of 70% (the similarities of the first image block i and the second image block j) and 80% (the similarities of the first image block i and the second image block k), respectively, it is determined that the two similarities have the same first image block i, and then the smaller 70% of the two similarities is deleted.

As can be seen from the numerical values of the first similarities, the probability that the objects corresponding to the first image block i and the second image block j are the same object is smaller than the probability that the objects corresponding to the first image block i and the second image block k are the same object. Since there is no possibility that i and j are the same object, and i and k are the same object, the similarity between the first image block i and the second image block j may be deleted to further narrow the search range of the same object.

In a specific implementation process of this step, the deletion operation may be performed on one or more groups of determined similarities meeting the requirement, where two image block groups corresponding to one group of determined similarities meeting the requirement have the same first image block or the same second image block.

Or traversing all the first similarities, and performing the deleting operation on all the groups of the judged similarities meeting the requirements until no group of the similarities meeting the requirements exists in the plurality of first similarities.

And step c3, determining the remaining first similarity as the similarity meeting the preset condition.

It can be understood that the remaining plurality of first similarities are a plurality of first similarities obtained by deleting the original plurality of first similarities which satisfy the requirement.

As explained in the above example, for the image block group corresponding to the similarity of 80% (i.e. the image block group consisting of the first image block i and the second image block k), the target corresponding to the first image block i and the target corresponding to the second image block k may be determined as the same target.

To facilitate understanding of step S302, a specific implementation of this step is illustrated:

referring to fig. 4, fig. 4(a) is a schematic view of a detection result of a gun camera image for example, and fig. 4(b) is a schematic view of a detection result of a ball camera image for example. There are three targets in fig. 4(a), X1, X2, and Y1, respectively, and two targets in fig. 4(b), X3 and Y2, respectively, wherein the categories of X1, X2, and X3 are human, and the categories of Y1 and Y2 are vehicle.

The method comprises the steps of firstly, determining a plurality of first image blocks and a plurality of second image blocks;

a first image block corresponding to X1, a first image block corresponding to X2, and a first image block corresponding to Y1 may be obtained; and a second image block corresponding to X3 and a second image block corresponding to Y2.

Second, it is determined that the same type of object as X3 exists in the gun camera image with respect to X3 in the dome camera image. Then, one target in the same category as that of the target X3 in the gun camera image is determined to be X1, the similarity between the second image block corresponding to X3 and the first image block corresponding to X1 is calculated, and the similarity ρ is obtained_X3,X180 percent; determine rho_X3,X1If the similarity is larger than the preset similarity threshold value by 60 percent and the similarity between the X3 and other targets in the image of the gunlock is judged to be not calculated, storing rho_X3,X1。

And continuously judging whether a target in the same category as that of the X3 exists in the gun image or not for the X3 in the dome image, and determining that another target in the same category as that of the X3 in the gun image is X2. Then, calculating the similarity between the second image block corresponding to X3 and the first image block corresponding to X2 to obtain the similarity ρ_X3,X285 percent; determine rho_X3,X2Greater than 60% of the preset similarity threshold value, and judging that the calculated similarity between the time X3 and the time X1 in the image of the gunlock is known, comparing rho_X3,X2And ρ_X3,X1，Finding ρ_X3,X2>ρ_X3,X1Then delete ρ_X3,X1Save rho_X3,X1。

And judging that the target with the same category as that of the X3 does not exist in the gun camera image, and selecting the next target in the dome camera image to repeatedly execute the step.

Namely: for Y2 in the dome camera image, determining the target of the same category in the gun camera image as Y1, calculating the similarity between the second image block corresponding to Y2 and the first image block corresponding to Y1 to obtain the similarity rho_Y2,Y170 percent; determining rho_Y2,Y1If the similarity is larger than the preset similarity threshold value by 60%, continuously judging that the similarity between the Y2 and the target in other gun images is not calculated, and storing rho_Y2,Y1。

Finally, from the saved rho_X3,X1And ρ_Y2,Y1It is determined that the targets corresponding to X3 and X1 are the same target, and the targets corresponding to Y2 and Y1 are the same target.

In addition, if it is determined that, for one target in the dome image, there is no target of the same type as the target in the gun camera image, the next target in the dome image is replaced. And if the similarity of a first image block and a second image block is not larger than a preset similarity threshold, deleting the similarity.

Since the shooting range of the gun camera is larger than that of the dome camera, the number of targets in the gun camera image is larger than that in the dome camera image. Therefore, in the above embodiment, in order to reduce the amount of calculation, it is not employed a manner of selecting one target in the gun camera image and determining whether the same target exists among a plurality of targets in the dome camera image, but rather a manner of selecting one target in the dome camera image and determining whether the same target exists among a plurality of targets in the gun camera image.

S303, determining a plurality of groups of matching points from coordinate points corresponding to at least one same target in the gun camera image and the dome camera image;

specific implementations of this step are described below for a different number of the same objectives.

1) When the at least one same target is a same target;

the determining a plurality of sets of matching points from coordinate points corresponding to at least one same target in the gun camera image and the dome camera image may include:

the method comprises the steps of firstly, determining coordinate points at a plurality of preset positions in coordinate points corresponding to the same target in a gun camera image, and determining coordinate points at the plurality of preset positions in coordinate points corresponding to the same target in a dome camera image; wherein the number of the plurality of preset positions is not less than four;

for ease of understanding, this step is illustrated: in the same target, among the coordinate points corresponding to the same target in the gun camera image, coordinate points at four vertices of a rectangular detection frame including the same target may be specified, and in the same manner, among the coordinate points corresponding to the same target in the dome camera image, coordinate points at four vertices of a rectangular detection frame including the same target may be specified.

Of course, coordinate points of center points of four edge lines of the rectangular detection frame containing the same target can also be determined; or, determining coordinate points of center points of three sidelines of the rectangular detection frame containing the same target, coordinate points of center points of the rectangular detection frame, and the like. Here, the coordinate points at the plurality of preset positions in the embodiment of the present invention are not limited.

And secondly, regarding each preset position, taking the coordinate point of the preset position in the gun camera image and the coordinate point of the preset position in the dome camera image as a group of matching points.

The following description will be given by taking a plurality of preset positions as four vertices of a rectangular detection box containing the same target: and regarding each vertex, taking a coordinate point at the vertex in the gun camera image and a coordinate point at the vertex in the dome camera image as a group of matching points. Thus, four sets of matching points can be obtained.

2) When the at least one same target is two same targets or three same targets;

the determining a plurality of sets of matching points from the coordinate points corresponding to at least one same target in the gun camera image and the dome camera image may be: and determining four groups of matching points from the coordinate points corresponding to the two same targets or the three same targets.

Specifically, in the gun camera image, coordinate points at four preset positions may be determined from coordinate points corresponding to the two identical targets or the three identical targets, and in the dome camera image, coordinate points at the four preset positions may be determined from coordinate points corresponding to the two identical targets or the three identical targets; and aiming at each preset position, taking the coordinate point at the preset position in the gun camera image and the coordinate point at the preset position in the dome camera image as a group of matching points.

Taking two identical objects as an example, the process of determining coordinate points at four preset positions is described: the coordinate points at two preset positions can be determined from the coordinate points corresponding to the first same target, and the coordinate points at the other two preset positions can be determined from the coordinate points corresponding to the second same target;

alternatively, it is reasonable to determine a coordinate point at one preset position among coordinate points corresponding to one identical target, determine coordinate points at three preset positions among coordinate points corresponding to another identical target, and the like.

Similarly, for three identical targets, the coordinate points at two preset positions may be determined in the coordinate points corresponding to one identical target, and the coordinate points at one preset position may be determined in the coordinate points corresponding to the other two identical targets, respectively.

It should be noted that the four preset positions may include: any four points on or inside the edge line of the rectangular detection frames containing the same target, for example, the preset positions may be: the vertex of the rectangular detection frame, the center point of the edge line, the center point of the rectangular detection frame containing the same target, and the like.

3) When the at least one identical target is at least four identical targets;

It is understood that at least four sets of matching points may be obtained.

Similarly, the preset position may be one of four vertices of a rectangular detection frame containing the same target, one of center points of four edges, a center point of the rectangular detection frame, any point on the edge of the rectangular detection frame except the vertices, or any point inside the rectangular detection frame except the center point.

For convenience of calculation, the central point of the rectangular detection frame may be selected as the preset position.

S304, calculating a coordinate mapping matrix about the image of the gun camera and the image of the ball machine based on the determined multiple groups of matching points.

Step S304 is the same as step S103, and is not described herein.

In the scheme provided by the embodiment of the invention, firstly, a pre-trained neural network is utilized to obtain a first output result of a gunlock image and a second output result of a dome camera image; secondly, determining a plurality of image block groups based on the position information included in the first output result and the position information included in the second output result, and determining at least one same target in the gunlock image and the dome camera image by calculating the similarity of a first image block and a second image block in each image block group; next, determining a plurality of groups of matching points by using position correspondence from coordinate points corresponding to at least one same target in the gun camera image and the dome camera image; and finally, calculating to obtain a coordinate mapping matrix related to the images of the gunlock and the dome camera by utilizing the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the dome camera image, and the same targets in the gun camera image and the dome camera image have the same characteristics, so that the number of the coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the position information is ensured to be more accurate, therefore, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the matching points are effectively determined, and the purpose of the gun and sphere coordinate association is effectively realized.

Meanwhile, for the gun camera and the ball camera which already comprise the detection function, the neural network provided by the embodiment of the invention can reuse the deep learning network of the original gun camera or ball camera without additionally increasing the network, and the memory and power consumption requirements of the gun camera or the ball camera cannot be increased; the embodiment of the invention utilizes the similarity of the measurement network to calculate the target, thereby improving the precision; in the embodiment of the invention, the time-consuming calculation process can be executed by the GPU or the deep learning/neural network acceleration module so as to improve the calculation speed; and the advantages of the framework can be fully utilized in the deep learning camera, so that the deep learning camera can carry on an acceleration module of a deep learning network to quickly execute network calculation, and coordinate association can be quickly and efficiently completed.

Corresponding to the above method embodiment, an embodiment of the present invention further provides a gun-sphere coordinate correlation apparatus, as shown in fig. 5, the apparatus includes:

an obtaining module 501, configured to input the image of the gun camera and the image of the dome camera to be associated with the coordinates into a pre-trained neural network, respectively, so as to obtain a first output result and a second output result; the neural network is used for identifying position information of targets in an image, the first output result comprises position information of each target in the gunlock image, and the second output result comprises position information of each target in the dome camera image;

a determining module 502, configured to determine multiple sets of matching points based on the position information included in the first output result and the position information included in the second output result; wherein, arbitrary group matching point includes a coordinate point in the rifle bolt image with a coordinate point in the ball machine image, and two coordinate points that arbitrary group matching point includes satisfy: the positions which belong to the same target and correspond to the same target in the image area have the correspondence;

a calculating module 503, configured to calculate a coordinate mapping matrix for the image of the gun camera and the image of the ball machine based on the determined sets of matching points.

Optionally, in this embodiment of the present invention, the determining module 502 includes:

Optionally, in an embodiment of the present invention, the first determining sub-module includes:

Optionally, in an embodiment of the present invention, the first output result further includes a category of each target in the image of the gun camera, and the second output result further includes a category of each target in the image of the ball machine;

the computing unit is specifically configured to:

Optionally, in an embodiment of the present invention, the calculating unit is specifically configured to:

Optionally, in an embodiment of the present invention, the second determining unit is specifically configured to:

Optionally, in the embodiment of the present invention, the at least one identical target is an identical target;

the second determining submodule is specifically configured to:

optionally, in the embodiment of the present invention, the at least one identical target is at least four identical targets;

the second determining submodule is specifically configured to:

Corresponding to the above method embodiment, the embodiment of the present invention further provides an electronic device, as shown in fig. 6, which may include a processor 601 and a memory 602, wherein,

the memory 602 is used for storing computer programs;

the processor 601 is configured to implement the steps of the gun-sphere coordinate association method provided by the embodiment of the present invention when executing the program stored in the memory 602.

The Memory may include a RAM (Random Access Memory) or an NVM (Non-Volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.

Through above-mentioned electronic equipment, can realize: firstly, obtaining a first output result of a gunlock image and a second output result of a dome camera image by utilizing a pre-trained neural network; secondly, determining a plurality of groups of matching points in coordinate points with corresponding positions in image areas corresponding to the same targets of the gunlock image and the dome camera image based on the position information of each target included in the first output result and the position information of each target included in the second output result; and finally, calculating to obtain a coordinate mapping matrix related to the images of the gunlock and the dome camera by utilizing the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the dome camera image, and the same targets in the gun camera image and the dome camera image have the same characteristics, so that the number of the coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the position information is ensured to be more accurate, therefore, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the matching points are effectively determined, and the purpose of the gun and sphere coordinate association is effectively realized.

In addition, corresponding to the gun and sphere coordinate association method provided in the foregoing embodiments, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the gun and sphere coordinate association method provided in the embodiment of the present invention.

The above-mentioned computer-readable storage medium stores an application program that executes the gun-sphere coordinate association method provided by the embodiment of the present invention when executed, and thus can implement: firstly, obtaining a first output result of a gunlock image and a second output result of a dome camera image by utilizing a pre-trained neural network; secondly, determining a plurality of groups of matching points in coordinate points with corresponding positions in image areas corresponding to the same targets of the gunlock image and the dome camera image based on the position information of each target included in the first output result and the position information of each target included in the second output result; and finally, calculating to obtain a coordinate mapping matrix related to the images of the gunlock and the dome camera by utilizing the determined multiple groups of matching points. In the scheme, the matching points based on the coordinate association are extracted from the image areas corresponding to the same targets in the gun camera image and the dome camera image, and the same targets in the gun camera image and the dome camera image have the same characteristics, so that the number of the coordinate points with the same characteristics is ensured, and the position information of each target is determined through a neural network with strong learning performance, so that the position information is ensured to be more accurate, therefore, the problems of insufficient characteristic points or excessive noise points and the like can be avoided, the matching points are effectively determined, and the purpose of the gun and sphere coordinate association is effectively realized.

For the embodiments of the electronic device and the computer-readable storage medium, since the contents of the related methods are substantially similar to those of the foregoing embodiments of the methods, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the embodiments of the methods.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A gun-sphere coordinate correlation method, comprising:

2. The method of claim 1, wherein determining a plurality of sets of matching points based on the location information included in the first output result and the location information included in the second output result comprises:

3. The method of claim 2, wherein determining at least one identical target in the bolt face image and the ball machine image based on the position information included in the first output result and the position information included in the second output result comprises:

4. The method of claim 3, wherein the first output further includes a category of each object in the image of the bolt face and the second output further includes a category of each object in the image of the ball machine;

5. The method according to claim 3 or 4, wherein calculating, for each group of blocks, the similarity between the first block and the second block in the group of blocks comprises:

6. The method according to claim 3 or 4, wherein the determining at least one similarity satisfying a preset condition among the similarities corresponding to the plurality of image block groups comprises:

7. The method of claim 2, wherein the at least one identical object is one identical object;

8. The method of claim 2, wherein the at least one identical target is at least four identical targets;

9. A gun-sphere coordinate correlation apparatus, comprising:

10. The apparatus of claim 9, wherein the determining module comprises:

11. The apparatus of claim 10, wherein the first determining submodule comprises:

12. The apparatus of claim 11, wherein the first output further comprises a category of each object in the image of the bolt face, and the second output further comprises a category of each object in the image of the ball machine;

the computing unit is specifically configured to:

13. The apparatus according to claim 11 or 12, wherein the computing unit is specifically configured to:

14. An electronic device comprising a processor and a memory, wherein,

the memory is used for storing a computer program;

the processor, when executing the program stored in the memory, implementing the method steps of any of claims 1-8.

15. A computer-readable storage medium, characterized in that,

the computer-readable storage medium has stored therein a computer program which, when being executed by a processor, carries out the method steps of any one of claims 1 to 8.