CN117354471A - Multi-camera collaborative monitoring method, device, equipment and storage medium - Google Patents
Multi-camera collaborative monitoring method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117354471A CN117354471A CN202311651214.0A CN202311651214A CN117354471A CN 117354471 A CN117354471 A CN 117354471A CN 202311651214 A CN202311651214 A CN 202311651214A CN 117354471 A CN117354471 A CN 117354471A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- features
- image
- feature
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 193
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012806 monitoring device Methods 0.000 claims abstract description 19
- 230000009466 transformation Effects 0.000 claims abstract description 18
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 9
- 238000009499 grossing Methods 0.000 claims description 7
- 230000004927 fusion Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 7
- 230000000694 effects Effects 0.000 description 9
- 239000013598 vector Substances 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 5
- 230000018109 developmental process Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/85—Camera processing pipelines; Components thereof for processing colour signals for matrixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/86—Camera processing pipelines; Components thereof for processing colour signals for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2624—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of whole input images, e.g. splitscreen
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of image processing, and discloses a multi-camera collaborative monitoring method, a multi-camera collaborative monitoring device, multi-camera collaborative monitoring equipment and a storage medium; the method comprises the following steps: extracting features of a plurality of original monitoring images in a target scene to obtain local features and global features; performing feature matching on the local features and the global features to obtain matching feature points; performing projection transformation on the original monitoring image based on the matched characteristic points to obtain a spliced image; fusing according to the spliced images to obtain a monitoring image, and cooperatively monitoring the target scene based on the monitoring image; according to the invention, the local features and the global features of each original monitoring image are extracted, the features are clustered, the clustered features are re-registered, the association among the features is finer and clearer due to the clustering division of the features, the features of different categories are matched from multiple layers, so that more accurate registration points are obtained, the image is spliced more accurately and rapidly, and the collaborative monitoring of multiple cameras is effectively realized.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for collaborative monitoring with multiple cameras.
Background
Public safety monitoring problems are becoming increasingly important in people's daily lives. With the rapid development of information science and technology, the price of monitoring equipment is continuously reduced, the monitoring equipment is widely applied to various scenes, but the requirement of people on security monitoring is not met by single monitoring equipment, and the requirement of people on monitoring images is gradually increased.
Most of the traditional video monitoring systems are based on videos provided by relatively independent monitoring cameras, and the independent cameras have large blind points of view due to the limitation of view angles, so that panoramic macroscopic views cannot be provided in some large-scene application environments. The wide-angle lens monitoring equipment can provide a wider observation range, large-scene imaging can better monitor the overall situation, but the monitoring image splicing effect of a plurality of cameras is poor, the monitoring effect is seriously affected, and the expansion of monitoring work is very unfavorable.
Disclosure of Invention
The invention mainly aims to provide a multi-camera collaborative monitoring method, device, equipment and storage medium, which aim to solve the technical problems that the splicing effect of monitoring images of a plurality of cameras is poor and the development of monitoring work is influenced in global monitoring in the prior art.
In order to achieve the above purpose, the present invention provides a multi-camera collaborative monitoring method, which comprises the following steps:
extracting features of a plurality of original monitoring images in a target scene to obtain local features and global features;
performing feature matching on the local features and the global features to obtain matching feature points;
performing projection transformation on the original monitoring image based on the matching feature points to obtain a spliced image;
and fusing the spliced images to obtain a monitoring image, and realizing cooperative monitoring in the target scene based on the monitoring image.
Optionally, the feature extraction of the plurality of original monitoring images in the target scene to obtain local features and global features includes:
extracting context information of an original monitoring image in a target scene, and obtaining a global feature sequence according to the context information;
smoothing the global feature sequence to obtain global features;
and comparing the pixel levels of the original monitoring images to obtain local extreme points, and obtaining local reference features based on the local extreme points.
Optionally, the comparing the pixel level of the original monitored image to obtain a local extremum point, and obtaining the local reference feature based on the local extremum point includes:
extracting key points of the original monitoring image, and comparing pixel levels of the key points to obtain initial local extremum points;
removing discrete pseudo extremum points in the initial local extremum points to obtain reference extremum points;
and removing edge extreme points in the reference extreme points to obtain local extreme points, and obtaining local reference features based on the local extreme points.
Optionally, the performing feature matching on the local feature and the global feature to obtain a matching feature point includes:
clustering the local features and the global features to obtain sub-features;
dividing the sub-features into different sub-regions;
and carrying out feature matching on the sub-features in different sub-regions to obtain matching feature points.
Optionally, the clustering the local feature and the global feature to obtain sub-features includes:
obtaining plane information of each global feature, and carrying out initial clustering according to the plane information to obtain initial clustering features;
obtaining the space distance between each local feature, and clustering the reference features according to the space distance to obtain reference clustering features;
and taking the coincident cluster characteristic of the initial cluster characteristic and the reference cluster characteristic as a sub-characteristic.
Optionally, the performing projective transformation on the original monitored image based on the matching feature points to obtain a stitched image includes:
constructing a world coordinate system according to camera calibration of each camera;
obtaining a splicing matrix of each original monitoring image according to the transformation parameters among the cameras;
and mapping the matching characteristic points into the world coordinate system based on the splicing matrix to obtain a spliced image.
Optionally, the fusing according to the stitched image to obtain a monitoring image, and implementing collaborative monitoring in the target scene based on the monitoring image includes:
decomposing the stitched image into a plurality of sub-images based on image factors of the stitched image;
fusing the plurality of sub-images based on different latitudes to obtain a fused image;
and carrying out image reconstruction on the fusion image to obtain a monitoring image, and realizing cooperative monitoring on the target scene based on the monitoring image.
In addition, in order to achieve the above object, the present invention further provides a multi-camera collaborative monitoring device, which includes:
the feature extraction module is used for extracting features of a plurality of original monitoring images in the target scene to obtain local features and global features;
the feature matching module is used for carrying out feature matching on the local features and the global features to obtain matching feature points;
the image stitching module is used for carrying out projection transformation on the original monitoring image based on the matching characteristic points to obtain a stitched image;
and the monitoring module is used for fusing the spliced images to obtain a monitoring image, and realizing cooperative monitoring in the target scene based on the monitoring image.
In addition, in order to achieve the above object, the present invention further provides a multi-camera collaborative monitoring apparatus, which includes: the system comprises a memory, a processor and a multi-camera collaborative monitoring program stored on the memory and capable of running on the processor, wherein the multi-camera collaborative monitoring program is configured to realize the steps of the multi-camera collaborative monitoring method.
In addition, in order to achieve the above object, the present invention further provides a storage medium, on which a multi-camera collaborative monitoring program is stored, which when executed by a processor, implements the steps of the multi-camera collaborative monitoring method as described above.
According to the invention, the local features and the global features of each original monitoring image are extracted, the matching of the local features and the global features to the original monitoring images can be carried out from a plurality of layers with high dimensionality and low latitude to obtain more accurate and fine registration points, and further, the clustered feature points are classified and then registered, so that the feature point classification is more detailed and definite, the image splicing can be more accurately and rapidly carried out, the cooperative monitoring of a plurality of cameras is more effectively realized, and the problems that the monitoring image splicing effect of a plurality of cameras is poor and the development of monitoring work is influenced during the global monitoring are solved.
Drawings
FIG. 1 is a schematic structural diagram of a multi-camera collaborative monitoring device for a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flow chart of a first embodiment of a multi-camera collaborative monitoring method according to the present invention;
FIG. 3 is a flowchart of a second embodiment of a multi-camera collaborative monitoring method according to the present invention;
FIG. 4 is a global feature matching diagram of an embodiment of a multi-camera collaborative monitoring method according to the present invention;
FIG. 5 is a schematic diagram of a pyramid differential image detection result according to an embodiment of the multi-camera collaborative monitoring method according to the present invention;
FIG. 6 is a flow chart of matching local and global features of an embodiment of a multi-camera collaborative monitoring method according to the present invention;
FIG. 7 is a flowchart of a third embodiment of a multi-camera collaborative monitoring method according to the present invention;
fig. 8 is a block diagram of a first embodiment of a multi-camera collaborative monitoring device according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a multi-camera collaborative monitoring device in a hardware operation environment according to an embodiment of the present invention.
As shown in fig. 1, the multi-camera collaborative monitoring device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the structure shown in fig. 1 is not limiting of a multi-camera collaborative monitoring device and may include more or fewer components than shown, or certain components may be combined, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a multi-camera collaborative monitor may be included in a memory 1005 as one storage medium.
In the multi-camera collaborative monitoring device shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the multi-camera collaborative monitoring device of the present invention may be disposed in the multi-camera collaborative monitoring device, where the multi-camera collaborative monitoring device invokes the multi-camera collaborative monitoring program stored in the memory 1005 through the processor 1001, and executes the multi-camera collaborative monitoring method provided by the embodiment of the present invention.
The embodiment of the invention provides a multi-camera collaborative monitoring method, and referring to fig. 2, fig. 2 is a flow chart of a first embodiment of the multi-camera collaborative monitoring method.
In this embodiment, the multi-camera collaborative monitoring method includes the following steps:
step S10: and extracting the characteristics of a plurality of original monitoring images in the target scene to obtain local characteristics and global characteristics.
It is understood that the target scene may be an area where the camera is to be manually installed, such as a parking lot of a district, a playground of a school, an office place inside an enterprise, or the like. The cameras can be arranged at different corners of the monitoring area according to requirements or in a scientifically calculated arrangement mode.
It should be understood that the original monitoring image may be an image acquired by each camera at the same time, and the original monitoring image may be the same frame monitoring image or the same second monitoring image, which is determined according to the processing speed of the image.
It should be noted that, in order to comprehensively extract the image information of the original monitoring image, the influence on the monitoring effect caused by poor splicing or fusion of a plurality of monitoring images due to neglecting of the detailed information in the image is avoided.
It can be understood that the local feature can be a feature obtained by focusing on the local content of the original monitoring image, but the local feature cannot obtain the whole content of the monitoring image, and the global feature is extracted again, so that the original monitoring image can be spliced and fused from the whole content and the local detail based on the local feature and the global feature, and a better monitoring effect is obtained.
It should be noted that, the execution main body of the embodiment is a multi-camera collaborative monitoring device, where the multi-camera collaborative monitoring device has functions of data processing, data communication, program running, and the like, and the multi-camera collaborative monitoring device may be an integrated controller, a control computer, and other devices with similar functions, and certainly may also be other devices with similar functions, which is not limited in this embodiment.
Step S20: and performing feature matching on the local features and the global features to obtain matching feature points.
It can be understood that the feature matching of the local features and the global features may be to match the local features of any two original monitoring images respectively, and at the same time, perform feature matching on the global features of the original monitoring images.
It can be appreciated that there is no requirement for local feature matching and global feature matching, and selection and adjustment can be performed according to actual data.
In specific implementation, the local features and the global features of any two original monitoring images are converted into local feature vectors and global feature vectors, and matching is performed based on the local feature vectors and the global feature vectors.
It should be noted that, the global feature matching may be understood as global matching, for example, if the global features extracted from the two original monitoring images all have automobiles, the global features of the automobiles are matched; the further extracted local features also comprise local features of windows, rearview mirrors, license plates and the like of the automobile, and feature matching is further carried out based on the local features to obtain matching feature points.
It should be emphasized that the matching of the two features may be comparing the similarity of the two feature vectors, and when the similarity is greater than a similarity threshold, the feature points corresponding to the two feature vectors are considered to be matching feature points, and image stitching may be performed based on the two matching feature points, where the similarity threshold may be 80% or 90%, or may be other similarity ratios, and may be adjusted according to the actual situation, and the embodiment is not limited.
Step S30: and carrying out projection transformation on the original monitoring image based on the matching characteristic points to obtain a spliced image.
It can be understood that the shooting angles of each original monitoring image are different, and after the matching feature points are obtained through feature point matching in the original monitoring images, projection transformation processing is still required to be carried out on two original images with two different angles, so that the two images are spliced and fused, and the viewing of human eyes is more met.
It should be understood that the stitched image may be understood as an image obtained by projectively transforming two original monitoring images based on matching feature points.
It should be noted that, the projective transformation of the original monitoring image based on the matching feature points may be to construct a world coordinate system according to camera calibration of each camera; obtaining a splicing matrix of each original monitoring image according to the transformation parameters among the cameras; and mapping the matching characteristic points into the world coordinate system based on the splicing matrix to obtain a spliced image.
It should be noted that, the world coordinate system may be constructed according to the camera calibration of each camera by taking the monitoring images of each camera, determining the coordinate transformation matrix between the monitoring images based on the space difference of the same object in the monitoring images of each monitoring camera, determining a world coordinate system based on the coordinate transformation matrix, and transforming and positioning the object in each monitoring image into the world coordinate system.
In a specific implementation, the position of the feature matching point in the world coordinate system can be obtained based on the feature matching point and the coordinate transformation matrix of the camera, and the spliced image is obtained by mapping the original monitoring image into the world coordinate system based on the position of the matching feature point.
Step S40: and fusing the spliced images to obtain a monitoring image, and realizing cooperative monitoring in the target scene based on the monitoring image.
It can be understood that the spliced image is a monitoring image from a plurality of different parameters with different angles, and the spliced image may have color errors, so that the spliced image needs to be fused, and the problem that the spliced image has color errors or is spliced misplacement is avoided.
It should be noted that, the fusing according to the stitched image to obtain a monitoring image, and implementing collaborative monitoring on the target scene based on the monitoring image may be to decompose the stitched image into a plurality of sub-images based on an image factor of the stitched image; fusing the plurality of sub-images based on different latitudes to obtain a fused image; and carrying out image reconstruction on the fusion image to obtain a monitoring image, and realizing cooperative monitoring on the target scene based on the monitoring image.
It is understood that the image factors may be factors such as resolution, frequency, or direction characteristics of the stitched image.
It should be understood that the decomposition of the stitched image based on the image factor may be based on resolution, for example, taking stitched images at different resolutions, and using the stitched images at each resolution as separate sub-images. The split image may be decomposed based on a direction characteristic factor of the image factor, for example, a light direction of each object in the room, so as to obtain a plurality of sub-images.
It can be understood that the merging of the plurality of sub-images based on different dimensions may be to first merge colors of the plurality of sub-images from a one-dimensional angle and then merge the sub-images from an overall two-dimensional angle, and the two-dimensional merging may include adjustment of angles and light rays.
It should be noted that, the image reconstruction may be to obtain the monitoring image after reconstructing the fused image based on different algorithms, where the reconstruction algorithm may be a two-dimensional fourier transform method, a back projection method, an iteration method, or a fitting approximation method, and may be selected or adjusted according to actual requirements, which is not limited in this embodiment.
According to the method, the local features and the global features of each original monitoring image are extracted, the local features and the global features are used for identifying and matching the original monitoring images, feature matching can be carried out from a plurality of layers with high dimensionality and low latitude, more accurate and fine registration points are obtained, further, the clustered feature points are subjected to registration after being divided, the feature point division is finer and clearer, therefore image splicing can be carried out more accurately and more quickly, the cooperative monitoring of a plurality of cameras is achieved more effectively, the problem that the monitoring image splicing effect of a plurality of cameras is poor during global monitoring, and the development of monitoring work is affected is solved.
Referring to fig. 3, fig. 3 is a flow chart of a second embodiment of a multi-camera collaborative monitoring method according to the present invention.
Based on the above first embodiment, the multi-camera collaborative monitoring method in this embodiment includes:
step S11: and extracting the context information of the original monitoring image in the target scene, and obtaining the global feature sequence according to the context information.
It is understood that extracting the context information of the original monitoring image may be global shape feature extraction of the original monitoring image.
It should be understood that extracting the context information of the original monitoring image in the target scene may be extracting a contour feature sequence of each object in the original monitoring image, and taking the contour feature sequence as a global feature sequence.
It can be understood that the i-th reference point of the original monitoring image is vi, the vi belongs to the monitoring image contour space containing N points, all the context information in the infrared image is connected according to the contour sequence, and an infrared image global contour feature sequence is obtained.
It is emphasized that a point v is taken in the reference polar coordinate system (which can be understood as the coordinate system established with the original monitoring image) i As an origin, the relative polar coordinates of other contour reference points are obtained, and the relative polar coordinates can be obtained by referring to the following formula:
the relative polar coordinates can be further changed according to the image shape sequence in the monitoring image, so as to obtain a global feature sequence, and the specific global feature sequence can refer to the following formula:
step S12: and carrying out smoothing treatment on the global feature sequence to obtain global features.
It can be understood that contradiction between precision and sensitivity of each feature in the global feature sequence can be avoided through smoothing, and robustness can be improved after smoothing so as to realize balance between precision and sensitivity.
The global feature sequence, F i The sequence is divided into N-1 sequences, and a plurality of mutually independent subsequences [1, t]，[t+1，2t]，[2t+1，3t]…, where the positive integer is t, solving for the mean of the sub-sequence components can be referred to by the following formula:
wherein, the sub sequence number after segmentation is C, c=1, 2, 3; c= (N-1)/t; η is a constant; establishing a global feature after smoothing treatment through the C subsequences; each sub-sequence component mean can be understood as a global feature.
The emphasis of the value is that after the global feature is extracted, the global feature can be matched with the shape feature of the object in the original monitoring image through the global feature, and the detailed global feature matching result can be referred to as fig. 4.
Step S13: and comparing the pixel levels of the original monitoring images to obtain local extreme points, and obtaining local reference features based on the local extreme points.
It is emphasized that the comparing the pixel levels of the original monitoring image to obtain local extremum points, and obtaining local reference features based on the local extremum points may be extracting key points of the original monitoring image, and comparing the pixel levels of the key points to obtain initial local extremum points; removing discrete pseudo extremum points in the initial local extremum points to obtain reference extremum points; and removing edge extreme points in the reference extreme points to obtain local extreme points, and obtaining local reference features based on the local extreme points.
Before comparing the pixel level of the monitored image, the Gaussian filters with different scale parameters and downsampling with the same scale parameter can be used for repeated operation to obtain images with different scales and different blurring degrees, the Gaussian blurred images in each layer are arranged from small to large according to the scale, each layer of image is the reduction of the image of the previous layer, and finally, the complete image scale space from large to small, namely the image pyramid, is constructed.
Further, a group of difference images is generated by performing difference operation on two adjacent pyramid Gaussian blur images. Wherein the image of each layer is the difference between two adjacent gaussian blurred images. And similarly, generating differential operation layer by layer and group by group, and constructing a complete pyramid differential image.
It should be noted that, extracting the key points of the original monitoring image, comparing the pixel levels of the key points, and obtaining the initial local extremum point may be obtaining the initial local extremum point by comparing the pixel levels of the image points in the original monitoring image with the pyramid differential image, where the pyramid differential image detection result of the local extremum point may refer to fig. 5, where each detection point may compare the pixel values of the neighborhood point of the current image layer 8 and the 9 points around the adjacent two-layer image, and find out the point where the pixel values are both greater than or both less than the central pixel value, and the point is the initial local extremum point.
It should be further noted that, eliminating the discrete pseudo extremum point in the initial local extremum points, obtaining the reference extremum point may be calculating the accurate position differential value of the initial extremum point according to the position of each initial local extremum point, and if the differential value is greater than or equal to 0.03, the discrete pseudo extremum point is considered to be eliminated.
The method further includes removing edge extremum points in the reference extremum points to obtain local extremum points, calculating curvature information of each initial extremum point according to the direction of feature vector of each initial extremum point, calculating edge distance threshold based on the curvature information, obtaining edge degree of each initial extremum point according to determinant sum trace of each initial extremum point, and considering the initial extremum point as the edge extremum point when the edge degree is smaller than the edge distance threshold. In a specific calculation process, the following formula may be referred to for the calculation mode of the edge degree:
wherein P represents a determinant of the initial extremum point, N represents a trace of the initial extremum point, where M is a preset value, which may be 0-50, and in this embodiment, the final edge degree threshold is calculated when M is 10. Further, a and B are the maximum eigenvalue and the minimum eigenvalue of the extreme point in the initial direction of the horizontal direction, wherein the determinant of the extreme point is the product of the maximum eigenvalue and the minimum eigenvalue, and the trace of the initial extreme point is the sum of the maximum eigenvalue and the minimum eigenvalue.
It should be noted that, after obtaining the local feature based on the local extremum, the local feature is matched again to obtain a local feature matching point, and the local feature matching point and the global feature matching point are spliced, where the matching result of the local feature matching point and the global feature matching point can refer to fig. 6.
According to the embodiment, the global feature is obtained based on the context information by extracting the context information of the original monitoring image, and the local reference feature is further obtained by comparing the pixel level of the original monitoring image; and therefore, the global features and the local features can be matched to obtain more comprehensive and accurate feature matching points, so that the splicing and fusion of a plurality of original monitoring images are realized, and the collaborative monitoring work of a plurality of cameras is smoothly developed.
Referring to fig. 7, fig. 7 is a flowchart of a third embodiment of a multi-camera collaborative monitoring method according to the present invention.
Based on the above-mentioned first embodiment, the multi-camera collaborative monitoring method in this embodiment includes:
step S21: and clustering the local features and the global features to obtain sub-features.
It should be noted that, the local features and the global features may be clustered to obtain multiple sub-features, and the multiple sub-features are processed to obtain multiple matching feature points; or clustering the local features and the global features simultaneously to obtain sub-features.
It is understood that the sub-feature may be a feature set, which is each clustered feature obtained by clustering the local feature and/or the global feature.
It should be noted that, the clustering of the features may be a hierarchical clustering method (AGNES, AGglomerative NESting algorithm is a hierarchical clustering method of aggregation, where AGNES initially uses each object as a cluster, and then the clusters are combined step by step according to some criteria) to cluster the local features and the global features. Other clustering methods are also possible, and the present embodiment is not limited to this, and may be limited according to actual situations.
The local features and the global features are clustered to obtain sub-features, namely plane information of each local feature and each global feature can be obtained, initial clustering is carried out according to the plane information, and initial clustering features are obtained; obtaining the space distance between each local feature and the global feature, and clustering according to the space distance to obtain a reference clustering feature; and taking the coincident cluster characteristic of the initial cluster characteristic and the reference cluster characteristic as a sub-characteristic.
Step S22: the sub-features are divided into different sub-regions.
It can be understood that the whole registered original image is divided into a plurality of subareas, each subarea comprises two or more original monitoring images to be spliced and fused, and each original monitoring image is divided based on clustered sub-features in each original monitoring image to obtain a plurality of subareas.
It should be noted that, because the sub-features are not necessarily all clustered together, there may be discrete sub-features, and thus, there may be multiple clustered sub-features for each sub-region.
It is emphasized that the relationship between the sub-regions may be inclusive, parallel or intersecting.
Step S23: and carrying out feature matching on the sub-features in different sub-regions to obtain matching feature points.
It is understood that the global features and the local features in the sub-features are further matched based on the sub-features in the respective sub-regions.
It should be noted that, matching is performed based on the local features and the global features in each sub-region, so that the same region in the two original images can be matched and spliced from details and the whole of the region, and a monitoring image with better splicing effect can be obtained.
According to the embodiment, the local features and the global features are clustered to obtain the sub-features, then the feature matching of each sub-region is carried out more specifically based on the sub-features, and the sub-features of each sub-region are matched, so that invalid repeated matching calculation of a plurality of feature points of the same region is avoided, the matching speed is improved while the accuracy is ensured, the spliced and fused image can be obtained more timely, and the monitoring work is realized and expanded better.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium is stored with a multi-camera collaborative monitoring program, and the multi-camera collaborative monitoring program realizes the steps of the multi-camera collaborative monitoring method when being executed by a processor.
Referring to fig. 8, fig. 8 is a block diagram illustrating a configuration of a first embodiment of a multi-camera collaborative monitoring device according to the present invention.
As shown in fig. 8, the multi-camera collaborative monitoring device provided by the embodiment of the invention includes:
the feature extraction module 10 is configured to perform feature extraction on a plurality of original monitoring images in a target scene to obtain local features and global features;
the feature matching module 20 is configured to perform feature matching on the local feature and the global feature to obtain a matching feature point;
the image stitching module 30 is configured to perform projective transformation on the original monitored image based on the matching feature points to obtain a stitched image;
and the monitoring module 40 is used for fusing the spliced images to obtain a monitoring image, and realizing cooperative monitoring in the target scene based on the monitoring image.
According to the method, the local features and the global features of each original monitoring image are extracted, the local features and the global features are used for identifying and matching the original monitoring images, feature matching can be carried out from a plurality of layers with high dimensionality and low latitude, more accurate and fine registration points are obtained, further, the clustered feature points are subjected to registration after being divided, the feature point division is finer and clearer, therefore image splicing can be carried out more accurately and more quickly, the cooperative monitoring of a plurality of cameras is achieved more effectively, the problem that the monitoring image splicing effect of a plurality of cameras is poor during global monitoring, and the development of monitoring work is affected is solved.
In an embodiment, the feature extraction module 10 is further configured to extract context information of an original monitoring image in the target scene, and obtain a global feature sequence according to the context information;
smoothing the global feature sequence to obtain global features;
and comparing the pixel levels of the original monitoring images to obtain local extreme points, and obtaining local reference features based on the local extreme points.
In an embodiment, the feature extraction module 10 is further configured to extract key points of the original monitored image, and compare pixel levels of the key points to obtain an initial local extremum point;
removing discrete pseudo extremum points in the initial local extremum points to obtain reference extremum points;
and removing edge extreme points in the reference extreme points to obtain local extreme points, and obtaining local reference features based on the local extreme points.
In an embodiment, the feature matching module 20 is further configured to cluster the local feature and the global feature to obtain sub-features;
dividing the sub-features into different sub-regions;
and carrying out feature matching on the sub-features in different sub-regions to obtain matching feature points.
In an embodiment, the feature matching module 20 is further configured to obtain plane information of each global feature, and perform initial clustering according to the plane information to obtain initial clustering features;
obtaining the space distance between each local feature, and clustering the reference features according to the space distance to obtain reference clustering features;
and taking the coincident cluster characteristic of the initial cluster characteristic and the reference cluster characteristic as a sub-characteristic.
In an embodiment, the image stitching module 30 is further configured to construct a world coordinate system according to camera calibration of each camera;
obtaining a splicing matrix of each original monitoring image according to the transformation parameters among the cameras;
and mapping the matching characteristic points into the world coordinate system based on the splicing matrix to obtain a spliced image.
In an embodiment, the monitoring module 40 is further configured to decompose the stitched image into a plurality of sub-images based on an image factor of the stitched image;
fusing the plurality of sub-images based on different latitudes to obtain a fused image;
and carrying out image reconstruction on the fusion image to obtain a monitoring image, and realizing cooperative monitoring on the target scene based on the monitoring image.
It should be understood that the foregoing is illustrative only and is not limiting, and that in specific applications, those skilled in the art may set the invention as desired, and the invention is not limited thereto.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.
Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of embodiments, it will be clear to a person skilled in the art that the above embodiment method may be implemented by means of software plus a necessary general hardware platform, but may of course also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory (ROM)/RAM, magnetic disk, optical disk) and comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
It should be understood that, although the steps in the flowcharts in the embodiments of the present application are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the figures may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily occurring in sequence, but may be performed alternately or alternately with other steps or at least a portion of the other steps or stages.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. The multi-camera collaborative monitoring method is characterized by comprising the following steps of:
extracting features of a plurality of original monitoring images in a target scene to obtain local features and global features;
performing feature matching on the local features and the global features to obtain matching feature points;
performing projection transformation on the original monitoring image based on the matching feature points to obtain a spliced image;
and fusing the spliced images to obtain a monitoring image, and realizing cooperative monitoring in the target scene based on the monitoring image.
2. The multi-camera collaborative monitoring method according to claim 1, wherein the feature extraction of the plurality of original monitoring images in the target scene to obtain the local feature and the global feature comprises:
extracting context information of an original monitoring image in a target scene, and obtaining a global feature sequence according to the context information;
smoothing the global feature sequence to obtain global features;
and comparing the pixel levels of the original monitoring images to obtain local extreme points, and obtaining local reference features based on the local extreme points.
3. The multi-camera collaborative monitoring method according to claim 2, wherein the comparing pixel levels of the original monitored image to obtain local extremum points, obtaining local reference features based on the local extremum points, comprises:
extracting key points of the original monitoring image, and comparing pixel levels of the key points to obtain initial local extremum points;
removing discrete pseudo extremum points in the initial local extremum points to obtain reference extremum points;
and removing edge extreme points in the reference extreme points to obtain local extreme points, and obtaining local reference features based on the local extreme points.
4. The method for collaborative monitoring by multiple cameras according to claim 1, wherein the performing feature matching on the local feature and the global feature to obtain matching feature points comprises:
clustering the local features and the global features to obtain sub-features;
dividing the sub-features into different sub-regions;
and carrying out feature matching on the sub-features in different sub-regions to obtain matching feature points.
5. The multi-camera collaborative monitoring method according to claim 4, wherein clustering the local feature and the global feature to obtain sub-features comprises:
obtaining plane information of each global feature, and carrying out initial clustering according to the plane information to obtain initial clustering features;
obtaining the space distance between each local feature, and clustering the reference features according to the space distance to obtain reference clustering features;
and taking the coincident cluster characteristic of the initial cluster characteristic and the reference cluster characteristic as a sub-characteristic.
6. The multi-camera collaborative monitoring method according to claim 1, wherein the performing projective transformation on the original monitoring image based on the matching feature points to obtain a stitched image comprises:
constructing a world coordinate system according to camera calibration of each camera;
obtaining a splicing matrix of each original monitoring image according to the transformation parameters among the cameras;
and mapping the matching characteristic points into the world coordinate system based on the splicing matrix to obtain a spliced image.
7. The multi-camera collaborative monitoring method according to any one of claims 1-6, wherein the fusing according to the stitched image results in a monitored image, and the collaborative monitoring in the target scene is implemented based on the monitored image, comprising:
decomposing the stitched image into a plurality of sub-images based on image factors of the stitched image;
fusing the plurality of sub-images based on different latitudes to obtain a fused image;
and carrying out image reconstruction on the fusion image to obtain a monitoring image, and realizing cooperative monitoring on the target scene based on the monitoring image.
8. The utility model provides a many cameras cooperate monitoring device which characterized in that, many cameras cooperate monitoring device includes:
the feature extraction module is used for extracting features of a plurality of original monitoring images in the target scene to obtain local features and global features;
the feature matching module is used for carrying out feature matching on the local features and the global features to obtain matching feature points;
the image stitching module is used for carrying out projection transformation on the original monitoring image based on the matching characteristic points to obtain a stitched image;
and the monitoring module is used for fusing the spliced images to obtain a monitoring image, and realizing cooperative monitoring in the target scene based on the monitoring image.
9. A multi-camera collaborative monitoring device, the device comprising: a memory, a processor, and a multi-camera co-monitoring program stored on the memory and executable on the processor, the multi-camera co-monitoring program configured to implement the multi-camera co-monitoring method of any one of claims 1 to 7.
10. A storage medium, wherein a multi-camera collaborative monitoring program is stored on the storage medium, and the multi-camera collaborative monitoring program when executed by a processor implements the multi-camera collaborative monitoring method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311651214.0A CN117354471B (en) | 2023-12-05 | 2023-12-05 | Multi-camera collaborative monitoring method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311651214.0A CN117354471B (en) | 2023-12-05 | 2023-12-05 | Multi-camera collaborative monitoring method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117354471A true CN117354471A (en) | 2024-01-05 |
| CN117354471B CN117354471B (en) | 2024-04-09 |
Family
ID=89367090
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311651214.0A Active CN117354471B (en) | 2023-12-05 | 2023-12-05 | Multi-camera collaborative monitoring method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117354471B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103336945A (en) * | 2013-06-10 | 2013-10-02 | 黑龙江大学 | Finger vein recognition method fusing local features and global features |
| CN109389555A (en) * | 2018-09-14 | 2019-02-26 | 复旦大学 | A kind of Panorama Mosaic method and device |
| CN111914921A (en) * | 2020-07-24 | 2020-11-10 | 山东工商学院 | Similarity image retrieval method and system based on multi-feature fusion |
| CN114091548A (en) * | 2021-09-23 | 2022-02-25 | 昆明理工大学 | Vehicle cross-domain re-identification method based on key point and graph matching |
| CN114547437A (en) * | 2022-01-04 | 2022-05-27 | 北京易华录信息技术股份有限公司 | Image retrieval method and device |
| CN115866213A (en) * | 2023-03-01 | 2023-03-28 | 深圳市微浦技术有限公司 | Multi-camera cooperative monitoring method, device, equipment and storage medium |
| CN116823905A (en) * | 2023-06-26 | 2023-09-29 | 阿里巴巴达摩院(杭州)科技有限公司 | Image registration method, electronic device, and computer-readable storage medium |
-
2023
- 2023-12-05 CN CN202311651214.0A patent/CN117354471B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103336945A (en) * | 2013-06-10 | 2013-10-02 | 黑龙江大学 | Finger vein recognition method fusing local features and global features |
| CN109389555A (en) * | 2018-09-14 | 2019-02-26 | 复旦大学 | A kind of Panorama Mosaic method and device |
| CN111914921A (en) * | 2020-07-24 | 2020-11-10 | 山东工商学院 | Similarity image retrieval method and system based on multi-feature fusion |
| CN114091548A (en) * | 2021-09-23 | 2022-02-25 | 昆明理工大学 | Vehicle cross-domain re-identification method based on key point and graph matching |
| CN114547437A (en) * | 2022-01-04 | 2022-05-27 | 北京易华录信息技术股份有限公司 | Image retrieval method and device |
| CN115866213A (en) * | 2023-03-01 | 2023-03-28 | 深圳市微浦技术有限公司 | Multi-camera cooperative monitoring method, device, equipment and storage medium |
| CN116823905A (en) * | 2023-06-26 | 2023-09-29 | 阿里巴巴达摩院(杭州)科技有限公司 | Image registration method, electronic device, and computer-readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117354471B (en) | 2024-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220222776A1 (en) | Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution | |
| CN110298884B (en) | Pose estimation method suitable for monocular vision camera in dynamic environment | |
| Wang et al. | Fast image upsampling via the displacement field | |
| CN110738601A (en) | low-resolution face image super-resolution reconstruction method based on three-dimensional face model | |
| Zhu et al. | Super-resolving commercial satellite imagery using realistic training data | |
| Frantc et al. | Machine learning approach for objective inpainting quality assessment | |
| EP2672424A1 (en) | Method and apparatus using adaptive face registration method with constrained local models and dynamic model switching | |
| CN112329616B (en) | Target detection method, device, equipment and storage medium | |
| CN112333468B (en) | Image processing method, device, equipment and storage medium | |
| Sakurai et al. | Restoring aspect ratio distortion of natural images with convolutional neural network | |
| CN117354471B (en) | Multi-camera collaborative monitoring method, device, equipment and storage medium | |
| Ahn et al. | Implement of an automated unmanned recording system for tracking objects on mobile phones by image processing method | |
| EP2672425A1 (en) | Method and apparatus with deformable model fitting using high-precision approximation | |
| EP2672423A1 (en) | Method and apparatus for locating features of an object using deformable models | |
| Kuo | Data augmentation | |
| WO2022206679A1 (en) | Image processing method and apparatus, computer device and storage medium | |
| CN116051736A (en) | Three-dimensional reconstruction method, device, edge equipment and storage medium | |
| Satiro et al. | Super-resolution of facial images in forensics scenarios | |
| Voronin et al. | A vision system using depth inpainting for virtual content reconstruction in Augmented Reality | |
| Viswanathan | Data fusion | |
| Koppal | Diffuse reflectance | |
| Koch et al. | Depth estimation | |
| Rao et al. | Global motion estimation–based method for nighttime video enhancement | |
| Campana et al. | Modification in the SAR super-resolution model using the fractal descriptor LMME in the term regularizer | |
| Tiwari et al. | Towards reduced dependency and faster unsupervised 3D face reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |