CN113938649A - Alarm message duplicate removal method and device - Google Patents
Alarm message duplicate removal method and device Download PDFInfo
- Publication number
- CN113938649A CN113938649A CN202111121444.7A CN202111121444A CN113938649A CN 113938649 A CN113938649 A CN 113938649A CN 202111121444 A CN202111121444 A CN 202111121444A CN 113938649 A CN113938649 A CN 113938649A
- Authority
- CN
- China
- Prior art keywords
- value
- data
- alarm
- layer
- alarm data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 25
- 239000011159 matrix material Substances 0.000 claims description 13
- 239000003086 colorant Substances 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000005549 size reduction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 5
- 230000002159 abnormal effect Effects 0.000 abstract description 4
- 238000007689 inspection Methods 0.000 abstract description 4
- 238000013500 data storage Methods 0.000 abstract description 2
- 238000011084 recovery Methods 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000007781 pre-processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000000593 degrading effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/18—Status alarms
- G08B21/182—Level alarms, e.g. alarms responsive to variables exceeding a threshold
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
- H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Power Engineering (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an alarm message duplicate removal method and device, which specifically comprise the following steps: s1, receiving alarm data sent by a third-party service platform at the current moment, and storing the alarm data into a database according to a time sequence; s2, analyzing the alarm data to generate first layer data and second layer data; s3, calculating the metadata value of the alarm data at the current moment; s4, calculating the metadata value of the alarm data in the optional time period in the database, calculating the similarity between the metadata value of the current time and the metadata value of the alarm data in the optional time period, judging the current time as new alarm data if the similarity is smaller than a preset threshold value, and turning to the step S5; and S5, inserting the alarm data at the current moment into the alarm data queue to be processed. The method has important significance in solving the problems of large consumption of data storage space, high data backup and recovery cost, abnormal event processing in the security inspection process and the like.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a device for removing duplicate of an alarm message.
Background
Although video monitoring has multiple advantages, in the process that a user invests a video monitoring system continuously and depends on the video monitoring system continuously deepens, the traditional video monitoring system cannot meet the use requirement of the modern industry, the overhead of mass data is bottomless hole, and if the mass data in cloud storage cannot be stored and managed efficiently, the use cost of storage resources can be increased to an unacceptable level rapidly along with the explosive increase of the storage data. Moreover, as more and more data are stored in the cloud, the efficiency of data access and data retrieval is also severely affected, thereby degrading the performance of the cloud storage. And the video abnormal event detection is used as important content in video monitoring, and the abnormal event is comprehensively detected, so that the accuracy and the correctness of the video monitoring are effectively improved, and the situations of missing report and false report are avoided. Therefore, the repeated storage of the alarm data of the traditional video monitoring system greatly occupies the network bandwidth of the system, and is more unfavorable for the real-time processing of the alarm data.
Disclosure of Invention
The invention aims to provide an alarm message duplicate removal method and device, firstly, extracting first-layer data from alarm data for comprehensively describing the attribute of an alarm picture; secondly, the similarity between the alarm data is measured and calculated by using the similarity, so that the repeated pictures are quickly positioned according to the hash value, the purposes of detecting and removing the repeated pictures are achieved, and the problems that the existing security inspection alarm data is high in repetition rate, occupies system resources and cannot be processed in real time are solved.
A method for removing duplicate alarm messages specifically comprises the following steps:
s1, receiving alarm data sent by a third-party service platform at the current moment, and storing the alarm data into a database according to a time sequence;
s2, analyzing the alarm data to generate first layer data and second layer data, wherein the first layer data is data describing the attribute of the alarm data, and the second layer data is an alarm picture;
s3, calculating a metadata value of the alarm data at the current moment according to the first layer data and the second layer data;
s4, obtaining the metadata value of the alarm data in the database in the optional time period, calculating the similarity between the metadata value of the current time and the metadata value of the alarm data in the optional time period, judging the current time alarm data to be new alarm data if the similarity is smaller than a preset threshold value, and turning to the step S5;
and S5, inserting the alarm data at the current moment into the alarm data queue to be processed.
Further, the step S3 specifically includes the following steps:
s301, calculating a hash value of the first layer data:
s302, calculating a fingerprint value of second-layer data;
and S303, summing the hash value of the first layer of data and the fingerprint value of the second layer of data to obtain the metadata value of the alarm data.
Further, the first layer data includes an alarm data type value, an alarm data content value, and a device id value of the alarm data, and the step S301 specifically includes the following steps:
s3011, summing the alarm data type value, the alarm data content value and the device id value of the alarm data to obtain a summation value of the first layer of data;
and S3012, performing hash operation on the summation value to obtain a hash value of the first layer of data.
Further, the step S302 specifically includes the following steps:
s3021, reducing the size, and reducing the alarm picture to a preset size to obtain an alarm picture with m × n pixels;
s3022, simplifying colors, and converting the alarm pictures of m × n pixels into grayscale pictures of m × n pixels;
s3023, calculating an average value, and calculating an average value of the m × n gray-scale pictures, that is, calculating a gray-scale average value of the m × n pixels of the gray-scale pictures;
s3024, comparing the gray levels of the pixels, traversing m × n pixels of the gray level picture, and comparing the gray level value of each pixel with the average value to generate a binary matrix;
and S3025, generating an m × n-bit integer value from the binary matrix according to a preset rule, and performing hash operation on the integer value to obtain a fingerprint value of the second-layer data.
Further, step S3024 specifically includes the following steps:
comparing the gray value of each pixel in the gray picture with the average gray value respectively; if the gray value of one pixel is larger than the average gray value, setting the value of the corresponding pixel in the binary image to be 1; otherwise, setting the value of the corresponding pixel in the binary image to 0.
Further, the step S4 further includes: if the similarity is larger than or equal to the preset threshold, judging that the alarm data at the current moment are similar alarm data, and deleting the alarm data at the current moment.
Further, the preset rule is from left to right, from top to bottom or big-endian.
Further, the selectable time period is a time period 2 minutes before the current time.
Further, the similarity in step S4 is: and calculating the ratio of the metadata value at the current moment to the metadata value in the database within the optional time period, wherein the preset threshold is 85%.
Further, the similarity in step S4 is: and calculating the Hamming distance between the metadata value at the current moment and the metadata value in the database within the optional time period, wherein the preset threshold is 5.
Further, before the step S3, preprocessing an alarm picture in the alarm data is further included, where the preprocessing includes denoising, illumination compensation, and contrast enhancement.
An alert message deduplication apparatus comprising:
an input module: the system comprises a database, a third-party service platform and a data processing module, wherein the database is used for receiving alarm data sent by the third-party service platform and storing the alarm data into the database according to a time sequence;
an analysis module: the alarm data processing device is used for analyzing the alarm data to generate first layer data and second layer data, wherein the first layer data is data describing the attribute of the alarm data, and the second layer data is an alarm picture;
a metadata module: the metadata value of the alarm data is calculated according to the first layer data and the second layer data;
a similarity determination module: obtaining a metadata value of alarm data in an optional time period in a database, calculating the similarity between the metadata value at the current time and the metadata value of the alarm data in the optional time period, judging the current time alarm data to be new alarm data if the similarity is smaller than a preset threshold value;
a queue module: the alarm data queue is used for inserting the alarm data at the current moment into the alarm data queue to be processed;
further, the metadata module specifically includes the following sub-modules:
the first layer data processing submodule is used for calculating the hash value of the first layer data;
the second layer data processing submodule is used for calculating a fingerprint value of the second layer data;
and the data addition submodule is used for summing the hash value of the first layer of data and the fingerprint value of the second layer of data to obtain a metadata value of the alarm data.
Further, the first layer data processing sub-module specifically includes the following sub-modules:
the sum value submodule is used for summing the alarm data type value, the alarm data content value and the device id value of the alarm data to obtain a sum value of the first layer of data;
and the Hash value submodule is used for carrying out Hash operation on the summation value to obtain a Hash value of the first layer of data.
Further, the second-layer data processing sub-module specifically includes the following sub-modules:
the size reduction submodule is used for reducing the alarm picture to a preset size to obtain the alarm picture with m multiplied by n pixels;
the simplified color submodule is used for converting the alarm picture of m multiplied by n pixels into a gray picture of m multiplied by n pixels;
the mean value calculating submodule is used for carrying out mean value calculation on the gray level picture of the m multiplied by n pixels, namely calculating the gray level mean value of the m multiplied by n pixels of the gray level picture;
the comparison pixel gray level sub-module is used for traversing m multiplied by n pixels of the gray level picture and comparing the gray level value of each pixel with the average value to generate a binary matrix;
and the fingerprint generation submodule is used for generating an m multiplied by n bit integer value from the binary matrix according to a preset rule, and performing hash operation on the integer value to obtain a fingerprint value of the second layer data.
In the prior art, deduplication is performed by using similarity between data, where similarity comparison is performed by performing calculation comparison of a character string or a field and the like through parameters such as an attribute value of metadata, before calculation comparison, a corresponding weight value may be provided along with input metadata, where the weight value is a product of a position value of a character in the character string and an ASC code value of the character, and is a position value of the character, and a composite similarity obtained by comparing metadata added with a weight value factor is compared with a preset threshold to perform a deletion operation on data with a higher similarity.
According to the invention, according to the particularity of the alarm picture data in the rail transit, the related content of the inherent characteristics of the alarm data is researched, and the related attributes including types, contents and equipment IDs are extracted as the first layer of factors for judging the repetition.
And processing the alarm picture and taking the fingerprint value of the alarm picture as a second layer factor for judging repetition. And taking the sum of the first layer factor and the second layer factor as a metadata value, judging the similarity between different alarm pictures, and increasing the accuracy of judging the similarity so as to judge whether the alarm pictures are repeated.
The invention has the following beneficial effects:
1. acquiring first layer data containing attribute information for comprehensively describing alarm data and second layer data containing the alarm picture by analyzing the alarm picture, respectively acquiring hash values of the first layer data and the second layer data, and acquiring metadata values, namely characteristic values, of the alarm data by summing the two hash values, wherein the metadata values are used for distinguishing the alarm data of a plurality of cameras;
2. the similarity judgment is obtained by comparing the ratio of the metadata values of the two alarm data with the preset threshold, and in the face of exponential increase of the alarm data in the current security inspection process, research on the duplicate removal method has important significance in solving the problems of large consumption of data storage space, high data backup and recovery cost, abnormal event processing in the security inspection process and the like.
Drawings
FIG. 1 is a schematic diagram of an alarm message deduplication process of the present invention;
FIG. 2 is a schematic diagram of an alarm message deduplication apparatus according to the present invention;
FIG. 3 is a block diagram of a metadata module according to the present invention;
FIG. 4 is a schematic structural diagram of a first-level data submodule according to the present invention;
FIG. 5 is a schematic structural diagram of a second layer data submodule according to the present invention;
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited to these examples.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "longitudinal", "lateral", "horizontal", "inner", "outer", "front", "rear", "top", "bottom", and the like indicate orientations or positional relationships that are based on the orientations or positional relationships shown in the drawings, or that are conventionally placed when the product of the present invention is used, and are used only for convenience in describing and simplifying the description, but do not indicate or imply that the device or element referred to must have a particular orientation, be constructed in a particular orientation, and be operated, and thus should not be construed as limiting the invention.
In the description of the present invention, it should also be noted that, unless otherwise explicitly specified or limited, the terms "disposed," "open," "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Example 1
S1, receiving alarm data sent by a third-party service platform at the current moment, and storing the alarm data into a database according to a time sequence;
s2, analyzing the alarm data to generate first layer data and second layer data, wherein the first layer data are data describing the attribute of the alarm data and comprise the type of the alarm data, the content of the alarm data and the equipment id of the alarm data, and the second layer data are alarm pictures;
s3, calculating a metadata value of the alarm data at the current moment according to the first layer data and the second layer data;
s301, calculating a hash value of the first layer data:
summing the alarm data type value, the alarm data content value and the device id value of the alarm data to obtain first-layer data and a first-layer value;
performing hash operation on the sum of the first layer of data to obtain a hash value of the first layer of data;
s302, calculating a fingerprint value of second-layer data;
s3001, reducing the size. The picture is downscaled to a size of 12 x 10 for a total of 120 pixels. The effect of this step is to remove the details of the picture, only keep basic information such as structure, light and shade, abandon the picture difference that different sizes, proportion bring.
S3002, simplifying colors. And converting the reduced picture into 120-level gray. That is, all pixels have 64 colors in total.
S3003, calculating an average value. The gray level average of all 120 pixels is calculated.
And S3004, comparing the gray levels of the pixels. The gray scale of each pixel is compared to the average. Greater than or equal to the average value, noted 1; less than the average, noted as 0.
And S3005, calculating the hash value. The comparison results from the previous step are combined to form a 120-bit integer, which is the fingerprint of the picture. The order of the combination is not important as long as it is guaranteed that all pictures are in the same order.
The 120-bit hash value sequence is converted into hexadecimal after every 4 partitions. After the alarm picture fingerprint is obtained, different alarm pictures can be compared through the alarm data fingerprint and the first layer data hash value.
And S303, summing the hash value of the first layer of data and the fingerprint value of the second layer of data to obtain a metadata value of the alarm data.
S4, obtaining the metadata value of the alarm data in the database in the optional time period, calculating the similarity between the metadata value of the current time and the metadata value of the alarm data in the optional time period, judging the current time alarm data to be new alarm data if the similarity is smaller than a preset threshold value, and turning to the step S5;
and S5, inserting the alarm data at the current moment into the alarm data queue to be processed.
The step S302 may also adopt the following method, and the specific method steps are:
s3001, reducing the size. The fastest way to remove high frequencies and details is to reduce the size by keeping the structure bright and dark. The picture is reduced to a size of 8x8 for a total of 64 pixels. The picture difference caused by different sizes and proportions is abandoned.
S3002, simplifying colors. And converting the reduced picture into 64-level gray. That is, all pixels have 64 colors in total.
S3003, calculating DCT (discrete cosine transform).
DCT is the frequency clustering and the ladder shape of the picture decomposition, although JPEG uses 8 × 8 DCT transform, here 32 × 32 DCT transform.
S3004, DCT reduction.
Although the result of DCT is a matrix of 32 x 32 size, we only need to retain the 8x8 matrix in the upper left corner, which part presents the lowest frequencies in the picture.
S3005, calculating an average value.
The average of all 64 values was calculated.
S3006, further reducing DCT.
This is the most important step, and based on the 8 × 8 DCT matrix, a hash value of 64 bits of 0 or 1 is set, and "1" is set for the DCT mean values greater than or equal to "1", and "0" is set for the DCT mean values smaller than "0". The results do not tell us about the low frequency of authenticity, but only roughly the relative proportion of the frequency we have with respect to the mean. As long as the overall structure of the picture remains unchanged, the hash result value is unchanged. The influence of gamma correction or color histogram adjustment can be avoided.
And S3007, calculating the hash value.
Setting 64bit to 64bit long integer, the order of combination is not important as long as it is guaranteed that all pictures are in the same order. The 32 x 32 DCT is converted to a 32 x 32 image.
The comparison results from the previous step are combined to form a 64-bit integer, which is the fingerprint of the picture. The order of the combination is not important as long as it is guaranteed that all pictures are in the same order (e.g., left to right, top to bottom, big-endian).
After the fingerprint is obtained, different pictures can be compared to see how many of the 64 bits are different. In theory, this is equivalent to calculating the "Hammingdistance" (Hammingdistance). If the different data bits do not exceed 5, the two pictures are very similar; if it is greater than 10, it is indicated that these are two different pictures.
Example 2
The steps of this embodiment are to provide an apparatus for removing duplicate alarm messages, including:
an input module: the system comprises a database, a third-party service platform and a data processing module, wherein the database is used for receiving alarm data sent by the third-party service platform and storing the alarm data into the database according to a time sequence;
an analysis module: the alarm data processing device is used for analyzing the alarm data to generate first layer data and second layer data, wherein the first layer data is data describing the attribute of the alarm data, and the second layer data is an alarm picture;
a metadata module: the metadata value of the alarm data is calculated according to the first layer data and the second layer data;
a similarity determination module: obtaining a metadata value of alarm data in an optional time period in a database, calculating the similarity between the metadata value at the current time and the metadata value of the alarm data in the optional time period, judging the current time alarm data to be new alarm data if the similarity is smaller than a preset threshold value;
a queue module: and the alarm data processing device is used for inserting the alarm data at the current moment into the alarm data queue to be processed.
Specifically, the metadata module specifically includes the following sub-modules:
the first layer data processing submodule is used for calculating the hash value of the first layer data;
the second layer data processing submodule is used for calculating a fingerprint value of the second layer data;
and the data addition submodule is used for summing the hash value of the first layer of data and the fingerprint value of the second layer of data to obtain a metadata value of the alarm data.
Specifically, the first layer data processing sub-module further includes the following sub-modules:
the sum value submodule is used for summing the alarm data type value, the alarm data content value and the device id value of the alarm data to obtain a sum value of the first layer of data;
and the Hash value submodule is used for carrying out Hash operation on the summation value to obtain a Hash value of the first layer of data.
Specifically, the second layer data processing sub-module further includes the following sub-modules:
the size reduction submodule is used for reducing the alarm picture to a preset size to obtain the alarm picture with m multiplied by n pixels;
the simplified color submodule is used for converting the alarm picture of m multiplied by n pixels into a gray picture of m multiplied by n pixels;
the mean value calculating submodule is used for carrying out mean value calculation on the gray level picture of the m multiplied by n pixels, namely calculating the gray level mean value of the m multiplied by n pixels of the gray level picture;
the comparison pixel gray level sub-module is used for traversing m multiplied by n pixels of the gray level picture and comparing the gray level value of each pixel with the average value to generate a binary matrix;
and the fingerprint generation submodule is used for generating an m multiplied by n bit integer value from the binary matrix according to a preset rule, and performing hash operation on the integer value to obtain a fingerprint value of the second layer data.
The foregoing is only a preferred embodiment of the present invention, and the present invention is not limited thereto in any way, and any simple modification, equivalent replacement and improvement made to the above embodiment within the spirit and principle of the present invention still fall within the protection scope of the present invention.
Claims (10)
1. A method for removing duplicate alarm messages is characterized by comprising the following steps:
s1, receiving alarm data sent by a third-party service platform at the current moment, and storing the alarm data into a database according to a time sequence;
s2, analyzing the alarm data to generate first layer data and second layer data, wherein the first layer data is data describing the attribute of the alarm data, and the second layer data is an alarm picture;
s3, calculating a metadata value of the alarm data at the current moment according to the first layer data and the second layer data;
s4, obtaining the metadata value of the alarm data in the database in the optional time period, calculating the similarity between the metadata value of the current time and the metadata value of the alarm data in the optional time period, judging the current time alarm data to be new alarm data if the similarity is smaller than a preset threshold value, and turning to the step S5;
and S5, inserting the alarm data at the current moment into the alarm data queue to be processed.
2. The method for eliminating duplicate alarm messages according to claim 1, wherein the step S3 specifically includes the following steps:
s301, calculating a hash value of the first layer of data;
s302, calculating a fingerprint value of second-layer data;
and S303, summing the hash value of the first layer of data and the fingerprint value of the second layer of data to obtain the metadata value of the alarm data.
3. The method according to claim 2, wherein the first layer data includes an alarm data type value, an alarm data content value, and a device id value of the alarm data, and the step S301 specifically includes the following steps:
s3011, summing the alarm data type value, the alarm data content value and the device id value of the alarm data to obtain a summation value of the first layer of data;
and S3012, performing hash operation on the summation value to obtain a hash value of the first layer of data.
4. The method for removing duplicate alarm messages according to claim 2, wherein the step S302 specifically includes the following steps:
s3021, reducing the size, and reducing the alarm picture to a preset size to obtain an alarm picture with m × n pixels;
s3022, simplifying colors, and converting the alarm pictures of m × n pixels into grayscale pictures of m × n pixels;
s3023, calculating an average value, and calculating an average value of the m × n gray-scale pictures, that is, calculating a gray-scale average value of the m × n pixels of the gray-scale pictures;
s3024, comparing the gray levels of the pixels, traversing m × n pixels of the gray level picture, and comparing the gray level value of each pixel with the average value to generate a binary matrix;
and S3025, generating an m × n-bit integer value from the binary matrix according to a preset rule, and performing hash operation on the integer value to obtain a fingerprint value of the second-layer data.
5. The method for alarm message repetition removal according to claim 4, wherein the step S3024 specifically includes the steps of:
comparing the gray value of each pixel in the gray picture with the average gray value respectively; if the gray value of one pixel is larger than the average gray value, setting the value of the corresponding pixel in the binary image to be 1; otherwise, setting the value of the corresponding pixel in the binary image to 0.
6. The method for removing duplicate alarm messages according to claim 1, wherein the step S4 further includes: if the similarity is larger than or equal to the preset threshold, judging that the alarm data at the current moment are similar alarm data, and deleting the alarm data at the current moment.
7. An alarm message deduplication apparatus, comprising:
an input module: the system comprises a database, a third-party service platform and a data processing module, wherein the database is used for receiving alarm data sent by the third-party service platform and storing the alarm data into the database according to a time sequence;
an analysis module: the alarm data processing device is used for analyzing the alarm data to generate first layer data and second layer data, wherein the first layer data is data describing the attribute of the alarm data, and the second layer data is an alarm picture;
a metadata module: the metadata value of the alarm data is calculated according to the first layer data and the second layer data;
a similarity determination module: obtaining a metadata value of alarm data in an optional time period in a database, calculating the similarity between the metadata value at the current time and the metadata value of the alarm data in the optional time period, judging the current time alarm data to be new alarm data if the similarity is smaller than a preset threshold value;
a queue module: and the alarm data processing device is used for inserting the alarm data at the current moment into the alarm data queue to be processed.
8. The apparatus according to claim 7, wherein the metadata module specifically includes the following sub-modules:
the first layer data processing submodule is used for calculating the hash value of the first layer data;
the second layer data processing submodule is used for calculating a fingerprint value of the second layer data;
and the data addition submodule is used for summing the hash value of the first layer of data and the fingerprint value of the second layer of data to obtain a metadata value of the alarm data.
9. The apparatus according to claim 8, wherein the first layer data processing sub-module further comprises the following sub-modules:
the sum value submodule is used for summing the alarm data type value, the alarm data content value and the device id value of the alarm data to obtain a sum value of the first layer of data;
and the Hash value submodule is used for carrying out Hash operation on the summation value to obtain a Hash value of the first layer of data.
10. The apparatus according to claim 8, wherein the second layer data processing sub-module further comprises the following sub-modules:
the size reduction submodule is used for reducing the alarm picture to a preset size to obtain the alarm picture with m multiplied by n pixels;
the simplified color submodule is used for converting the alarm picture of m multiplied by n pixels into a gray picture of m multiplied by n pixels;
the mean value calculating submodule is used for carrying out mean value calculation on the gray level picture of the m multiplied by n pixels, namely calculating the gray level mean value of the m multiplied by n pixels of the gray level picture;
the comparison pixel gray level sub-module is used for traversing m multiplied by n pixels of the gray level picture and comparing the gray level value of each pixel with the average value to generate a binary matrix;
and the fingerprint generation submodule is used for generating an m multiplied by n bit integer value from the binary matrix according to a preset rule, and performing hash operation on the integer value to obtain a fingerprint value of the second layer data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111121444.7A CN113938649A (en) | 2021-09-24 | 2021-09-24 | Alarm message duplicate removal method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111121444.7A CN113938649A (en) | 2021-09-24 | 2021-09-24 | Alarm message duplicate removal method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113938649A true CN113938649A (en) | 2022-01-14 |
Family
ID=79276676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111121444.7A Pending CN113938649A (en) | 2021-09-24 | 2021-09-24 | Alarm message duplicate removal method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113938649A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115378791A (en) * | 2022-08-22 | 2022-11-22 | 平安银行股份有限公司 | Data management method, device, storage medium and electronic equipment |
| CN116701610A (en) * | 2023-08-03 | 2023-09-05 | 成都大成均图科技有限公司 | Effective alarm condition identification method and device based on emergency multisource alarm |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678702A (en) * | 2013-12-30 | 2014-03-26 | 优视科技有限公司 | Video duplicate removal method and device |
| CN108229574A (en) * | 2018-01-18 | 2018-06-29 | 维沃移动通信有限公司 | A kind of method, apparatus and mobile terminal of picture screening |
| CN110275975A (en) * | 2019-06-26 | 2019-09-24 | 北京深醒科技有限公司 | A kind of method for quickly retrieving of similar pictures |
| CN111428122A (en) * | 2020-03-20 | 2020-07-17 | 南京中孚信息技术有限公司 | Picture retrieval method and device and electronic equipment |
| CN111629165A (en) * | 2020-07-28 | 2020-09-04 | 天津中新智冠信息技术有限公司 | Alarm video processing method, device, equipment and storage medium |
| WO2021175040A1 (en) * | 2020-03-02 | 2021-09-10 | Oppo广东移动通信有限公司 | Video processing method and related device |
-
2021
- 2021-09-24 CN CN202111121444.7A patent/CN113938649A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678702A (en) * | 2013-12-30 | 2014-03-26 | 优视科技有限公司 | Video duplicate removal method and device |
| CN108229574A (en) * | 2018-01-18 | 2018-06-29 | 维沃移动通信有限公司 | A kind of method, apparatus and mobile terminal of picture screening |
| CN110275975A (en) * | 2019-06-26 | 2019-09-24 | 北京深醒科技有限公司 | A kind of method for quickly retrieving of similar pictures |
| WO2021175040A1 (en) * | 2020-03-02 | 2021-09-10 | Oppo广东移动通信有限公司 | Video processing method and related device |
| CN111428122A (en) * | 2020-03-20 | 2020-07-17 | 南京中孚信息技术有限公司 | Picture retrieval method and device and electronic equipment |
| CN111629165A (en) * | 2020-07-28 | 2020-09-04 | 天津中新智冠信息技术有限公司 | Alarm video processing method, device, equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115378791A (en) * | 2022-08-22 | 2022-11-22 | 平安银行股份有限公司 | Data management method, device, storage medium and electronic equipment |
| CN116701610A (en) * | 2023-08-03 | 2023-09-05 | 成都大成均图科技有限公司 | Effective alarm condition identification method and device based on emergency multisource alarm |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113938649A (en) | Alarm message duplicate removal method and device | |
| WO2022105019A1 (en) | Snapshot quality evaluation method and apparatus for vehicle bayonet device, and readable medium | |
| Andriotis et al. | JPEG steganography detection with Benford's Law | |
| US8107670B2 (en) | Scanning images for pornography | |
| CN113344826B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN110096945B (en) | Indoor monitoring video key frame real-time extraction method based on machine learning | |
| KR20090006397A (en) | System and method for multi-stage filtering of malicious videos in video distribution environment | |
| Kumar et al. | Near lossless image compression using parallel fractal texture identification | |
| US7885474B2 (en) | Solution for detecting a streak in a compressed gray scale image | |
| CN111144455B (en) | Data processing method, device, equipment and readable storage medium | |
| CN110532772B (en) | File detection method, model, device and computer readable storage medium | |
| US8238601B2 (en) | System and method for removing digital watermarks from a watermarked image | |
| CN112182329B (en) | Network picture infringement monitoring and automatic evidence obtaining method | |
| CN117854402A (en) | Abnormal display detection method and device of display screen and terminal equipment | |
| CN112468509A (en) | Deep learning technology-based automatic flow data detection method and device | |
| CN112613396B (en) | Task emergency degree processing method and system | |
| CN113743235B (en) | Electric power inspection image processing method, device and equipment based on edge calculation | |
| EP2725553A1 (en) | DCT based black frame detection using varying blocksize | |
| CN115424179A (en) | Real-time video monitoring method and device based on edge calculation and storage medium | |
| CN113542771A (en) | Video high-efficiency compression processing method based on content weight | |
| CN113971229A (en) | Frame comparison method analysis method and device | |
| George et al. | Speeding Up Fractal Image Compression Using DCT | |
| CN110705518A (en) | Video detection method and system | |
| CN118057474A (en) | Copyright early warning method, related device and computer medium | |
| Parjuangan et al. | Forensics Investigation on Image Files using Quantization Value |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |