CN109359213A - A kind of security protection video intelligent retrieval system and its search method - Google Patents
A kind of security protection video intelligent retrieval system and its search method Download PDFInfo
- Publication number
- CN109359213A CN109359213A CN201810983328.8A CN201810983328A CN109359213A CN 109359213 A CN109359213 A CN 109359213A CN 201810983328 A CN201810983328 A CN 201810983328A CN 109359213 A CN109359213 A CN 109359213A
- Authority
- CN
- China
- Prior art keywords
- video
- node
- feature
- security protection
- retrieval
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
Abstract
A kind of security protection video intelligent retrieval system, including the MQ node for memory scan video request message queue, for inquiring storage location search range, the segmentation video features file, the Master node for distributing search mission of video features file, and the Worker node being compared for video decoding, feature extraction, characteristic storage with extraction feature.Video frequency searching target is carried out tag file by the present invention, convenient for being distributed in destination host progress distributed search not, is improved working efficiency, is avoided the inefficiencies of traversal retrieval.Working host of the invention had both been responsible for the video decoding and feature extraction of conventional resolution server, it is responsible for the conventional characteristic storage and aspect ratio pair for comparing server again, save the limited server resource of prototype system, it is possible thereby to first do video parsing, aspect ratio pair is done again, is staggered, is not interfere with each other completely with server in time.
Description
Technical field
The invention belongs to video intelligent retrieval technique fields, and in particular to a kind of security protection video intelligent retrieval system and its inspection
Suo Fangfa.
Background technique
Safety monitoring video is one of maximum data class of the scale of construction.City video monitoring is usually constructed with thousands of monitoring
Point, HD video are becoming increasingly popular, and high-definition monitoring point 1 month of the 1 8Mbps code rate video generated just has a 2.5TB, and 1,000
Video caused by a monitoring point just has 2.5PB.Different monitoring scenes, there is also differences for the tag file size therefrom extracted
It is different, according to experience, be typically sized to original video 1/2 to 1/3 of tag file also means that the original view of 2.5PB
Frequently extracted tag file about 1PB will handle such huge quantity, and carrying out parallel computation using server cluster is
Inevitable choice.
Traditional security protection video intelligent retrieval system is frequently with following group system framework, and wherein resolution server cluster is negative
Video decoding and feature extraction are blamed, server cluster is compared and is responsible for characteristic storage and aspect ratio pair.The simple principle of picture search
Exactly pass through the characteristic of traversal specified range and be compared one by one, to be matched to the target figure within similarity threshold
Piece.
In terms of actual test result, there are serious performance bottleneck, reasons to be in architecture design for earlier picture search system
Processing server and certain amount of video channel will be bundled that (this is the common practice of security industry, referring to DVR/
NVR video cassette recorder, coding and decoding video server etc.), specifically when operating, tens of comparison servers are executing specific retrieval times
Very likely only have several comparison servers to participate in calculating when business, even only several pieces of disks participate in data throughput, utilization rate
It is extremely low.Moreover, parsing is mutually separated with comparing, when parsing, it is idle to compare server, and when comparison, resolution server leaves unused, and causes
A large amount of wastes.
Summary of the invention
In view of the defects existing in the prior art, the present invention provides a kind of security protection video intelligent retrieval system, as far as possible will view
Frequency tag file is uniformly distributed on all comparison servers, and it is multiple to carry out timesharing to resolution server and comparison server
Effective integration can effectively solve the above problems.
The technical solution adopted by the invention is as follows:
The present invention provides a kind of security protection video intelligent retrieval system, including
For the MQ node of memory scan video request message queue,
For inquiring video features file storage location search range, segmentation video features file, distributing search mission
Master node,
With decoded for video, feature extraction, characteristic storage with extract the Worker node that is compared of feature.
Preferably, mode is hashed on the Worker node be uniformly distributed with safety monitoring video features；It is described
The quantity of Worker node is 2 or more；The safety monitoring video features include the tag file for storing video.
Preferably, the safety monitoring video features include video feature vector data and key frame of video image data.
A kind of security protection video intelligent search method, includes the following steps:
Retrieval video request is obtained, and is stored in message queue, as MQ node；
The retrieval video request in MQ node is obtained, as Master node；Master node by the retrieval video into
Row segmentation, and the video after segmentation is allocated task；
The distribution task is obtained, as Worker node；It is uniform that tag file for storing video hashes mode
It is distributed on the Worker node；The distribution task is extracted feature by the Worker node, and by the extraction
Feature is compared with the storage feature of server；If comparing result is consistent, MQ node is fed back to, completes retrieval.
Preferably, the retrieval video is split the tag file for fixed size, each feature by Master node
File distributes globally unique filename.
It is further preferred that the distribution task is carried out feature extraction by Worker node, and carry out characteristic storage.
It is further preferred that including search time range, camera numbers information in the tag file block.
A kind of security protection video intelligent retrieval system provided by the invention has the advantage that
(1) video frequency searching target is carried out tag file by the present invention, and the destination host convenient for being distributed in different is divided
Cloth retrieval, improves working efficiency, avoids the inefficiencies of traversal retrieval.
(2) working host of the invention had not only been responsible for the video decoding and feature extraction of resolution server, but also was responsible for comparing clothes
The characteristic storage and aspect ratio pair of business device, save the limited server resource of prototype system, it is possible thereby to video parsing is first done, then
Aspect ratio pair is done, is staggered, do not interfere with each other completely with server in time.
(3) present invention hashes it fifty-fifty on different operating host as far as possible, is mentioned using tag file localization storage
The high efficiency of network transmission.
Detailed description of the invention
Fig. 1 is intelligent retrieval system architecture diagram of the present invention；
Fig. 2 is that feature of present invention stores schematic diagram；
Fig. 3 is that feature of present invention searches for schematic diagram.
Specific embodiment
Below in conjunction with attached drawing, the present invention is described in detail:
The present invention as shown in Figure 1 provides a kind of security protection video intelligent retrieval system, by Master node, MQ node,
Worker node and Client client etc. are constituted.Master node can be single node, be also possible to the more of mutually standby coordination
A node；There are two MQ nodes or more than two, to construct distributed MQ；The support of Worker node is extending transversely, from minority
It is several to be likely to hundreds and thousands of；Multiple client can concurrently access this system.
The a set of communication protocol of our communication schemes between Master, Worker, Client, defines user and steps on
Record, user log off, feature extraction, signature search, the addition of Worker node, Worker knot removal, Worker node exit, are special
Levy storage location acquisition etc. and business and the relevant communication information of management.
Worker node had not only been responsible for the video decoding and feature extraction of resolution server, but also was responsible for comparing the feature of server
Storage and aspect ratio pair, in order to the limited server resource of prototype system is saved, we can first do video parsing, then
Aspect ratio pair is done, is staggered, do not interfere with each other completely in time.
Video features are divided into the tag file block of fixed size, when tag file creates, Master node meeting
Globally unique filename is distributed to each file.Master node is also responsible for the management of tag file metadata, feature extraction
With the management of the distributed scheduling Worker node of signature search task.
Message queue can simplify communication complicated between numerous nodes such as Master, Worker, Client, and distributed
Message queue can further provide for stablizing, reliable communication environment.
As shown in Fig. 2, target signature is more equally distributed in each Worker when in order to search for, characteristic storage host also by
Master is distributed unitedly.The more typical search condition of video search is camera numbers and period, for this business demand,
Tag file carrys out tissue according to camera numbers and period.Firstly, it is necessary to which guarantee to store in same tag file is same
The data of camera continuous period, secondly, same camera tag file needs in different time periods fifty-fifty hash as far as possible
On different Worker, Master carrys out the storage location of assigned characteristics file according to this Distribution Principles.Worker is starting newly
When the feature extraction of tag file, first to Master inquire this feature file should store where.Tag file is using fixation
Design of length is transmitted to destination host when Worker, which extracts feature, reaches file size, and transmission channel should after closing
The file information informs Master.
As shown in figure 3, the feature pool for guaranteeing that Worker is compared is stored in local when assigned characteristics search mission, it can be effective
Reduce network transmission, improves search efficiency.When Master receives the signature search request of Client, according to signature search item
Part from the tag file in tag file table within the scope of query search, and will be searched such as search time range, camera numbers
Rope Task-decomposing distributes to the Worker where tag file.Worker reads the local tag file for needing to compare, by target
Aspect ratio pair is carried out with it, the record that similarity reaches requirement is returned into Master.
We deploy system prototype on small-scale cluster: realizing distribution using open source big data technology HDFS
Master and Worker, distributed MQ message queue is realized using open source message-oriented middleware Kafka, using image data
Feature contrast conting is replaced, using SHA calculating to simulate ratio with the scale that simulation feature stores instead of video frequency feature data
To the time-consuming of algorithm.This experimental situation is able to satisfy initial stage verifying demand substantially.
Simulated experiment is tested using 8 PC servers, is communicated between server using gigabit networking interconnection.
Hadoop uses 5.3 version of Cloudera cdh, and HDFS data backup number is set as 1, block size and is set as 128M.Respectively
Software distribution situation is as shown in the table:
It writes program simulation manufacture picture and characteristic, storage condition is as follows:
In the case where upper table tests environment, picture search experiment is carried out.Worker carries out feature reading using single thread in experiment
With compare, acquired results are as shown in the table.
Judging from the experimental results, when the scan data volume of video search changes according to different space-time search conditions
When, the response time is also linear increase, and it is unrelated with the search space-time unique of input, it really gets rid of and is regarded because server bundlees
Frequency channel bring accesses hot issue.When search range is small, search time will not be consumed since total amount of data is very big
Take the excessive time, this is mainly due to the tag files that Master can navigate to search range quickly.When search range expands directly
When to full scan, handling capacity also complies with our expection.If necessary to support bigger handling capacity, it is only necessary to expand cluster increase
Worker.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
Depending on protection scope of the present invention.
Claims (7)
1. a kind of security protection video intelligent retrieval system, which is characterized in that including
For the MQ node of memory scan video request message queue,
For inquiring video features file storage location search range, segmentation video features file, distributing search mission
Master node,
With decoded for video, feature extraction, characteristic storage with extract the Worker node that is compared of feature.
2. security protection video intelligent retrieval system according to claim 1, which is characterized in that hashed on the Worker node
Mode is uniformly distributed with safety monitoring video features；The quantity of the Worker node is 2 or more；The safety monitoring view
Frequency feature includes the tag file for storing video.
3. security protection video intelligent retrieval system according to claim 2, which is characterized in that the safety monitoring video features
Including video feature vector data and key frame of video image data.
4. a kind of security protection video intelligent search method, which comprises the steps of:
Retrieval video request is obtained, and is stored in message queue, as MQ node；
The retrieval video request in MQ node is obtained, as Master node；Master node is divided the retrieval video
It cuts, and the video after segmentation is allocated task；
The distribution task is obtained, as Worker node；Tag file for storing video hashes mode and is uniformly distributed
On the Worker node；The distribution task is extracted feature by the Worker node, and by the extraction feature
It is compared with the storage feature of server；If comparing result is consistent, MQ node is fed back to, completes retrieval.
5. security protection video intelligent search method according to claim 4, which is characterized in that Master node is by the retrieval
The tag file that video is split as fixed size, each tag file distribute globally unique filename.
6. security protection video intelligent search method according to claim 5, which is characterized in that Worker node is by the distribution
Task carries out feature extraction, and carries out characteristic storage.
7. security protection video intelligent search method according to claim 5, which is characterized in that include in the tag file block
Search time range, camera numbers information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810983328.8A CN109359213A (en) | 2018-08-27 | 2018-08-27 | A kind of security protection video intelligent retrieval system and its search method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810983328.8A CN109359213A (en) | 2018-08-27 | 2018-08-27 | A kind of security protection video intelligent retrieval system and its search method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109359213A true CN109359213A (en) | 2019-02-19 |
Family
ID=65349933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810983328.8A Pending CN109359213A (en) | 2018-08-27 | 2018-08-27 | A kind of security protection video intelligent retrieval system and its search method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109359213A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978081A (en) * | 2019-05-05 | 2019-07-05 | 星环信息科技（上海）有限公司 | Determination method, apparatus, equipment and the medium of eigentransformation mode |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101137027A (en) * | 2006-11-23 | 2008-03-05 | 中兴通讯股份有限公司 | Media slicing memory method |
| CN101188569A (en) * | 2006-11-16 | 2008-05-28 | 饶大平 | Method for constructing data quanta space in network and distributed file storage system |
| CN101840422A (en) * | 2010-04-09 | 2010-09-22 | 江苏东大金智建筑智能化系统工程有限公司 | Intelligent video retrieval system and method based on target characteristic and alarm behavior |
| CN102495904A (en) * | 2011-12-22 | 2012-06-13 | 刘翔 | Distributed intelligent face video retrieval system |
| CN106534344A (en) * | 2016-12-07 | 2017-03-22 | 高新兴科技集团股份有限公司 | Cloud platform video processing system and application method thereof |
-
2018
- 2018-08-27 CN CN201810983328.8A patent/CN109359213A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101188569A (en) * | 2006-11-16 | 2008-05-28 | 饶大平 | Method for constructing data quanta space in network and distributed file storage system |
| CN101137027A (en) * | 2006-11-23 | 2008-03-05 | 中兴通讯股份有限公司 | Media slicing memory method |
| CN101840422A (en) * | 2010-04-09 | 2010-09-22 | 江苏东大金智建筑智能化系统工程有限公司 | Intelligent video retrieval system and method based on target characteristic and alarm behavior |
| CN102495904A (en) * | 2011-12-22 | 2012-06-13 | 刘翔 | Distributed intelligent face video retrieval system |
| CN106534344A (en) * | 2016-12-07 | 2017-03-22 | 高新兴科技集团股份有限公司 | Cloud platform video processing system and application method thereof |
Non-Patent Citations (1)
| Title |
|---|
| 陈燕，李桃迎: "《信息系统集成技术与方法教程》", 31 October 2015, 大连海事大学出版社 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978081A (en) * | 2019-05-05 | 2019-07-05 | 星环信息科技（上海）有限公司 | Determination method, apparatus, equipment and the medium of eigentransformation mode |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11620308B2 (en) | Adaptive distribution method for hash operations | |
| Lohrmann et al. | Nephele streaming: stream processing under QoS constraints at scale | |
| CN106407463A (en) | Hadoop-based image processing method and system | |
| Sun et al. | {SmartCuckoo}: A Fast and {Cost-Efficient} Hashing Index Scheme for Cloud Storage Systems | |
| CN105205169A (en) | Distributed image index and retrieval method | |
| KR101622810B1 (en) | System and method for distributing big data | |
| CN108228709B (en) | Data storage method and system, electronic device, program, and medium | |
| Im et al. | Pinot: Realtime olap for 530 million users | |
| CN107092686B (en) | File management method and device based on cloud storage platform | |
| WO2017097124A1 (en) | Method, apparatus and system for transmitting tasks based on sub-libraries and sub-tables | |
| You et al. | Spatial join query processing in cloud: Analyzing design choices and performance comparisons | |
| Plimpton et al. | Streaming data analytics via message passing with application to graph algorithms | |
| US10048991B2 (en) | System and method for parallel processing data blocks containing sequential label ranges of series data | |
| CN109359213A (en) | A kind of security protection video intelligent retrieval system and its search method | |
| Yuan et al. | Efficient processing of streaming graphs for evolution-aware clustering | |
| Nguyen et al. | An efficient similar image search framework for large-scale data on cloud | |
| Fu et al. | Efficient shuffle management with scache for DAG computing frameworks | |
| Yahyaoui et al. | CloudFC: Files clustering for storage space optimization in clouds | |
| Dai et al. | The Data Platform for Large-Scale Video Surveillance Systems | |
| Cattaneo et al. | Achieving efficient source camera identification on Hadoop | |
| Johnson et al. | Big data processing using Hadoop MapReduce programming model | |
| Li et al. | An agricultural data storage mechanism based on HBase | |
| Fernandez Casani et al. | Designing Alternative Transport Methods for the Distributed Data Collection of ATLAS EventIndex Project | |
| Wang et al. | A Massive Data Retrieval Method for Power Information Collection Systems | |
| CN115658677A (en) | Distributed face similarity searching system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190219 |
|
| RJ01 | Rejection of invention patent application after publication |