CN104301671B - Traffic Surveillance Video storage method based on event closeness in HDFS - Google Patents
Traffic Surveillance Video storage method based on event closeness in HDFS Download PDFInfo
- Publication number
- CN104301671B CN104301671B CN201410490195.2A CN201410490195A CN104301671B CN 104301671 B CN104301671 B CN 104301671B CN 201410490195 A CN201410490195 A CN 201410490195A CN 104301671 B CN104301671 B CN 104301671B
- Authority
- CN
- China
- Prior art keywords
- event
- data
- video
- file
- closeness
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
Traffic Surveillance Video storage method based on event closeness in HDFS, it is characterised in that specifically include following steps：Intelligent video camera head obtains traffic monitoring data, while carrying out the detection of traffic events, the event type description information included in traffic video flow data and data then is sent into video server simultaneously.Video server receives and video data is encoded and split after the video stream data that camera is sent, and then generates video file, and is that file indicates event type according to the event description information of video data.Video data is uploaded into HDFS；NameNode nodes in HDFS are received after data storage request, and it is the data file selection target back end that will be deposited to call the data Placement Strategy based on event closeness；File is stored in the target data node chosen in step (3).It can solve the problem that HDFS stores the unbalanced problem of node load produced during magnanimity Traffic Surveillance Video, so as to realize the efficient storage of traffic monitoring data.
Description
Technical field
The present invention relates to the traffic prison based on event closeness in the storage of massive video data, more particularly to a kind of HDFS
Control video storage method.
Background technology
With continuing to develop for video compression technology and network transmission technology, intelligent video monitoring system is gradually popularized.City
Intelligent traffic video monitoring in city is due to monitoring device increase and high Qinghua, and the monitor video data volume of generation is very huge, with
Exemplified by 2,000,000 pixel high-definition cameras of the functions such as violation event detection, its maximum video code check is 4Mbps/s, then 8
The monitor area at crossing (each crossing set 4 cameras) will produce 40TB video data in 30 days.The number of rapid growth
Acid test is proposed to the storage performance of monitoring system according to amount.Simultaneously the various of generation are contained in the monitor video of storage
Traffic events, these traffic events later stages can carry out retrieval verification by relevant department.Preserve the monitoring newly produced in real time in guarantee
Need to make quick response to event inquiry demand in time while video, will this again improves the performance for monitoring system
Ask.
Except Large Copacity, monitor video also has high reliability, and (video data is correctly stored within least 30 days and deposited
Store up space in, in case evidence obtaining) and scalability in terms of requirement.Meanwhile, urban traffic video monitoring storage again with it is non-forever
The features such as long property stores (typically only requiring 15~30d of storage), coherence request is relatively low, video access time is long.With
The simulating of video monitoring experience, numeral, network three phases, in video monitoring storage the Main Morphology that stores include DVR,
Tri- kinds of NVR, IPSAN, although disclosure satisfy that system for the demand in terms of storage, but the deployment of system and management service cost
It is higher, and it is unfavorable for the expansion of the applications such as video data analysis.
With GFS (Googlefilesystem), Hadoop distributed data processing architectures appearance, cloud storage technology
Be developing progressively ripe, it has, and Storage Virtualization, enhanced scalability, cost are low, be easily managed with mode flexibly etc. advantage, break through
The performance and capacity bottleneck of conventional store mode, have great importance in field of storage, are especially the great Rong such as video class
Measure, there is provided important solution for the storage of non-structured data.Wherein HDFS is GFS realization of increasing income, and is Hadoop
One of two big cores of framework, are designed to be deployed on cheap hardware, using the teaching of the invention it is possible to provide high transmission rates DDM.
Built using Hadoop and be applied to the distributed memory system that Traffic Surveillance Video is stored, can carried for intelligent traffic monitoring field
For an inexpensive, high performance distributed storage scheme.
But HDFS employs random data Placement Strategy in the placement of data block.This is likely to result in excessive number
According to being relatively concentrated on some nodes so as to produce storage focus, cause system load unbalanced, influence the handling capacity of system.Though
Right Traffic Surveillance Video data scale is larger, but the access request of user is often concentrated on comprising traffic events among practical application
Video segment.If excessive event video data is left concentratedly in some back end because of random placement,
These nodes may turn into storage focus due to loading higher.Therefore HDFS be used for Traffic Surveillance Video storage relative to
Other data-storage applications, more load imbalance situations are may result in using the frame perceptual strategy of acquiescence.
Annotation：Hadoop Distributed File System, abbreviation HDFS, are a distributed file systems.
The content of the invention
It is an object of the invention to provide the Traffic Surveillance Video storage method based on event closeness, its energy in a kind of HDFS
The unbalanced problem of node load produced during HDFS storage magnanimity Traffic Surveillance Videos is enough solved, so as to realize traffic monitoring data
Efficient storage.
To achieve the above object, the present invention is adopted the following technical scheme that：
The Traffic Surveillance Video storage method based on event closeness, specifically includes following steps in HDFS：
(1) intelligent video camera head obtains traffic monitoring data, while the detection of traffic events is carried out, then by traffic video stream
Event type description information included in data and data is sent to video server simultaneously.
(2) video server receives and video data is encoded and split after the video stream data that camera is sent, so
After generate video file, and be that file indicates event type according to the event description information of video data.Video data is uploaded
To HDFS.
(3) the NameNode nodes in HDFS are received after data storage request, call the data based on event closeness
Placement Strategy is the data file selection target back end that will be deposited.
(4) file is stored in the target data node chosen in step (3).
The data Placement Strategy of the step (3) based on event closeness, is comprised the following steps that：
1) when system receives the file operation requests of client, judge operation why type；
If 2) operation is file write operations, the event disturbance degree for each event type safeguarded immediately in acquisition system；
Event disturbance degree represents the video file comprising different type event because the visit capacity of user is different to back end where it
Load effect degree.The present invention defines visit capacity of the e type files in time interval T and the text of e type files in systems
The ratio between part sum is the event disturbance degree of e types of events, and updated once every time T.If aeRepresent event type e influence
Degree, calculation formula is as follows：
3) the event closeness of each back end is calculated；If the collection of the back end in cluster is combined into D, back end d
∈ D, F are the file set in node d, and e (f) is the event type of file f.ae(f)For the event disturbance degree of file f.The present invention
Defined in back end d event closeness be calculated as follows：
4) factor such as event closeness, the real time load of back end, disk size of comprehensive each back end chooses mesh
Mark places node；If it is g that target, which places node, d is as follows for the choosing method of any data node g in cluster：
a)Lg≤Ld, i.e. g is the node of event closeness minimum in all back end.
B) the node d load at current time is set as Nd,NdTable is come with the client operation number of requests for being currently connected to d
Show.If 1) node selected in is multiple, then g is present load NdNode that is minimum and having enough memory capacity.
If c) 2) in equally obtain multiple nodes, d is the section selected from these nodes according to frame perceptual strategy
Point.
5) the event type e of video will be stored by obtaining, and update the stored number of e type files in destination node e and system；
6) if operation is File read operation, the access number of e type files is updated.System is at every fixed time
T, is every kind of each event type update event disturbance degree.
The innovative point of the present invention is embodied in：
Mass transportation monitor video storage is carried out using HDFS, can carrying out classification by event using traffic video, this is special
Levy, the video data content that back end is placed as data place when one of primary reference point, placing new
During video data, the load that the file that pre-estimation back end is stored may be caused to it, in combination with the reality of back end
When the factor such as loading condition and space utilisation node is evaluated, selection optimal node carries out data placements, reduces
System load is unbalance caused by depositing the excessively video file of high attention rate due to node, improves the handling capacity of storage system.
Brief description of the drawings
Fig. 1：The overall structure figure of method；
Fig. 2：Data Placement Strategy workflow diagram based on event closeness.
Embodiment
The present invention basic fundamental thinking be：Comprehensively utilize storage performance advantage and the friendship of Hadoop distributed memory systems
Feature that logical monitor video possesses solves the problems, such as the efficient storage of mass transportation monitor video.Traffic Surveillance Video data
It is larger, but the access request of user often concentrates on the video segment for including traffic events among practical application.HDFS's
Excessive event video data may be concentrated because of random data placement method and be stored in some data by frame perceptual strategy
Node, so as to cause storage focus to occur.Some, of the invention to be carried out using traffic video by event type consider comprehensively more than
Classify this feature, when data are placed by all types of event videos stored in back end it may be caused it is negative
Carry as one of primary evaluation factor of node, the combined factors such as real time load, disk size in combination with node are commented
Valency, selects optimal data to place node, so that the load of equilibrium criterion node.
The Traffic Surveillance Video storage method based on event closeness, specifically includes the following steps in HDFS：
(1) traffic incidents detection during crossing intelligent video camera head carries out video while monitor video is generated, if
Event occurs, then camera generates event description information, and video server is together issued together with video data.
(2) video server receives and video data is encoded and split after video original data, then generates video
File, and be that file indicates event type according to the event description information of data.Video data is uploaded into HDFS.
(3) present invention realizes the data Placement Strategy based on event closeness, the frame given tacit consent to for replacing in HDFS
Perceptual strategy.Different from the method for randomly selecting back end of frame perceptual strategy, the data based on event closeness are placed
Strategy is to calculate back end according to all kinds of event number of videos stored in back end when choosing back end
Event closeness, present load and disk size in combination with back end are evaluated back end.When HDFS is received
Store and ask to file, selecting back end by the data Placement Strategy based on event closeness carries out file storage.
(4) HDFS safeguards the stored number of each all types of files of back end, and target data is updated in storage file
The quantity of the type video file in node.
(5) when user accesses video data, HDFS safeguards user's access number per class event video, when fixing
Between calculate the access temperature for updating all kinds of event videos, be used as load effect degree of all kinds of event video councils for back end.
Technical solution of the present invention is described further below in conjunction with accompanying drawing.
As shown in figure 1, provided herein is the Traffic Surveillance Video storage method based on event closeness in a kind of HDFS, specifically
Comprise the following steps：
(1) intelligent video camera head obtains traffic monitoring data, while the detection of traffic events is carried out, then by traffic video stream
Event type description information included in data and data is sent to video server simultaneously.
(2) video server receives and video data is encoded and split after the video stream data that camera is sent, so
After generate video file, and be that file indicates event type according to the event description information of video data.Video data is uploaded
To HDFS.
(3) the NameNode nodes in HDFS are received after data storage request, call the data based on event closeness
Placement Strategy is the data file selection target back end that will be deposited.
(4) file is stored in the target data node chosen in (3).
The idiographic flow of the data Placement Strategy based on event closeness proposed in the present invention is as shown in Fig. 2 specific step
It is rapid as follows：
1) when system receives the file operation requests of client, judge operation why type；
If 2) operation is file write operations, the event disturbance degree for each event type safeguarded immediately in acquisition system；
Event disturbance degree represents the video file comprising different type event because the visit capacity of user is different to back end where it
Load effect degree.The present invention defines visit capacity of the e type files in time interval T and the text of e type files in systems
The ratio between part sum is the event disturbance degree of e types of events, and updated once every time T.If aeRepresent event type e influence
Degree, calculation formula is as follows：
3) the event closeness of each back end is calculated；If the collection of the back end in cluster is combined into D, back end d
∈ D, F are the file set in node d, and e (f) is the event type of file f.ae(f)For the event disturbance degree of file f.The present invention
Defined in back end d event closeness be calculated as follows：
4) factor such as event closeness, the real time load of back end, disk size of comprehensive each back end chooses mesh
Mark places node；If it is g that target, which places node, d is as follows for the choosing method of any data node g in cluster：
d)Lg≤Ld, i.e. g is the node of event closeness minimum in all back end.
E) the node d load at current time is set as Nd,NdTable is come with the client operation number of requests for being currently connected to d
Show.If 1) node selected in is multiple, then g is present load NdNode that is minimum and having enough memory capacity.
If f) 2) in equally obtain multiple nodes, d is the section selected from these nodes according to frame perceptual strategy
Point.
5) the event type e of video will be stored by obtaining, and update the stored number of e type files in destination node e and system；
6) if operation is File read operation, the access number of e type files is updated.System is at every fixed time
T, is every kind of each event type update event disturbance degree.
Claims (1)
- Traffic Surveillance Video storage method based on event closeness in 1.HDFS, it is characterised in that specifically include following steps：(1) intelligent video camera head obtains traffic monitoring data, while the detection of traffic events is carried out, then by traffic video flow data Video server is sent to the event type description information included in data simultaneously；(2) video server receives and video data is encoded and split after the video stream data that camera is sent, Ran Housheng It is that file indicates event type into video file, and according to the event description information of video data；Video data is uploaded to HDFS；(3) the NameNode nodes in HDFS are received after data storage request, call the data based on event closeness to place Strategy is the data file selection target back end that will be deposited；(4) file is stored in the target data node chosen in step (3)；The data Placement Strategy of the step (3) based on event closeness, is comprised the following steps that：1) when system receives the file operation requests of client, judge operation why type；If 2) operation is file write operations, the event disturbance degree for each event type safeguarded immediately in acquisition system；Event Disturbance degree represents the video file comprising different type event and back end where it is loaded because the visit capacity of user is different Influence degree；It is total that the present invention defines visit capacity file with e type file in systems of the e type files in time interval T The ratio between number is the event disturbance degree of e types of events, and updated once every time T；If aeEvent type e disturbance degree is represented, Calculation formula is as follows：3) the event closeness of each back end is calculated；If the collection of the back end in cluster is combined into D, back end d ∈ D, F For the file set in node d, e (f) is the event type of file f；ae(f)For the event disturbance degree of file f；Institute in the present invention The event closeness for defining back end d is calculated as follows：<mrow> <mi>L</mi> <mrow> <mo>(</mo> <mi>d</mi> <mo>)</mo> </mrow> <mo>=</mo> <munder> <mo>&Sigma;</mo> <mrow> <mi>f</mi> <mo>&Element;</mo> <mi>F</mi> </mrow> </munder> <msub> <mi>a</mi> <mrow> <mi>e</mi> <mrow> <mo>(</mo> <mi>f</mi> <mo>)</mo> </mrow> </mrow> </msub> </mrow>4) factor such as event closeness, the real time load of back end, disk size of comprehensive each back end is chosen target and put Put node；If it is g that target, which places node, d is as follows for the choosing method of any data node g in cluster：A) L (g)≤L (d), i.e. g are the nodes of event closeness minimum in all back end；B) the node d load at current time is set as Nd,NdRepresented with being currently connected to d client operation number of requests；Such as The node selected in fruit a) is multiple, then g is present load NdNode that is minimum and having enough memory capacity；If c) b) in equally obtain multiple nodes, d is the node selected from these nodes according to frame perceptual strategy；5) the event type e of video will be stored by obtaining, and update the stored number of e type files in destination node g and system；6) if operation is File read operation, the access number of e type files is updated；System T at every fixed time, be Every kind of each event type update event disturbance degree.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410490195.2A CN104301671B (en) | 2014-09-23 | 2014-09-23 | Traffic Surveillance Video storage method based on event closeness in HDFS |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410490195.2A CN104301671B (en) | 2014-09-23 | 2014-09-23 | Traffic Surveillance Video storage method based on event closeness in HDFS |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104301671A CN104301671A (en) | 2015-01-21 |
| CN104301671B true CN104301671B (en) | 2017-09-29 |
Family
ID=52321207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410490195.2A Active CN104301671B (en) | 2014-09-23 | 2014-09-23 | Traffic Surveillance Video storage method based on event closeness in HDFS |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104301671B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN106506665B (en) * | 2016-11-18 | 2019-09-24 | 郑州云海信息技术有限公司 | A kind of load-balancing method and platform of distributed video monitoring system |
| CN106686108A (en) * | 2017-01-13 | 2017-05-17 | 中电科新型智慧城市研究院有限公司 | Video monitoring method based on distributed detection technology |
| CN107592506B (en) * | 2017-09-26 | 2020-06-30 | 英华达（上海）科技有限公司 | Monitoring method, monitoring device and monitoring system |
| CN110309223B (en) * | 2018-03-08 | 2023-08-22 | 华为技术有限公司 | Method, system and related equipment for selecting data node |
| CN109089075A (en) * | 2018-07-10 | 2018-12-25 | 浙江工商大学 | Embedded across cloud intelligence memory method and system |
| CN109379563B (en) * | 2018-10-30 | 2020-10-23 | 华南师范大学 | Method and system for monitoring video data storage management |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007067974A (en) * | 2005-08-31 | 2007-03-15 | Toshiba Corp | Video monitoring system and video monitoring method |
| CN202663496U (en) * | 2011-10-28 | 2013-01-09 | 唐玉勇 | Intelligent surveillance system by utilizing video monitoring |
| CN103294701A (en) * | 2012-02-24 | 2013-09-11 | 联想(北京)有限公司 | Distributed file system and data processing method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2473984A1 (en) * | 2009-09-01 | 2012-07-11 | Demaher Industrial Cameras Pty Limited | Video camera system |
-
2014
- 2014-09-23 CN CN201410490195.2A patent/CN104301671B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007067974A (en) * | 2005-08-31 | 2007-03-15 | Toshiba Corp | Video monitoring system and video monitoring method |
| CN202663496U (en) * | 2011-10-28 | 2013-01-09 | 唐玉勇 | Intelligent surveillance system by utilizing video monitoring |
| CN103294701A (en) * | 2012-02-24 | 2013-09-11 | 联想(北京)有限公司 | Distributed file system and data processing method |
Non-Patent Citations (1)
| Title |
|---|
| 一种改进的Hadoop数据负载均衡算法;刘琨，钮文良;《河南理工大学学报（自然科学版）》;20130630;第32卷(第3期);第332-336页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104301671A (en) | 2015-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104301671B (en) | Traffic Surveillance Video storage method based on event closeness in HDFS | |
| Balouek-Thomert et al. | Towards a computing continuum: Enabling edge-to-cloud integration for data-driven workflows | |
| US10560465B2 (en) | Real time anomaly detection for data streams | |
| US11979433B2 (en) | Highly scalable four-dimensional web-rendering geospatial data system for simulated worlds | |
| KR102481906B1 (en) | Apparatus and method for providing a virtual device | |
| CN105791431A (en) | On-line distributed monitoring video processing task scheduling method and device | |
| US9647904B2 (en) | Customer-directed networking limits in distributed systems | |
| US20180052744A1 (en) | Tiered cloud storage for different availability and performance requirements | |
| TW202131661A (en) | Device and method for network optimization and non-transitory computer-readable medium | |
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| US20150149611A1 (en) | Centralized Resource Usage Visualization Service For Large-Scale Network Topologies | |
| CN108076317B (en) | Video data storage method and system, video access method and node | |
| US9436692B1 (en) | Large scale video analytics architecture | |
| CN106506665B (en) | A kind of load-balancing method and platform of distributed video monitoring system | |
| CA2926293A1 (en) | Systems and methods for mapping and routing based on clustering | |
| CN108363553A (en) | A kind of data processing method, apparatus and system | |
| US9792334B2 (en) | Large-scale processing and querying for real-time surveillance | |
| CN103702053A (en) | Video storage and search method and system as well as monitoring system | |
| US20160173620A1 (en) | Time-based data placement in a distributed storage system | |
| CN104182487A (en) | Unified storage method supporting various storage modes | |
| CN107249135A (en) | Video data storage systems and its operating method and index server | |
| Raj et al. | Big data analytics processes and platforms facilitating smart cities | |
| US20170142189A1 (en) | Attachment of cloud services to big data services | |
| CN110505444A (en) | Safety defense monitoring system based on big data | |
| CN114048186A (en) | Data migration method and system based on mass data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |