CN109218366A - Monitor video temperature cloud storage method based on k mean value - Google Patents
Monitor video temperature cloud storage method based on k mean value Download PDFInfo
- Publication number
- CN109218366A CN109218366A CN201710540167.0A CN201710540167A CN109218366A CN 109218366 A CN109218366 A CN 109218366A CN 201710540167 A CN201710540167 A CN 201710540167A CN 109218366 A CN109218366 A CN 109218366A
- Authority
- CN
- China
- Prior art keywords
- video
- temperature
- cluster
- cloud storage
- mean value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of monitor video temperature cloud storage method based on k mean value, the analysis of video temperature is carried out to the video resource on network according to classical k-means clustering algorithm, predicted city camera temperature, and the higher video of temperature is stored in the high cluster of performance node, the lower video of temperature is stored in the low cluster of performance node, to accelerate video frequency searching speed and later period intelligent processing.
Description
Technical field
The present invention relates to technical field of video monitoring, and in particular to a kind of storage method of monitor video.
Background technique
With popularizing for digital security technology, for monitoring technology gradually toward high Qinghua, web development, consequent is sea
The data storage problem of amount.Mass data, which must possess, to be able to carry out reliable, certifiable efficiency and possesses fast reading and writing and sound
Should be able to power storage.Traditional video monitoring storage is broadly divided into three phases, respectively the HD recording storage equipment of early stage,
Direct Attached Storage (DAS) and network-based network attached storage (NAS).In order to reduce video monitoring system storage equipment
The problems such as cost and solution massive store, Hadoop distribution cloud storage cluster is constructed, framework is as shown in Figure 1.Mesh
Preceding cloud storage technology development is swift and violent, and more well-known has Amazon cloud storage, Google's cloud storage, IBM, Baidu's cloud, Huawei's Dropbox
Deng.However unitized Hadoop platform is to there is the following disadvantage:
Platform intelligent process performance is played 1. can not maximize.Video on demand playback temperature is different, and user is general only
High temperature video is selected to carry out intelligent processing, if video frequency abstract, panorama merge, and video intelligentization processing is to computer node
Performance requirement is higher, and high temperature video, which is stored in low performance node, can reduce intelligent analysis speed.
2. generic video cloud storage framework wastes memory space.The monitor video overwhelming majority is all redundant data, and is regarded
Frequency storage timeliness is one month.Hadoop distributed storage cluster backup number is fixed, for the monitor video indifference of different temperatures
Different storage, wastes memory space, causes utilization rate of equipment and installations low.
3. general video cloud storage architecture is to bandwidth requirement height.Feature is monitored for safe city, camera is tens of thousands of
Road, even hundreds of thousands road, are distributed in each corner in city, and all video flowings converge to a cluster, cause to network bandwidth
Enormous pressure, or even will appear network congestion, so that losing video data packet.
Summary of the invention
It is an object of the invention to by proposing one kind towards intelligence to the video storage architecture analysis based on Hadoop
Double Hadoop cluster-based storage frameworks.Algorithm is based on k mean prediction monitor video temperature, and monitors feature pair according to safe city
Video backup number is adjusted, so that utilization rate of equipment and installations can be improved, accelerates video file retrieval and intelligent analysis speed.
Technical scheme is as follows:
A kind of monitor video temperature cloud storage method based on k mean value, it is characterised in that: poly- according to classical k-means
Class algorithm carries out the analysis of video temperature to the video resource on network, predicted city camera temperature, and by the higher view of temperature
Frequency is stored in the high cluster of performance node, and the lower video of temperature is stored in the low cluster of performance node.
The video temperature Analysis Service of high temperature monitor video is analyzed once daily, and video cloud is entered according to user
The intensity distribution of storage system issues the video temperature index database time again.
The invention proposes one kind towards intelligentized double Hadoop cluster-based storage frameworks, and temperature will be used in video monitoring
High video is stored in High-Performance Computing Cluster, to accelerate video frequency searching speed and later period intelligent processing；Double Hadoop collection simultaneously
Backup number can be respectively set for the monitor video of different temperatures in group, to save memory space, improve utilization rate of equipment and installations.
Detailed description of the invention
Fig. 1 is the existing video storage rack composition based on Hadoop；
Fig. 2 is Hadoop storage rack composition of the invention；
Fig. 3 is k-means algorithm flow chart of the invention.
Specific embodiment
Fig. 1 is the existing video storage rack composition based on Hadoop.The present invention is the video based on existing Hadoop
Thinking is stored, the api interface provided by Hadoop is realized and uploads to the video stream file received in HDFS from local.
During this, front-end camera or encoder continuously forward video flowing, then acquire and converge in server-side
(being equivalent to streaming media service or the video acquisition service of NVR), locally carries out caching packaged data, then in real time with the shape of stream
Formula docks " buffer area " with HDFS, is later uploaded file by way of stream.
For the storage strategy of video information, at present there are many, the present invention minimizes with system overall response time, improves and sets
Standby utilization rate proposes a kind of cloud storage child node service memory strategy using monitor video temperature as foundation for target.The storage
Strategy mainly considers following three Factors Factor: video uses temperature, the process performance of each node server, network bandwidth；
In video on demand playback behavior, it may appear that the phenomenon that " video resource of 80% user application 20% ", or even occur it is big
Occurs the phenomenon that " video resource of 90% user's application 10% " in the case where piece or backing.Therefore the high view of storage utilization rate
Frequency server will receive more user request certainly so that the load of these video servers comparatively can than heavier,
Load imbalance is caused, therefore, preferentially the video of high utilization rate is stored on multiple high performance server nodes, and will be low
The video of utilization rate is stored on the lower server node of performance.Meanwhile video cloud storage is the basis of cloud computing, high-performance
The cluster of node greatly accelerates its analysis processing speed for later period video intelligentization processing such as video frequency abstract, to scheme to search figure.
The present invention improves existing framework according to the above analysis, as shown in Figure 2.
Framework shown in Fig. 2 includes video temperature Analysis Service, and query video service, video temperature index database, video connects
Enter service.Safe city camera temperature is predicted by video temperature Analysis Service, so that the higher video of temperature is stored in
The lower video of temperature is stored in the low cluster of performance node by the high cluster of performance node.For high temperature monitor video
Prediction video temperature Analysis Service analyze daily once.Again the video temperature index database time is issued, view is entered according to user
The intensity distribution of frequency cloud storage system determines that daily morning is the Best Times of redistribution, and 2:00 AM and 6 points of the morning are also
The relatively inning updated.The core of improved video storage architecture technology is predicted monitor video, judges importance i.e.
Video temperature.The present invention predicts certain video all the way using k-means algorithm from machine learning, according to existing characteristic value
Temperature.
The present invention is based on the clusterings in machine learning, and monitor video is divided into high temperature and low-heat degree video.Cluster
Analysis is also known as cluster analysis, it is a kind of statistical analysis technique for studying (sample or index) classification problem.Cluster is by data point
Class is to different such a processes of class or cluster, so the object in the same cluster has a very big similitude, and different clusters
Between object have very big diversity.Clustering can classify automatically from sample data.From the angle of practical application
Degree sees that clustering is one of main task of data mining.The present invention uses most classic k-means clustering algorithm,
Referred to as k- mean algorithm.It is calculated using the mean value of all data samples in each cluster subset as the representative of cluster point
Method main thought is by iterative process so that data set is divided into different classifications, so that the criterion letter of evaluation clustering performance
Number is optimal, to make compact in independent, class between each cluster class generated.
The input of cluster is one group of data for not having classification to mark, in advance it is known that these data be polymerized to several clusters can also be with
Do not know and is polymerized to several clusters.By analyzing these data, according to certain clustering criteria, classifying rationally set of records ends, to make phase
As record and be divided into the same cluster, dissimilar data are divided into different clusters.Assuming that given data set X=
{ xm | m=1,2 ..., total }, the sample in X indicate with d description attribute A1, A2 ..., Ad (dimension), data sample xi
=(xi1, xi2 ..., xid), xj=(xj1, xj2 ..., xjd), wherein xi1, xi2 ..., xid and xj1, xj2 ..., xj3 points
It is not the specific value of the corresponding d description attribute A1, A2 ..., Ad of sample xi and xj.Similarity between sample xi and xj is logical
The distance between they d (xi, xj) is commonly used to indicate, apart from smaller, sample xi and xj is more similar, and diversity factor is smaller；Distance is got over
Large sample xi and xj is more dissimilar, and diversity factor is bigger.Euclidean distance formula is as follows:
K-means clustering algorithm evaluates clustering performance using error sum of squares criterion function.Data-oriented collection X, wherein
Only comprising description attribute, category attribute is not included.Assuming that X includes k cluster subset X 1, X2 ..., Xk, in each cluster subset
Sample size be respectively n1, n2 ..., nk, it is respectively m1 that the mean value of each cluster subset, which represents point (also referred to as cluster centre),
m2,…,mk.Error sum of squares criterion function formula are as follows:
Algorithm description is as follows:
For center vector c1, c2 ..., k seed when ck is initial；
Grouping: it assigns samples to away from nearest center vector, by the disjoint cluster of these sample architectures；
It determines center: using the center vector of each cluster as new center；
The step of repeated packets and determining center, until algorithmic statement.
The specific algorithm flow chart of the embodiment of the present invention is as shown in Fig. 3, and detailed process is as follows for algorithm:
1. k are arbitrarily chosen from data set { xn } n=1 is assigned to initial cluster centre c1, c2 ..., ck；
2. each sample xi that pair data are concentrated, calculates the Euclidean distance of itself and each cluster centre cj and obtains its classification
Label:
Label (i)=arg min | | xi-cj | |, i=1 ..., N, j=1 ..., k；
3. recalculating k cluster centre as the following formula:
Step 2 and step 3 are repeated, until reaching maximum number of iterations.
The present embodiment has chosen six characteristic values according to the characteristics of safe city video monitoring, respectively three days nearly (day before yesterday,
Yesterday, today) video play back number, region importance, video camera attribute, video marker respectively.It puts within video nearly three days and asks time
Number is that the history on the day of recording all cameras respectively has access to total degree；Region importance be according to user's specific business need into
Row divides, and can change according to demand and change region importance；Video camera attribute is video camera functional attributes itself, camera
It there is high definition, SD, ball machine, gunlock, night market, infrared, the different attribute according to possessed by video camera divides its significance level；
Video marker is from software using level, and for specific business, user can mark certain interested road video.Chart is such as
Under:
1 camera characteristic value of table
Data normalization (normalization) processing is an element task of data mining, and different evaluation index often has not
With dimension and dimensional unit, such situation influence whether data analysis as a result, in order to eliminate the dimension shadow between index
It rings, needs to carry out data normalization processing, to solve the comparativity between data target.Initial data is by data normalization
After reason, each index is in the same order of magnitude, is appropriate for Comprehensive Correlation evaluation, it is therefore desirable to carry out to six characteristic values of selection
Data normalization carries out linear transformation to initial data (carrying out to arrange), is mapped to end value between [0-1].With the first via
Day before yesterday access times for, convert it is as follows:
Wherein max is the maximum value in table sample data column, and min is the minimum value of table sample data column.
Claims (4)
1. a kind of monitor video temperature cloud storage method based on k mean value, it is characterised in that: clustered according to classical k-means
Algorithm carries out the analysis of video temperature to the video resource on network, predicted city camera temperature, and by the higher video of temperature
It is stored in the high cluster of performance node, the lower video of temperature is stored in the low cluster of performance node.
2. the monitor video temperature cloud storage method according to claim 1 based on k mean value, it is characterised in that: for height
The video temperature Analysis Service of temperature monitor video is analyzed once daily, and enters the intensity of video cloud storage system according to user
Distribution issues the video temperature index database time again.
3. the monitor video temperature cloud storage method according to claim 1 based on k mean value, which is characterized in that use k-
Means clustering algorithm on network video resource carry out the analysis of video temperature specifically includes the following steps:
(1) all videos are divided into n data set, k is arbitrarily chosen from n >=1 data set { xn } and is assigned in initial cluster
The heart c1, c2 ..., ck；
(2) each sample xi concentrated to data, calculates the Euclidean distance of itself and each cluster centre cj and obtains its classification mark
Number:
Label (i)=arg min | | xi-cj | |, i=1 ..., N, j=1 ..., k；
(3) k cluster centre is recalculated as the following formula:
(4) step (2) and step (3) are repeated, until reaching maximum number of iterations.
4. the monitor video temperature cloud storage method according to claim 3 based on k mean value, it is characterised in that: regarded
Frequency temperature analysis before, first video is standardized, the video standard be to each video choose six characteristic values, including
The day before yesterday, yesterday, today, totally 3 days videos played back number, region importance, video camera attribute, video marker respectively；Wherein, depending on
Certain day access times of frequency are that the history on the day of recording all cameras respectively has access to total degree；Region importance is according to user
Specific business need is divided, and can be changed according to demand and be changed region importance；Video camera attribute is video camera itself
Functional attributes, camera are divided into high definition, SD, ball machine, gunlock, night market, infrared, the different attribute according to possessed by video camera
Divide its significance level；Video marker be from software using level, for specific business, user can mark it is interested certain
Road video；Linear transformation is carried out to six characteristic values per video data all the way, is mapped to end value between [0-1].
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710540167.0A CN109218366A (en) | 2017-07-04 | 2017-07-04 | Monitor video temperature cloud storage method based on k mean value |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710540167.0A CN109218366A (en) | 2017-07-04 | 2017-07-04 | Monitor video temperature cloud storage method based on k mean value |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109218366A true CN109218366A (en) | 2019-01-15 |
Family
ID=64992004
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710540167.0A Pending CN109218366A (en) | 2017-07-04 | 2017-07-04 | Monitor video temperature cloud storage method based on k mean value |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109218366A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110401847A (en) * | 2019-07-17 | 2019-11-01 | 咪咕文化科技有限公司 | Compression storage method, electronic equipment and system for cloud DVR video |
| CN111475506A (en) * | 2020-03-30 | 2020-07-31 | 广州虎牙科技有限公司 | Data storage and query method, device, system, equipment and storage medium |
| CN113705979A (en) * | 2021-08-03 | 2021-11-26 | 海尔数字科技（上海）有限公司 | Logistics-based cargo transportation batch information tracing system and method |
| CN113918100A (en) * | 2021-11-15 | 2022-01-11 | 深圳潮数软件科技有限公司 | Multi-protocol multifunctional cloud storage gateway |
-
2017
- 2017-07-04 CN CN201710540167.0A patent/CN109218366A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110401847A (en) * | 2019-07-17 | 2019-11-01 | 咪咕文化科技有限公司 | Compression storage method, electronic equipment and system for cloud DVR video |
| CN111475506A (en) * | 2020-03-30 | 2020-07-31 | 广州虎牙科技有限公司 | Data storage and query method, device, system, equipment and storage medium |
| CN111475506B (en) * | 2020-03-30 | 2024-03-01 | 广州虎牙科技有限公司 | Method, device, system, equipment and storage medium for data storage and query |
| CN113705979A (en) * | 2021-08-03 | 2021-11-26 | 海尔数字科技（上海）有限公司 | Logistics-based cargo transportation batch information tracing system and method |
| CN113918100A (en) * | 2021-11-15 | 2022-01-11 | 深圳潮数软件科技有限公司 | Multi-protocol multifunctional cloud storage gateway |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108460121B (en) | Little file merging method for space-time data in smart city | |
| JP7119630B2 (en) | Information processing device, information exchange system, information processing method and information processing program | |
| CN109218366A (en) | Monitor video temperature cloud storage method based on k mean value | |
| Guan et al. | Video summarization with global and local features | |
| JP5155025B2 (en) | Similar data search system | |
| CN106933511A (en) | Consider the GML data storage method for organizing and system of load balancing and disk efficiency | |
| KR20220070482A (en) | Image incremental clustering method, apparatus, electronic device, storage medium and program product | |
| Mythily et al. | Clustering models for data stream mining | |
| CN116166191A (en) | Integrated system of lake and storehouse | |
| US11599577B2 (en) | System and method for content-hashed object storage | |
| CN110059148A (en) | The accurate searching method that spatial key applied to electronic map is inquired | |
| CN111581420B (en) | Flink-based medical image real-time retrieval method | |
| US20220035872A1 (en) | Direct cloud storage intake and upload architecture | |
| CN108090182B (en) | A kind of distributed index method and system of extensive high dimensional data | |
| Zhang et al. | A scalable approach for content-based image retrieval in peer-to-peer networks | |
| Anuradha et al. | Suggested techniques for clustering and mining of data streams | |
| Zhang et al. | Research on the construction of university data platform based on hybrid architecture | |
| Dhana Lakshmi et al. | Efficient techniques for clustering of users on web log data | |
| Dai et al. | The Data Platform for Large-Scale Video Surveillance Systems | |
| Devi et al. | Hybridized harmony search method for text clustering using concept factorization | |
| Parveen et al. | Data processing for large database using feature selection | |
| Chiroma et al. | A theoretical framework for big data analytics based on computational intelligent algorithms with the potential to reduce energy consumption | |
| Patil et al. | Categorizing documents by support vector machine trained using self-organizing maps clustering approach | |
| Mokashi et al. | Agricultural Sensor Captured Data Analysis for Optimum Storage | |
| Pelekis et al. | The case of big mobility data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190115 |
|
| WD01 | Invention patent application deemed withdrawn after publication |