CN107066581B - The storage of distributed traffic monitor video data and quick retrieval system - Google Patents
The storage of distributed traffic monitor video data and quick retrieval system Download PDFInfo
- Publication number
- CN107066581B CN107066581B CN201710242755.6A CN201710242755A CN107066581B CN 107066581 B CN107066581 B CN 107066581B CN 201710242755 A CN201710242755 A CN 201710242755A CN 107066581 B CN107066581 B CN 107066581B
- Authority
- CN
- China
- Prior art keywords
- video data
- data
- module
- distributed
- semantic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a kind of storage of distributed traffic monitor video data and quick retrieval systems, comprising: video data memory module is connect with HBase distribution columnar database, for video data to be stored into HBase distribution columnar database；Distributed video data semantic retrieval module, for establishing structured index model memory-based to video data semanteme；Data communication module, for the data communication between multiple video data producers, multiple video data memory modules and the distributed video data semantic retrieval module.The present invention quickly can be carried out data retrieval according to video semanteme information and can be obtained retrieved video data with rapid batch, be obviously improved recall precision while realizing video data storage.
Description
Technical field
The present invention relates to video data storage and retrieval technical fields, particularly relate to a kind of distributed traffic monitor video number
According to storage and quick retrieval system.
Background technique
In recent years, with the raising of people's quality of the life, automobile is no longer unreachable object, is increasingly becoming every family
The essential consumer goods of each household.Along with the fast development of auto manufacturing and country for pushing automobile industry to send out energetically
The policy of exhibition, China's car ownership rise year by year.Video data is usually stored in point by traditional monitor video storage system
In cloth database, it is high that extensive video data can be provided by distributed data base redundant storage and the characteristic of deblocking
The stable and high storage and retrieval service handled up.
But the distributed data base for storing large-scale data can only support the data service of OLAP mode, they are for multiple
Miscellaneous inquiry request can not be answered in time, and the Millisecond response that major key can only be supported to inquire.Traditional monitor video data are deposited
Storage system generally requires to carry out recall precision of a large amount of Optimization Works to improve system, nevertheless, recall precision is still lower.
Presently, there are some researchs for video data storage, but its most of otherwise be only absorbed in a specific field
Optimization of Information Retrieval, such as the retrieval of geographic information；It focuses on not being able to satisfy video counts to the optimization of particular memory system
It can not be bonded completely according to storage and retrieval for the demand of performance and with monitor video data.And more existing it is absorbed in
, there is retrieval performance, the needs to video data quick-searching can not be competent in the system of video data storage.There are also one
The searching system based on HBase does not have the function of retrieval video semanteme a bit, some to provide the system of video semanteme search function
The index structure of HBase is not optimized.In addition, the method requirement extracting characteristics of image using MapReduce task and retrieving
Inputting information is image, cannot be directly to text semantic information retrieval.
Summary of the invention
In view of this, it is an object of the invention to propose a kind of storage of distributed traffic monitor video data and quick-searching
System quickly can carry out data retrieval according to video semanteme information and can obtain the video figure retrieved with rapid batch
As data.
Based on a kind of above-mentioned purpose distributed traffic monitor video data storage provided by the invention and quick retrieval system,
Include:
Video data memory module is connect, for video data to be stored into HBase distribution columnar database
HBase distribution columnar database；
Distributed video data semantic retrieval module, for establishing structured index memory-based to video data semanteme
Model；
Data communication module, for multiple video data producers, multiple video data memory modules and described point
Data communication between cloth video data semantic retrieval module.
In some embodiments, the video data memory module incorporates video semanteme information in RowKey, makes more
Secondary HBase is converted to the request of one or more RowKey range scans based on the data access request of RowKey.
In some embodiments, the distributed video data semantic retrieval module is stored same using skip list structure
The timestamp information of all video datas of camera same type semanteme；For the video data of image type, to image
Time of origin is ranked up；For the video data of video type, the initial time of video is ranked up.
In some embodiments, the Probability Choice Model of the skip list structure based on random function, the random of skip list are put down
Equal search length is C (log1/pN-1)=(log1/pN-1)/p, wherein n is skip list storing data number, and p is to choose probability；It should
Function obtains minimum near p=0.5, chooses Probability p and is set as 0.5.
In some embodiments, the queried access of the skip list structure is filtered using bit array.
In some embodiments, the semantic information with timestamp is mapped, each semanteme of each camera
A corresponding bit array, and the value that the minimum time of the semanteme time of occurrence and the semanteme subtracts each other is that the video data is stored in
Its corresponding bit array position is then 1, is otherwise 0 by the subscript of bit array when there is video data in some timestamp.
In some embodiments, trigger-type compression is carried out to the bit array using compressing lower calibration method, when than
Special array capacity is more than the threshold value of setting, will be compressed to bit array.
In some embodiments, using cuckoo Hash hash function to the distributed video data semantic retrieval mould
Hash table in block optimizes, firstly, the data of character string type are passed through mapping functionIt reflects
It penetrates as integer type, then passes through mapping functionMapping is carried out to obtain
Take the subscript position hashed twice.
In some embodiments, the data communication module serializes message using binary mode；Institute
It states data communication module and the video data producer and the distributed video number is also transmitted using the subscribing mode based on Redis
According to the message between semantic retrieval module；The data communication module is also by video data by the way of history message persistence
All message durations communicated between the producer and the distributed video data semantic retrieval module into ordered queue, when
When having new node to be added or need to rebuild index, the data pulled in history message queue carry out retrieval building.
From the above it can be seen that distributed traffic monitor video data storage provided by the invention and quick-searching system
System quickly can be carried out data retrieval according to video semanteme information and can be tied with quick obtaining video and image type retrieval
Fruit collects data, is obviously improved recall precision while realizing video data storage.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is the storage of distributed traffic monitor video data and quick retrieval system structural representation of the embodiment of the present invention
Figure；
Fig. 2 is video data memory module RowKey conceptual schematic drawing schematic diagram in the embodiment of the present invention；
Fig. 3 is the skip list structural schematic diagram in the embodiment of the present invention；
Fig. 4 is to test skip list structure average length of search in the embodiment of the present invention to change point diagram with random chance；
Fig. 5 is the bit array schematic diagram in the embodiment of the present invention；
Fig. 6 is during the present invention is implemented using the hash structure schematic diagram for improving cuckoo hashing algorithm；
Fig. 7 is the data communication module structure chart during the present invention is implemented；
Fig. 8 is the RowKey retrieval flow schematic diagram during the present invention is implemented.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with specific embodiment, and reference
Attached drawing, the present invention is described in more detail.
The embodiment of the invention provides a kind of storage of distributed traffic monitor video data and quick retrieval systems.With reference to figure
1, it is the storage of distributed traffic monitor video data and the quick retrieval system structural schematic diagram of the embodiment of the present invention.
The distributed traffic monitor video data storage and quick retrieval system, comprising: with HBase distribution column number
Video data memory module, distributed video data semantic retrieval module memory-based and the data communication mould connected according to library
Block.Wherein:
Video data memory module is used to for video data being stored into HBase distribution columnar database, and guarantees to deposit
The high efficiency of storage and access.
Distributed video data semantic retrieval module, for establishing structured index memory-based to video data semanteme
Model accelerates the efficiency based on semantic retrieval.
Data communication module, for multiple video data producers, multiple video data memory modules and described point
Data communication between cloth video data semantic retrieval module.
Specifically, video data memory module includes following operation content:
(11) it to break random RowKey and incremental RowKey design method based on the inefficiencies on semantic retrieval, uses
Semantic-based RowKey design method, semantic information is dissolved among RowKey, so that obtaining RowKey from retrieval module
When list, these RowKey are physically data access request of adjacent, the multiple in this way HBase based on RowKey
The request of one or several RowKey range scans can be converted to, database loads is reduced and promotes retrieval performance simultaneously.The module
RowKey design scheme is as shown in Figure 2.Such RowKey design scheme makes the video data of the identical semanteme of identical camera
It is distributed in HBase database sequentially in time, what retrieval request each in this way was obtained by retrieval module is physically to connect
Continuous RowKey range only needs successively to obtain the video data within the scope of this in this way, it is only necessary to carry out primary or several
Secondary database access, the process are as shown in Figure 8.
Specifically, distributed video data semantic retrieval module includes following operation content:
(21) it to accelerate semantic information time range effectiveness of retrieval, combines when semantic information data volume is huge
The expense for waiting system maintenance, all video datas of same camera same type semanteme are stored using orderly skip list structure
Timestamp information is ranked up the video data of image type to the time of origin of image, and for the view of video type
Frequency evidence is ranked up the initial time of video.Image and video type video data skip list structure store schematic diagram such as Fig. 3
It is shown.
(22) due to hash table in the data structure of storage in the biggish situation of unknown and data volume, it is low that there are storage efficiencies
Under problem, be to improve to the storage efficiency and recall precision of a large amount of categorical datas, using cuckoo Hash hash function to dissipating
List optimizes.
Wherein, the operation content (21) includes following operation content:
(21a) since the average length of search of skip list structure and the selection Probability p setting of skip list are related, in order to ensure skip list
Recall precision it is optimal, need to choose Probability p optimize setting.Used skip list structure is selected based on the probability of random function
Model is selected, the stochastic averagina search length of skip list is C (log1/pN-1)=(log1/pN-1)/p, wherein n is skip list storing data
Number, p are to choose probability.Skip list structure average length of search is as shown in Figure 4 with random chance variation point diagram.The function is in p=
0.5 nearby obtains minimum, chooses Probability p and is set as 0.5.
(21b) will cause inquiry idle running twice, consumption systematicness since skip list structure is when data are not present in query context
Can, it is empty retrieval scene search efficiency for query structure collection to accelerate skip list structure, using bit array come to skip list
The queried access of structure is filtered.Bit array schematic diagram is as shown in Figure 5.
Further, the operation content (21b) includes following operation content:
(21ba) believes since bit array is a kind of binary structure of arrays in order to use bit array storage time to stab
Breath maps the semantic information with timestamp, and each of each camera semanteme can correspond to a bit array,
And the value that the minimum time of the semanteme time of occurrence and the semanteme subtracts each other is the subscript that the video data is stored in bit array, when
There is video data in some timestamp, is then 1 by its corresponding bit array position, is otherwise then 0.
(21bb) be solves the problems, such as bit array had when data volume is larger storage occupied space it is larger, weigh
High efficiency and space efficiency are retrieved, trigger-type compression is carried out to bit array using lower calibration method is compressed, when bit array is held
Amount is more than the threshold value of setting, will be compressed to bit array, and the time range that each in bit array indicates increases, such as
Each can only indicate this second either with or without data before, each expression now two seconds either with or without data.Each is indicated
Range increasing be twice, the occupied space of bit array declines one times.
Wherein, the operation content (22) includes following operation content:
(22a) since hash table is in the data structure of storage in the biggish situation of unknown and data volume, there are storage efficiencies
Low problem uses cuckoo Hash hash function pair to improve storage efficiency and recall precision to a large amount of categorical datas
Hash table optimizes.Since hash structure stores character string type data, there are two hash letter for the requirement of cuckoo hash algorithm
The data of character string type are passed through mapping function first by the way of mapping twice by number calculating elements storage locationIt is mapped as integer type.Next, passing through mapping functionIt carries out mapping and obtains the subscript position hashed twice.Use improvement
The hash structure of cuckoo hashing algorithm is as shown in Figure 6.
Specifically, data communication module includes following operation content:
(31) it in order to guarantee the high efficiency communicated between data producer and retrieval module, is offseted using binary mode
Breath is serialized, so that small volume is suitble to network transmission after message sequence.Use Protocol Buffer data exchange
Agreement is as underlying protocol.
(32) due to the relationship that data producer and retrieval module are multi-to-multis, in order to guarantee the video data producer and divide
Space between cloth video data semantic retrieval module it is non-coupled with it is synchronous non-coupled, using the subscribing mode based on Redis
To transmit the message between the video data producer and distributed video data semantic retrieval module.The video data producer with point
Cloth video data semantic retrieval module is by subscribing to designated key and sending the message to achieve the purpose that communication to designated key.
(33) in order to guarantee that the time between data producer and retrieval module is non-coupled, using history message persistence
Mode is by all message durations communicated between the video data producer and distributed video data semantic retrieval module to having
In sequence queue, when thering is new node to be added or when need to rebuild index, pull data in history message queue into
Row retrieval building.Data communication module structure chart is as shown in Figure 7.
It should be understood by those ordinary skilled in the art that: the discussion of any of the above embodiment is exemplary only, not
It is intended to imply that the scope of the present disclosure (including claim) is limited to these examples；Under thinking of the invention, above embodiments
Or can also be combined between the technical characteristic in different embodiments, step can be realized with random order, and be existed such as
Many other variations of the upper different aspect of the invention, for simplicity, they are not provided in details.
Elaborating detail (such as color and license plate type semantic data) to describe exemplary embodiment of the present invention
In the case where, it will be apparent to those skilled in the art that can without these specific details or
These details implement the present invention in the case where changing.Therefore, these descriptions should be considered as illustrative rather than limit
Property processed.
Although having been incorporated with specific embodiments of the present invention, invention has been described, according to retouching for front
It states, many replacements of these embodiments, modifications and variations will be apparent for those of ordinary skills.Example
Such as, it stores other kinds of semantic data and is retrieved by these semantic datas (for example, the type of vehicle, vehicle
Size) discussed embodiment can be used.
The embodiment of the present invention be intended to cover fall into all such replacements within the broad range of appended claims,
Modifications and variations.Therefore, all within the spirits and principles of the present invention, any omission, modification, equivalent replacement, the improvement made
Deng should all be included in the protection scope of the present invention.
Claims (7)
1. a kind of distributed traffic monitor video data storage and quick retrieval system characterized by comprising
Video data memory module is connect with HBase distribution columnar database, for video data to be stored into HBase points
Cloth columnar database；The video data memory module incorporates semantic information in RowKey, is based on multiple HBase
The data access request of RowKey is converted to a RowKey range scans request；
Distributed video data semantic retrieval module, for establishing structured index mould memory-based to video data semanteme
Type；The hash table in the distributed video data semantic retrieval module is optimized using cuckoo hash function；By word
The data of symbol string type pass through mapping functionIt need to be mapped as integer type；Pass through mapping functionIt carries out mapping and obtains the subscript position hashed twice；
Data communication module is used for multiple video data producers, multiple video data memory modules and distributed video
Data communication between data semantic retrieval module.
2. system according to claim 1, which is characterized in that the distributed video data semantic retrieval module is using jump
Table structure stores the timestamp informations of all video datas of same camera same type semanteme；For the view of image type
Frequency evidence is ranked up the time of origin of image；For the video data of video type, the initial time of video is arranged
Sequence.
3. system according to claim 2, which is characterized in that probability selection mould of the skip list structure based on random function
Type, the stochastic averagina search length of skip list are C (log1/pN-1)=(log1/pN-1)/p, wherein n is skip list storing data number,
P is to choose probability；The function obtains minimum near p=0.5, chooses Probability p and is set as 0.5.
4. system according to claim 2, which is characterized in that visited using bit array come the inquiry to the skip list structure
It asks and is filtered.
5. system according to claim 4, which is characterized in that map the semantic information with timestamp, each
The semantic corresponding bit array of each of camera, and the value that the minimum time of the semanteme time of occurrence and the semanteme subtracts each other is
The video data is stored in the subscript of bit array, when there is video data in some timestamp, then by its corresponding bit array
Position is 1, is otherwise 0.
6. system according to claim 4, which is characterized in that carried out using lower calibration method is compressed to the bit array
Trigger-type compression will compress bit array when threshold value of the bit array capacity more than setting.
7. system according to claim 1, which is characterized in that the data communication module is offseted using binary mode
Breath is serialized；The data communication module also using based on Redis subscribing mode come transmit the video data producer with
Message between the distributed video data semantic retrieval module；The data communication module also uses history message persistence
Mode all message communicated between the video data producer and the distributed video data semantic retrieval module are lasting
Change into ordered queue, when thering is new node to be added or need to rebuild index, pulls in history message queue
Data carry out retrieval building.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710242755.6A CN107066581B (en) | 2017-04-14 | 2017-04-14 | The storage of distributed traffic monitor video data and quick retrieval system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710242755.6A CN107066581B (en) | 2017-04-14 | 2017-04-14 | The storage of distributed traffic monitor video data and quick retrieval system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107066581A CN107066581A (en) | 2017-08-18 |
| CN107066581B true CN107066581B (en) | 2019-10-08 |
Family
ID=59600596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710242755.6A Active CN107066581B (en) | 2017-04-14 | 2017-04-14 | The storage of distributed traffic monitor video data and quick retrieval system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107066581B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679185A (en) * | 2017-09-30 | 2018-02-09 | 深圳市晟达机械设计有限公司 | A kind of Intelligent traffic video searching system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984745A (en) * | 2014-05-23 | 2014-08-13 | 何震宇 | Distributed video vertical searching method and system |
| CN104850576A (en) * | 2015-03-02 | 2015-08-19 | 武汉烽火众智数字技术有限责任公司 | Fast characteristic extraction system based on mass videos |
-
2017
- 2017-04-14 CN CN201710242755.6A patent/CN107066581B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984745A (en) * | 2014-05-23 | 2014-08-13 | 何震宇 | Distributed video vertical searching method and system |
| CN104850576A (en) * | 2015-03-02 | 2015-08-19 | 武汉烽火众智数字技术有限责任公司 | Fast characteristic extraction system based on mass videos |
Non-Patent Citations (1)
| Title |
|---|
| 云环境下监控视频结构化分析方法研究与实现;赵晓萌;《中国优秀硕士学位论文全文数据库信息科技辑》;20150830;31-47页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107066581A (en) | 2017-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106528773B (en) | Map computing system and method based on Spark platform supporting spatial data management | |
| CN104252536B (en) | A kind of internet log data query method and device based on hbase | |
| US8078394B2 (en) | Indexing large-scale GPS tracks | |
| CN107169083A (en) | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment | |
| CN105303456A (en) | Method for processing monitoring data of electric power transmission equipment | |
| CN100458784C (en) | Researching system and method used in digital labrary | |
| CN110825733B (en) | Multi-sampling-stream-oriented time series data management method and system | |
| CN103067461A (en) | Metadata management system of document and metadata management method thereof | |
| CN105159845A (en) | Memory reading method | |
| CN102739622A (en) | Expandable data storage system | |
| CN101247271A (en) | Performance data storage method and device | |
| CN103326925B (en) | A kind of information push method and device | |
| CN105096174A (en) | Transaction matching method and transaction matching system | |
| CN104111924A (en) | Database system | |
| CN112148731B (en) | Data paging query method, device and storage medium | |
| Von der Weth et al. | Multiterm keyword search in NoSQL systems | |
| CN113811928B (en) | Distributed memory space data storage for K nearest neighbor search | |
| CN107066581B (en) | The storage of distributed traffic monitor video data and quick retrieval system | |
| CN112559459B (en) | Cloud computing-based self-adaptive storage layering system and method | |
| CN116756253B (en) | Data storage and query methods, devices, equipment and media of relational database | |
| CN112181302A (en) | Data multilevel storage and access method and system | |
| CN112015733A (en) | Method for storing and rapidly inquiring mass data of electric power customer service operation and distribution service | |
| CN114338725B (en) | Distributed storage scheduling method for improving upper limit of large-scale cluster rendering | |
| CN105095224A (en) | Method, apparatus and system for carrying out OLAP analysis in mobile communication network | |
| Qin et al. | Massive AIS data management based on HBase and Spark |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |