CN107066581A - Distributed traffic monitor video data storage and quick retrieval system - Google Patents
Distributed traffic monitor video data storage and quick retrieval system Download PDFInfo
- Publication number
- CN107066581A CN107066581A CN201710242755.6A CN201710242755A CN107066581A CN 107066581 A CN107066581 A CN 107066581A CN 201710242755 A CN201710242755 A CN 201710242755A CN 107066581 A CN107066581 A CN 107066581A
- Authority
- CN
- China
- Prior art keywords
- video data
- semantic
- distributed
- data
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a kind of distributed traffic monitor video data storage and quick retrieval system, including：Video data memory module, is connected with the distributed columnar databases of HBase, for video data to be stored into the distributed columnar databases of HBase；Distributed video data semantic retrieval module, for setting up the structured index model based on internal memory to video data semanteme；Data communication module, for the data communication between multiple video data producers, multiple video data memory modules and the distributed video data semantic retrieval module.The present invention quickly can be carried out data retrieval according to video semanteme information and retrieved video data can be obtained with rapid batch, and recall precision is obviously improved while video data storage is realized.
Description
Technical field
The present invention relates to video data storage and retrieval technical field, a kind of distributed traffic monitor video number is particularly related to
According to storage and quick retrieval system.
Background technology
In recent years, with the raising of people's quality of the life, automobile has no longer been unreachable object, is increasingly becoming every family
The essential consumer goods of each household.Fast development and country along with auto manufacturing for promoting automobile industry to send out energetically
The policy of exhibition, China's car ownership rises year by year.Video data is generally stored in point by traditional monitor video storage system
In cloth database, extensive video data can be provided by distributed data base redundant storage and the characteristic of deblocking high
The stable and high storage and retrieval service handled up.
But the distributed data base of storage large-scale data can only support the data, services of OLAP modes, they are for multiple
Miscellaneous inquiry request can not be answered in time, and the Millisecond response that major key can only be supported to inquire about.Traditional monitor video data are deposited
Storage system generally requires to carry out a large amount of Optimization Works to improve the recall precision of system, nevertheless, recall precision is still relatively low.
Presently, there are some for video data store researchs, or but its most of be only absorbed in a specific field
Optimization of Information Retrieval, the retrieval of such as geographic information；Focus on the optimization to particular memory system, it is impossible to meet video counts
Can not be completely fitted for the demand of performance and with monitor video data according to storage and retrieval.And more existing be absorbed in
, there is retrieval performance issue in the system of video data storage, it is impossible to the need for being competent to video data quick-searching.Also one
A little searching systems based on HBase do not possess the function of retrieval video semanteme, and some provide the system of video semanteme search function
HBase index structure is not optimized.In addition, the method requirement extracted characteristics of image using MapReduce tasks and retrieved
It is image to input information, it is impossible to directly to text semantic information retrieval.
The content of the invention
In view of this, it is an object of the invention to propose a kind of distributed traffic monitor video data storage and quick-searching
System, quickly can carry out data retrieval according to video semanteme information and can obtain the video figure retrieved with rapid batch
As data.
A kind of distributed traffic monitor video data storage and quick retrieval system provided based on the above-mentioned purpose present invention,
Including：
Video data memory module, is connected with the distributed columnar databases of HBase, for video data to be stored into
HBase distribution columnar databases；
Distributed video data semantic retrieval module, for setting up the structured index based on internal memory to video data semanteme
Model；
Data communication module, for multiple video data producers, multiple video data memory modules and described point
Data communication between cloth video data semantic retrieval module.
In some embodiments, the video data memory module incorporates video semanteme information in RowKey, makes many
Data access requests of the secondary HBase based on RowKey is converted to the request of one or more RowKey range scans.
In some embodiments, the distributed video data semantic retrieval module stores same using skip list structure
The timestamp information of the semantic all video datas of camera same type；For the video data of image type, to image
Time of origin is ranked up；For the video data of video type, the initial time to video is ranked up.
In some embodiments, the Probability Choice Model of the skip list structure based on random function, the random of skip list is put down
Equal search length is C (log1/pN-1)=(log1/pN-1)/p, wherein n are skip list data storage number, and p is selection probability；Should
Function obtains minimum near p=0.5, chooses Probability p and is set to 0.5.
In some embodiments, filtered using bit array come the queried access to the skip list structure.
In some embodiments, the semantic information with timestamp is mapped, each semanteme of each camera
One bit array of correspondence, and the value that the semantic time of occurrence minimum time semantic with this subtracts each other is stored in for the video data
The subscript of bit array, when there is video data in some timestamp, is then 1 by its corresponding bit array position, is otherwise 0.
In some embodiments, trigger-type compression is carried out to the bit array using the lower calibration method of compression, when than
Special array capacity exceedes the threshold value set, and bit array will be compressed.
In some embodiments, using cuckoo Hash hash function to the distributed video data semantic retrieval mould
Hash table in block is optimized, first, and the data of character string type are passed through into mapping functionReflect
Penetrate as integer type, then pass through mapping functionMapping is carried out to obtain
Take the subscript position hashed twice.
In some embodiments, the data communication module is serialized using binary mode to message；Institute
State data communication module and the video data producer and the distributed video number are also transmitted using the subscribing mode based on Redis
According to the message between semantic retrieval module；The data communication module is also by video data by the way of history message persistence
All message durations communicated between the producer and the distributed video data semantic retrieval module into ordered queue, when
When having new node addition or need to rebuild index, the data pulled in history message queue carry out retrieval structure.
From the above it can be seen that the distributed traffic monitor video data storage of the invention provided and quick-searching system
System, quickly can carry out data retrieval and can retrieve with quick obtaining video and image type to tie according to video semanteme information
Fruit collection data, recall precision is obviously improved while video data storage is realized.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Distributed traffic monitor video data storages and quick retrieval system structural representation of the Fig. 1 for the embodiment of the present invention
Figure；
Fig. 2 is video data memory module RowKey conceptual schematic drawing schematic diagrames in the embodiment of the present invention；
Fig. 3 is the skip list structural representation in the embodiment of the present invention；
Fig. 4 changes point diagram for test skip list structure average length of search in the embodiment of the present invention with random chance；
Fig. 5 is the bit array schematic diagram in the embodiment of the present invention；
Fig. 6 uses the hash structure schematic diagram for improving cuckoo hashing algorithm in implementing for the present invention；
Fig. 7 is the data communication module structure chart in present invention implementation；
Fig. 8 is the RowKey retrieval flow schematic diagrames in present invention implementation.
Embodiment
For the object, technical solutions and advantages of the present invention are more clearly understood, below in conjunction with specific embodiment, and reference
Accompanying drawing, the present invention is described in more detail.
The embodiments of the invention provide a kind of distributed traffic monitor video data storage and quick retrieval system.With reference to figure
1, it is the distributed traffic monitor video data storage and quick retrieval system structural representation of the embodiment of the present invention.
The distributed traffic monitor video data storage and quick retrieval system, including：With the distributed column numbers of HBase
The video data memory module connected according to storehouse, the distributed video data semantic retrieval module based on internal memory and data communication mould
Block.Wherein：
Video data memory module is used to video data being stored into the distributed columnar databases of HBase, and ensures to deposit
Storage and the high efficiency accessed.
Distributed video data semantic retrieval module, for setting up the structured index based on internal memory to video data semanteme
Model, accelerates the efficiency based on semantic retrieval.
Data communication module, for multiple video data producers, multiple video data memory modules and described point
Data communication between cloth video data semantic retrieval module.
Specifically, video data memory module includes following operation content：
(11) to break random RowKey and being incremented by inefficiencies of the RowKey design methods on based on semantic retrieval, use
Based on semantic RowKey design methods, semantic information is dissolved among RowKey so that obtain RowKey from retrieval module
When list, these RowKey are physically data access requests of adjacent, the so multiple HBase based on RowKey
The request of one or several RowKey range scans can be converted to, reduction database loads lift retrieval performance simultaneously.The module
RowKey designs are as shown in Figure 2.Such RowKey designs cause the identical semantic video data of identical camera
It is distributed in sequentially in time in HBase databases, so each retrieval request is physically to connect by retrieve module acquisition
Continuous RowKey scopes, so only need to obtain the video data in the range of this successively, it is only necessary to carry out once or several
Secondary database access, the process is as shown in Figure 8.
Specifically, distributed video data semantic retrieval module includes following operation content：
(21) to accelerate semantic information time range effectiveness of retrieval, while taking into account when semantic information data volume is huge
The expense of system maintenance is waited, the semantic all video datas of same camera same type are stored using orderly skip list structure
Timestamp information, for the video data of image type, is ranked up to the time of origin of image, and regarding for video type
Frequency evidence, the initial time to video is ranked up.Image and video type video data skip list structure storage schematic diagram such as Fig. 3
It is shown.
(22) because hash table is unknown and in the case that data volume is larger in the data structure of storage, there is storage efficiency low
Lower the problem of, to improve storage efficiency and recall precision to a large amount of categorical datas, using cuckoo Hash hash function to dissipating
List is optimized.
Wherein, the operation content (21) includes following operation content：
(21a) is relevant due to the average length of search of skip list structure and the selection Probability p setting of skip list, in order to ensure skip list
Recall precision it is optimal, it is necessary to choose Probability p optimize setting.Probability choosing of the used skip list structure based on random function
Model is selected, the stochastic averagina search length of skip list is C (log1/pN-1)=(log1/pN-1)/p, wherein n are skip list data storage
Number, p is selection probability.It is as shown in Figure 4 that skip list structure average length of search changes point diagram with random chance.The function is in p=
0.5 nearby obtains minimum, chooses Probability p and is set to 0.5.
(21b) can cause to inquire about idle running twice, consumption systematicness because skip list structure is when data are not present in query context
Can, it is empty retrieval scene search efficiency for query structure collection to accelerate skip list structure, using bit array come to skip list
The queried access of structure is filtered.Bit array schematic diagram is as shown in Figure 5.
Further, the operation content (21b) includes following operation content：
(21ba) is a kind of binary structure of arrays due to bit array, is believed in order that being stabbed with bit array storage time
Breath, maps the semantic information with timestamp, each semanteme of each camera can correspond to a bit array,
And the value that the semantic time of occurrence subtracts each other with the semantic minimum time is stored in the subscript of bit array for the video data, when
There is video data in some timestamp, be then 1 by its corresponding bit array position, be otherwise then 0.
(21bb) has the problem of storage space-consuming is larger, balance to solve bit array when data volume is larger
High efficiency and space efficiency are retrieved, trigger-type compression is carried out to bit array using the lower calibration method of compression, when bit array is held
Amount will be compressed more than the threshold value set to bit array, the time range increase that each in bit array is represented, such as
Each can only represent this second either with or without data before, and each represents two seconds either with or without data now.Each is represented
Scope increase and be twice, the space-consuming of bit array declines one times.
Wherein, the operation content (22) includes following operation content：
(22a) is unknown and in the case that data volume is larger in the data structure of storage due to hash table, exists
The problem of storage efficiency is low, to improve storage efficiency and recall precision to a large amount of categorical datas, is breathed out using cuckoo
Uncommon hash function is optimized to hash table.Because hash structure stores character string type data, cuckoo Hash is calculated
Method requirement has two hash function calculating elements storage locations, by the way of mapping twice, first by character string class
The data of type pass through mapping functionIt is mapped as integer type.Next, passing through mapping functionCarry out mapping and obtain the subscript position hashed twice.Use improvement
The hash structure of cuckoo hashing algorithm is as shown in Figure 6.
Specifically, data communication module includes following operation content：
(31) in order to ensure the high efficiency communicated between data producer and retrieval module, offseted using binary mode
Breath is serialized so that small volume is adapted to network transmission after message sequence.Use Protocol Buffer data exchanges
Agreement is used as underlying protocol.
(32) because data producer and retrieval module are the relations of multi-to-multi, in order to ensure the video data producer with dividing
Space between cloth video data semantic retrieval module is non-coupled and synchronous non-coupled, using the subscribing mode based on Redis
To transmit the message between the video data producer and distributed video data semantic retrieval module.The video data producer is with dividing
Cloth video data semantic retrieval module is by subscribing to designated key and sending the message to reach the purpose of communication to designated key.
(33) in order to ensure that the time between data producer and retrieval module is non-coupled, using history message persistence
Mode is by all message durations communicated between the video data producer and distributed video data semantic retrieval module to having
In sequence queue, when having new node to add or need to rebuild index, the data pulled in history message queue are entered
Row retrieval is built.Data communication module structure chart is as shown in Figure 7.
Those of ordinary skills in the art should understand that：The discussion of any of the above embodiment is exemplary only, not
It is intended to imply that the scope of the present disclosure (including claim) is limited to these examples；Under the thinking of the present invention, above example
Or can also not be combined between the technical characteristic in be the same as Example, step can be realized with random order, and be existed such as
Many other changes of upper described different aspect of the invention, for simplicity, they are provided not in details.
Elaborating detail (such as color and car plate type semantic data) with describe the present invention exemplary embodiment
In the case of, it will be apparent to those skilled in the art that can in the case of these details or
These details implement the present invention in the case of changing.Therefore, these descriptions are considered as illustrative rather than limit
Property processed.
Although having been incorporated with specific embodiment of the invention, invention has been described, according to retouching above
State, many replacements of these embodiments, modifications and variations will be apparent for those of ordinary skills.Example
Such as, store other kinds of semantic data and retrieved by these semantic datas (for example, the type of vehicle, vehicle
Size) discussed embodiment can be used.
Embodiments of the invention be intended to fall within the broad range of appended claims it is all it is such replace,
Modifications and variations.Therefore, within the spirit and principles of the invention, any omission, modification, equivalent substitution, the improvement made
Deng should be included in the scope of the protection.
Claims (10)
1. a kind of distributed traffic monitor video data storage and quick retrieval system, it is characterised in that including：
Video data memory module, is connected with the distributed columnar databases of HBase, for video data to be stored into HBase points
Cloth columnar database；
Distributed video data semantic retrieval module, for setting up the structured index mould based on internal memory to video data semanteme
Type；
Data communication module, for multiple video data producers, multiple video data memory modules and distributed video
Data communication between data semantic retrieval module.
2. system according to claim 1, it is characterised in that the video data memory module incorporates semantic information
In RowKey, data access requests of the multiple HBase based on RowKey is set to be converted to a RowKey range scans request.
3. system according to claim 1, it is characterised in that the distributed video data semantic retrieval module is using jump
Table structure stores the timestamp information of the semantic all video datas of same camera same type；For regarding for image type
Frequency evidence, is ranked up to the time of origin of image；For the video data of video type, the initial time to video is arranged
Sequence.
4. system according to claim 3, it is characterised in that probability selection mould of the skip list structure based on random function
Type, the stochastic averagina search length of skip list isWherein n is skip list data storage
Number, p is selection probability；The function obtains minimum near p=0.5, chooses Probability p and is set to 0.5.
5. system according to claim 3, it is characterised in that visited using bit array the inquiry of the skip list structure
Ask and filtered.
6. system according to claim 5, it is characterised in that map the semantic information with timestamp, each
One bit array of each semantic correspondence of camera, and the value that the semantic time of occurrence minimum time semantic with this subtracts each other is
The video data is stored in the subscript of bit array, when there is video data in some timestamp, then by its corresponding bit array
Position is 1, is otherwise 0.
7. system according to claim 5, it is characterised in that carried out using the lower calibration method of compression to the bit array
Trigger-type is compressed, and when bit array capacity exceedes the threshold value set, bit array will be compressed.
8. system according to claim 1, it is characterised in that using cuckoo hash function to the distributed video number
Optimized according to the hash table in semantic retrieval module.
9. system according to claim 8, it is characterised in that the data of character string type are passed through into mapping functionInteger type need to be mapped as；Pass through mapping function
Carry out mapping and obtain the subscript position hashed twice.
10. system according to claim 1, it is characterised in that the data communication module uses binary mode pair
Message is serialized；The data communication module also transmits the video data producer using the subscribing mode based on Redis
With the message between the distributed video data semantic retrieval module；The data communication module is also lasting using history message
The mode of change holds all message communicated between the video data producer and the distributed video data semantic retrieval module
Longization when having new node to add or need to rebuild index, is pulled in history message queue into ordered queue
Data carry out retrieval structure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710242755.6A CN107066581B (en) | 2017-04-14 | 2017-04-14 | The storage of distributed traffic monitor video data and quick retrieval system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710242755.6A CN107066581B (en) | 2017-04-14 | 2017-04-14 | The storage of distributed traffic monitor video data and quick retrieval system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107066581A true CN107066581A (en) | 2017-08-18 |
| CN107066581B CN107066581B (en) | 2019-10-08 |
Family
ID=59600596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710242755.6A Active CN107066581B (en) | 2017-04-14 | 2017-04-14 | The storage of distributed traffic monitor video data and quick retrieval system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107066581B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679185A (en) * | 2017-09-30 | 2018-02-09 | 深圳市晟达机械设计有限公司 | A kind of Intelligent traffic video searching system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984745A (en) * | 2014-05-23 | 2014-08-13 | 何震宇 | Distributed video vertical searching method and system |
| CN104850576A (en) * | 2015-03-02 | 2015-08-19 | 武汉烽火众智数字技术有限责任公司 | Fast characteristic extraction system based on mass videos |
-
2017
- 2017-04-14 CN CN201710242755.6A patent/CN107066581B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984745A (en) * | 2014-05-23 | 2014-08-13 | 何震宇 | Distributed video vertical searching method and system |
| CN104850576A (en) * | 2015-03-02 | 2015-08-19 | 武汉烽火众智数字技术有限责任公司 | Fast characteristic extraction system based on mass videos |
Non-Patent Citations (1)
| Title |
|---|
| 赵晓萌: "云环境下监控视频结构化分析方法研究与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679185A (en) * | 2017-09-30 | 2018-02-09 | 深圳市晟达机械设计有限公司 | A kind of Intelligent traffic video searching system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107066581B (en) | 2019-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106528773B (en) | Map computing system and method based on Spark platform supporting spatial data management | |
| CN103714134B (en) | Network flow data index method and system | |
| CN102163195B (en) | Query optimization method based on unified view of distributed heterogeneous database | |
| CN103067461B (en) | A kind of metadata management system of file and metadata management method | |
| CN109299113B (en) | Range query method with storage-aware mixed index | |
| CN107169083A (en) | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment | |
| CN104424258A (en) | Multidimensional data query method and system, query server and column storage server | |
| CN106503243A (en) | Electric power big data querying method and system based on HBase secondary indexs | |
| CN107506464A (en) | A kind of method that HBase secondary indexs are realized based on ES | |
| CN107766445B (en) | Efficient and rapid data retrieval method supporting multi-dimensional retrieval | |
| CN103617276A (en) | Method for storing distributed hierarchical RDF data | |
| CN101594319A (en) | List item lookup method and device | |
| CN104111924A (en) | Database system | |
| CN103326925B (en) | A kind of information push method and device | |
| Von der Weth et al. | Multiterm keyword search in NoSQL systems | |
| CN113811928B (en) | Distributed memory space data storage for K nearest neighbor search | |
| CN105635278A (en) | Method for managing metadata of storage system and metadata server | |
| CN102737061A (en) | Distributed ticket query management system and method | |
| CN103425694B (en) | The searching method of relational data and device | |
| CN107066581B (en) | The storage of distributed traffic monitor video data and quick retrieval system | |
| CN110716933B (en) | Novel urban rail train big data-oriented high-flexibility distributed index method | |
| CN110365728A (en) | A kind of grouping cooperative caching system of Information central site network | |
| CN114338725B (en) | Distributed storage scheduling method for improving upper limit of large-scale cluster rendering | |
| CN103020300B (en) | Method and device for information retrieval | |
| Qin et al. | Massive AIS data management based on HBase and Spark |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |