CN104050271A - Bayonet data processing method based on HBase - Google Patents
Bayonet data processing method based on HBase Download PDFInfo
- Publication number
- CN104050271A CN104050271A CN201410284598.1A CN201410284598A CN104050271A CN 104050271 A CN104050271 A CN 104050271A CN 201410284598 A CN201410284598 A CN 201410284598A CN 104050271 A CN104050271 A CN 104050271A
- Authority
- CN
- China
- Prior art keywords
- data
- systems
- bit representation
- hbase
- bayonet socket
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The invention relates to a bayonet data processing method based on an HBase. The bayonet data processing method includes the steps of obtaining data records to be stored, judging the data size, storing small-number temporary data through a relational data base SQL server if data are the small-number temporary data, judging the structural state of mass data if the data are the mass data, storing the data through the distributed data base HBase if the data are structural data, and storing the data through a HDFS if the data are unstructured data. Compared with the prior art, the bayonet data processing method has the advantages that the data can be rapidly accessed, the data processing efficiency can be improved, and the mass data processing problem can be solved.
Description
Technical field
The present invention relates to computer realm, particularly a kind of bayonet socket data processing method based on HBase.
Background technology
Recent years is along with public security strengthens the construction of Gate System (being arranged on the supervisory system that there is defence on traffic main artery etc. and checks the gateway of facility), more and more important for the networking on a large scale of Gate System and integrated application change.Vehicle by bayonet socket put when position system need to extract and registration card dot information, by details such as time, car plate, the speed of a motor vehicle, car plate colors.In traditional relevant database, each attribute conventionally recording as major key, taking vehicle pass-through taking vehicle pass-through record number (vehicle of unique identification bayonet socket point position is through overwriting), as row, creates vehicle pass-through record information list.But the increase of quantity of building along with Gate System, also rapid growth of the data volume of bayonet socket information of vehicles, reach the rank of billions of records, traditional relevant database cannot meet the demand of system, must adopt new technological means to carry out the fast processing of mass data.
Summary of the invention
Technical matters to be solved by this invention is to provide a kind of fast access data, promotes data-handling efficiency, solves the bayonet socket data processing method based on HBase of a mass data processing difficult problem.
The technical scheme that the present invention solves the problems of the technologies described above is as follows: a kind of bayonet socket data processing method based on HBase, comprises the following steps:
Step 1: obtain data recording to be stored;
Step 2: judge data volume size, if a small amount of provisional data (data that data recording is less than 100,000,000) perform step 3; If mass data (data recording is more than or equal to the data of 100,000,000) performs step 4;
Step 3: a small amount of provisional data are stored by relational database SQL server;
Step 4: judge the configuration state of mass data, structural data (data at once, are stored in lane database, can carry out the data that logical expression is realized by bivariate table structure) if, by distributed data base, Hbase stores; Unstructured data (data of being inconvenient to use database two dimension logical table to show) if, by file system HDFS (Hadoop Distributed File System, it is a distributed file system, HDFS has the feature of high fault tolerance, and design is used for being deployed on cheap hardware) store.
The invention has the beneficial effects as follows: by the classification processing to bayonet socket data, promoted the data-handling efficiency of bayonet socket, solved a difficult problem for mass data processing in current Gate System.
On the basis of technique scheme, the present invention can also do following improvement.
Further technical scheme, described distributed data base HBase (Hadoop Database, it is a high reliability, high-performance, towards row, telescopic distributed memory system, utilizing HBase technology on cheap PC Server, to erect large-scale structure storage cluster) the Row Key of list structure (Row key is the major key in the collecting structure of a series of two-dimensional arrays of distributed data base HBase) adopts 36 scale codings, organizational structure's numbering corresponding to 1 to 4 bit representation camera of 36 systems, corresponding stuck point numbering in 36 system 5 to 6 bit representation camera place organizational structures, corresponding camera lens numbering in the 7th bit representation camera place stuck point of 36 systems, 8 to the 10 bit representation bayonet socket vehicle pass-through dates of 36 systems, 11 to the 14 bit representation bayonet socket vehicle pass-through times of 36 systems, 15 to 22 bit representation bayonet socket vehicle license plate numbers of 36 systems, the speed of 23 to 24 bit representation bayonet socket vehicles of 36 systems, the 25th bit representation car plate kind of 36 systems, the 26th bit representation license plate number color of 36 systems, the 27th bit representation body color of 36 systems, 28 to 29 bit representation vehicle emblems of 36 systems, the 30th bit representation encryption identification information of 36 systems.
Adopt the beneficial effect of above-mentioned further scheme to be: to facilitate data search, can make the data that newly write be found fast in searching.
The Row Key of described distributed data base HBase list structure stores multiple Region server (domain server) into simultaneously.
Adopt the beneficial effect of above-mentioned further scheme to be: the pressure that disperses multi-user to ask by multiple Region server.
Described current date and time uses the timestamp (character string, the time at unique mark a moment) of backward to carry out binary storage.
Adopt the beneficial effect of above-mentioned further scheme to be: each bayonet socket point camera is captured up-to-date data in data and is stored in foremost, the using value maximum of up-to-date data, also be the more interested data of most of user, can have greatly accelerated in this way the speed of visit data.
Brief description of the drawings
Fig. 1 is the process flow diagram of a kind of bayonet socket data processing method based on HBase of the present invention.
Embodiment
Below in conjunction with accompanying drawing, principle of the present invention and feature are described, example, only for explaining the present invention, is not intended to limit scope of the present invention.
As shown in Figure 1, a kind of bayonet socket data processing method based on HBase, comprises the following steps:
Step 1: obtain data recording to be stored;
Step 2: judge data volume size, if a small amount of provisional data perform step 3; If mass data performs step 4;
Step 3: a small amount of provisional data are stored by relational database SQL server;
Step 4: judge the configuration state of mass data, structural data if, by distributed data base, Hbase stores; Unstructured data if, by distributed file system, HDFS stores.
The Row Key of described distributed data base HBase list structure adopts 36 scale codings, organizational structure's numbering corresponding to 1 to 4 bit representation camera of 36 systems, corresponding stuck point numbering in 36 system 5 to 6 bit representation camera place organizational structures, corresponding camera lens numbering in the 7th bit representation camera place stuck point of 36 systems, 8 to the 10 bit representation bayonet socket vehicle pass-through dates of 36 systems, 11 to the 14 bit representation bayonet socket vehicle pass-through times of 36 systems, 15 to 22 bit representation bayonet socket vehicle license plate numbers of 36 systems, the speed of 23 to 24 bit representation bayonet socket vehicles of 36 systems, the 25th bit representation car plate kind of 36 systems, the 26th bit representation license plate number color of 36 systems, the 27th bit representation body color of 36 systems, 28 to 29 bit representation vehicle emblems of 36 systems, the 30th bit representation encryption identification information of 36 systems.
The Row Key of described distributed data base HBase list structure stores multiple Region server into simultaneously.
Described current date and time uses the timestamp of backward to carry out binary storage.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any amendment of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (4)
1. the bayonet socket data processing method based on HBase, is characterized in that: comprise the following steps:
Step 1: obtain data recording to be stored;
Step 2: judge data volume size, if a small amount of provisional data perform step 3; If mass data performs step 4;
Step 3: a small amount of provisional data are stored by relational database SQL server;
Step 4: judge the configuration state of mass data, structural data if, by distributed data base, Hbase stores; Unstructured data if, by file system, HDFS stores.
2. a kind of bayonet socket data processing method based on HBase according to claim 1, it is characterized in that: the Row Key of described distributed data base HBase list structure adopts 36 scale codings, organizational structure's numbering corresponding to 1 to 4 bit representation camera of 36 systems, corresponding stuck point numbering in 36 system 5 to 6 bit representation camera place organizational structures, corresponding camera lens numbering in the 7th bit representation camera place stuck point of 36 systems, 8 to the 10 bit representation bayonet socket vehicle pass-through dates of 36 systems, 11 to the 14 bit representation bayonet socket vehicle pass-through times of 36 systems, 15 to 22 bit representation bayonet socket vehicle license plate numbers of 36 systems, the speed of 23 to 24 bit representation bayonet socket vehicles of 36 systems, the 25th bit representation car plate kind of 36 systems, the 26th bit representation license plate number color of 36 systems, the 27th bit representation body color of 36 systems, 28 to 29 bit representation vehicle emblems of 36 systems, the 30th bit representation encryption identification information of 36 systems.
3. according to a kind of bayonet socket data processing method based on HBase described in claim 1 or 2, it is characterized in that: the Row Key of described distributed data base HBase list structure stores multiple Region server into simultaneously.
4. according to a kind of bayonet socket data processing method based on HBase described in claim 1 or 2, it is characterized in that: described current date and time uses the timestamp of backward to carry out binary storage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410284598.1A CN104050271A (en) | 2014-06-23 | 2014-06-23 | Bayonet data processing method based on HBase |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410284598.1A CN104050271A (en) | 2014-06-23 | 2014-06-23 | Bayonet data processing method based on HBase |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104050271A true CN104050271A (en) | 2014-09-17 |
Family
ID=51503103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410284598.1A Pending CN104050271A (en) | 2014-06-23 | 2014-06-23 | Bayonet data processing method based on HBase |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104050271A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104794226A (en) * | 2015-04-30 | 2015-07-22 | 北京奇艺世纪科技有限公司 | Method and device for writing data based on HBase database |
| CN104952251A (en) * | 2015-06-18 | 2015-09-30 | 安徽四创电子股份有限公司 | Urban viaduct traffic state sensing method based on bayonet and HADOOP technology |
| CN105374210A (en) * | 2015-12-03 | 2016-03-02 | 中山大学 | Gate detection rate obtaining method based on floating vehicle data |
| CN105426437A (en) * | 2015-11-05 | 2016-03-23 | 西安翔迅科技有限责任公司 | HBase-based storage method of intelligent transportation field checkpost data |
| CN105788264A (en) * | 2016-04-28 | 2016-07-20 | 泰华智慧产业集团股份有限公司 | Hbase-based barrier data area collision analysis method and Hbase-based barrier data area collision analysis system |
| CN106294499A (en) * | 2015-06-09 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of database data querying method and equipment |
| CN107862867A (en) * | 2017-11-08 | 2018-03-30 | 泰华智慧产业集团股份有限公司 | Based on big data for the first time enter the method and system of city vehicle analysis |
| CN107993444A (en) * | 2017-11-22 | 2018-05-04 | 紫光捷通科技股份有限公司 | The suspicion car identification of car big data analysis is crossed based on bayonet |
| CN112003956A (en) * | 2020-10-27 | 2020-11-27 | 武汉中科通达高新技术股份有限公司 | Traffic management system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130103658A1 (en) * | 2011-10-19 | 2013-04-25 | Vmware, Inc. | Time series data mapping into a key-value database |
| CN103514273A (en) * | 2013-09-17 | 2014-01-15 | 宁波东冠科技有限公司 | Data collection and monitoring control system and data processing method of system |
| CN103700256A (en) * | 2012-09-28 | 2014-04-02 | 中国电信股份有限公司 | Method and system for gate management |
| CN103778184A (en) * | 2013-12-26 | 2014-05-07 | 深圳市华仁达技术有限公司 | Intelligent transportation massive checkpoint passage data transmission system |
-
2014
- 2014-06-23 CN CN201410284598.1A patent/CN104050271A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130103658A1 (en) * | 2011-10-19 | 2013-04-25 | Vmware, Inc. | Time series data mapping into a key-value database |
| CN103700256A (en) * | 2012-09-28 | 2014-04-02 | 中国电信股份有限公司 | Method and system for gate management |
| CN103514273A (en) * | 2013-09-17 | 2014-01-15 | 宁波东冠科技有限公司 | Data collection and monitoring control system and data processing method of system |
| CN103778184A (en) * | 2013-12-26 | 2014-05-07 | 深圳市华仁达技术有限公司 | Intelligent transportation massive checkpoint passage data transmission system |
Non-Patent Citations (1)
| Title |
|---|
| 陈吉荣等: "基于Hadoop生态系统的大数据解决方案综述", 《计算机工程与科学》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104794226A (en) * | 2015-04-30 | 2015-07-22 | 北京奇艺世纪科技有限公司 | Method and device for writing data based on HBase database |
| CN106294499A (en) * | 2015-06-09 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of database data querying method and equipment |
| CN104952251A (en) * | 2015-06-18 | 2015-09-30 | 安徽四创电子股份有限公司 | Urban viaduct traffic state sensing method based on bayonet and HADOOP technology |
| CN105426437A (en) * | 2015-11-05 | 2016-03-23 | 西安翔迅科技有限责任公司 | HBase-based storage method of intelligent transportation field checkpost data |
| CN105374210A (en) * | 2015-12-03 | 2016-03-02 | 中山大学 | Gate detection rate obtaining method based on floating vehicle data |
| CN105374210B (en) * | 2015-12-03 | 2017-08-04 | 中山大学 | Bayonet socket verification and measurement ratio acquisition methods based on floating car data |
| CN105788264A (en) * | 2016-04-28 | 2016-07-20 | 泰华智慧产业集团股份有限公司 | Hbase-based barrier data area collision analysis method and Hbase-based barrier data area collision analysis system |
| CN107862867A (en) * | 2017-11-08 | 2018-03-30 | 泰华智慧产业集团股份有限公司 | Based on big data for the first time enter the method and system of city vehicle analysis |
| CN107862867B (en) * | 2017-11-08 | 2019-08-27 | 泰华智慧产业集团股份有限公司 | The method and system for entering city vehicle analysis for the first time are carried out based on big data |
| CN107993444A (en) * | 2017-11-22 | 2018-05-04 | 紫光捷通科技股份有限公司 | The suspicion car identification of car big data analysis is crossed based on bayonet |
| CN112003956A (en) * | 2020-10-27 | 2020-11-27 | 武汉中科通达高新技术股份有限公司 | Traffic management system |
| CN112003956B (en) * | 2020-10-27 | 2021-01-15 | 武汉中科通达高新技术股份有限公司 | Traffic management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104050271A (en) | Bayonet data processing method based on HBase | |
| CN103631909B (en) | System and method for combined processing of large-scale structured and unstructured data | |
| CN107590250A (en) | A kind of space-time orbit generation method and device | |
| CN106933833B (en) | Method for quickly querying position information based on spatial index technology | |
| CN106484808B (en) | A kind of traffic object holography electronic record Data Modeling Method | |
| CN103577440A (en) | Data processing method and device in non-relational database | |
| CN104834693A (en) | Depth-search-based visual image searching method and system thereof | |
| US10002142B2 (en) | Method and apparatus for generating schema of non-relational database | |
| CN103605651A (en) | Data processing showing method based on on-line analytical processing (OLAP) multi-dimensional analysis | |
| CN103412925A (en) | System and method for integrated searching of structured data and unstructured data | |
| CN105550314A (en) | Data statistics method, device and system based on combined major key | |
| CN110750599B (en) | Associated information extraction and display method based on entity modeling | |
| CN104462421A (en) | Multi-tenant expanding method based on Key-Value database | |
| CN108170752A (en) | metadata management method and system based on template | |
| Ramesh et al. | Data modelling for discrete time series data using Cassandra and MongoDB | |
| CN103425653A (en) | Method and system for realizing DICOM (digital imaging and communication in medicine) image quadratic search | |
| CN105095436A (en) | Automatic modeling method for data of data sources | |
| CN105069094A (en) | Semantic understanding based space keyword indexing method | |
| CN106326358A (en) | Big data-based vehicle passing information inquiry method and device | |
| CN101963993B (en) | Method for fast searching database sheet table record | |
| CN106294792A (en) | The method for building up of correlation inquiry system and set up system | |
| CN112363996B (en) | Method, system and medium for establishing physical model of power grid knowledge graph | |
| CN103425789B (en) | The querying method of a kind of space-time data and device | |
| CN104881475A (en) | Method and system for randomly sampling big data | |
| CN101916260A (en) | Method for establishing semantic mapping between disaster body and relational database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140917 |