CN103793493A - Method and system for processing car-mounted terminal mass data - Google Patents
Method and system for processing car-mounted terminal mass data Download PDFInfo
- Publication number
- CN103793493A CN103793493A CN201410027650.5A CN201410027650A CN103793493A CN 103793493 A CN103793493 A CN 103793493A CN 201410027650 A CN201410027650 A CN 201410027650A CN 103793493 A CN103793493 A CN 103793493A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- cluster
- mounted terminal
- car
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and system for processing car-mounted terminal mass data. The method and system simplify operation and improve query performance and efficiency. The method comprises the steps that an Hadoop server cluster stores real-time service data of a car-mounted terminal to an Hbase data base cluster; a relational data base server cluster stores basic data of the car-mounted terminal to a relational data base cluster; the Hadoop server cluster enables the real-time service data to be cooperatively processed by all region servers corresponding to a management region; the Hadoop server cluster sends the processed data to the relational data base cluster in the relational data base server cluster in an interaction mode. On the one hand, the advantages of a traditional relational data base and naturally-distributed Hadoop cloud storage are effectively and reasonably combined, storage and retrieval problems of various characteristics of mass data of Internet of Things are solved, and the problems of query performance of a traditional solving scheme are merged; on the other hand, load balances of servers in the Hadoop server cluster is achieved, and the performance of the whole system is improved.
Description
Technical field
The present invention relates to cloud computing field, be specifically related to a kind of method and system of processing car-mounted terminal mass data.
Background technology
Some car-mounted terminal, because depended on vehicle most time is in the middle of travelling, often produces quantity and crosses hundred million grades, data volume and reach the record of TB level.For example, produce data p.s. with a car-mounted terminal, the conservative estimation of 100 bytes of every data, so, hour 3600 records, 0.34MB data, the every chassis of conservative estimation on average starts 4 hours every day, the quantity that records producing for one day is: 4*60*60=14400 bar, the i.e. data of (14400*100)/(1024*1024)=2.04MB.1,000,000 car-mounted terminals will produce 14,400,000,000 records every day so, and the data of 2TB, produce 5,000,000,000,000 record every year, the data of TB more than 600.Because data volume is huge, how to store and how to process, how to guarantee high handling property, guarantee that again high fault tolerance energy etc. is all the problem that industry is urgently to be resolved hurrily and process.
Due to traditional relational database cannot be in the single table in single storehouse Storage and Processing number in 10,000,000,000 record, single storehouse multilist also cannot store number with 10,000,000,000 and data volume reach the record of 2TB, a kind of method of processing car-mounted terminal mass data that prior art provides is to carry out the data of storage of collected with traditional relational database point storehouse submeter.Divide the strategy of storehouse submeter to be processed by the data access layer (Data Access Layout, DAL) of database completely, comprise that the backup of data and the disaster recovery of database etc. all rely on manual processing, the analysis of data goes to realize one's very own.
The major defect of above-mentioned prior art is: the first, after data volume and load increase, add the operation more complicated of the equipment such as server; The second, some range query need to be accessed nearly all subregion, performance and the efficiency of impact inquiry.
Summary of the invention
The embodiment of the present invention provides a kind of method and system of processing car-mounted terminal mass data, to simplify the operation and to improve query performance and efficiency.
The embodiment of the present invention provides a kind of method of processing car-mounted terminal mass data, and described method comprises:
The real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster;
Relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal;
Described real time business data are transferred to each and the corresponding region server associated treatment in management area by described Hadoop server cluster;
Described Hadoop server cluster is the relational database cluster to described relational database server cluster by data interaction after treatment.
Another embodiment of the present invention provides a kind of system of processing car-mounted terminal mass data, and described system comprises Hadoop server cluster and relational database server cluster;
Described Hadoop server cluster, for being stored in Hbase data-base cluster by the real time business data from car-mounted terminal;
Described relational database server cluster, for being stored in relational database cluster from the basic data of car-mounted terminal;
Described Hadoop server cluster is also for described real time business data being transferred to each and the corresponding region server associated treatment in management area, the relational database cluster by data interaction after treatment to described relational database server cluster.
From the invention described above embodiment, on the one hand, because the real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster, relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal, by effectively and reasonably store this advantage of two kinds and solved storage and the search problem of the various features of Internet of Things mass data in conjunction with the Hadoop cloud of traditional relational database and natural distributed formula, merger the performance difficult problem of inquiry for traditional solution; On the other hand, real time business data are transferred to each and the corresponding region server associated treatment in management area by Hadoop server cluster, realized the equilibrium of server load in Hadoop server cluster, improved the performance of whole system.
Accompanying drawing explanation
Fig. 1 is the basic procedure schematic diagram of the method for the processing car-mounted terminal mass data that provides of the embodiment of the present invention;
Fig. 2 is the system architecture schematic diagram of the processing car-mounted terminal mass data that provides of the embodiment of the present invention;
Fig. 3 is the embodiment of the present invention corresponding logic function configuration diagram of system that provide and processing car-mounted terminal mass datas accompanying drawing 2 examples;
Fig. 4 is the embodiment of the present invention corresponding physical equipment configuration diagram of system that provide and processing car-mounted terminal mass datas accompanying drawing 2 examples;
Fig. 5 is the system architecture schematic diagram of the processing car-mounted terminal mass data that provides of another embodiment of the present invention.
Embodiment
The embodiment of the present invention provides a kind of method of processing car-mounted terminal mass data, and described method comprises: the real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster; Relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal; Described real time business data are transferred to each and the corresponding region server associated treatment in management area by described Hadoop server cluster; Described Hadoop server cluster is the relational database cluster to described relational database server cluster by data interaction after treatment.The embodiment of the present invention also provides the system of corresponding processing car-mounted terminal mass data.Below be elaborated respectively.
The basic procedure of the method for the processing car-mounted terminal mass data of the embodiment of the present invention can be with reference to figure 1, mainly comprises that step S101 is to step S104:
S101, the real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster.
Hadoop is the project of increasing income below Apache foundation, wherein comprise a fairly large number of sub-project, Hadoop server cluster is a distributed server cluster, including Hbase data-base cluster wherein, the system of formation is a cloud storage system, Hbase database in Hbase data-base cluster is one of Hadoop sub-project, for distributed, towards row the database of increasing income.In embodiments of the present invention, comprise GPS information, vehicle real-time status data and remote diagnosis information etc. from the real time business data of car-mounted terminal, these information, as basic data, can be stored in Hbase data-base cluster.
S102, relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal.
Relational database server cluster is the set of a group relational database server, and each relational database server is equipped with relational database, for example, and MySQL database.Comprise software, time-zone information and the system journal etc. of facility information, user profile, user's automobile threshold value (comprising the frequency of electronics barrier, hypervelocity threshold value and various data acquisitions etc.), download from the basic data of car-mounted terminal.
S103, real time business data are transferred to each and the corresponding region server associated treatment in management area by Hadoop server cluster.
Particularly, as one embodiment of the invention, Hadoop server cluster transfers to each and the corresponding region server associated treatment in management area to comprise the steps that S1031 is to step S1034 real time business data:
S1031, region server obtains after the described real time business data from car-mounted terminal, transfers to region example to carry out I/O I/O operation described real time business data.
In the distributed cloud storage system being formed by Hadoop server cluster, comprise a master server (Master Server), in each moment, Hbase data-base cluster only has a master server program HMaster of master server in operation.In Hadoop server cluster, carry out management area server (being RegionServer) with master server, each region (being Region) distributed to corresponding region server by master server, and the load of coordination area domain server and the state of cluster.From start to finish, master server can externally not provide data, services, as for the read-write requests of data, is solely responsible for processing by region server.
Work as car-mounted terminal, for example RCU-U, the real time business data of collecting are sent to after Hadoop server cluster by network, first associated component goes to obtain in HMaster the position at place, region, then these data are transferred to relevant district's server to process, district's server is taken after request of data, gives corresponding region example carry out I/O (Input/Output, I/O) operation by these data.
For after can playback of data and synchrodata when abnormal, in embodiments of the present invention, region server is transferred to real time business data after region example carries out I/O operation, the result of I/O operation is write User operation log by region server.
S1032, region server is by the result write memory of I/O operation.
In embodiments of the present invention, internal memory is MenStore, is a region of memory that is specifically designed to the file of storage hbase specific format.
S1033, according to the size of described data in EMS memory amount, region server forms the second storage file by the Refresh Data in described internal memory to disk and by Piece file mergence.
Particularly, comprise step S10331 and step S10332,
S10331, in the time that data in EMS memory amount exceedes Second Threshold, the Refresh Data in internal memory to disk is formed the first storage file by region server.
Particularly, when the data volume accumulative total in internal memory exceedes Second Threshold, for example, when a predefined threshold values, independent thread of region server unlatching in disk, forms a storage file StoreFile by the Refresh Data in internal memory.In order to distinguish, the storage file now forming is called to the first storage file here.
S10332, in the time that the quantity of the first storage file exceedes the 3rd threshold value, merges into the second storage file by multiple the first storage files.
In the time that the quantity of multiple the first storage files in step S10331 exceedes the 3rd threshold value, just this is multiplely merged into a new storage file while exceeding the 3rd threshold value.In order to distinguish with aforesaid the first storage file, the new storage file after merging is called to the second storage file here.
Certainly, in the process merging, can carry out versions merging and data deletion etc., that is to say that what in Hbase database, deposit is last incremental data, so upgrade and deletion action be all after merging process in carry out, this is also advantage of the present invention and feature, because raw data only has incremental data, there is no the logic of revising and deleting, its reason is that Data Source is car-mounted terminal, for example RCU-U, and car-mounted terminal real-time data inserting of meeting in operational process does not have the logic of revising and deleting.
S1034, in the time that the capacity of the second storage file exceedes first threshold, current management area is divided into two management areas by the master server in Hadoop server cluster, and corresponding two region server processing are transferred to respectively in two management areas.
When the size of single storage file is when the capacity of the second storage file exceedes first threshold, current management area is divided into two management areas by master server in Hadoop server cluster, then by HMaster, corresponding two region server processing are transferred to respectively in described two management areas cutting apart gained, thus the load balancing of realization.
S104, Hadoop server cluster is the relational database cluster to relational database server cluster by data interaction after treatment.
Hadoop server cluster is after data processing, can be in business scenario data interaction after treatment to the relational database cluster of relational database server cluster.For example, think to check from certain dimension frequently the historical statistics information of certain vehicle later, for the consideration of efficiency, Hadoop server cluster is the relational database cluster to relational database server cluster by data interaction after treatment, for example, the relational database cluster being formed by numerous MySQL databases.
As for the position at the place, all districts (being sublist) of each table, leave in metadata (meta) table.Along with the increase of sublist, the data volume of the meta table of deposit position relation also can become very huge thereupon.For the consideration of performance, also meta table can be divided into multiple sublists and it is put into corresponding management region.In order to navigate to meta sublist, HBase database can show to deposit with Root the positional information of meta sublist.As for Root table somewhere, can in ZooKeeper, record.For example, so each storing cart mounted terminal, when the recording of RCU-U, ZooKeeper is the only way which must be passed, only in ZooKeeper, finds relevant positional information, could specified data where act on, and the ZooKeeper generally acknowledged coordination service instrument that is industry is quite reliable.
The method of the processing car-mounted terminal mass data providing from the invention described above embodiment, on the one hand, because the real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster, relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal, by effectively and reasonably store this advantage of two kinds and solved storage and the search problem of the various features of Internet of Things mass data in conjunction with the Hadoop cloud of traditional relational database and natural distributed formula, merger the performance difficult problem of inquiry for traditional solution, on the other hand, real time business data are transferred to each and the corresponding region server associated treatment in management area by Hadoop server cluster, realized the equilibrium of server load in Hadoop server cluster, improved the performance of whole system.
The system of the processing car-mounted terminal mass data of the embodiment of the present invention to the method for carrying out above-mentioned processing car-mounted terminal mass data describes below, and its basic structure is with reference to figure 2.For convenience of explanation, only show the part relevant to the embodiment of the present invention.The system of the processing car-mounted terminal mass data of accompanying drawing 2 examples mainly comprises Hadoop server cluster 201 and relational database server cluster 202, is described in detail as follows:
Hadoop server cluster 201, for being stored in Hbase data-base cluster by the real time business data from car-mounted terminal.
Relational database server cluster 202, for being stored in relational database cluster from the basic data of car-mounted terminal.
Hadoop server cluster 201 is also for described real time business data being transferred to each and the corresponding region server associated treatment in management area, the relational database cluster by data interaction after treatment to described relational database server cluster.
Accompanying drawing 3 has provided the corresponding logic function framework of system with the processing car-mounted terminal mass data of accompanying drawing 2 examples, in this framework, cloud storage system, for example the data of RCU cloud storage system storage comprise GPS information, vehicle real-time status data, remote diagnosis information, User operation log record, cloud storage system, the for example application on RCU cloud storage system comprises facility information storage, user profile storage, user's automobile threshold values storage (electronics barrier, the frequency of hypervelocity threshold values and various data acquisitions etc.), software is downloaded, API of system journal storage and third party's application etc.
Accompanying drawing 4 has provided the corresponding physical equipment framework of system with the processing car-mounted terminal mass data of accompanying drawing 2 examples.In the corresponding physical equipment framework of system of the processing car-mounted terminal mass data of accompanying drawing 4 examples, communication cluster is by cloud storage system layer, for example, the api interface that RCU cloud storage system layer provides by car-mounted terminal (for example, the data of the real time business RCU-U) gathering are inserted into Hbase database, cloud storage, for example RCU cloud storage door adopts classical MVC three-decker, and the one deck changing wherein just can meet the change of applying.The model layer that the operation flow of an application or the change of business rule only need be changed MVC just can realize separating of display module and functional module.These features have improved maintainability, portability, extensibility and the reusability of program, have reduced the development difficulty of program.The concept of key-course is also very effective, and because it has been combined different requests different models and different sets of views, therefore, key-course can be described as the concept that has comprised user's request permissions.Finally, it also helps software engineering management.Due to different layers, Each performs its own functions, and the different application of every one deck has some identical feature, is conducive to produce supervisory routine code by through engineering approaches, tool.According to user's request, decision is obtain data or obtain data or both have from Hbase data-base cluster from relational database cluster.
In the system of the processing car-mounted terminal mass data of accompanying drawing 2 examples, Hadoop server cluster 201 comprises master server 501 and at least one region server 502, the system of the processing car-mounted terminal mass data that another embodiment of the present invention provides as shown in Figure 5.In the system of the processing car-mounted terminal mass data of accompanying drawing 5 examples, Hadoop server cluster 201 is also for transferring to real time business data each and the corresponding region server associated treatment in management area to be specially: region server 502 is for obtaining after the described real time business data from car-mounted terminal, transfer to region example to carry out I/O I/O operation described real time business data, by the result write memory of described I/O operation, according to the size of described data in EMS memory amount, Refresh Data in described internal memory is formed to the second storage file to disk and by Piece file mergence, master server 501 is in the time that the capacity of described the second storage file exceedes first threshold, current management area is divided into two management areas, corresponding two region server processing are transferred to respectively in two management areas.
In the system of the processing car-mounted terminal mass data of accompanying drawing 5 examples, region server 502 is for according to the size of described data in EMS memory amount, by the Refresh Data in described internal memory to disk and form the second storage file by Piece file mergence and be specially: region server 502 is in the time that described data in EMS memory amount exceedes Second Threshold, Refresh Data in described internal memory to disk is formed to the first storage file, in the time that the quantity of described the first storage file exceedes the 3rd threshold value, multiple described the first storage files are merged into described the second storage file.
In the system of the processing car-mounted terminal mass data of accompanying drawing 5 examples, region server 502 also when multiple described the first storage files are merged into described the second storage file, merges and data is deleted version.
In the system of the processing car-mounted terminal mass data of accompanying drawing 5 examples, region server 502 is also for obtaining after the described real time business data from car-mounted terminal, described real time business data are transferred to after region example carries out I/O I/O operation, the result of described I/O operation to be write to User operation log.
As on the other hand, yet another embodiment of the invention also provides a kind of computer-readable recording medium, and this computer-readable recording medium can be the computer-readable recording medium comprising in the storer in above-described embodiment; Also can be individualism, be unkitted the computer-readable recording medium of allocating in terminal.Described computer-readable recording medium stores more than one or one program, and described more than one or one program is used for carrying out a method of processing car-mounted terminal mass data by one or more than one processor, and described method comprises:
The real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster;
Relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal;
Described real time business data are transferred to each and the corresponding region server associated treatment in management area by described Hadoop server cluster;
Described Hadoop server cluster is the relational database cluster to described relational database server cluster by data interaction after treatment.
Suppose that above-mentioned is the possible embodiment of the first, in the possible embodiment of the first embodiment possible as the second basic and that provide, described real time business data are transferred to each and the corresponding region server associated treatment in management area by described Hadoop server cluster, comprising:
Described region server obtains after the described real time business data from car-mounted terminal, transfers to region example to carry out I/O I/O operation described real time business data;
Described region server is by the result write memory of described I/O operation;
According to the size of described data in EMS memory amount, described region server forms the second storage file by the Refresh Data in described internal memory to disk and by Piece file mergence;
In the time that the capacity of described the second storage file exceedes first threshold, current management area is divided into two management areas by the master server in described Hadoop server cluster, and corresponding two region server processing are transferred to respectively in described two management areas.
Suppose that above-mentioned is the possible embodiment of the second, in the third the possible embodiment providing as basis at the possible embodiment of the second, described according to the size of described data in EMS memory amount, described region server forms the second storage file by the Refresh Data in described internal memory to disk and by Piece file mergence, comprising:
In the time that described data in EMS memory amount exceedes Second Threshold, the Refresh Data in described internal memory to disk is formed the first storage file by described region server;
In the time that the quantity of described the first storage file exceedes the 3rd threshold value, multiple described the first storage files are merged into described the second storage file.
Suppose above-mentionedly for the third possible embodiment, in the 4th kind of possible embodiment providing as basis at the third possible embodiment, describedly multiple described the first storage files are merged into described the second storage file also comprise:
Version is merged and data are deleted.
Suppose that above-mentioned is the possible embodiment of the second, in the 5th kind of possible embodiment providing as basis at the possible embodiment of the second, described region server obtains after the described real time business data from car-mounted terminal, transfers to region example to carry out also comprising after I/O I/O operation described real time business data:
The result of described I/O operation is write User operation log by described region server.
It should be noted that, the content such as information interaction, implementation between the each module/unit of said apparatus, due to the inventive method embodiment based on same design, its technique effect bringing is identical with the inventive method embodiment, particular content can, referring to the narration in the inventive method embodiment, repeat no more herein.
One of ordinary skill in the art will appreciate that all or part of step in the whole bag of tricks of above-described embodiment is can carry out the hardware that instruction is relevant by program to complete, this program can be stored in a computer-readable recording medium, storage medium can comprise: ROM (read-only memory) (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or CD etc.
A kind of method and system of processing car-mounted terminal the mass data above embodiment of the present invention being provided is described in detail, applied specific case herein principle of the present invention and embodiment are set forth, the explanation of above embodiment is just for helping to understand method of the present invention and core concept thereof; , for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention meanwhile.
Claims (10)
1. a method of processing car-mounted terminal mass data, is characterized in that, described method comprises:
The real time business data from car-mounted terminal are stored in Hbase data-base cluster by Hadoop server cluster;
Relational database server cluster will be stored in relational database cluster from the basic data of car-mounted terminal;
Described real time business data are transferred to each and the corresponding region server associated treatment in management area by described Hadoop server cluster;
Described Hadoop server cluster is the relational database cluster to described relational database server cluster by data interaction after treatment.
2. method according to claim 1, is characterized in that, described real time business data are transferred to each and the corresponding region server associated treatment in management area by described Hadoop server cluster, comprising:
Described region server obtains after the described real time business data from car-mounted terminal, transfers to region example to carry out I/O I/O operation described real time business data;
Described region server is by the result write memory of described I/O operation;
According to the size of described data in EMS memory amount, described region server forms the second storage file by the Refresh Data in described internal memory to disk and by Piece file mergence;
In the time that the capacity of described the second storage file exceedes first threshold, current management area is divided into two management areas by the master server in described Hadoop server cluster, and corresponding two region server processing are transferred to respectively in described two management areas.
3. method according to claim 2, is characterized in that, described according to the size of described data in EMS memory amount, and described region server forms the second storage file by the Refresh Data in described internal memory to disk and by Piece file mergence, comprising:
In the time that described data in EMS memory amount exceedes Second Threshold, the Refresh Data in described internal memory to disk is formed the first storage file by described region server;
In the time that the quantity of described the first storage file exceedes the 3rd threshold value, multiple described the first storage files are merged into described the second storage file.
4. method according to claim 3, is characterized in that, describedly multiple described the first storage files are merged into described the second storage file also comprises:
Version is merged and data are deleted.
5. method according to claim 2, is characterized in that, described region server obtains after the described real time business data from car-mounted terminal, transfers to region example to carry out also comprising after I/O I/O operation described real time business data:
The result of described I/O operation is write User operation log by described region server.
6. a system of processing car-mounted terminal mass data, is characterized in that, described system comprises Hadoop server cluster and relational database server cluster;
Described Hadoop server cluster, for being stored in Hbase data-base cluster by the real time business data from car-mounted terminal;
Described relational database server cluster, for being stored in relational database cluster from the basic data of car-mounted terminal;
Described Hadoop server cluster is also for described real time business data being transferred to each and the corresponding region server associated treatment in management area, the relational database cluster by data interaction after treatment to described relational database server cluster.
7. system according to claim 6, it is characterized in that, described Hadoop server cluster comprises master server and at least one region server, and described Hadoop server cluster is also for transferring to described real time business data each and the corresponding region server associated treatment in management area to be specially:
Described region server, for obtaining after the described real time business data from car-mounted terminal, transfer to region example to carry out I/O I/O operation described real time business data, by the result write memory of described I/O operation, according to the size of described data in EMS memory amount, the Refresh Data in described internal memory is formed to the second storage file to disk and by Piece file mergence;
Described master server, in the time that the capacity of described the second storage file exceedes first threshold, is divided into two management areas by current management area, and corresponding two region server processing are transferred to respectively in described two management areas.
8. system according to claim 7, it is characterized in that, described region server is used for according to the size of described data in EMS memory amount, by the Refresh Data in described internal memory to disk and form the second storage file by Piece file mergence and be specially: described region server is in the time that described data in EMS memory amount exceedes Second Threshold, Refresh Data in described internal memory to disk is formed to the first storage file, in the time that the quantity of described the first storage file exceedes the 3rd threshold value, multiple described the first storage files are merged into described the second storage file.
9. system according to claim 8, is characterized in that, described region server also when multiple described the first storage files are merged into described the second storage file, merges and data are deleted version.
10. system according to claim 7, it is characterized in that, described region server is also for obtaining after the described real time business data from car-mounted terminal, described real time business data are transferred to after region example carries out I/O I/O operation, the result of described I/O operation to be write to User operation log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410027650.5A CN103793493B (en) | 2014-01-21 | 2014-01-21 | A kind of method and system for handling car-mounted terminal mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410027650.5A CN103793493B (en) | 2014-01-21 | 2014-01-21 | A kind of method and system for handling car-mounted terminal mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103793493A true CN103793493A (en) | 2014-05-14 |
| CN103793493B CN103793493B (en) | 2017-12-29 |
Family
ID=50669159
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410027650.5A Active CN103793493B (en) | 2014-01-21 | 2014-01-21 | A kind of method and system for handling car-mounted terminal mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103793493B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599092A (en) * | 2015-02-25 | 2015-05-06 | 北京嘀嘀无限科技发展有限公司 | Order business monitoring method and equipment |
| CN105243109A (en) * | 2015-09-25 | 2016-01-13 | 杭州华为数字技术有限公司 | Data backup method and data processing system |
| CN106021574A (en) * | 2016-05-27 | 2016-10-12 | 安徽四创电子股份有限公司 | Data storage replication method and system |
| CN106126553A (en) * | 2016-06-16 | 2016-11-16 | 西安科技大市场有限公司 | A kind of storage method based on the big data of scientific and technological resources |
| CN107092691A (en) * | 2017-04-24 | 2017-08-25 | 中科院微电子研究所昆山分所 | A kind of date storage method and system based on car networking |
| CN108304142A (en) * | 2017-12-29 | 2018-07-20 | 杭州华为数字技术有限公司 | A kind of data managing method and device |
| CN108460054A (en) * | 2017-02-22 | 2018-08-28 | 北京京东尚科信息技术有限公司 | A kind of mthods, systems and devices improving cloud storage system performance |
| CN108763562A (en) * | 2018-06-04 | 2018-11-06 | 广东京信软件科技有限公司 | A kind of construction method based on big data skill upgrading data exchange efficiency |
| CN109343500A (en) * | 2018-11-28 | 2019-02-15 | 上海风语筑展示股份有限公司 | A kind of centralization shows monitoring nodes management system and method |
| CN110019528A (en) * | 2017-12-26 | 2019-07-16 | 中国移动通信集团湖北有限公司 | Database manipulation load-balancing method, device, equipment and medium |
| US11132260B2 (en) | 2015-09-25 | 2021-09-28 | Huawei Technologies Co., Ltd. | Data processing method and apparatus |
| US20220012213A1 (en) * | 2016-03-08 | 2022-01-13 | International Business Machines Corporation | Spatial-temporal storage system, method, and recording medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102739775A (en) * | 2012-05-29 | 2012-10-17 | 宁波东冠科技有限公司 | Method for monitoring and managing Internet of Things data acquisition server cluster |
| US20120271903A1 (en) * | 2011-04-19 | 2012-10-25 | Michael Luna | Shared resource and virtual resource management in a networked environment |
| CN102929933A (en) * | 2012-09-21 | 2013-02-13 | 北京世纪高通科技有限公司 | Data processing method and device |
-
2014
- 2014-01-21 CN CN201410027650.5A patent/CN103793493B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120271903A1 (en) * | 2011-04-19 | 2012-10-25 | Michael Luna | Shared resource and virtual resource management in a networked environment |
| CN102739775A (en) * | 2012-05-29 | 2012-10-17 | 宁波东冠科技有限公司 | Method for monitoring and managing Internet of Things data acquisition server cluster |
| CN102929933A (en) * | 2012-09-21 | 2013-02-13 | 北京世纪高通科技有限公司 | Data processing method and device |
Non-Patent Citations (3)
| Title |
|---|
| HTTP://WWW.BINOSPACE.COM/INDEX.PHP/HBASE-IN-DEPTH-ANALYSIS-OF-TH: "HBase深入分析之RegionServer", 《HTTP://WWW.BINOSPACE.COM/INDEX.PHP/HBASE-IN-DEPTH-ANALYSIS-OF-THE-REGIONSERVER/?UTM_SOURCE=TUICOOL&UTM_MEDIUM=REFERRAL》 * |
| 刘晓晓: "《网络系统集成》", 31 January 2012, 清华大学出版社 * |
| 王仁阳等: "基于HBase的车辆监控系统", 《中国科技论文在线》 * |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599092B (en) * | 2015-02-25 | 2019-02-12 | 北京嘀嘀无限科技发展有限公司 | For monitoring the method and apparatus of order business |
| CN104599092A (en) * | 2015-02-25 | 2015-05-06 | 北京嘀嘀无限科技发展有限公司 | Order business monitoring method and equipment |
| US11132260B2 (en) | 2015-09-25 | 2021-09-28 | Huawei Technologies Co., Ltd. | Data processing method and apparatus |
| US11119863B2 (en) | 2015-09-25 | 2021-09-14 | Huawei Technologies Co., Ltd. | Data backup method and data processing system |
| CN105243109B (en) * | 2015-09-25 | 2021-10-15 | 华为技术有限公司 | Data backup method and data processing system |
| CN105243109A (en) * | 2015-09-25 | 2016-01-13 | 杭州华为数字技术有限公司 | Data backup method and data processing system |
| US20220012213A1 (en) * | 2016-03-08 | 2022-01-13 | International Business Machines Corporation | Spatial-temporal storage system, method, and recording medium |
| CN106021574A (en) * | 2016-05-27 | 2016-10-12 | 安徽四创电子股份有限公司 | Data storage replication method and system |
| CN106126553A (en) * | 2016-06-16 | 2016-11-16 | 西安科技大市场有限公司 | A kind of storage method based on the big data of scientific and technological resources |
| CN108460054A (en) * | 2017-02-22 | 2018-08-28 | 北京京东尚科信息技术有限公司 | A kind of mthods, systems and devices improving cloud storage system performance |
| CN107092691A (en) * | 2017-04-24 | 2017-08-25 | 中科院微电子研究所昆山分所 | A kind of date storage method and system based on car networking |
| CN110019528A (en) * | 2017-12-26 | 2019-07-16 | 中国移动通信集团湖北有限公司 | Database manipulation load-balancing method, device, equipment and medium |
| CN108304142A (en) * | 2017-12-29 | 2018-07-20 | 杭州华为数字技术有限公司 | A kind of data managing method and device |
| CN108304142B (en) * | 2017-12-29 | 2021-10-15 | 华为技术有限公司 | Data management method and device |
| CN108763562A (en) * | 2018-06-04 | 2018-11-06 | 广东京信软件科技有限公司 | A kind of construction method based on big data skill upgrading data exchange efficiency |
| CN109343500A (en) * | 2018-11-28 | 2019-02-15 | 上海风语筑展示股份有限公司 | A kind of centralization shows monitoring nodes management system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103793493B (en) | 2017-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103793493A (en) | Method and system for processing car-mounted terminal mass data | |
| JP7410181B2 (en) | Hybrid indexing methods, systems, and programs | |
| US10942813B2 (en) | Cloud object data layout (CODL) | |
| US10013440B1 (en) | Incremental out-of-place updates for index structures | |
| Sumbaly et al. | The big data ecosystem at linkedin | |
| CN107544984B (en) | Data processing method and device | |
| US11314717B1 (en) | Scalable architecture for propagating updates to replicated data | |
| CN106484906B (en) | Distributed object storage system flash-back method and device | |
| CN107423422B (en) | Spatial data distributed storage and search method and system based on grid | |
| CN107657049B (en) | Data processing method based on data warehouse | |
| US11977532B2 (en) | Log record identification using aggregated log indexes | |
| CN107835983A (en) | Backup-and-restore is carried out in distributed data base using consistent database snapshot | |
| CN113535856B (en) | Data synchronization method and system | |
| CN104714878B (en) | A kind of method and device of collector journal data | |
| CN105303456A (en) | Method for processing monitoring data of electric power transmission equipment | |
| CN111881223B (en) | Data management method, device, system and storage medium | |
| CN110347651A (en) | Method of data synchronization, device, equipment and storage medium based on cloud storage | |
| CN103064933A (en) | Data query method and system | |
| CN102890678A (en) | Gray-code-based distributed data layout method and query method | |
| CN113468199B (en) | Index updating method and system | |
| CN106802928B (en) | Power grid historical data management method and system | |
| CN106708911A (en) | Method and device for synchronizing data files in cloud environment | |
| CN115858488A (en) | Parallel migration method and device based on data governance and readable medium | |
| CN115599871A (en) | Lake and bin integrated data processing system and method | |
| CN110769062A (en) | Distributed storage remote disaster recovery method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |