CN104090901A - Method, device and server for processing data - Google Patents
Method, device and server for processing data Download PDFInfo
- Publication number
- CN104090901A CN104090901A CN201310756095.5A CN201310756095A CN104090901A CN 104090901 A CN104090901 A CN 104090901A CN 201310756095 A CN201310756095 A CN 201310756095A CN 104090901 A CN104090901 A CN 104090901A
- Authority
- CN
- China
- Prior art keywords
- data
- checked
- storage engines
- stored
- database table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
Abstract
The invention discloses a method, a device and a server for processing data, which belongs to the field of computers. The method comprises the following steps: receiving a checking sentence transmitted from a terminal, wherein the checking sentence carries storage position information of data to be checked and a processing mode for processing data to be checked; according to the storage position information of the data to be checked, confirming whether a databank for storing the data to be checked is a data warehouse or a storage engine; if the databank is a data warehouse, transmitting the checking sentence to the data warehouse, and enabling the data warehouse to acquire the data to be checked and process the data to be checked in the processing mode; if the databank is a storage engine, transmitting the checking sentence to the storage engine, and enabling the storage engine to acquire the data to be checked and process the data to be checked in the processing mode. Therefore, the storage engine and the data warehouse can bring the advantages of selves into play respectively, the steps and the time for processing the data can be saved, and the waiting time of a user and the system resource are saved to a great extent.
Description
Technical field
The present invention relates to computer realm, particularly a kind of method that data are processed, device and server.
Background technology
Data warehouse is a kind of for storing the data system of data and having the function of data analysis, and Database Systems using it as backstage, internet usually at present, for storing the data of magnanimity.And the data analysis function that data warehouse has can provide data processing service for user.
At present, user need to inquire about and during deal with data, user sends to data warehouse by the memory location of these data and processing mode, and data warehouse obtains this data according to this memory location, and according to this processing mode, these data is carried out to data.For example, the wage that the data Wei Mou company of user's inquiry provided in 2013, the table name of storing the payroll of these data is the memory location of these data, and processing mode is the total wages that each employee of statistics got in 2013, data warehouse obtains according to this table name the wage that the said firm provided in 2013, and counts according to this processing mode the total wages that each employee got in 2013.
Wherein, the time rank that data warehouse is processed data is minute rank, when the larger large data of quantity are processed, need to plant somewhat or the longer time, when the less small data of data volume is processed, also need to plant somewhat or the longer time, this processing time concerning small data is oversize, cannot meet the requirement of small data to the processing time, treatment effeciency is lower.
Summary of the invention
In order to improve the efficiency that data are processed, the invention provides a kind of method that data are processed, device and server.Described technical scheme is as follows:
The method that data are processed, described method comprises:
The query statement that receiving terminal sends, described query statement carries the stored position information of data to be checked and processes the processing mode of described data to be checked, and described stored position information at least comprises the table name of described data place database table to be checked;
According to the stored position information of described data to be checked, determine the database of the described data to be checked of storage, described database comprises data warehouse or storage engines;
If determined as data warehouse, described query statement is sent to described data warehouse, make described data warehouse according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed;
If determined as storage engines, the storage engines that described query statement is sent to the described data to be checked of storage, makes described storage engines according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed.
The device that data are processed, described device comprises:
The first receiver module, the query statement sending for receiving terminal, described query statement carries the stored position information of data to be checked and processes the processing mode of described data to be checked, and described stored position information at least comprises the table name of described data place database table to be checked;
The first determination module, for according to the stored position information of described data to be checked, determines the database of the described data to be checked of storage, and described database comprises data warehouse or storage engines;
The first sending module, if for determining as data warehouse, described query statement is sent to described data warehouse, make described data warehouse according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed;
The second sending module, if for determining as storage engines, the storage engines that described query statement is sent to the described data to be checked of storage, makes described storage engines according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed.
The server that data are processed, described server comprises:
One or more processors; With
Storer;
Described memory stores has one or more programs, and described one or more programs are configured to be carried out by described one or more processors, and described one or more routine packages contain for carrying out the instruction of a kind of described method that data are processed.
In embodiments of the present invention, when the larger large data of data volume are processed, on the processing time, data warehouse has great advantage with respect to storage engines tool, and when the less small data of data volume is processed, on the processing time, storage engines has great advantage with respect to data warehouse tool; Load store engine in data warehouse, make storage engines and data warehouse can bring into play advantage separately, short slab while having solved the less small data of data warehouse deal with data amount on the processing time, can save treatment step and processing time that data are processed, greatly degree has reduced user's stand-by period and has saved system resource.
Accompanying drawing explanation
Fig. 1 is a kind of storage system that the embodiment of the present invention provides;
Fig. 2 is a kind of method flow diagram that data are processed that the embodiment of the present invention 1 provides;
Fig. 3-1st, a kind of method flow diagram of storing small data that the embodiment of the present invention 2 provides;
Fig. 3-2nd, a kind of method flow diagram of storing large data that the embodiment of the present invention 2 provides;
Fig. 3-3rd, a kind of method flow diagram of storing data that the embodiment of the present invention 2 provides;
Fig. 3-4th, a kind of inquiry that the embodiment of the present invention 2 provides the method flow diagram of deal with data;
Fig. 3-5th, a kind of inquiry that the embodiment of the present invention 2 provides the method flow diagram of deal with data;
Fig. 4 is a kind of apparatus structure schematic diagram that data are processed that the embodiment of the present invention 3 provides;
Fig. 5 is a kind of server architecture schematic diagram that data are processed that the embodiment of the present invention 4 provides.
Embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing, embodiment of the present invention is described further in detail.
Referring to Fig. 1, the embodiment of the present invention provides a kind of storage system, comprising:
Control end 1, data warehouse 2 and at least one storage engines 3, control end 1 can be integrated in data warehouse 2.
Data warehouse is a kind of for storing the data system of data and having the function of data analysis, it is when processing the data of storage, processing procedure is loaded down with trivial details, be specially: first by the SQL(Structured Query Language receiving, Structured Query Language (SQL)) statement is translated as a plurality of MapReduce(mappings and stipulations programming model) program; Each MapReduce program is process separately of simultaneously initialization start initialized process respectively, from the database table of data warehouse, obtain again pending data, then by pending data importing process, afterwards the data that import are carried out to Map computing and Reduce computing, wherein Map computing is that the data that import are divided into groups by predetermined way, Reduce computing is for processing the data after grouping by processing mode, final Output rusults, the interim result that constantly clean up process obtains in calculating process simultaneously.
Data warehouse can move a plurality of processes simultaneously, and when processing mass data, on the required time, it has very large advantage with respect to storage engines; But no matter data volume is much, all need to carry out according to above-mentioned steps, so required time needs a few minutes at least.
Storage engines can be stored data and can to the data of storage, process according to processing mode.Storage engines has very powerful Statistical information system and query optimizer, the two is in conjunction with can and sorting the Data classification of storage, to allow to clearly obtain the memory location of each class data, speed when data query is very fast, can obtain with the very short time memory location the quick obtaining data to be checked of the data to be checked that need inquiry, the speed that its speed of obtaining data is obtained data than data warehouse is a lot of soon, and storage engines can directly be carried out SQL statement, and can directly according to processing mode, to the data of obtaining, process, and do not need the SQL statement of reception to be translated as a plurality of MapReduce programs, do not need initialization process start initialized process separately, do not need the data of importing to carry out Map computing and Reduce computing, and do not need the interim result that constantly clean up process obtains in calculating process, therefore the speed of obtaining data to be checked of storage engines processing procedure very fast and that the data of obtaining are processed is easy, when the less small data of deal with data amount, expend time in and only need several milliseconds, but storage engines only has a compute node, can only start a process simultaneously, the data that deal with data amount is larger do not possess advantage, conventionally need to spend dozens of minutes.
Therefore, when the less small data of deal with data amount, the required time of data warehouse 2 deal with data is greater than the required time of storage engines 3 deal with data, for example, the required time of data warehouse 2 deal with data is a few minutes, is greater than storage engines 3 deal with data required time in several seconds; When therefore the less small data of data volume is processed, on the processing time, storage engines 3 has great advantage with respect to data warehouse 2 tools.
When the larger large data of deal with data amount, the required time of storage engines 3 deal with data is greater than the required time of data warehouse 2 deal with data, for example, the required time of storage engines 3 deal with data is dozens of minutes, is greater than data warehouse 2 deal with data required a few minutes time; Therefore,, when the larger large data of data volume are processed, on the processing time, data warehouse 2 has great advantage with respect to storage engines 3 tools.
Load store engine 3 in data warehouse 2, make storage engines 3 and data warehouse 2 can bring into play advantage separately, short slab while having solved the less small data of data warehouse 2 deal with data amounts on the processing time, can save treatment step and processing time that data are processed, greatly degree reduces user's stand-by period and saves system resource.
If user need to store the small data that data volume is less, as data volume is less than or equal to the small data of predetermined threshold value, predetermined threshold value can, for 8GB, 10GB or 12GB etc., can be stored in small data in storage engines 3.
If user need to store the large data that data volume is larger, as data volume is greater than the large data of predetermined threshold value, large data can be stored in data warehouse 2.
In the storage system shown in Fig. 1, data warehouse 2 and storage engines 3 be all a kind of for store data data system and can to storage data analysis.
Storage engines 3 is when the less small data of deal with data amount, and the processing time is very short and generally only need several milliseconds, and data warehouse 2 is when the less small data of deal with data amount, and the processing time is very long generally at least needs a few minutes.Therefore,, when the less small data of data volume is processed, on the processing time, storage engines 3 has great advantage with respect to data warehouse 2 tools, can save a large amount of time and system resource.
Data warehouse 2 is when the larger large data of deal with data amount, and the processing time is very short and generally only need a few minutes, and storage engines 3 is when the larger large data of deal with data amount, the very long dozens of minutes that generally at least needs of processing time.Therefore,, when the larger large data of data volume are processed, on the processing time, data warehouse 2 has great advantage with respect to storage engines 3 tools, can save a large amount of time and system resource.
Embodiment 1
Referring to Fig. 2, the embodiment of the present invention provides a kind of method that data are processed, and the method is for the storage system shown in Fig. 1, and the method flow process comprises:
Step 101: the query statement that receiving terminal sends, query statement carries the stored position information of data to be checked and processes the processing mode of data to be checked, and stored position information at least comprises the table name of data place database table to be checked;
Step 102: according to the stored position information of data to be checked, determine the database of the data that storage is to be checked, database comprises data warehouse or storage engines;
Step 103: if determined as data warehouse, query statement is sent to data warehouse, make data warehouse obtain data to be checked according to stored position information and according to processing mode, data to be checked be processed;
Step 104: if determined as storage engines, query statement is sent to the storage engines of storage data to be checked, make storage engines obtain data to be checked according to stored position information and according to processing mode, data to be checked be processed.
In embodiments of the present invention, when the larger large data of data volume are processed, on the processing time, data warehouse has great advantage with respect to storage engines tool, and when the less small data of data volume is processed, on the processing time, storage engines has great advantage with respect to data warehouse tool; Load store engine in data warehouse, make storage engines and data warehouse can bring into play advantage separately, short slab while having solved the less small data of data warehouse deal with data amount on the processing time, can save treatment step and processing time that data are processed, greatly degree has reduced user's stand-by period and has saved system resource.
Embodiment 2
Referring to Fig. 3-1, if user need to store the small data that data volume is less than or equal to predetermined threshold value, this small data is data to be stored, first in storage engines, set up a database table, again data to be stored are stored in the database table of foundation, specifically can realize by following 201 to 209 step.
201: terminal is obtained the sign of the storage engines of user's selection, send first and build predicative sentence to control end; First builds the sign that predicative sentence at least carries this storage engines, the table name of database table to be created, row name and the data type of each row that database table to be created comprises;
When user need to be stored on storage engines by data to be stored, in terminal, select the sign of a storage engines, and row name and the data type of the table name of the database table to be created that need to create on this storage engines to terminal input and each row that database table to be created comprises.
Terminal is obtained the sign of the storage engines of user's selection again, and the table name of database table to be created and row name and data type of each row that database table to be created comprises of receiving user's input, and then transmission first is built predicative sentence to control end.
For example, suppose that predetermined threshold value is 10GB, user need to store the wage data that AA company provided in the January, these wage data are data to be stored, data to be stored only account for 100MB capacity, the data volume of data to be stored is less than predetermined threshold value 10GB, therefore user can be stored in data to be stored in storage engines, be specifically as follows: first user selects the ID1 that is designated of a storage engines in terminal, the table name of the payroll that need to create in this storage engines to terminal input is again " AA company-January-payroll ", and payroll comprise each row row name and data type, this payroll comprises two row, row " employee's name " by name and the data type of first row are " string ", row " wage number " by name and the data type of secondary series are " int ".
Then terminal sends first and builds predicative sentence to control end, and first builds the sign ID1 that predicative sentence at least carries this storage engines, the table name of payroll, row name and the data type of each row that payroll comprises.
202: control end receives first and builds predicative sentence, obtain the first sign of building the storage engines that predicative sentence carries;
Further, control end also obtains first data type of building each row that database table to be created that predicative sentence carries comprises, from the data type of each row, determine the data type that this storage engines is not supported, the data type that the data type conversion of determining is supported for this storage engines, is implemented as follows:
Control end obtains first data type of building each row that database table to be created that predicative sentence carries comprises, according to the first sign of building this storage engines that predicative sentence carries, obtain the data type conversion relation table that this storage engines is corresponding, this data type conversion relation table is for storing the corresponding relation of the data type of data type to be converted and the support of this storage engines, for example, table 1 is the data type conversion relation table that certain storage engines is corresponding, first classifies data type to be converted as, second classifies the data type that this storage engines is supported as, and all data types that this storage engines is supported in secondary series, have been stored.
Again according to this data type conversion relation table, from the data type of each row, determine the data type that this storage engines is not supported, using each data type of determining as data type to be converted, and from this data type conversion relation table, obtain data type corresponding to each data type of determining, each data type of determining is converted to respectively to each self-corresponding data type.For example, a data type of determining is int, using data type int as data type to be converted, and from data type conversion table as shown in table 1, to obtain the data type that data type int is corresponding be integer, and data type int to be converted is converted to data type integer.
Table 1
| The data type that storage engines is supported | Data type to be converted |
| integer | int |
| varchar | string |
| numeric | double |
| bit?boolean | boolean |
| tinyint | tinyint |
| bigint | bigint |
203: control end, according to the sign of storage engines, sends first and builds predicative sentence to storage engines;
Further, control end can also be built the table name of database table to be created and the sign of storage engines that predicative sentence carries by first and be stored in the corresponding relation of the table name of database table and the sign of storage engines, and tool entity is achieved as follows:
Control end is searched and is comprised that first builds the record of the sign of the storage engines that predicative sentence carries in the corresponding relation of the table name of database table and the sign of storage engines, if found, do not comprise that first builds the record of the sign of the storage engines that predicative sentence carries, by first, building the table name of database table to be created and the sign of storage engines that predicative sentence carries is stored in the corresponding relation of the table name of database table and the sign of storage engines, if found, comprise that first builds the record of the sign of the storage engines that predicative sentence carries, in this record, add the first table name of building the database table to be created that predicative sentence carries.
For example, control end is searched the record of the sign ID1 that comprises storage engines in the corresponding relation of the table name of the database table shown in table 2 and the sign of storage engines, and find the record of the sign ID1 that comprises storage engines, the table name " AA company-January-payroll " that adds payroll in this record, obtains table 3
The corresponding relation of the table name of shown database table and the sign of storage engines.
Table 2
| The table name of database table | The sign of storage engines |
| " class " | ID1 |
| " student performance ", " school ", " age per capita " | ID2 |
| " per tour average achievement ", " order commodity " | ID3 |
Table 3
| The table name of database table | The sign of storage engines |
| " class ", " AA company-January-payroll " | ID1 |
| " student performance ", " school ", " age per capita " | ID2 |
| " per tour average achievement ", " order commodity " | ID3 |
204: storage engines receives first and builds predicative sentence, builds predicative sentence create database table to be created according to first;
Particularly, storage engines is determined the columns of database table to be created according to first row name of building each row that database table to be created that predicative sentence carries comprises, according to definite columns, create blank database table, according to first row name of building the table name of the database table to be created that predicative sentence carries and each row that database table to be created comprises, this blank database table is set.
Further, storage engines sends the first notification message to control end, and the first notification message is used for notifying user's success creation database table; Control end receives the first notification message and is transmitted to terminal; Terminal receives the first notification message, and prompting user storage engines has successfully created database table to be created.
Terminal shows that after this information, user submits memory command to terminal, and terminal is stored in the data to be stored in terminal in storage engines by following flow process.
205: the table name of terminal transmission database table and data to be stored are to control end, and the table name of this database table is the table name of above-mentioned database table to be created;
For example, data to be stored are employee's name " Zhang San " and wage " 1000 yuan " corresponding to Zhang San, the wage " 2000 yuan " that employee's name " Li Si " and Li Si are corresponding, and the wage " 3000 yuan " of employee's name " king five " and king's five correspondences, and the table name of data place database table to be stored is " AA company-January-payroll "; The row " employee's name " by name of the row that " Zhang San ", " Li Si " and " king five " are corresponding, the row " wage number " by name of the row that " 1000 yuan ", " 2000 yuan " and " 3000 yuan " are corresponding.
206: control end receives the table name of this database table and data to be stored;
For example, control end receives the table name " AA company-January-payroll " of data to be stored and data place database table to be stored.
207: control end, according to the table name of this database table, obtains for storing the sign of the storage engines of data to be stored;
Particularly, control end is according to the table name of this database table, look-up table list of file names, table name list is for storing the table name of each database table that data warehouse comprises, and find out the table name that does not have this database table in table name list, determine and data to be stored need to be stored in storage engines, according to the table name of this database table, from the table name of database table and the sign of storage engines, obtain the sign of corresponding storage engines, using the sign of the storage engines obtaining as for storing the sign of the storage engines of data to be stored.
208: according to the sign of the storage engines obtaining, the table name of this database table and data to be stored are sent to corresponding storage engines;
209: storage engines receives the table name of this database table and data to be stored, data to be stored are stored in the database table that the table name of this database table is corresponding.
Particularly, storage engines receives the table name of this database table and data to be stored, according to the table name of this database table, obtain corresponding database table, the row name of obtaining the row at every data place that data to be stored comprise, is stored in corresponding row according to every data of row star of the row at every data place.
For example, employee's name " Zhang San " and wage " 1000 yuan " corresponding to Zhang San are stored in respectively in the row of row " employee's name " and " wage number " by name, employee's name " Li Si " and wage " 2000 yuan " corresponding to Li Si are stored in respectively in the row of row " employee's name " and " wage number " by name, and the wage " 3000 yuan " of employee's name " king five " and king's five correspondences are stored in respectively in the row of row " employee's name " and " wage number " by name.
Referring to Fig. 3-2, if user need to store the large data that data volume is greater than predetermined threshold value, these large data are data to be stored, first in data warehouse, set up a database table, again data to be stored are stored in the database table of foundation, specifically by following 301 to 305 step, realize.
301: terminal sends second and builds predicative sentence to control end, and second builds the table name that predicative sentence at least carries database table to be created, row name and the data type of each row that database table to be created comprises;
When user need to be stored on data warehouse by data to be stored, row name and the data type of the table name of the database table to be created that directly need to create on data warehouse to terminal input and each row that database table to be created comprises.
Terminal receives only the table name of database table to be created and row name and the data type of each row that database table to be created comprises of user's input, and definite user need to be stored in data to be stored in data warehouse, and send second and build predicative sentence to control end.
For example, user need to store BB company the sales figure data of 2013, these sales figure data are data to be stored, data to be stored account for 15GB capacity, the data volume of data to be stored is greater than predetermined threshold value 10GB, therefore user can be stored in data to be stored in data warehouse by designated terminal, be specially: the table name of the sales figure table that user need to create to terminal input in data warehouse is " BB company-2013 year-sales figure table ", and sales figure table comprise each row row name and data type, this sales figure table comprises two row, row " trade name " by name and the data type of first row are " string ", row " sales volume " by name and the data type of secondary series are " int ".
302: control end receives second and builds predicative sentence, build predicative sentence according to second, in data warehouse, create database table to be created;
Particularly, control end is determined the columns of database table to be created according to second row name of building each row that database table to be created that predicative sentence carries comprises, according to definite columns, create blank database table, according to second row name and data type of building the table name of the database table to be created that predicative sentence carries and each row that database table to be created comprises, this blank database table is set.
Further, control end is stored in the second table name of building the database table to be created that predicative sentence carries in table name list.
Further, control end sends the second notification message to terminal, and the second notification message is for notifying user successfully to create database table to be created; Terminal receives the second notification message, and prompting user data warehouse has successfully created database table to be created.
Terminal shows that after this information, user submits memory command to terminal, and terminal is stored in the data to be stored in terminal in data warehouse by following flow process.
303: the table name of terminal transmission database table and data to be stored are to control end, and the table name of this database table is the table name of above-mentioned database table to be created;
For example, data to be stored are the sales volume " 4000 " that trade name " shampoo " and shampoo are corresponding, the sales volume " 5000 " that trade name " toothpaste " and toothpaste are corresponding, and trade name " perfumed soap " and sales volume " 6000 " corresponding to perfumed soap, and the table name of data place database table to be stored is " BB company-2013 year-sales figure table "; The row " trade name " by name of the row that " shampoo ", " toothpaste " and " perfumed soap " are corresponding, the row " sales volume " by name of the row that " 4000 ", " 5000 " and " 6000 " are corresponding.
304: control end receives the table name of this database table and data to be stored, according to the table name of this database table, determine data to be stored are stored in data warehouse;
For example, control end receives the table name " BB company-2013 year-sales figure table " of data to be stored and data place database table to be stored, determines data to be stored are stored in data warehouse.
305: control end is stored in data to be stored in the database table that the table name of this database table is corresponding in data warehouse.
Particularly, control end obtains corresponding database table according to the table name of this database table, obtains the row name of the row at every data place that data to be stored comprise, according to every data of row star of the row at every data place, is stored in corresponding row.
For example, trade name " shampoo " and sales volume " 4000 " corresponding to shampoo are stored in respectively in the row of row " trade name " and " wage number " by name; Trade name " toothpaste " and sales volume " 5000 " corresponding to toothpaste are stored in respectively in the row of row " employee's name " and " wage number " by name; Trade name " perfumed soap " and sales volume " 6000 " corresponding to perfumed soap are stored in respectively in the row of row " employee's name " and " wage number " by name.
In embodiments of the present invention, also flow process is stored in data to be stored in data warehouse or storage engines by the following method.User selects the data of a needs storage in terminal, and table name from database table to be created to terminal that input, row name and the data type of each row that database table to be created comprises.The data that terminal is selected user are as data to be stored, obtain the data volume size of data to be stored, and the table name that receives the database table to be created of user's input, row name and the data type of each row that database table to be created comprises, transmission is built predicative sentence to control end, this builds the data volume size that predicative sentence carries data to be stored, the table name of database table to be created, row name and the data type of each row that database table to be created comprises.
Control end receives this and builds predicative sentence, according to the data volume size of data to be stored, be identified for storing the database of data to be stored, (for example, if the data volume size of data to be stored is greater than predetermined threshold value, the database that is identified for storing data to be stored is data warehouse, if the population size of data to be stored is less than or equal to predetermined threshold value, the database that is identified for storing data to be stored is storage engines).
If determined as data warehouse, control end creates database table to be created in data warehouse, and in data warehouse, stores data to be stored, and specific implementation process can, referring to the step of above-mentioned steps 302 to 305, no longer describe in detail at this.
If determined as storage engines, (for example select the sign of a storage engines, select the sign of the storage engines of residual memory space maximum), according to the sign of this storage engines, send this and build predicative sentence to this storage engines, this storage engines creates database table to be created and stores data to be stored, and specific implementation process can, referring to the step of above-mentioned steps 204 to 209, no longer describe in detail at this.
For example, referring to Fig. 3-3, next with a concrete example, the process of above-mentioned storage data to be stored is elaborated, data warehouse can be TDW(Tencent Distributed Warehouse, Tengxun's Distributed Data Warehouse), storage engines can be PostgreSQL(storage engines).
The database table that establishment that control end receiving terminal sends is to be created build predicative sentence create stored as pgdata, this builds the sign that predicative sentence carries storage engines PostgreSQL, the table name of database table to be created, row name and the data type of each row that database table to be created comprises, send this and build predicative sentence to PostgreSQL; PostgreSQL receives this and builds predicative sentence and attempt according to building predicative sentence creation database table;
If create unsuccessfully, notify the unsuccessful creation database table of control end PostgreSQL; Control end is the unsuccessful creation database table of notification terminal PostgreSQL then, end operation.
If created successfully, notify control end PostgreSQL success creation database table; Control end is stored in the sign of the table name of database table to be created and PostgreSQL in the corresponding relation of the table name of database table and the sign of storage engines, and notification terminal PostgreSQL success creation database table; The table name of the data to be stored that terminal is stored needs and data place database table to be stored sends to control end; Control end receives the table name of data to be stored and data place database table to be stored, according to the table name of data place database table to be stored, in the corresponding relation of the table name of database table and the sign of storage engines, obtain the sign of corresponding storage engines PostgreSQL, according to the sign of PostgreSQL, send data to be stored to PostgreSQL; PostgreSQL receives data to be stored, stores data to be stored, end operation in the database table having created.
Referring to Fig. 3-4, if user need to inquire about and process the data of having stored, user can send query statement to control end by designated terminal, so that control end obtains data according to this query statement, the data of obtaining are processed, and result is returned to terminal, specifically can realize by following 401 to 408 step.
401: terminal is obtained the query statement of user's input, send this query statement to control end, this query statement carries the stored position information of data to be checked and processes the processing mode of data to be checked, and stored position information at least comprises the table name of data place database table to be checked;
When user need to inquire about and process the data of having stored, directly to terminal input inquiry statement, this query statement carries the stored position information of the data to be checked that need inquiry and processes the processing mode of data to be checked, and stored position information at least comprises the table name of data place database table to be checked.
Terminal is obtained the query statement of user's input and is sent this query statement to control end.
For example, user need to inquire about the average salary number that AA company provided in the January, can be in terminal input inquiry statement, this query statement carries needs the data AA company to be checked of inquiry at the stored position information of the wage data of January granting, and the average salary number provided in the January for calculating AA company of processing mode, this stored position information at least comprises the table name " AA company-January-payroll " of the payroll at data place to be checked.
402: the query statement that control end receiving terminal sends, the table name of the data place database table to be checked carrying according to query statement, look-up table list of file names, if table name list comprises the table name of data place database table to be checked, determine that data to be checked are stored in data warehouse, execution step 403; If do not comprise the table name of data place database table to be checked in table name list, determine that data to be checked are stored in storage engines, execution step 406;
For example, the query statement that control end receiving terminal sends, obtain the table name " AA company-January-payroll " of the payroll at the data place to be checked that query statement comprises, search table name list as shown in table 4, and in the table name list shown in table 4, do not comprise that name is called the table name of " AA company-January-payroll ", execution step 406.
Table 4
| " student performance " |
| " class " |
| " employee's age " |
403: control end sends to data warehouse by this query statement;
404: data warehouse receives this query statement, according to the stored position information of data to be checked, obtain data to be checked, the processing mode of carrying according to this query statement is processed data to be checked, and result is sent to control end;
405: control end reception & disposal result also sends to terminal, end operation;
406: control end, according to the table name of data place database table to be checked, obtains for storing the sign of the storage engines of data to be checked, and send query statement to the storage engines of the sign correspondence of the storage engines obtaining;
Particularly, control end is according to the table name of data place database table to be checked, from the corresponding relation of the table name of database table stored and the sign of storage engines, find out the sign of corresponding storage engines, using the sign of the storage engines finding out as for storing the sign of the storage engines of data to be checked; And send query statement to the storage engines of the sign correspondence of the storage engines finding out.
For example, in the corresponding relation of the table name of the database table shown in table 2 and the sign of storage engines, search the record that comprises table name " AA company-January-payroll ", the payroll of the wage data that database table WeiAA company corresponding to this table name provided in the January, and find the record that comprises table name " AA company-January-payroll ", obtain the sign ID1 of the storage engines that this record comprises, send query statement to storage engines corresponding to the sign ID1 of storage engines.
407: storage engines receives this query statement, according to the stored position information of data to be checked, obtain data to be checked, the processing mode of carrying according to this query statement is processed data to be checked, and result is sent to control end;
For example, storage engines receives query statement, obtain data place database table to be checked, in this tables of data, obtain wage number " 1000 yuan ", " 2000 yuan " and " 3000 yuan " that AA company provided to each employee in the January, and calculate the average salary number " 2000 yuan " of providing, the average salary number calculating " 2000 yuan " is sent to control end.
408: control end reception & disposal result is also transmitted to terminal, end operation.
Wherein, it should be noted that, if storage engines cannot successfully be processed data to be checked, the data to be checked that transmission is obtained are to control end; Control end receives data to be checked, and the processing mode of carrying according to this query statement is processed data to be checked, and result is sent to terminal.
For example, referring to Fig. 3-5, next with a concrete example to above-mentioned inquiry and the process of processing the data stored be elaborated, the query statement that control end receiving terminal sends, query statement carries the table name of the data place database table to be checked that needs inquiry and processes the processing mode of data to be checked, searches the record of the table name of the database table that comprises that query statement carries in the corresponding relation of the table name of database table and the sign of storage engines; If find out the record of the table name of the database table that comprises that query statement carries, obtain the sign of the storage engines PostgreSQL in this record, the sign according to PostgreSQL, sends to PostgreSQL by query statement; PostgreSQL receives query statement, and the table name of the data place database table to be checked comprising according to query statement obtains data to be checked, and attempts according to the processing mode of processing data to be checked, data to be checked being processed; If process successfully, result returned to control end; Control end reception & disposal result also sends to terminal, end operation; If process unsuccessful, the data to be checked of obtaining are sent to control end, control end receives data to be checked, and according to the processing mode of processing data to be checked, data to be checked is processed, result is sent to terminal to end operation.
Wherein, data warehouse is when upgrading the data of its storage, only support the renewal of subregion granularity formula, for example, in the time of need to deleting some data in it a certain database table comprising, first need the data of determining the data of needs deletion and not needing to delete, then all data in this database table are deleted, finally will not need the data of deleting again to write in this database table; Whole process steps is loaded down with trivial details, and required time is very long and wasted a large amount of system resource.And storage engines is when upgrading the data of its storage, the data of needs deletion directly can be deleted from database table, compared with data warehouse, at aspect tools such as operation steps, times, have great advantage.
In addition, the outer computer language interface of data warehouse is not abundant, only has the interface of a few computers language, only can obtain by the program of being write by the computerese of minority the data of data warehouse storage; But the outer computer language interface of storage engines is very abundant, almost can obtain by the program of being write by any computerese the data of storage engines storage; By the interface of data warehouse and the interface of storage engines, between the two, connect, when user uses the program of being write by computerese to obtain the data of storing in data warehouse, even if the computerese data warehouse that user uses does not possess the computerese interface that user uses, but because storage engines possesses the computerese interface that user uses, user also can obtain the data in data warehouse, is specifically as follows: user inputs to storage engines by program; Storage engines obtains the program of user's input, by the connection between storage engines and data warehouse, sends program to data warehouse; Data warehouse reception program also obtains data according to program, sends data to storage engines; User just can obtain the data of storing in data warehouse, has indirectly enriched the external language interface of data warehouse.
In embodiments of the present invention, when the larger large data of data volume are processed, on the processing time, data warehouse has great advantage with respect to storage engines tool, and when the less small data of data volume is processed, on the processing time, storage engines has great advantage with respect to data warehouse tool; Load store engine on data warehouse, so that user can be stored in the less small data of data volume in storage engines, the large data that data volume is larger are stored in data warehouse; When user need to inquire about and process the data of having stored, if these data be small data and be stored in storage engines, query statement is sent to storage engines so that storage engines obtains these data and these data are processed; If these data are large data and are stored in data warehouse, query statement is sent to data warehouse so that data warehouse obtains these data and these data are processed; So make storage engines and data warehouse can bring into play advantage separately; Short slab when load store engine has solved the less small data of data warehouse deal with data amount in data warehouse on the processing time, can save treatment step and processing time that data are processed, greatly degree has reduced user's stand-by period and has saved system resource.
Embodiment 3
Referring to Fig. 4, the embodiment of the present invention provides a kind of device that data are processed, and comprising:
The first receiver module 501, the query statement sending for receiving terminal, query statement carries the stored position information of data to be checked and processes the processing mode of data to be checked, and stored position information at least comprises the table name of data place database table to be checked;
The first determination module 502, for according to the stored position information of data to be checked, determines the database of the data that storage is to be checked, and database comprises data warehouse or storage engines;
The first sending module 503, if for determining as data warehouse, query statement is sent to data warehouse, makes data warehouse according to stored position information, obtain data to be checked and according to processing mode, data to be checked be processed;
The second sending module 504, if for determining as storage engines, the storage engines that query statement is sent to storage data to be checked, makes storage engines according to stored position information, obtain data to be checked and according to processing mode, data to be checked be processed.
Preferably, the first determination module 502 comprises:
Search unit, for according to the table name look-up table list of file names of data place database table to be checked, table name list is for the table name of the database table storing data warehouse and comprise;
The first determining unit, if comprise the table name of data place database table to be checked for table name list, determine that data to be checked are stored in data warehouse, if do not comprise the table name of data place database table to be checked in table name list, determine that data to be checked are stored in storage engines.
Preferably, the second sending module 504 comprises:
Acquiring unit for according to the table name of data place database table to be checked, obtains the sign of the storage engines of storage data to be checked from the corresponding relation of the table name of database table stored and the sign of storage engines;
Transmitting element, for according to the sign of storage engines, sends to query statement the storage engines of storage data to be checked.
Further, this device also comprises:
Processing module if cannot process data to be checked for storage engines, according to the stored position information of data to be checked, is obtained data to be checked from storage engines, according to processing mode, data to be checked is processed.
Further, this device also comprises:
The second receiver module, what for receiving terminal, send builds predicative sentence, builds the sign that predicative sentence at least carries storage engines, the table name of database table to be created, row name and data type of each row that database table to be created comprises;
The 3rd sending module, for according to the sign of storage engines, sends and builds predicative sentence to storage engines, makes storage engines create database table to be created.
Further, this device also comprises:
The 3rd receiver module, what for receiving terminal, send builds predicative sentence, builds the data volume size that predicative sentence at least carries data to be stored, the table name of database table to be created, row name and data type of each row that database table to be created comprises;
The second determination module, for according to data volume size, is identified for storing the database of data to be stored;
The 4th sending module, if for determining as data warehouse, create database table to be created in data warehouse;
The 5th sending module, if for determining as storage engines, select the sign of a storage engines, according to the sign of storage engines, sends and builds predicative sentence to storage engines, makes storage engines create database table to be created.
Preferably, the second determination module comprises:
The second determining unit, if be greater than predetermined threshold value for data volume size, the database that is identified for storing data to be stored is data warehouse;
The 3rd determining unit, if be less than or equal to predetermined threshold value for data volume size, the database that is identified for storing data to be stored is storage engines.
Further, this device also comprises:
The 3rd determination module, the data type of each row that comprise for the database table to be created is determined the data type that storage engines is not supported;
Modular converter, for being the data type that storage engines is supported building predicative sentence by the data type conversion of determining.
Further, this device also comprises:
Memory module, for by the corresponding relation of building the table name of database table to be created that predicative sentence carries and the sign of storage engines and being stored in the table name of database table and the sign of storage engines.
In embodiments of the present invention, when the larger large data of data volume are processed, on the processing time, data warehouse has great advantage with respect to storage engines tool, and when the less small data of data volume is processed, on the processing time, storage engines has great advantage with respect to data warehouse tool; Load store engine in data warehouse, make storage engines and data warehouse can bring into play advantage separately, short slab while having solved the less small data of data warehouse deal with data amount on the processing time, can save treatment step and processing time that data are processed, greatly degree has reduced user's stand-by period and has saved system resource.
Embodiment 4
Referring to Fig. 5, the embodiment of the present invention provides a kind of server that data are processed, and comprising:
CPU (central processing unit) (CPU) 601, comprise the system storage 604 of random-access memory (ram) 602 and ROM (read-only memory) (ROM) 603 and the system bus 605 of connected system storer 604 and CPU (central processing unit) 601.Described server 600 also comprises the basic input/output (I/O system) 606 of transmission information between each device helping in computing machine, and for the mass-memory unit 607 of storage operation system 613, application program 610 and other program modules 614.
Described basic input/output 606 includes for showing the display 608 of information and for the input equipment such as mouse, keyboard 609 of user's input information.Wherein said display 608 and input equipment 609 are all connected to CPU (central processing unit) 601 by being connected to the input/output control unit 610 of system bus 605.Described basic input/output 606 can also comprise that input/output control unit 610 is for receiving and process the input from a plurality of other equipment such as keyboard, mouse or electronic touch pens.Similarly, input/output control unit 610 also provides the output device that outputs to display screen, printer or other types.
Described mass-memory unit 607 is connected to CPU (central processing unit) 601 by being connected to the bulk memory controller (not shown) of system bus 605.Described mass-memory unit 607 and the computer-readable medium that is associated thereof provide non-volatile memories for server 600.That is to say, described mass-memory unit 607 can comprise the computer-readable medium (not shown) such as hard disk or CD-ROM drive.
Without loss of generality, described computer-readable medium can comprise computer-readable storage medium and communication media.Computer-readable storage medium comprises volatibility and non-volatile, the removable and irremovable medium for any method of information such as computer-readable instruction, data structure, program module or other data of storage or technology realization.Computer-readable storage medium comprises RAM, ROM, EPROM, EEPROM, flash memory or its technology of other solid-state storage, CD-ROM, DVD or other optical memory, tape cassete, tape, disk storage or other magnetic storage apparatus.Certainly, the known described computer-readable storage medium of those skilled in the art is not limited to above-mentioned several.Above-mentioned system storage 604 and mass-memory unit 607 can be referred to as storer.
According to various embodiments of the present invention, described server 600 can also move by the remote computer being connected on network such as networks such as the Internets.Also be that server 600 can be connected to network 612 by the network interface unit 611 being connected on described system bus 605, in other words, also can be connected to by network interface unit 611 network or the remote computer system (not shown) of other types.
Described storer also comprises one or more than one program, and the above program of described or is stored in storer, and described one or above routine package are containing for carrying out the instruction of a kind of method that data are processed provided by the invention.
One of ordinary skill in the art will appreciate that all or part of step that realizes above-described embodiment can complete by hardware, also can come the hardware that instruction is relevant to complete by program, described program can be stored in a kind of computer-readable recording medium, the above-mentioned storage medium of mentioning can be ROM (read-only memory), disk or CD etc.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any modification of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (19)
1. a method of data being processed, is characterized in that, described method comprises:
The query statement that receiving terminal sends, described query statement carries the stored position information of data to be checked and processes the processing mode of described data to be checked, and described stored position information at least comprises the table name of described data place database table to be checked;
According to the stored position information of described data to be checked, determine the database of the described data to be checked of storage, described database comprises data warehouse or storage engines;
If determined as data warehouse, described query statement is sent to described data warehouse, make described data warehouse according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed;
If determined as storage engines, the storage engines that described query statement is sent to the described data to be checked of storage, makes described storage engines according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed.
2. the method for claim 1, is characterized in that, described according to the stored position information of described data to be checked, determines the database of the described data to be checked of storage, comprising:
According to the table name look-up table list of file names of described data place database table to be checked, described table name list is for storing the table name of the database table that described data warehouse comprises;
If described table name list comprises the table name of described data place database table to be checked, determine that described data to be checked are stored in described data warehouse, if do not comprise the table name of described data place database table to be checked in described table name list, determine that described data to be checked are stored in storage engines.
3. the method for claim 1, is characterized in that, the described storage engines that described query statement is sent to the described data to be checked of storage, comprising:
According to the table name of described data place database table to be checked, from the corresponding relation of the table name of database table stored and the sign of storage engines, obtain the sign of the storage engines of the described data to be checked of storage;
According to the sign of described storage engines, described query statement is sent to the storage engines of the described data to be checked of storage.
4. the method for claim 1, is characterized in that, described method also comprises:
If described storage engines cannot be processed described data to be checked, according to the stored position information of described data to be checked, from described storage engines, obtain described data to be checked, according to described processing mode, described data to be checked are processed.
5. the method for claim 1, is characterized in that, described method also comprises:
What receive that described terminal sends builds predicative sentence, described in build the sign that predicative sentence at least carries storage engines, the table name of database table to be created, row name and data type of each row that described database table to be created comprises;
According to the sign of described storage engines, build predicative sentence to described storage engines described in transmission, make described storage engines create described database table to be created.
6. the method for claim 1, is characterized in that, described method also comprises:
What receive that described terminal sends builds predicative sentence, described in build the data volume size that predicative sentence at least carries data to be stored, the table name of database table to be created, row name and data type of each row that described database table to be created comprises;
According to described data volume size, be identified for storing the database of described data to be stored;
If determined as data warehouse, in described data warehouse, create described database table to be created;
If determined as storage engines, select the sign of a storage engines, according to the sign of described storage engines, build predicative sentence to described storage engines described in transmission, make described storage engines create described database table to be created.
7. method as claimed in claim 6, is characterized in that, described big or small according to described data volume, is identified for storing the database of described data to be stored, comprising:
If described data volume size is greater than predetermined threshold value, the database that is identified for storing described data to be stored is data warehouse;
If described data volume size is less than or equal to described predetermined threshold value, the database that is identified for storing described data to be stored is storage engines.
8. the method as described in claim 5 or 6, is characterized in that, described according to the sign of described storage engines, builds predicative sentence to before described storage engines described in transmission, also comprises:
The data type of each row that comprise from described database table to be created, determine the data type that described storage engines is not supported;
Described by the described data type conversion of determining, be the data type that described storage engines is supported in building predicative sentence.
9. method as claimed in claim 5, is characterized in that, described receiving terminal sends build predicative sentence after, also comprise:
By described, build the table name of database table to be created and the sign of storage engines that predicative sentence carries and be stored in the corresponding relation of the table name of database table and the sign of storage engines.
10. a device of data being processed, is characterized in that, described device comprises:
The first receiver module, the query statement sending for receiving terminal, described query statement carries the stored position information of data to be checked and processes the processing mode of described data to be checked, and described stored position information at least comprises the table name of described data place database table to be checked;
The first determination module, for according to the stored position information of described data to be checked, determines the database of the described data to be checked of storage, and described database comprises data warehouse or storage engines;
The first sending module, if for determining as data warehouse, described query statement is sent to described data warehouse, make described data warehouse according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed;
The second sending module, if for determining as storage engines, the storage engines that described query statement is sent to the described data to be checked of storage, makes described storage engines according to described stored position information, obtain described data to be checked and according to described processing mode, described data to be checked be processed.
11. devices as claimed in claim 10, is characterized in that, described the first determination module comprises:
Search unit, for according to the table name look-up table list of file names of described data place database table to be checked, described table name list is for the table name of the database table storing described data warehouse and comprise;
The first determining unit, if comprise the table name of described data place database table to be checked for described table name list, determine that described data to be checked are stored in described data warehouse, if do not comprise the table name of described data place database table to be checked in described table name list, determine that described data to be checked are stored in storage engines.
12. devices as claimed in claim 10, is characterized in that, described the second sending module comprises:
Acquiring unit for according to the table name of described data place database table to be checked, obtains the sign of the storage engines of the described data to be checked of storage from the corresponding relation of the table name of database table stored and the sign of storage engines;
Transmitting element, for according to the sign of described storage engines, sends to described query statement the storage engines of the described data to be checked of storage.
13. devices as claimed in claim 10, is characterized in that, described device also comprises:
Processing module, if cannot process described data to be checked for described storage engines,, according to the stored position information of described data to be checked, from described storage engines, obtain described data to be checked, according to described processing mode, described data to be checked are processed.
14. devices as claimed in claim 10, is characterized in that, described device also comprises:
The second receiver module, builds predicative sentence for what receive that described terminal sends, described in build the sign that predicative sentence at least carries storage engines, the table name of database table to be created, row name and data type of each row that described database table to be created comprises;
The 3rd sending module, for according to the sign of described storage engines, builds predicative sentence to described storage engines described in transmission, makes described storage engines create described database table to be created.
15. devices as claimed in claim 10, is characterized in that, described device also comprises:
The 3rd receiver module, for what receive that described terminal sends, build predicative sentence, the described data volume size that predicative sentence at least carries data to be stored, the table name of database table to be created, row name and the data type of each row that described database table to be created comprises of building;
The second determination module, for according to described data volume size, is identified for storing the database of described data to be stored;
The 4th sending module if for determining as data warehouse, creates described database table to be created in described data warehouse;
The 5th sending module, if for determining as storage engines, select the sign of a storage engines, according to the sign of described storage engines, builds predicative sentence to described storage engines described in transmission, makes described storage engines create described database table to be created.
16. devices as claimed in claim 15, is characterized in that, described the second determination module comprises:
The second determining unit, if be greater than predetermined threshold value for described data volume size, the database that is identified for storing described data to be stored is data warehouse;
The 3rd determining unit, if be less than or equal to described predetermined threshold value for described data volume size, the database that is identified for storing described data to be stored is storage engines.
17. devices as described in claims 14 or 15, is characterized in that, described device also comprises:
The 3rd determination module, determines for each data type being listed as comprising from described database table to be created the data type that described storage engines is not supported;
Modular converter, for being the data type that described storage engines is supported at the described predicative sentence of building by the described data type conversion of determining.
18. devices as claimed in claim 14, is characterized in that, described device also comprises:
Memory module, for building by described the corresponding relation that the table name of database table to be created that predicative sentence carries and the sign of storage engines are stored in the table name of database table and the sign of storage engines.
19. 1 kinds of servers that data are processed, is characterized in that, described server comprises:
One or more processors; With
Storer;
Described memory stores has one or more programs, and described one or more programs are configured to be carried out by described one or more processors, and described one or more routine packages contain for carrying out the instruction of method as described in claim 1 to 9 any one claim.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310756095.5A CN104090901B (en) | 2013-12-31 | 2013-12-31 | A kind of method that data are processed, device and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310756095.5A CN104090901B (en) | 2013-12-31 | 2013-12-31 | A kind of method that data are processed, device and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104090901A true CN104090901A (en) | 2014-10-08 |
| CN104090901B CN104090901B (en) | 2017-06-13 |
Family
ID=51638617
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310756095.5A Active CN104090901B (en) | 2013-12-31 | 2013-12-31 | A kind of method that data are processed, device and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104090901B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105930506A (en) * | 2016-05-10 | 2016-09-07 | 乐视控股（北京）有限公司 | Object retrieval method and object storage system |
| CN106202084A (en) * | 2015-04-30 | 2016-12-07 | 阿里巴巴集团控股有限公司 | Date storage method and data storage device |
| CN106294581A (en) * | 2016-07-29 | 2017-01-04 | 苏州商信宝信息科技有限公司 | A kind of based on wages variation tendency exhibiting method during searching enterprise |
| CN106547807A (en) * | 2015-09-23 | 2017-03-29 | 财团法人工业技术研究院 | Data analysis method and device |
| CN107077511A (en) * | 2015-03-28 | 2017-08-18 | 华为技术有限公司 | The apparatus and method that User Defined variable-size label is created on the record in RDBMS |
| CN107463693A (en) * | 2017-08-11 | 2017-12-12 | 深圳乐信软件技术有限公司 | A kind of data processing method, device, terminal and computer-readable recording medium |
| CN108228606A (en) * | 2016-12-14 | 2018-06-29 | 北京国双科技有限公司 | The wiring method and device of data |
| CN108268517A (en) * | 2016-12-30 | 2018-07-10 | 希姆通信息技术（上海）有限公司 | The management method and system of label in database |
| CN108280230A (en) * | 2018-02-27 | 2018-07-13 | 北京中关村科金技术有限公司 | A kind of method, apparatus, equipment and the storage medium of analysis data |
| CN109542953A (en) * | 2018-11-29 | 2019-03-29 | 成都四方伟业软件股份有限公司 | Data processing method and device based on presto |
| CN111143399A (en) * | 2019-12-19 | 2020-05-12 | 微民保险代理有限公司 | Data processing method, data processing device, storage medium and computer equipment |
| CN111858644A (en) * | 2020-07-08 | 2020-10-30 | 联思智云(北京)科技有限公司 | Method, device and system for data fusion and query |
| CN114817338A (en) * | 2022-06-28 | 2022-07-29 | 杭州湖畔网络技术有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7007029B1 (en) * | 1999-01-15 | 2006-02-28 | Metaedge Corporation | System for visualizing information in a data warehousing environment |
| US20060235838A1 (en) * | 2005-04-19 | 2006-10-19 | International Business Machines Corporation | System and method for managing complex relationships over distributed heterogeneous data sources |
| US20070094286A1 (en) * | 2005-10-20 | 2007-04-26 | Ravi Murthy | Managing relationships between resources stored within a repository |
| CN102214236A (en) * | 2011-06-30 | 2011-10-12 | 北京新媒传信科技有限公司 | Method and system for processing mass data |
| CN102915373A (en) * | 2012-11-06 | 2013-02-06 | 无锡江南计算技术研究所 | Data storage method and device |
| CN103268336A (en) * | 2013-05-13 | 2013-08-28 | 刘峰 | Fast data and big data combined data processing method and system |
-
2013
- 2013-12-31 CN CN201310756095.5A patent/CN104090901B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7007029B1 (en) * | 1999-01-15 | 2006-02-28 | Metaedge Corporation | System for visualizing information in a data warehousing environment |
| US20060235838A1 (en) * | 2005-04-19 | 2006-10-19 | International Business Machines Corporation | System and method for managing complex relationships over distributed heterogeneous data sources |
| US20070094286A1 (en) * | 2005-10-20 | 2007-04-26 | Ravi Murthy | Managing relationships between resources stored within a repository |
| CN102214236A (en) * | 2011-06-30 | 2011-10-12 | 北京新媒传信科技有限公司 | Method and system for processing mass data |
| CN102915373A (en) * | 2012-11-06 | 2013-02-06 | 无锡江南计算技术研究所 | Data storage method and device |
| CN103268336A (en) * | 2013-05-13 | 2013-08-28 | 刘峰 | Fast data and big data combined data processing method and system |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107077511A (en) * | 2015-03-28 | 2017-08-18 | 华为技术有限公司 | The apparatus and method that User Defined variable-size label is created on the record in RDBMS |
| CN106202084A (en) * | 2015-04-30 | 2016-12-07 | 阿里巴巴集团控股有限公司 | Date storage method and data storage device |
| CN106547807B (en) * | 2015-09-23 | 2021-01-22 | 财团法人工业技术研究院 | Data analysis method and device |
| CN106547807A (en) * | 2015-09-23 | 2017-03-29 | 财团法人工业技术研究院 | Data analysis method and device |
| US11086881B2 (en) | 2015-09-23 | 2021-08-10 | Industrial Technology Research Institute | Method and device for analyzing data |
| CN105930506A (en) * | 2016-05-10 | 2016-09-07 | 乐视控股（北京）有限公司 | Object retrieval method and object storage system |
| CN106294581A (en) * | 2016-07-29 | 2017-01-04 | 苏州商信宝信息科技有限公司 | A kind of based on wages variation tendency exhibiting method during searching enterprise |
| CN108228606A (en) * | 2016-12-14 | 2018-06-29 | 北京国双科技有限公司 | The wiring method and device of data |
| CN108268517A (en) * | 2016-12-30 | 2018-07-10 | 希姆通信息技术（上海）有限公司 | The management method and system of label in database |
| CN107463693A (en) * | 2017-08-11 | 2017-12-12 | 深圳乐信软件技术有限公司 | A kind of data processing method, device, terminal and computer-readable recording medium |
| CN107463693B (en) * | 2017-08-11 | 2020-05-01 | 深圳乐信软件技术有限公司 | Data processing method, device, terminal and computer readable storage medium |
| CN108280230A (en) * | 2018-02-27 | 2018-07-13 | 北京中关村科金技术有限公司 | A kind of method, apparatus, equipment and the storage medium of analysis data |
| CN109542953A (en) * | 2018-11-29 | 2019-03-29 | 成都四方伟业软件股份有限公司 | Data processing method and device based on presto |
| CN111143399A (en) * | 2019-12-19 | 2020-05-12 | 微民保险代理有限公司 | Data processing method, data processing device, storage medium and computer equipment |
| CN111143399B (en) * | 2019-12-19 | 2024-04-02 | 微民保险代理有限公司 | Data processing method, device, storage medium and computer equipment |
| CN111858644A (en) * | 2020-07-08 | 2020-10-30 | 联思智云(北京)科技有限公司 | Method, device and system for data fusion and query |
| CN111858644B (en) * | 2020-07-08 | 2022-11-18 | 联思智云(北京)科技有限公司 | Method, device and system for data fusion and query |
| CN114817338A (en) * | 2022-06-28 | 2022-07-29 | 杭州湖畔网络技术有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104090901B (en) | 2017-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104090901A (en) | Method, device and server for processing data | |
| CN107247808B (en) | Distributed NewSQL database system and picture data query method | |
| CN108874971B (en) | Tool and method applied to mass tagged entity data storage | |
| CN104536904B (en) | A kind of method of data management, equipment and system | |
| CN109241159B (en) | Partition query method and system for data cube and terminal equipment | |
| WO2016011904A1 (en) | Database access method and apparatus and database system | |
| CN103440288A (en) | Big data storage method and device | |
| CN103064933A (en) | Data query method and system | |
| CN102479223A (en) | Data query method and system | |
| CN101996067A (en) | Data export method and device | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN101556678A (en) | Processing method of batch processing services, system and service processing control equipment | |
| CN103544153A (en) | Data updating method and system based on database | |
| CN103778133A (en) | Database object changing method and device | |
| US20230024345A1 (en) | Data processing method and apparatus, device, and readable storage medium | |
| CN105117433A (en) | Method and system for statistically querying HBase based on analysis performed by Hive on HFile | |
| CN102902763A (en) | Method and device for relating and retrieving information processing data and processing information tasks | |
| CN111258978A (en) | Data storage method | |
| CN103810219A (en) | Line storage database-based data processing method and device | |
| CN106471501A (en) | The method of data query, the storage method data system of data object | |
| CN106055678A (en) | Hadoop-based panoramic big data distributed storage method | |
| CN114297173A (en) | Knowledge graph construction method and system for large-scale mass data | |
| CN114820080A (en) | User grouping method, system, device and medium based on crowd circulation | |
| CN103440302B (en) | The method and system of Real Data Exchangs | |
| CN112131214A (en) | Method, system, equipment and storage medium for data writing and data query |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |