CN105022763A - Method and system for implementing data query - Google Patents
Method and system for implementing data query Download PDFInfo
- Publication number
- CN105022763A CN105022763A CN201410183883.4A CN201410183883A CN105022763A CN 105022763 A CN105022763 A CN 105022763A CN 201410183883 A CN201410183883 A CN 201410183883A CN 105022763 A CN105022763 A CN 105022763A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- bucket
- raw data
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The present invention provides a method and system for implementing data query. The method for implementing data query comprises the following steps of: acquiring a query request; positioning a bucket corresponding to the query request in a data file partition based on Hive and a column in the bucket; and reading data corresponding to the column in the positioned bucket. The system comprises: a request acquisition module, used for acquiring a query request; a positioning module, used for positioning the bucket corresponding to the query request in the data file partition based on the Hive and the column in the bucket; and a reading module, used for reading the data corresponding to the column in the positioned bucket. By adopting the method and system for implementing the data query, the query efficiency of the data can be improved.
Description
Technical field
The present invention relates to data processing technique, particularly relate to a kind of method and system realizing data query.
Background technology
Along with the continuous growth of data volume and data value, traditional data warehouse technology all encounters huge obstacle in various, cannot meet the demand of large data processing.
Hive is for building data warehouse, processing the most frequently used Open Framework of mass data in current Internet enterprises, but owing to being without any optimization carrying out data storage, cause it can not well embody its operational efficiency under a lot of scene, particularly greatly have impact on the search efficiency in later stage.
Summary of the invention
Based on this, be necessary to provide a kind of method realizing data query that can improve search efficiency.
In addition, there is a need to provide a kind of system realizing data query that can improve search efficiency.
Realize a method for data query, comprise the steps:
Obtain inquiry request;
Locate the row in the bucket and described bucket that described in the data file subregion based on Hive, inquiry request is corresponding;
Read the data corresponding to row in the bucket that described location obtains.
Wherein in an embodiment, described location based on Hive data file subregion described in the step of row in bucket corresponding to inquiry request and described bucket comprise:
Change described inquiry request into MapReduce task;
Obtain metadata, obtain the data file subregion based on Hive relevant to described MapReduce task according to described metadata;
Data store organisation according to definition calculates to obtain corresponding informative abstract value to the inquiry field in described MapReduce task, and obtaining the data storage location corresponding with described inquiry field by the delivery between described informative abstract value and default barrelage amount, described data storage location is used to indicate the row in bucket corresponding to described inquiry field and described bucket.
Wherein in an embodiment, the step of the data corresponding to row in the bucket that the described location of described reading obtains comprises:
Carry out Data import according to the row in the bucket that described inquiry field is corresponding, and the data of described loading are processed.
Wherein in an embodiment, described location based on Hive data file subregion described in row in bucket corresponding to inquiry request and described bucket step before, described method also comprises:
Receive the raw data of input, and described raw data is stored as the first data list structure;
Process is optimized to the described raw data being stored as the first data list structure, with the data that optimization process is obtained stored in configuration file configure based in the data file subregion of Hive.
Wherein in an embodiment, the described raw data receiving input, and the step described raw data being stored as the first data list structure comprises:
Configured by described raw data stored in comprising in the data file of partition information by configuration file, wherein, the raw data being stored into described data file stores with JSON form.
Wherein in an embodiment, be describedly optimized process to the raw data that described load store is the first data list structure, what configure stored in configuration file with data optimization process obtained comprises based on the step in the data file subregion of Hive:
Extract the data of the every a line JSON form be stored as in the raw data of the first data list structure one by one;
The district location to the data file based on Hive that the raw data of described first data list structure stores is obtained by the data file comprising partition information;
Calculate to obtain corresponding informative abstract value to the raw data of described extraction, and obtain the data storage location of raw data described in the subregion of data file by described informative abstract value and the delivery of the barrelage amount preset;
The data of described extraction are carried out compression process, and store according to described data storage location after compression process.
Realize a system for data query, comprising:
Acquisition request module, for obtaining inquiry request;
Locating module, for locating the row in bucket and described bucket that described in the data file subregion based on Hive, inquiry request is corresponding;
Read module, for reading the data corresponding to the row in bucket that described location obtains.
Wherein in an embodiment, described locating module comprises:
Converting unit, for changing described inquiry request into MapReduce task;
Partitions of file acquiring unit, for obtaining metadata, obtains the data file subregion based on Hive relevant to described MapReduce task according to described metadata;
Position acquisition unit, for calculating to obtain corresponding informative abstract value to the inquiry field in described MapReduce task according to the data store organisation of definition, and obtaining the data storage location corresponding with described inquiry field by the delivery between described informative abstract value and default barrelage amount, described data storage location is used to indicate the row in bucket corresponding to described inquiry field and described bucket.
Wherein in an embodiment, described read module also for carrying out Data import according to the row in bucket corresponding to inquiry field, and is output in data to the data of described loading and processes.
Wherein in an embodiment, described system also comprises:
Raw data memory module, for receiving the raw data of input, and is stored as the first data list structure by described raw data;
Optimization process module, for being optimized process to the described raw data being stored as the first data list structure, with the data that optimization process is obtained stored in configuration file configure based in the data file subregion of Hive.
Wherein in an embodiment, described raw data memory module is also for configuring by described raw data stored in comprising in the data file of partition information by configuration file, and wherein, the raw data being stored into described data file stores with JSON form.
Wherein in an embodiment, described optimization process module comprises:
Extraction unit, for extracting the data of the every a line JSON form be stored as in the raw data of the first data list structure one by one;
Subregion acquiring unit, obtains the district location to the data file based on Hive that the raw data of described first data list structure stores for the data file by comprising partition information;
Position arithmetic element, obtains corresponding informative abstract value for calculating the raw data of described extraction, and obtains the data storage location of raw data described in the subregion of data file by described informative abstract value and the delivery of the barrelage amount preset;
Storage unit, carries out compression process for the raw data that the data of described extraction are relevant, and stores according to described data storage location after compression process.
The above-mentioned method and system realizing data query, after the inquiry request getting data, by location based on the bucket corresponding to inquiry request in the data file subregion of Hive and the row in described bucket, and then the data exported corresponding to these row, owing to not needing to search in all data, and the data obtained after the bucket found corresponding to inquiry request and row are the data of current required inquiry, substantially increase search efficiency and inquiry velocity, even if inquire about the fast finding that also can complete data to the mass data stored.
Accompanying drawing explanation
Fig. 1 is the method flow diagram realizing data query in an embodiment;
Fig. 2 is the method flow diagram of the row of locating in Fig. 1 in bucket and this barrel that in the data file subregion based on Hive, inquiry request is corresponding;
Fig. 3 is the method flow diagram realizing data query in another embodiment;
Fig. 4 is optimized process to the raw data being stored as the first data list structure in Fig. 3, with the data that optimization process is obtained stored in configuration file configure based on the method flow diagram in the data file subregion of Hive;
Fig. 5 is the structural representation realizing the system of data query in an embodiment;
Fig. 6 is the structural representation of locating module in Fig. 5;
Fig. 7 is the structural representation of the system realizing data query in another embodiment;
Fig. 8 is the structural representation of optimization process module in Fig. 7.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.
Unless context separately has the description of specific distinct, the element in the present invention and assembly, the form that quantity both can be single exists, and form that also can be multiple exists, and the present invention does not limit this.Although the step in the present invention arranges with label, and be not used in and limit the precedence of step, the order of step or the execution of certain step need based on other steps unless expressly stated, otherwise the relative rank of step is adjustable.Be appreciated that term "and/or" used herein relates to and contains the one or more any and all possible combination in the Listed Items be associated.
As shown in Figure 1, in one embodiment, a kind of method realizing data query, comprises the steps:
Step S110, obtains inquiry request.
In the present embodiment, visualization interface will be provided, to obtain the inquiry request that user triggers, such as, the query interface input inquiry request of data of user by existing with the form such as the java DataBase Control of JDBC/ODBC standard and web interface, wherein, inquiry request can comprise the information such as the file name corresponding to data file, priority level, inquiry field, export path that user's appointment is carried out inquiring about.
In a preferred embodiment, the inquiry request obtained will be one or more, therefore, the inquiry request of acquisition will be saved to database table, such as, in mysql table, to perform in order as several executable tasks.
Concrete, the scheduling of task will be realized by arranging timer.By the timer triggering timing arranged, with the unenforced task of preserving in automatic cycle monitor database table, judge whether to there is unenforced task, if yes, then according to the sequence still unenforced task in performing database table successively of priority level, and be executed by the state updating of this task after running succeeded.
Step S130, locates the row in the bucket and this barrel that in the data file subregion based on Hive, inquiry request is corresponding.
In the present embodiment, the data file based on Hive will be present in the catalogue of HDFS (Hadoop distributed file system), and each data file is the form of tables of data, and possesses identical data list structure, so that carry out the fast query of data.
Concrete, the data stored in data file are table data, these data carry out subregion in Hive, bucket is divided to store, that is, subregion creates a directory in Hive, its catalogue is called partition information, then data divided bucket stored in this catalogue, to realize the optimizing process that data store, and then make the bucket only needing to find data place in the query script of data, again the data in bucket are searched and can obtain required data, and do not need to search one by one the data of all storages, effectively improve search efficiency, decrease the demand of internal memory, reduce data in internetwork transmission.
Step S150, reads the data corresponding to row in the bucket of locating and obtaining.
In the present embodiment, carry out data search according to the inquiry field in inquiry request locating in the bucket that obtains, to obtain the row relevant to this inquiry field, and then read the data corresponding to these row, these data are the current Query Result carrying out data query.
Further, the data that obtained and in inquiry request inquiry field is relevant are mostly more, therefore, the data just read are merged into a file, and the data search of pointing out user current carried out runs succeeded, in the query interface of data, provide Query Result to derive link, derive the file containing a large amount of Query Result to facilitate user.
Further, be also saved in compressed file by merging by the data read the file obtained, to save storage space, in a preferred embodiment, this compressed file is cvs (Comma Separated Value) compressed file.
As shown in Figure 2, in one embodiment, the detailed process of above-mentioned steps S130 is:
Step S131, conversion query request is MapReduce task.
In the present embodiment, inquiry request is HQL statement, obtain after inquiry request, be converted into corresponding MapReduce task at bottom, and then parallel processing carried out to the data being distributed in each node, effectively will improve treatment effeciency.
Step S133, obtains metadata, obtains the data file subregion based on Hive relevant to MapReduce task according to metadata.
In the present embodiment, metadata includes the file name of data place data file, this file name will indicate the information such as the subregion at data place in data file, therefore, the subregion to be checked at the data file place based on Hive relevant to MapReduce task can be obtained according to metadata, and then realize searching roughly of data.
Step S135, data store organisation according to definition calculates to obtain corresponding informative abstract value to the inquiry field in MapReduce task, and obtaining the data storage location corresponding with inquiring about field by the delivery of informative abstract value and default barrelage amount, this data storage location is used to indicate the row in bucket corresponding to inquiry field and bucket.
In the present embodiment, extract the inquiry field in MapReduce task, to carry out the calculating of informative abstract value to this inquiry field, such as, by Hash (Hash) algorithm realization, to obtain corresponding informative abstract value.The inquiry field extracted is specified being according to carrying out the data store organisation of definition when data store, therefore, does not need to extract the calculating that all inquiry fields carry out informative abstract value.Concrete, specify according to the field of definition when carrying out data and storing, such as, the storage of data will be realized by statement " create tableip_login (`uid`string; `ip`string; `time`int) PARTITIONED BY (tm int) CLUSTERED BY (`uid`) SORTED BY (`uid`) INTO2BUCKETS STORED ASORC ", wherein, the inquiry field of carrying out the calculating of informative abstract value is needed will to be specified by CLUSTERED BY.
After calculating corresponding informative abstract value, point bucket of data will be carried out by informative abstract value and default barrelage amount, and then obtain the bucket at the data place relevant to this inquiry field.Concrete, by with calculating gained to informative abstract value and preset barrelage measure the cutting that mould carries out data storage location, accurately to obtain the data storage location corresponding to the data relevant to inquiring about field, namely store the bucket of the data relevant to inquiring about field.
Such as, in the statement enumerated as mentioned above, that is specified by CLUSTERED BY needs the inquiry field of carrying out the calculating of informative abstract value to be uid field, now, will by uid field computing information digest value, then by this value and barrelage amount, " 2 " delivery namely in " 2BUCKETS " obtains corresponding data storage location.
Now, the region limited being carried out the inquiry of data at the bucket storing the data relevant to inquiring about field, obtaining storing the row of the data relevant to inquiring about field to search.
By data query as above, by the meticulous inquiry not needing the inquiry carrying out data to all data files can realize data, and then make the inquiry of data arbitrarily can be applied to mass data, while improve search efficiency and inquiry velocity, also improve the degree of accuracy of inquiry.
In one embodiment, the detailed process of above-mentioned steps S150 is: carry out Data import according to the row in the bucket that inquiry field is corresponding, and process the data loaded.
In the present embodiment, load the data of the row be recorded in bucket corresponding to inquiry field, and suitably processed, so that provide corresponding Query Result to user by the data loaded.Concrete, the process carried out the data loaded can be can be shown in the query interface of data one by one by the data of loading; Also can merge into a file, and store according to the path of presetting, and then in the query interface of data, provide Query Result to derive link, i.e. the derivation path of this file, user carries out checking of Query Result by the mode of export; Also can be that these row are calculated accordingly, such as, be associated with other data file, train value duplicate removal etc.
As shown in Figure 3, in one embodiment, before above-mentioned steps S130, the method also comprises the steps:
Step S210, receives the raw data of input, and raw data is stored as the first data list structure.
In the present embodiment, the first data list structure defines the storage organization of raw data.To the raw data input in different pieces of information source be received, now, raw data will directly be stored in the respective directories of HDFS according to the first data list structure.
Concrete, the loading of raw data can not carry out the Optimum Operations such as any verification, compression to this raw data, and just raw data directly be uploaded in the respective directories of HDFS according to the first data list structure.
This data file corresponding to the first data list structure will be also the form of tables of data, and the raw data of also will only preserve this and load, will directly cover original raw data when again carrying out the loading of raw data.
Further, in this data file corresponding to the first data list structure, its file name will be configured by configuration file, such as, can be that the physical file name affix corresponding to raw data identifies accordingly, such as, user_gambling_text, wherein, user_gambling is the physical file name corresponding to raw data, and text is additional mark.
Step S230, is optimized process to the raw data being stored as the first data list structure, with the data that optimization process is obtained stored in configuration file configure based in the data file subregion of Hive.
In the present embodiment, to the raw data being stored as the first data list structure be carried out to subregion, point optimization process such as bucket, sequence, with the storage mode of optimization data, and then data optimization process to be obtained are stored in based in the data file subregion of Hive, current-period data and historical data are separated, greatly promote Storage and Processing efficiency.
Further, the corresponding data file of data configuration obtained for optimization process by configuration file, to obtain corresponding metadata, so be convenient to carry out data stored in and inquiry.
In one embodiment, the detailed process of above-mentioned steps S210 is:
By configuration file by raw data stored in comprising in the data file of partition information, wherein, be stored into each row raw data in the raw data of this data file and store with JSON form.
In the present embodiment, by configuration file for current carried out raw data is stored in the corresponding metadata of configuration, namely the filename corresponding to raw data configuration data file of the first data list structure is loaded as, according to the definition of configuration file, the physical file name affix that now filename of data file is corresponding to raw data identifies accordingly, contain corresponding partition information, the subregion of the data file that raw data stores can be known according to filename.
Further, the raw data of storing data files by with row be unit realize raw data stored in, namely each row raw data all will store with JSON form.In raw data, every a line all contains multiple row, therefore, when raw data is stored according to the first data list structure, the raw data extracted corresponding to each row is stored according to JSON form, and separate between each row raw data with character or other form, such as "-" or tab key, so that be able to store according to being classified as unit in follow-up storing process.As shown in Figure 4, in one embodiment, the detailed process of above-mentioned steps S230 is:
Step S231, extracts the data of the every a line JSON form be stored as in the raw data of the first data list structure one by one.
Step S232, obtains the district location to the data file based on Hive that the raw data of the first data list structure stores by the data file comprising partition information.
In the present embodiment, the raw data being the first data list structure by the rule defined in configuration file configures corresponding metadata, namely be used to indicate stored in the filename corresponding to the data file of the raw data being loaded as the first data list structure, because data file is tables of data, therefore, this file star include raw data be about to stored in data table name and the partition information that is stored in tables of data of raw data.Such as, in " user_gambling-plat=604, tm=20130930-20131001-0 " this filename, user_gambling is data table name, plat=604, tm=20130930 then represent that raw data is by the child partition tm=20130930 that is stored under subregion plat=604.
Further, obtained the partition information of the storage of current carried out raw data by filename, i.e. " plat " and " tm " this numerical value corresponding to two fields.Such as, the 3rd index and first index can be got according to the actual file name of this raw data, obtain the numerical value corresponding to field " plat " according to the 3rd index, obtain the numerical value corresponding to field " tm " according to first index, and then form relevant partition information.
Step S233, calculates to obtain corresponding informative abstract value to the raw data extracted, and obtains the data storage location of raw data in the subregion of data file by informative abstract value and the delivery of the barrelage amount preset.
In the present embodiment, as mentioned above, this data storage location by indicate be about to stored in the subregion of data file and bucket, and then realize accurately storing.
The data of extraction are carried out compression process by step S234, and store according to data storage location after compression process.
In the present embodiment, obtain the data of this extraction, compression process is carried out to it, to save storage space, and then carry out stored in data file according to the data storage location obtained after completing compression process.
Further, for realize data-optimized stored in data file be column storage file, carry out compression and process choosing optimum compressed format according to the type of raw data in each row, with the storage space shared by maximized minimizing.
Such as, this column storage file can be ORC file, each ORC file is made up of multiple stripes, wherein, stripes is made up of the index of a lightweight, column data and stripe script, the maximal value arranged in the index default storage capable data of predetermined number and minimum value, for the query context of locator data accurate in the query script of data, thus save a large amount of input and output operations.
Further, also by stored in raw data according to extract raw data sort, to accelerate the inquiry velocity of data further.
In actual operation process, by reservation two parts of data files, and be placed in two different machines, to guarantee can automatically recover after any machine breakdown in cluster, be both the concurrency also improving process.
As shown in Figure 5, in one embodiment, a kind of system realizing data query, comprises acquisition request module 110, locating module 130 and output module 150.
Acquisition request module 110, for obtaining inquiry request.
In the present embodiment, acquisition request module 110 will provide visualization interface, to obtain the inquiry request that user triggers, such as, the query interface input inquiry request of data of user by existing with the form such as the java DataBase Control of JDBC/ODBC standard and web interface, wherein, inquiry request can comprise the information such as the file name corresponding to data file, priority level, inquiry field, export path that user's appointment is carried out inquiring about.
In a preferred embodiment, the inquiry request that acquisition request module 110 obtains will be one or more, therefore, the inquiry request of acquisition will be saved to database table, such as, in mysql table, to perform in order as several executable tasks.
Concrete, the scheduling of task will be realized by arranging timer.By the timer triggering timing arranged, with the unenforced task of preserving in automatic cycle monitor database table, judge whether to there is unenforced task, if yes, then according to the sequence still unenforced task in performing database table successively of priority level, and be executed by the state updating of this task after running succeeded.
Locating module 130, for locating the row in bucket and this barrel that in the data file subregion based on Hive, inquiry request is corresponding.
In the present embodiment, the data file based on Hive will be present in the catalogue of HDFS, and each data file is the form of tables of data, and possesses identical data list structure, so that carry out the fast query of data.
Concrete, the data stored in data file are table data, these data carry out subregion in Hive, bucket is divided to store, that is, subregion creates a directory in Hive, its catalogue is called partition information, then data divided bucket stored in this catalogue, to realize the optimizing process that data store, and then make locating module 130 in the query script of data only need to find the bucket at data place, again the data in bucket are searched and can obtain required data, and do not need to search one by one the data of all storages, effectively improve search efficiency, decrease the demand of internal memory, reduce data in internetwork transmission.
Read module 150, for reading the data corresponding to the row of locating in the bucket that obtains.
In the present embodiment, data search is carried out locating in the bucket that obtains according to the inquiry field in inquiry request, to obtain the row relevant to this inquiry field, and then read module 150 reads the data corresponding to these row, and these data are the current Query Result carrying out data query.
Further, the data that obtained and in inquiry request inquiry field is relevant are mostly more, therefore, the data that read module 150 just reads are merged into a file, and the data search of pointing out user current carried out runs succeeded, in the query interface of data, provide Query Result to derive link, derive the file containing a large amount of Query Result to facilitate user.
Further, read module 150 is also saved in compressed file by merging by the data read the file obtained, and to save storage space, in a preferred embodiment, this compressed file is cvs compressed file.
As shown in Figure 6, in one embodiment, above-mentioned locating module 130 comprises converting unit 131, partitions of file acquiring unit 133 and position acquisition unit 135.
Converting unit 131 is MapReduce task for conversion query request.
In the present embodiment, inquiry request is HQL statement, obtain after inquiry request, converting unit 131 is converted into corresponding MapReduce task at bottom, and then carries out parallel processing to the data being distributed in each node, effectively will improve treatment effeciency.
Partitions of file acquiring unit 133, for obtaining metadata, obtains the data file based on Hive relevant to MapReduce task according to metadata.
In the present embodiment, metadata includes the file name of data place data file, this file name will indicate the information such as the subregion at data place in data file, therefore, partitions of file acquiring unit 133 can obtain the subregion to be checked at the data file place based on Hive relevant to MapReduce task according to metadata, and then realizes searching roughly of data.
Position acquisition unit 135, for calculating to obtain corresponding informative abstract value to the inquiry field in MapReduce task according to the data store organisation of definition, and obtaining the data storage location corresponding with inquiring about field by the delivery of informative abstract value and default barrelage amount, data storage location is used to indicate the row in bucket corresponding to inquiry field and bucket.
In the present embodiment, position acquisition unit 135 extracts the inquiry field in MapReduce task, to carry out the calculating of informative abstract value to this inquiry field, such as, by Hash (Hash) algorithm realization, to obtain corresponding informative abstract value.
The inquiry field that position acquisition unit 135 extracts is specified being according to carrying out the data store organisation of definition when data store, therefore, does not need to extract the calculating that all inquiry fields carry out informative abstract value.Concrete, specify according to the field of definition when carrying out data and storing, such as, the storage of data will be realized by statement " createtable ip_login (`uid`string; `ip`string; `time`int) PARTITIONED BY (tm int) CLUSTERED BY (`uid`) SORTED BY (`uid`) INTO2BUCKETS STORED ASORC ", wherein, the inquiry field of carrying out the calculating of informative abstract value is needed will to be specified by CLUSTERED BY.
After calculating corresponding informative abstract value, position acquisition unit 135 will carry out point bucket of data by informative abstract value and default barrelage amount, and then obtains the bucket at the data place relevant to this inquiry field.Concrete, position acquisition unit 135 by with calculating gained to informative abstract value and preset barrelage measure the cutting that mould carries out data storage location, accurately to obtain the data storage location corresponding to the data relevant to inquiring about field, namely store the bucket of the data relevant to inquiring about field.
Such as, in the statement enumerated as mentioned above, that is specified by CLUSTERED BY needs the inquiry field of carrying out the calculating of informative abstract value to be uid field, now, will by uid field computing information digest value, then by this value and barrelage amount, " 2 " delivery namely in " 2BUCKETS " obtains corresponding data storage location.
Now, the inquiry of data is carried out in the region limited at the bucket storing the data relevant to inquiring about field by position acquisition unit 135, obtains storing the row of the data relevant to inquiring about field to search.
By data query as above, by the meticulous inquiry not needing the inquiry carrying out data to all data files can realize data, and then make the inquiry of data arbitrarily can be applied to mass data, while improve search efficiency and inquiry velocity, also improve the degree of accuracy of inquiry.
In one embodiment, read module 150 also for carrying out Data import according to the row inquired about in bucket corresponding to field, and processes the data loaded.
In the present embodiment, read module 150 loads the data of the row be recorded in bucket corresponding to inquiry field, and is suitably processed, so that provide corresponding Query Result to user by the data loaded.Concrete, the process that read module 150 carries out the data loaded can be can be shown in the query interface of data one by one by the data of loading; Also can merge into a file, and store according to the path of presetting, and then in the query interface of data, provide Query Result to derive link, i.e. the derivation path of this file, user carries out checking of Query Result by the mode of export; Also can be that these row are calculated accordingly, such as, be associated with other data file, train value duplicate removal etc.
As shown in Figure 7, in another embodiment, said system also comprises raw data memory module 210 and optimization process module 230.
Raw data memory module 210, for receiving the raw data of input, and is stored as the first data list structure by raw data.
In the present embodiment, the first data list structure defines the storage organization of raw data.Raw data memory module 210 will receive the raw data input in different pieces of information source, now, raw data will directly be stored in the respective directories of HDFS according to the first data list structure.
Concrete, the loading of raw data can not carry out the Optimum Operations such as any verification, compression to this raw data, and just raw data directly be uploaded in the respective directories of HDFS according to the first data list structure.
This data file corresponding to the first data list structure will be also the form of tables of data, and the raw data of also will only preserve this and load, will directly cover original raw data when again carrying out the loading of raw data.
Further, in this data file corresponding to the first data list structure, its file name will be configured by configuration file, such as, can be that the physical file name affix corresponding to raw data identifies accordingly, such as, user_gambling_text, wherein, user_gambling is the physical file name corresponding to raw data, and text is additional mark.
Optimization process module 230, for being optimized process to the raw data being stored as the first data list structure, with the data that optimization process is obtained stored in configuration file configure based in the data file subregion of Hive.
In the present embodiment, optimization process module 230 will carry out subregion, point optimization process such as bucket, sequence to the raw data being stored as the first data list structure, with the storage mode of optimization data, and then data optimization process to be obtained are stored in based in the data file subregion of Hive, current-period data and historical data are separated, greatly promote Storage and Processing efficiency.
Further, the corresponding data file of data configuration that optimization process module 230 is obtained for optimization process by configuration file, to obtain corresponding metadata, so be convenient to carry out data stored in and inquiry.
In one embodiment, above-mentioned raw data load-on module 210 also for by configuration file by raw data stored in comprising in partition information data file, wherein, the source document being stored into data file stores with JSON form.
In the present embodiment, raw data memory module 210 by configuration file for current carried out raw data stored in configuration corresponding metadata, namely the filename corresponding to raw data configuration data file of the first data list structure is loaded as, according to the definition of configuration file, the physical file name affix that now filename of data file is corresponding to raw data identifies accordingly, contain corresponding partition information, the subregion of the data file that raw data stores can be known according to filename.
Further, the raw data of raw data memory module 210 storing data files by with row be unit realize raw data stored in, namely each row raw data all will store with JSON form.In raw data, every a line all contains multiple row, therefore, when raw data is stored according to the first data list structure, the raw data extracted corresponding to each row stores according to JSON form by raw data memory module 210, and separate between each row raw data with character or other form, such as "-" or tab key, so that be able to store according to being classified as unit in follow-up storing process.
As shown in Figure 8, in one embodiment, above-mentioned optimization process module 230 comprises extraction unit 231, subregion acquiring unit 233, position arithmetic element 235 and storage unit 237.
Extraction unit 231, for extracting the data of the every a line JSON form be stored as in the raw data of the first data list structure one by one.
Subregion acquiring unit 233, obtains the district location to the data file based on Hive that the raw data of the first data list structure stores for the data file by comprising partition information.
In the present embodiment, the raw data that subregion acquiring unit 233 is the first data list structure by the rule defined in configuration file configures corresponding metadata, namely be used to indicate stored in the filename corresponding to the data file of the raw data being loaded as the first data list structure, because data file is tables of data, therefore, this file star include raw data be about to stored in data table name and the partition information that is stored in tables of data of raw data.Such as, in " user_gambling-plat=604, tm=20130930-20131001-0 " this filename, user_gambling is data table name, plat=604, tm=20130930 then represent that raw data is by the child partition tm=20130930 that is stored under subregion plat=604.
Further, subregion acquiring unit 233 obtains the partition information of the storage of current carried out raw data by filename, i.e. " plat " and " tm " this numerical value corresponding to two fields.Such as, the 3rd index and first index can be got according to the actual file name of this raw data, obtain the numerical value corresponding to field " plat " according to the 3rd index, obtain the numerical value corresponding to field " tm " according to first index, and then form relevant partition information.
Position arithmetic element 25, for calculating to obtain corresponding informative abstract value to the raw data extracted, and obtains the data storage location of raw data in the subregion of data file by informative abstract value and the delivery of the barrelage amount preset.
In the present embodiment, as mentioned above, this data storage location by indicate be about to stored in the subregion of data file and bucket, and then realize accurately storing.
Storage unit 237, carries out compression process for the raw data that the data of extraction are relevant, and stores according to data storage location after compression process.
In the present embodiment, storage unit 237 obtains the data of this extraction, carries out compression process to it, to save storage space, and then carries out stored in data file according to the data storage location obtained after completing compression process.
Further, for realize data-optimized stored in data file be column storage file, storage unit 237 is carried out compression and is processed choosing optimum compressed format according to the type of raw data in each row, with the storage space shared by maximized minimizing.
Such as, this column storage file can be ORC file, each ORC file is made up of multiple stripes, wherein, stripes is made up of the index of a lightweight, column data and stripe script, the maximal value arranged in the index default storage capable data of predetermined number and minimum value, for the query context of locator data accurate in the query script of data, thus save a large amount of input and output operations.
Further, storage unit 237 also by stored in raw data according to extract raw data sort, to accelerate the inquiry velocity of data further.
In actual operation process, by reservation two parts of data files, and be placed in two different machines, to guarantee can automatically recover after any machine breakdown in cluster, be both the concurrency also improving process.
The above embodiment only have expressed several embodiment of the present invention, and it describes comparatively concrete and detailed, but therefore can not be interpreted as the restriction to the scope of the claims of the present invention.It should be pointed out that for the person of ordinary skill of the art, without departing from the inventive concept of the premise, can also make some distortion and improvement, these all belong to protection scope of the present invention.Therefore, the protection domain of patent of the present invention should be as the criterion with claims.
Claims (12)
1. realize a method for data query, comprise the steps:
Obtain inquiry request;
Locate the row in the bucket and described bucket that described in the data file subregion based on Hive, inquiry request is corresponding;
Read the data corresponding to row in the bucket that described location obtains.
2. method according to claim 1, is characterized in that, described location based on Hive data file subregion described in the step of row in bucket corresponding to inquiry request and described bucket comprise:
Change described inquiry request into MapReduce task;
Obtain metadata, obtain the data file subregion based on Hive relevant to described MapReduce task according to described metadata;
Data store organisation according to definition calculates to obtain corresponding informative abstract value to the inquiry field in described MapReduce task, and obtaining the data storage location corresponding with described inquiry field by the delivery between described informative abstract value and default barrelage amount, described data storage location is used to indicate the row in bucket corresponding to described inquiry field and described bucket.
3. method according to claim 2, is characterized in that, the step of the data corresponding to row in the bucket that the described location of described reading obtains comprises:
Carry out Data import according to the row in the bucket that described inquiry field is corresponding, and the data of described loading are processed.
4. method according to claim 1, is characterized in that, described location based on Hive data file subregion described in row in bucket corresponding to inquiry request and described bucket step before, described method also comprises:
Receive the raw data of input, and described raw data is stored as the first data list structure;
Process is optimized to the described raw data being stored as the first data list structure, with the data that optimization process is obtained stored in configuration file configure based in the data file subregion of Hive.
5. method according to claim 4, is characterized in that, the described raw data receiving input, and the step described raw data being stored as the first data list structure comprises:
Configured by described raw data stored in comprising in the data file of partition information by configuration file, wherein, the raw data being stored into described data file stores with JSON form.
6. method according to claim 4, it is characterized in that, describedly be optimized process to the described raw data being stored as the first data list structure, what configure stored in configuration file with data optimization process obtained comprises based on the step in the data file subregion of Hive:
Extract the data of the every a line JSON form be stored as in the raw data of the first data list structure one by one;
The district location to the data file based on Hive that the raw data of described first data list structure stores is obtained by the data file comprising partition information;
Calculate to obtain corresponding informative abstract value to the raw data of described extraction, and obtain the data storage location of raw data described in the subregion of data file by described informative abstract value and the delivery of the barrelage amount preset;
The data of described extraction are carried out compression process, and store according to described data storage location after compression process.
7. realize a system for data query, it is characterized in that, comprising:
Acquisition request module, for obtaining inquiry request;
Locating module, for locating the row in bucket and described bucket that described in the data file subregion based on Hive, inquiry request is corresponding;
Read module, for reading the data corresponding to the row in bucket that described location obtains.
8. system according to claim 7, is characterized in that, described locating module comprises:
Converting unit, for changing described inquiry request into MapReduce task;
Partitions of file acquiring unit, for obtaining metadata, obtains the data file subregion based on Hive relevant to described MapReduce task according to described metadata;
Position acquisition unit, for calculating to obtain corresponding informative abstract value to the inquiry field in described MapReduce task according to the data store organisation of definition, and obtaining the data storage location corresponding with described inquiry field by the delivery between described informative abstract value and default barrelage amount, described data storage location is used to indicate the row in bucket corresponding to described inquiry field and described bucket.
9. system according to claim 8, is characterized in that, described read module also for carrying out Data import according to the row in bucket corresponding to inquiry field, and is output in data to the data of described loading and processes.
10. system according to claim 7, is characterized in that, described system also comprises:
Raw data memory module, for receiving the raw data of input, and is stored as the first data list structure by described raw data;
Optimization process module, for being optimized process to the described raw data being stored as the first data list structure, with the data that optimization process is obtained stored in configuration file configure based in the data file subregion of Hive.
11. systems according to claim 10, it is characterized in that, described raw data memory module is also for configuring by described raw data stored in comprising in the data file of partition information by configuration file, and wherein, the raw data being stored into described data file stores with JSON form.
12. systems according to claim 10, is characterized in that, described optimization process module comprises:
Extraction unit, for extracting the data of the every a line JSON form be stored as in the raw data of the first data list structure one by one;
Subregion acquiring unit, obtains the district location to the data file based on Hive that the raw data of described first data list structure stores for the data file by comprising partition information;
Position arithmetic element, obtains corresponding informative abstract value for calculating the raw data of described extraction, and obtains the data storage location of raw data described in the subregion of data file by described informative abstract value and the delivery of the barrelage amount preset;
Storage unit, carries out compression process for the raw data that the data of described extraction are relevant, and stores according to described data storage location after compression process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410183883.4A CN105022763B (en) | 2014-04-30 | 2014-04-30 | Realize the method and system of data query |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410183883.4A CN105022763B (en) | 2014-04-30 | 2014-04-30 | Realize the method and system of data query |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105022763A true CN105022763A (en) | 2015-11-04 |
| CN105022763B CN105022763B (en) | 2019-03-26 |
Family
ID=54412742
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410183883.4A Active CN105022763B (en) | 2014-04-30 | 2014-04-30 | Realize the method and system of data query |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105022763B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105808661A (en) * | 2016-02-29 | 2016-07-27 | 浪潮通信信息系统有限公司 | Data query method and device |
| CN106682838A (en) * | 2016-12-30 | 2017-05-17 | 北京水兵壹号科技有限公司 | Barreled water distribution method and device based on big data |
| CN108322331A (en) * | 2017-12-28 | 2018-07-24 | 山东鲁能智能技术有限公司 | A kind of communication means and system being dynamically adapted interactive measuring point |
| CN109902126A (en) * | 2019-02-18 | 2019-06-18 | 国家计算机网络与信息安全管理中心 | Support the loading system and its implementation of HIVE auto-partition |
| CN110362577A (en) * | 2019-07-10 | 2019-10-22 | 星环信息科技（上海）有限公司 | A kind of data insertion method, device, equipment and storage medium |
| CN112231351A (en) * | 2020-10-14 | 2021-01-15 | 广东亿迅科技有限公司 | Real-time query method and device for PB-level mass data |
| CN112286587A (en) * | 2020-06-29 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data visualization page loading method and device, server, client and medium |
| CN113284573A (en) * | 2021-06-02 | 2021-08-20 | 山东健康医疗大数据有限公司 | Method and device for searching document database |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110295855A1 (en) * | 2010-05-31 | 2011-12-01 | Microsoft Corporation | Graph-Processing Techniques for a MapReduce Engine |
| CN102609487A (en) * | 2012-01-20 | 2012-07-25 | 东华大学 | Column-storage-oriented Hash joint method for indexes in barrels |
| CN102685221A (en) * | 2012-04-29 | 2012-09-19 | 华北电力大学（保定） | Distributed storage and parallel mining method for state monitoring data |
| CN103425762A (en) * | 2013-08-05 | 2013-12-04 | 南京邮电大学 | Telecom operator mass data processing method based on Hadoop platform |
| CN103440288A (en) * | 2013-08-16 | 2013-12-11 | 曙光信息产业股份有限公司 | Big data storage method and device |
-
2014
- 2014-04-30 CN CN201410183883.4A patent/CN105022763B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110295855A1 (en) * | 2010-05-31 | 2011-12-01 | Microsoft Corporation | Graph-Processing Techniques for a MapReduce Engine |
| CN102609487A (en) * | 2012-01-20 | 2012-07-25 | 东华大学 | Column-storage-oriented Hash joint method for indexes in barrels |
| CN102685221A (en) * | 2012-04-29 | 2012-09-19 | 华北电力大学（保定） | Distributed storage and parallel mining method for state monitoring data |
| CN103425762A (en) * | 2013-08-05 | 2013-12-04 | 南京邮电大学 | Telecom operator mass data processing method based on Hadoop platform |
| CN103440288A (en) * | 2013-08-16 | 2013-12-11 | 曙光信息产业股份有限公司 | Big data storage method and device |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105808661B (en) * | 2016-02-29 | 2019-03-08 | 浪潮天元通信信息系统有限公司 | A kind of method and device of data query |
| CN105808661A (en) * | 2016-02-29 | 2016-07-27 | 浪潮通信信息系统有限公司 | Data query method and device |
| CN106682838A (en) * | 2016-12-30 | 2017-05-17 | 北京水兵壹号科技有限公司 | Barreled water distribution method and device based on big data |
| CN108322331B (en) * | 2017-12-28 | 2020-11-06 | 国网智能科技股份有限公司 | Communication method and system capable of dynamically adjusting interactive measuring points |
| CN108322331A (en) * | 2017-12-28 | 2018-07-24 | 山东鲁能智能技术有限公司 | A kind of communication means and system being dynamically adapted interactive measuring point |
| CN109902126A (en) * | 2019-02-18 | 2019-06-18 | 国家计算机网络与信息安全管理中心 | Support the loading system and its implementation of HIVE auto-partition |
| CN109902126B (en) * | 2019-02-18 | 2021-12-07 | 国家计算机网络与信息安全管理中心 | Loading system supporting HIVE automatic partition and implementation method thereof |
| CN110362577B (en) * | 2019-07-10 | 2020-06-09 | 星环信息科技（上海）有限公司 | Data insertion method, device, equipment and storage medium |
| WO2021004266A1 (en) * | 2019-07-10 | 2021-01-14 | 星环信息科技（上海）有限公司 | Data insertion method and apparatus, device and storage medium |
| CN110362577A (en) * | 2019-07-10 | 2019-10-22 | 星环信息科技（上海）有限公司 | A kind of data insertion method, device, equipment and storage medium |
| CN112286587A (en) * | 2020-06-29 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data visualization page loading method and device, server, client and medium |
| CN112231351A (en) * | 2020-10-14 | 2021-01-15 | 广东亿迅科技有限公司 | Real-time query method and device for PB-level mass data |
| CN113284573A (en) * | 2021-06-02 | 2021-08-20 | 山东健康医疗大数据有限公司 | Method and device for searching document database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105022763B (en) | 2019-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105022763A (en) | Method and system for implementing data query | |
| US11423082B2 (en) | Methods and apparatus for subgraph matching in big data analysis | |
| US8756237B2 (en) | Scalable distributed processing of RDF data | |
| CN102693302B (en) | Quick file comparison method, system and client side | |
| CN104298760B (en) | A kind of data processing method and data processing equipment applied to data warehouse | |
| CN106959963B (en) | Data query method, device and system | |
| CN108694195B (en) | Management method and system of distributed data warehouse | |
| CN105224658B (en) | A kind of Query method in real time and system of big data | |
| US11093461B2 (en) | Method for computing distinct values in analytical databases | |
| CN105956123A (en) | Local updating software-based data processing method and apparatus | |
| Xu et al. | TSAaaS: Time series analytics as a service on IoT | |
| US20180129708A1 (en) | Query processing management in a database management system | |
| US20140201192A1 (en) | Automatic data index establishment method | |
| CN105303456A (en) | Method for processing monitoring data of electric power transmission equipment | |
| CN106970929B (en) | Data import method and device | |
| CN105653592A (en) | Small file merging tool and method based on HDFS | |
| US10664460B2 (en) | Index B-tree maintenance for linear sequential insertion | |
| CN106503008B (en) | File storage method and device and file query method and device | |
| CN104778182A (en) | Data import method and system based on HBase (Hadoop Database) | |
| US8266180B2 (en) | Validation of database schema upgrades using a data differencing tool | |
| CN114911830A (en) | Index caching method, device, equipment and storage medium based on time sequence database | |
| US9971793B2 (en) | Database management system and database management method | |
| CN103279529A (en) | Unstructured data retrieval method and system | |
| CN104156385A (en) | Method and device for processing time series data | |
| CN105069101A (en) | Distributed index construction and search method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20161114 Address after: 518000 Guangdong city of Shenzhen province Nanshan District Xili liuxiandong Zhongshan Road No. 1001 TCL Science Park R & D building D3 8 storey building A room 801 unit Applicant after: Shenzhen City, Oriental Boya Technology Co. Ltd. Address before: 518057 Guangdong city of Shenzhen province Nanshan District Zhongshan Road No. 1001 TCL Industrial Park International City building D3 9B-C E Applicant before: Burson Marsteller network game development (Shenzhen) Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |