CN102332030A - Data storing, managing and inquiring method and system for distributed key-value storage system - Google Patents
Data storing, managing and inquiring method and system for distributed key-value storage system Download PDFInfo
- Publication number
- CN102332030A CN102332030A CN201110314927A CN201110314927A CN102332030A CN 102332030 A CN102332030 A CN 102332030A CN 201110314927 A CN201110314927 A CN 201110314927A CN 201110314927 A CN201110314927 A CN 201110314927A CN 102332030 A CN102332030 A CN 102332030A
- Authority
- CN
- China
- Prior art keywords
- data
- type
- storage system
- character string
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a data storing, managing and inquiring method and system for a distributed key-value storage system. The method comprises the following steps: before storing data, respectively converting data into ordered character strings according to different data classifications and then storing the ordered character strings into the distributed key-value storage system, wherein storage sequences of the converted character strings sequenced according a dictionary sequence are consistent with numerical value sequences before conversion; and during data retrieval, the same conversion is carried out on the numerical values in retrieval conditions. The method provided by the invention supports storage and retrieval of complex data classifications and includes complex data inquiring methods, such as multi-field attribute inquiring, multi-data classification inquiring, numerical value statistic analysis and the like.
Description
Technical field
The invention belongs to information security field, relate in particular to data storage, management and inquiry mechanism in distributed key-value (KEY-VALUE) text data store system.
Background technology
Ecommerce at present, system's running log, web are used to wait and have all been accumulated a large amount of text class data, and it is huge, even the data volume of last TB has also been collected by small business, for the application background of this large data sets, the traditional data storehouse can't be suitable for.To such scene, successful solution is to utilize distributed KEY-VALUE type NO-SQL database to deposit structure through setting up row, improves the compression efficiency and the storage size of data in conjunction with technology such as compression, piecemeals at present.Because the text data type simple, intuitive, be easy in the group system of isomery, move, in the at present typical large-scale distributed mass data storage system directly employing text data as the storage data source of bottom.The Bigtable of typical distributed NO-SQL database such as Google; The Hbase of Apache and the PNUTS of Yahoo etc.; All directly preserve the data source of text, set up the row storage organization and further realize distributed mass data storage management system on this basis.In text class data-storage system, text data is stored according to the ordering of dictionary preface; Can navigate to the record position that satisfies condition fast; Return this record then or return all records thereafter according to the order of sequence from this start-of-record; Because the expense of having avoided a large amount of random read operation to bring, accurately inquiry and interval query have very high search efficiency in text class data storage management system.
But in text class data-storage system; For non-character type data; Such as integer (INTEGER) type etc., inconsistent according to the dictionary preface ordering back storage order of text with the numerical value order, when data retrieval, can't give play to the batch of text class data-storage system and retrieve advantage.And existing text class data storage management system externally only can provide accurately search, the interval searches two kinds of methods.Search VALUE through KEY, promptly under the KEY=K1 condition, search corresponding VALUE; Search the VALUE in an interval through KEY, promptly in K1＜KEY＜K2 scope, search all VALUE.It is thus clear that this distributed text storage system of depositing based on row is no longer suitable for the application scenarios of complex query, operation such as multiple attribute retrieval that can't the implementation structure data, numerical value class statistics, analysis.This limitation has reduced text class distributed data base range of application in large data sets, structural data.
Summary of the invention
Therefore; The objective of the invention is to overcome the defective of above-mentioned prior art; A kind of data storage, management and querying method that is used for distributed KEY-VALUE data-storage system is provided, supports the structural data storage and inquiry of multiword section attribute, multidata type.
The objective of the invention is to realize through following technical scheme:
According to an aspect of the present invention, the invention provides a kind of date storage method that is used for distributed KEY-VALUE data-storage system, this method comprises:
Step 1) is corresponding character string according to the conversion regime of different data types with data-switching, so that the character string after the conversion is consistent with the numerical value order of the data before the order after the dictionary preface ordering and the conversion;
Step 2) character string that step 1) is obtained is stored by the ordering of dictionary preface.
In the above-mentioned date storage method, said data type comprises that value type, text, memory point divide metric IP the data type of address, the data type of storage time.
In the above-mentioned date storage method, the conversion regime of said value type is for converting the data of value type into the digit strings of fixed length through the mode of left side filling 0.
In the above-mentioned date storage method, the conversion regime of the data type of said storage time is for being 14 character string with the data-switching of the type for the length of " date Hour Minute Second " form, and wherein " year " is 4 character lengths, and all the other are 2 character lengths.
In the above-mentioned date storage method, said memory point divides the conversion regime of the data type of metric IP address to be:
At first, convert the data of the type into integer from the form of " a.b.c.d " by following computing formula: a * k
3+ b * k
2+ c * k
1+ d, wherein k is the natural number greater than 255;
The integer that will after formula calculates, obtain then converts the character string of fixed length into the mode of left side filling 0.
In the above-mentioned date storage method, the conversion regime of said text saves as character string for direct data with the type.
According to a further aspect of the invention, the invention provides a kind of data enquire method that is used to adopt the distributed KEY-VALUE data-storage system of above-mentioned date storage method, said method comprises:
Receive the search condition of user's input, and be corresponding character string with the data-switching in the search condition according to the conversion regime of different data types;
Based on said character string, search through accurate inquiry and/or interval that the KEY-VALUE data-storage system provides, locate concrete memory location, and return Query Result;
Conversion regime according to said different data type carries out inverse operation to obtain the data of respective type to Query Result.
According to a further aspect of the invention, the invention provides a kind of structural data management method that is used for distributed KEY-VALUE data-storage system, said method comprises:
Step 101 is come loading data with the unit that writes down; The form of said record is limited form, and the essential information of form is kept in the form metadata, and said form essential information comprises table name, a plurality of field names and corresponding data type thereof, and said record comprises all fields in the form;
Step 102 is a data files with each n recorded and stored, for each field of record, stores according to the above-mentioned date storage method of corresponding data type information employing of the preservation in the form metadata; Wherein, The corresponding content of different field is saved in the diverse location of identical file respectively; Adopt the interior index of piece to be identified at the deviation post that a plurality of fields are preserved in the same file in this document; Said interior index is made up of the memory location pointer of designation data in file, preserves index in the piece at each data file rear portion with the mode of appending.
In the said structure data managing method, said data file is stored in the distributed storage medium, carries out index with the B+ tree between the data file.
In the said structure data managing method, also comprise the step of inquiry, the step of said inquiry may further comprise the steps:
Whether the search condition according to the information judges of the preservation in the form metadata is imported is legal;
If legal, then based on the data type information of the field of the preservation in the form metadata, would be corresponding character string with the data-switching in the search condition according to the conversion regime of different data types;
Based on said character string, search through accurate inquiry and/or interval that the KEY-VALUE data-storage system provides, locate concrete memory location, and return qualified record;
Conversion regime according to said different data type carries out inverse operation to obtain the data of respective type to each field of record.
According to a further aspect of the invention, the invention provides a kind of distributed KEY-VALUE data-storage system, said system comprises:
The conversion regime that is used for according to different data types is the device of corresponding character string with data-switching, said after conversion the numerical value of the data before the character string that the obtains order after with the ordering of dictionary preface and the conversion in proper order consistent;
Being used for will be through the character string that the is converted to device by dictionary preface ordering storage.
In the above-mentioned data-storage system, said data type comprises that value type, text, memory point divide metric IP the data type of address, the data type of storage time.
In the above-mentioned data-storage system, the conversion regime of said value type is for converting the data of value type into the digit strings of fixed length through the mode of left side filling 0.The conversion regime of the data type of said storage time is for being 14 character string with the data-switching of the type for the length of " date Hour Minute Second " form, and wherein " year " is 4 character lengths, and all the other are 2 character lengths.The conversion regime of said text saves as character string for direct data with the type.
In the above-mentioned data-storage system, said memory point divides the conversion regime of the data type of metric IP address to be:
At first, convert the data of the type into integer from the form of " a.b.c.d " by following computing formula: a * k
3+ b * k
2+ c * k
1+ d, wherein k is the natural number greater than 255;
The integer that will after formula calculates, obtain then converts the character string of fixed length into the mode of left side filling 0.
In the above-mentioned data-storage system; Also comprise the data query device; It is used for according to the conversion regime of different data types is corresponding character string with the data-switching of search condition; And be used for searching through accurate inquiry and/or interval that the KEY-VALUE data-storage system provides, locate concrete memory location, and return Query Result.
In the above-mentioned data-storage system, also comprise:
Be used for coming the device of loading data with the unit of record; The form of said record is limited form, and the essential information of form is kept in the form metadata, and said form essential information comprises table name, a plurality of field names and corresponding data type thereof, and said record comprises all fields in the form;
Being used for each n recorded and stored is a data files, for each field of record, and the device that adopts above-mentioned date storage method to store according to the corresponding data type information of the preservation in the form metadata; Wherein, The corresponding content of different field is saved in the diverse location of identical file respectively; Adopt the interior index of piece to be identified at the deviation post that a plurality of fields are preserved in the same file in this document; Said interior index is made up of the memory location pointer of designation data in file, preserves index in the piece at each data file rear portion with the mode of appending.
Wherein said data file is stored in the distributed storage medium, carries out index with the B+ tree between the data file.
Compared with prior art, the invention has the advantages that:
The present invention carries out Data Format Transform when data storage, guarantee that the ordering of data value is consistent with text storage dictionary preface; When data retrieval, the numerical value in the search condition is carried out identical conversion.Making full use of on text data store, the retrieval advantage basis, through conversion, support the storage and retrieval of complex data type to data, comprise complicated data enquire methods such as multiword section attribute query, the inquiry of multidata type and numerical statistic analysis; Realize the function of numerous types of data management.This method can be given full play to the potentiality of distributed text data management system, the potential value of excavation mass data, and more rich data querying method externally is provided.
Description of drawings
Followingly the embodiment of the invention is described further with reference to accompanying drawing, wherein:
Fig. 1 is the synoptic diagram according to a plurality of records that meet form definition of the embodiment of the invention;
Fig. 2 is the schematic flow sheet according to the structural data management method that is used for distributed KEY-VALUE data-storage system of the embodiment of the invention;
Fig. 3 is according to the data load of the embodiment of the invention and transformation flow synoptic diagram;
Fig. 4 is according to the data storage of the embodiment of the invention and the configuration diagram of index;
Fig. 5 is the structured data query method flow diagram that is used for distributed KEY-VALUE data-storage system according to the embodiment of the invention;
Fig. 6 is the configuration diagram according to the distributed KEY-VALUE data-storage system of the embodiment of the invention.
Embodiment
In order to make the object of the invention, technical scheme and advantage are clearer, pass through specific embodiment to further explain of the present invention below in conjunction with accompanying drawing.Should be appreciated that specific embodiment described herein only in order to explanation the present invention, and be not used in qualification the present invention.
In one embodiment, a kind of date storage method that is used for distributed KEY-VALUE data-storage system is provided.This method is before the storage data; Be orderly character string with data-switching respectively according to different data types earlier; Then it is saved in the distributed KEY-VALUE data-storage system, storage order and the numerical value order before the conversion after the character string that obtains after conversion sorts with the dictionary preface are consistent, so that support complex data type such as INTEGER (integer); And the storage of custom data type and retrieval, for example be used for the IPFIELD type of storing IP address.The data type that adopts in the present embodiment comprises value type, IPFIELD (IP address field), text (TEXT), TIMESTAMP (timestamp).
More specifically, for the data of value type, be example with INTEGER (integer), need to support greater than, less than waiting mathematics compare operation, numerical value class statistical study operation, like sum operation SUM, the operation A of averaging VG etc.In the present embodiment; With maximum character number is that 10 decimal digit character string is represented the INTEGER categorical data; The integer range of its representative is (0-9999999999), when the numerical character number of input less than 10 the time, a left side fills 0, and to take length be 10 character string.Character string as a result such as numeral 1000 conversions is " 0000001000 ".In other embodiments, can be according to user's request or system configuration, be the digit strings of other length with the data-switching of integer type.Also can the data of value types such as real number, decimal be converted into the digit strings of fixed length through the mode of left side filling 0 according to user's request or system configuration.
The IPFIELD type is used for memory point and divides metric IPV4 address date, is a kind of data type that is provided with according to application scenarioss such as network data flows.The search rule of IPFIELD data type is retrieval accurate IP address or retrieves an IP address in the interval.At first, convert the data of the type into integer from the form of " a.b.c.d " by following computing formula: a * k
3+ b * k
2+ c * k
1+ d, wherein k is the natural number greater than 255; Resulting then integer converts the digit strings of fixed length into the mode of left side filling 0.Such as 10.0.0.1, it can convert integer 167772161 into, and computing method are: 10 * 256^3+0 * 256^2+0 * 256^1+1=167772161.Do the integer storage to 167772161 then, according to replenishing 0 on the left of the transformation rule of above-mentioned integer, the character string of generation is " 0167772161 ".
The content of text that the TEXT categorical data directly keeps input gets final product.Utilize the Retrieval Interface of existing text class data-storage system can directly support the query manipulation of text class, for example realize accurately searching, prefix matching is searched etc.
The TIMESTAMP class record is the timestamp information of unit with the second, supports accurately searching and the time interval search operation of timestamp.Its data layout is the digit strings of " date Hour Minute Second "; Such as the current time be " 20: 54: 18 on the 28th July in 2011 "; The string format of then preserving is " 20110728205418 "; Thereby the TIMESTAMP type stores is become the character string of fixed length, in the text storage system, can realize according to dictionary preface ordering storage.
After respectively data-switching being orderly character string according to different data types, it is saved in the distributed KEY-VALUE data-storage system.Can overcome like this in existing text class data-storage system for dictionary preface ordering back storage order and the numerical value order inconsistent problem of non-character type data, thereby when data retrieval, give full play of the retrieval advantage of text class data-storage system according to text.
For example, for the search condition of TIMESTAMP type and the search condition of TEXT type, can directly accurately search according to what text class data-storage system provided, interval lookup method is retrieved.For the INTEGER type field; Because when storage INTEGER categorical data mentioned above, all will carry out left zero padding fills to convert orderly character string into; Numerical value to the INTEGER type in the querying condition carries out identical transformation when data query so; And then utilize the accurate inquiry of text data store system, be positioned concrete memory location, and return Query Result.For special statistical operation, such as SUM, aggregate functions such as AVG are the result set that returns the statistical operation of the laggard line correlation of value corresponding according to the target string format conversion.The IPFIELD categorical data is mainly supported the accurate inquiry of IP address class, subnet inquiry (referring to inquire about the IP address in the specific network segment) or the like.Similarly, before beginning retrieval, switch process as indicated above that IPFIELD field in the search condition is changed.Such as inquiry all IP addresses in 192.16.18.1 arrives the 192.16.18.255 network segment, can transfer the integer that satisfies condition in the retrieval integer range [3222278657,3222278911] to.
In yet another embodiment, a kind of structural data management method that is used for distributed KEY-VALUE data-storage system also is provided.In this embodiment, adopted the way to manage of structure-oriented data, the management of data and storage are all carried out with the unit of being recorded as.Wherein, the form of record can be limited form, for example, can pass through definition tables, becomes user-defined record to a plurality of KEY-VALUE type data integration.The essential information of form and supplementary are kept in the form metadata.The form essential information comprises table name, field name, and information such as field type, these information are being carried out standard to the data of input, use when transforming and encapsulating.Supplementary is the query manipulation for the accelerating structure data, and information recorded comprises record sum, the current maximum of each field, minimum value etc.
Each form can comprise many records; Every record comprises defined all fields of form, and each field can have data, also can be sky; When being empty, use ' MULL ' key word to indicate; That is to say when when carrying out data processing, running into ' MULL ' key word, do not process or do not show that representing does not have content in this field.
Gather at present embodiment, the form essential information is that the user calls establishment and preservation when creating the form order.Typically the order of establishment form for example: CREATE TABLE DNS_TABLE (DOMAIN TEXT, VALUE IPFIELD, COUNT INTEGER, TIME TIMESTAMP).Wherein CREATE TABLE is the order of creating form, and DNS_TABLE is user-defined title.The implication of mentioned order is: create the form that table name is DNS_TABLE, comprise field DOMAIN, its data type is TEXT; Field VALUE, its data type IPFIELD; Field COUNT, its data type is INTEGER; Field TIME, its data type is TIMESTAMP.Wherein DOMAIN deposits domain-name information, like ' www.xx.com '; VALUE is the corresponding IP address date of domain name, and like ' 192.16.18.10 ', COUNT is the number of times of being visited in a period of time, adopts integer digit, and TIME is the timestamp information that this record produces, and is unit with the second.As shown in Figure 1, for example, " www.xx.com, 192.16.18.10,10,20110628101010 ", " www.yy.com, 192.16.18.8,1,20110628101011 " all are the complete records that meet defined tableau format.
Should be understood that the needs of user, can define the form of other type, such as the form of a storage of definition student information according to oneself; As CREATE TABLE STUDENT_INFO (NAME TEXT, AGE INTEGER, CLASSTEXT); Wherein, the title of definition is STUDENT_INFO, and three fields are arranged in the form, is respectively student's name (NAME), the TEXT type; Students ' age (AGE), INTEGER type and student class (CLASS), TEXT type.
The form metadata generates when creating form, can preserve always, when the deletion form, deletes.After form is created, all can carry out according to the form metadata to the operation that form is all.When data load, need about the data type in the essential information of form each field of input to be judged according to what in the form metadata, preserve; Such as for a field; If it is recorded as INTEGER in the form essential information; But imported character string ' aa ' expression do not meet type, makes the fault reason, returns the input error prompting.When data retrieval, also to transform according to the numerical value that the form essential information of in the form metadata, preserving and supplementary provide in to search condition and judge.
Should point out that above-mentioned list structure only plays example rather than restriction, concrete list structure can be provided with according to demand by the user, can comprise a plurality of fields that the user is required.In addition, in the data load process, some of each form supplementarys commonly used, such as total record count, information such as maximum, minimum time stamp also can regularly be saved in the form metadata, quicken the retrieving in later stage.
Shown in Figure 2 is the schematic flow sheet that is used for the structural data management method of distributed KEY-VALUE data-storage system.This structural data management method mainly may further comprise the steps:
Shown in Figure 3 is data load and transformation flow synoptic diagram.Wherein, data load is meant and reads raw data, according to the form metadata it is packaged into record.Can import data one by one with the unit of being recorded as, also can use data file to import data in batches, use between many records ' n ' cut apart, use separator ", " to cut apart between a plurality of fields.Then, comprise the content of each field,, carry out type and judge that do not meet the data type that defines in the tableau format, reporting errors also returns according to the form essential information of preserving in the form metadata to every record; The record that meets define styles carries out the conversion of data layout according to method mentioned above.Data load is for example ordered:
INSERT?INTO?DNS_TABLE?VALUES(‘www.google.com’，‘192.16.18.10’，‘10’，‘20110628101010’)；
The implication of mentioned order uses ', ' to cut apart for loading the record with four fields to the DNS_TABLE form between the field.Respectively the data of each field of this record are carried out format conversion according to method mentioned above, for example, its left side is replenished 0, the corresponding character string " 0000000010 " in order of generation for the data " 10 " of INGERTER type.
Step 2, data storage.
Shown in Figure 4 is the synoptic diagram of data storage and index.Wherein, Each field for record; To pass through field value after the conversion according to dictionary preface ordering storage, the corresponding content of different field is saved in the diverse location of file respectively, in order in same file, to distinguish the deviation post that a plurality of field reality are preserved hereof; Introduce the interior index of piece and identify, index is to be made up of the memory location pointer of designation data in file in the piece.Preserve index in the piece at each data file rear portion with the mode of appending.In the piece for example shown in Figure 4 < DOMAIN Pointer >, its pointed is with first field value < www.google.com, Record1>of the DOMAIN field of dictionary preface arrangement.The size of index is used the data block store of regular length in the piece, is convenient to disposable being loaded into and carries out data retrieval in the internal memory, for example can be the 4K size.In addition; Can also on memory node, adopt the gzip compression algorithm to carry out data compression to the data file; Because data value identical or close field is adjacent storage, and carry out the ordering of dictionary preface, therefore introduce the storage efficiency that can significantly improve data after the compress technique.
Compared with prior art; Index is that structurized record data are set up index in the file in the piece of present embodiment; Accelerate the structured data query of user-defined form; What other KEY-VALUE type database was set up is index in the KEY-VALUE type piece, supports KEY-VALUE type data query, and does not support the inquiry of structuring data.In addition; In the present embodiment with the data managing method of the unit of being recorded as both can the accelerating structure data query script; Also can utilize the data-storage system of other KEY-VALUE type database as bottom; Utilize its Retrieval Interface that provides, the encapsulation, conversion of carrying out data in conjunction with the described form metadata of preceding text are to the management of user's implementation structure data.
When data quantity stored more after a little while, under stand-alone environment, can realize corresponding data management function.But along with the prolongation of holding time, system's data quantity stored can increase gradually, and during such as TB even PB magnitude, therefore the processing power of unit also need carry out distributed storage with data head and shoulders above.
In the present embodiment, adopted the fixed length block dividing method, adopted 2,000,000 records, when record count reaches 2,000,000, be spliced into a new file through said method as the threshold value of separating.With the file is that unit carries out distributed store with data.For the retrieving of expedited data, the index mechanism that has adopted distributed index to combine with centralized index is set up between data file simultaneously, the data file that unified each distribution of index is deposited, and this index unification is deposited; And at inner field index in the piece mentioned above, the retrieving that quickens each data block internal data physical record data of using of each data file.As shown in Figure 4, between a plurality of data files, set up distributed B+Tree index and carry out unified tissue.Each leaf node of distributed B+Tree is concrete data file.Because in daily record class data; Application scenarios such as flow data; Time is main retrieval sign; Can also give tacit consent to the standard of employing time, promptly store the interior at interval record that loads of certain hour as a data files, set up distributed B+Tree based on time attribute simultaneously as file division.
Fig. 5 has described the schematic flow sheet of the data enquire method that is used for distributed KEY-VALUE data-storage system.This method is based on the form metadata, and the data item that provides in the search condition of method to user's input of introducing according to preceding text is carried out format conversion, is converted into the data mode of method actual storage in storage system of being introduced with preceding text; According to the search condition after the conversion, the data file of searching system storage; Before the return results collection is showed, carry out inverse operation to result set according to data conversion rule, convert the expression way that the user understands into.This method is supported the storage and retrieval of complex data type through the conversion to data, comprises accurate inquiry, interval query, fuzzy query, numerical statistic, the data enquire method that packet sequencing, multiword section attribute query, the inquiry of multidata type etc. are complicated.Discuss in more detail in the face of this method down.
In this embodiment, for example, the user can describe querying condition with sql like language.After receiving the querying condition of user's input; Need resolve it, extract corresponding querying condition and relevant data value, come the judgment data value whether legal according to the form metadata; If legal, then the data that relate in the querying condition are carried out format conversion.With DNS_TABLE is example; The retrieval of multiword section attribute is meant to a plurality of fields in the form search condition is set; As to retrieve domain name be www.google.com; Resolve number of times in a period of time greater than 10, and the DNS Visitor Logs of time in 2011 06 month 10: 10: 10 on the 28th to 2011 06 month 10: 10: 10 on the 29th; Can be expressed as:
SELECT*FROM?DNS_TABLE?WHERE?DOMAIN＝‘www.google.com’AND?COUNT＞10AND?TIME＝[20110628101010TO20110629101010]；
Wherein, for the querying condition of the TEXT type of the querying condition of the TIMESTAMP type of TIME field and DOMAIN field, can directly accurately search according to what text class data-storage system provided, interval lookup method is retrieved.For the INTEGER type field; Because when storage INTEGER categorical data mentioned above, all will carry out left zero padding fills to convert orderly character string into; Numerical value to the INTEGER type in the querying condition carries out identical transformation when data query so; And then utilize the accurate inquiry of text data store system, be positioned concrete memory location, and return Query Result.For special statistical operation, such as SUM, aggregate functions such as AVG are the result set that returns the statistical operation of the laggard line correlation of value corresponding according to the target string format conversion.
Again for example, inquiry IP address and is done descending sort to result set according to COUNT in 192.16.18.1 access times in the 192.16.18.255 network segment less than 100 record, can use following querying condition:
SELECT*FROM?DNS_TABLE?WHERE?VALUE＝[192.16.18.1TO192.16.18.255]AND?COUNT＜100ORDER?BY?COUNT?DESC；
Wherein before beginning retrieval, switch process as indicated above that the IPFIELD field is changed.Such as the record that satisfies condition that [192.16.18.1TO 192.16.18.255] is converted in the retrieval integer range [3222278657,3222278911].
In yet another embodiment, a kind of distributed KEY-VALUE data-storage system also is provided.The synoptic diagram of an embodiment of distributed KEY-VALUE data-storage system shown in Figure 6.As shown in Figure 6, this system comprises: data importing device M, data let-off gear(stand) N; Input data type conversion and charger A; Meta data management device B, data retrieval and output data type conversion device C, data storage and management devices D and distributed storage media management devices E.
Wherein, data importing device M reads the raw data source that is loaded by user program or data file, imports one by one with the unit of being recorded as.Every record comprises a plurality of fields, and each field can have data, also can be sky, when being empty, uses ' MULL ' key word to indicate.Data importing device M also can use data file to import data in batches.Use between many records this moment ' n ' cut apart, use separator ", " to cut apart between a plurality of fields.
Input data type conversion and charger A receive the data source that is read by data importing device M.Comprise the content of each field to every record,, carry out type and judge that if do not meet the data type that defines in the tableau format, reporting errors also returns according to the form metadata information among the meta data management device C; If meet the record of define styles, the conversion regime according to the different types of data of data type and the preceding text introduction of definition carries out Data Format Transform.Record through after the conversion is loaded in the distributed storage environment through data storage and management devices D.
Meta data management device B definition and the essential information of preserving form comprise table name, field name, metadata informations such as field type.Input data type conversion and charger A can change according to the metadata information of preserving in the meta data management device B when loaded data source is changed accordingly.The data item that provides in the search condition of metadata to user's input that data retrieval and output data type conversion device C preserve in also can operative installations B when carrying out data conversion is judged, is transformed.
Data retrieval and output data type conversion device C receive the search condition that the user provides; To the data that provide in the search condition; The conversion regime of the different types of data of introducing according to preceding text; Convert the data retrieval interface of calling data storage and management devices D behind the data layout of preserving in the bottom storage system to and carry out the data query operation, and the query results that returns of reception, combined data storage and management device D, the transformation of being correlated with in case of necessity.
The managing distributed stored text data of data storage and management devices D are set up Distributed Storage and rule to a large amount of text datas.Basic skills is through data are carried out piecemeal, distributed store, and each data block is a basic storage cell.In order to quicken retrieving, independent index in the centralized indexes of introducing data file-level and the piece of each file inside through the method that centralized indexes combines with distribution index, is supported the fast query operation of text data.
The distributed storage media management devices E provide the hardware store resource, in extensive mass data storage system, can use distributed file system, like the integration of realization hardware resources such as HDFS.Make system possess data and write, read and basic function such as online dilatation, replica management is provided in case of necessity, functions such as data disaster tolerance.
Data let-off gear(stand) N receives the Query Result data set that is returned by data retrieval and output data type conversion device C, carries out the encapsulation of data layout, converts User Defined and discernible data layout to.The result set of retrieval is directly showed on the interface usually or is exported in the file or catalogue of assigned address.
This system can be deployed in the distributed environment; Each device runs on independently on the server apparatus; Data importing device M and data let-off gear(stand) N are user oriented interface sections, and distributed storage media management devices E can use realizations such as present distributed file system or cluster file system.
Though the present invention is described through preferred embodiment, yet the present invention is not limited to described embodiment here, also comprises various changes and the variation done without departing from the present invention.
Claims (19)
1. date storage method that is used for distributed KEY-VALUE data-storage system, said method comprises:
Step 1) is corresponding character string according to the conversion regime of different data types with data-switching, so that the character string after the conversion is consistent with the numerical value order of the data before the order after the dictionary preface ordering and the conversion;
Step 2) character string that step 1) is obtained is stored by the ordering of dictionary preface.
2. date storage method according to claim 1, said data type comprise that value type, text, memory point divide metric IP the data type of address, the data type of storage time.
3. date storage method according to claim 2, the conversion regime of said value type is for converting the data of value type into the digit strings of fixed length through the mode of left side filling 0.
4. data managing method according to claim 2; The conversion regime of the data type of said storage time is for the data-switching of the type for the length of " date Hour Minute Second " form being 14 character string; Wherein " year " is 4 character lengths, and all the other are 2 character lengths.
5. data managing method according to claim 2, said memory point divide the conversion regime of the data type of metric IP address to be:
At first, convert the data of the type into integer from the form of " a.b.c.d " by following computing formula: a * k
3+ b * k
2+ c * k
1+ d, wherein k is the natural number greater than 255;
The integer that will after formula calculates, obtain then converts the character string of fixed length into the mode of left side filling 0.
6. date storage method according to claim 2, the conversion regime of said text saves as character string for direct data with the type.
7. one kind is used to adopt the data enquire method like the distributed KEY-VALUE data-storage system of the described date storage method of above-mentioned arbitrary claim, and said method comprises:
Receive the search condition of user's input;
Conversion regime according to different data types is corresponding character string with the data-switching in the search condition;
Based on said character string, search through accurate inquiry and/or interval that the KEY-VALUE data-storage system provides, locate concrete memory location, and return Query Result;
Conversion regime according to said different data type carries out inverse operation to obtain the data of respective type to Query Result.
8. structural data management method that is used for distributed KEY-VALUE data-storage system, said method comprises:
Step 101 is come loading data with the unit that writes down; The form of said record is limited form, and the essential information of form is kept in the form metadata, and said form essential information comprises table name, a plurality of field names and corresponding data type thereof, and said record comprises all fields in the form;
Step 102 is a data files with each n recorded and stored, for each field of record, stores like the described date storage method of claim 1-6 according to the corresponding data type information employing of the preservation in the form metadata; Wherein, The corresponding content of different field is saved in the diverse location of identical file respectively; Adopt the interior index of piece to be identified at the deviation post that a plurality of fields are preserved in the same file in this document; Said interior index is made up of the memory location pointer of designation data in file, preserves index in the piece at each data file rear portion with the mode of appending.
9. structural data management method according to claim 8, wherein said data file are stored in the distributed storage medium, carry out index with the B+ tree between the data file.
10. structural data management method according to claim 8 also comprises the step of inquiry, and the step of said inquiry may further comprise the steps:
Whether the search condition according to the information judges of the preservation in the form metadata is imported is legal;
If legal, then based on the data type information of the field of the preservation in the form metadata, would be corresponding character string with the data-switching in the search condition according to the conversion regime of different data types;
Based on said character string, search through accurate inquiry and/or interval that the KEY-VALUE data-storage system provides, locate concrete memory location, and return qualified record;
Conversion regime according to said different data type carries out inverse operation to obtain the data of respective type to each field of record.
11. a distributed KEY-VALUE data-storage system, said system comprises:
The conversion regime that is used for according to different data types is the device of corresponding character string with data-switching, said after conversion the numerical value of the data before the character string that the obtains order after with the ordering of dictionary preface and the conversion in proper order consistent;
Being used for will be through the character string that the is converted to device by dictionary preface ordering storage.
12. data-storage system according to claim 11, wherein, said data type comprises that value type, text, memory point divide metric IP the data type of address, the data type of storage time.
13. data-storage system according to claim 12, the conversion regime of said value type is for converting the data of value type into the digit strings of fixed length through the mode of left side filling 0.
14. data-storage system according to claim 12; The conversion regime of the data type of said storage time is for the data-switching of the type for the length of " date Hour Minute Second " form being 14 character string; Wherein " year " is 4 character lengths, and all the other are 2 character lengths.
15. data-storage system according to claim 12, said memory point divide the conversion regime of the data type of metric IP address to be:
At first, convert the data of the type into integer from the form of " a.b.c.d " by following computing formula: a * k
3+ b * k
2+ c * k
1+ d, wherein k is the natural number greater than 255;
The integer that will after formula calculates, obtain then converts the character string of fixed length into the mode of left side filling 0.
16. data-storage system according to claim 12, the conversion regime of said text saves as character string for direct data with the type.
17. data-storage system according to claim 12; Also comprise the data query device; It is used for according to the conversion regime of different data types is corresponding character string with the data-switching of search condition; And be used for searching through accurate inquiry and/or interval that the KEY=VALUE data-storage system provides, locate concrete memory location, and return Query Result.
18. data-storage system according to claim 11 also comprises:
Be used for coming the device of loading data with the unit of record; The form of said record is limited form, and the essential information of form is kept in the form metadata, and said form essential information comprises table name, a plurality of field names and corresponding data type thereof, and said record comprises all fields in the form;
Being used for each n recorded and stored is a data files, for each field of record, adopts the device of storing like the described date storage method of claim 1-6 according to the corresponding data type information of the preservation in the form metadata; Wherein, The corresponding content of different field is saved in the diverse location of identical file respectively; Adopt the interior index of piece to be identified at the deviation post that a plurality of fields are preserved in the same file in this document; Said interior index is made up of the memory location pointer of designation data in file, preserves index in the piece at each data file rear portion with the mode of appending.
19. data-storage system according to claim 18, wherein said data file are stored in the distributed storage medium, carry out index with the B+ tree between the data file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110314927A CN102332030A (en) | 2011-10-17 | 2011-10-17 | Data storing, managing and inquiring method and system for distributed key-value storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110314927A CN102332030A (en) | 2011-10-17 | 2011-10-17 | Data storing, managing and inquiring method and system for distributed key-value storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102332030A true CN102332030A (en) | 2012-01-25 |
Family
ID=45483806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110314927A Pending CN102332030A (en) | 2011-10-17 | 2011-10-17 | Data storing, managing and inquiring method and system for distributed key-value storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102332030A (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750356A (en) * | 2012-06-11 | 2012-10-24 | 清华大学 | Construction and management method for secondary indexes of key value library |
| CN103744952A (en) * | 2014-01-02 | 2014-04-23 | 北京集奥聚合网络技术有限公司 | Method and system for implementing multi-table property of NoSQL database |
| CN103823850A (en) * | 2014-02-13 | 2014-05-28 | 南京梅山冶金发展有限公司 | Storage method of multilayer structural relationship database |
| WO2014079282A1 (en) * | 2012-11-23 | 2014-05-30 | 腾讯科技（深圳）有限公司 | Method and apparatus for storing and verifying redeem code |
| CN103885721A (en) * | 2012-12-24 | 2014-06-25 | 中国移动通信集团公司 | Data storing or reading method and device for key-value system |
| CN103914483A (en) * | 2013-01-07 | 2014-07-09 | 深圳市腾讯计算机系统有限公司 | File storage method and device and file reading method and device |
| CN104008134A (en) * | 2014-05-06 | 2014-08-27 | 武汉邮电科学研究院 | Efficient storage method and system based on Hbase |
| CN104008158A (en) * | 2014-05-27 | 2014-08-27 | 青岛海信移动通信技术股份有限公司 | Data verification method, verification device and mobile terminal |
| WO2014161261A1 (en) * | 2013-07-24 | 2014-10-09 | 中兴通讯股份有限公司 | Data storage method and apparatus |
| CN104699815A (en) * | 2015-03-24 | 2015-06-10 | 北京嘀嘀无限科技发展有限公司 | Data processing method and system |
| CN105187568A (en) * | 2015-08-12 | 2015-12-23 | 广东睿江科技有限公司 | IPV4 address conversion method and device |
| CN105260464A (en) * | 2015-10-16 | 2016-01-20 | 北京奇虎科技有限公司 | Data storage structure conversion method and apparatus |
| CN105608103A (en) * | 2015-10-16 | 2016-05-25 | 北京奇虎科技有限公司 | Transforming method and device of data storing structure |
| CN106407442A (en) * | 2016-09-28 | 2017-02-15 | 中国银行股份有限公司 | Massive text data processing method and apparatus |
| WO2017028514A1 (en) * | 2015-08-18 | 2017-02-23 | 华为技术有限公司 | Method and device for storing and reading data |
| WO2017092470A1 (en) * | 2015-12-01 | 2017-06-08 | 中兴通讯股份有限公司 | Data storage method and device |
| CN107301017A (en) * | 2017-06-06 | 2017-10-27 | 北京云知声信息技术有限公司 | A kind of date storage method and device |
| CN107767132A (en) * | 2016-08-23 | 2018-03-06 | 百度在线网络技术（北京）有限公司 | Method and apparatus for generating prepaid card password |
| CN107766494A (en) * | 2017-10-19 | 2018-03-06 | 北京科技大学 | The storage method and system of material genetic engineering data |
| WO2018059430A1 (en) * | 2016-09-27 | 2018-04-05 | 新华三技术有限公司 | Database searching |
| CN109145081A (en) * | 2018-07-27 | 2019-01-04 | 安康市惠企财税服务有限公司 | A kind of financial data search method and system |
| CN109542908A (en) * | 2018-11-23 | 2019-03-29 | 中科驭数（北京）科技有限公司 | Data compression method, storage method, access method and system in key-value database |
| CN109918426A (en) * | 2018-12-21 | 2019-06-21 | 航天信息股份有限公司 | Eat the building method of drug administration data management platform |
| CN110069452A (en) * | 2019-04-26 | 2019-07-30 | 北京字节跳动网络技术有限公司 | Date storage method, device and computer readable storage medium |
| CN110554878A (en) * | 2019-09-04 | 2019-12-10 | 网易（杭州）网络有限公司 | data conversion method, game data processing method and device and server |
| CN110837510A (en) * | 2019-11-13 | 2020-02-25 | 北京深演智能科技股份有限公司 | Data processing method, device, storage medium and processor |
| WO2020041928A1 (en) * | 2018-08-27 | 2020-03-05 | 深圳市锐明技术股份有限公司 | Data storage method and system and terminal device |
| CN111061698A (en) * | 2019-12-30 | 2020-04-24 | 语联网（武汉）信息技术有限公司 | Storage method and device of Ether house contract data |
| CN112632075A (en) * | 2020-12-25 | 2021-04-09 | 创新科技术有限公司 | Storage and reading method and device of cluster metadata |
| CN110059153B (en) * | 2019-03-29 | 2023-09-26 | 中国人民解放军63791部队 | Data exchange method for real-time data processing system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1588366A (en) * | 2004-08-02 | 2005-03-02 | 中国科学院计算机网络信息中心 | Ciphertext data base search technology |
| CN101136035A (en) * | 2007-10-15 | 2008-03-05 | 金蝶软件（中国）有限公司 | Ordering method and system for different attribute haracter |
| CN101136014A (en) * | 2006-09-01 | 2008-03-05 | 北大方正集团有限公司 | Method for supporting full text retrieval system, and searching numerical value categorical data domain meanwhile |
| WO2009012604A1 (en) * | 2007-07-20 | 2009-01-29 | Ferag Ag | Method and device for inserting, collecting or collating a plurality of flexible, planar products |
| CN101477555A (en) * | 2009-01-09 | 2009-07-08 | 南京联创科技股份有限公司 | Fast retrieval and generation display method for task tree based on SQL database |
| WO2009126041A1 (en) * | 2008-04-11 | 2009-10-15 | Alfatroll As | Reson - dynamic search key for knowledge based information management |
| WO2010150750A1 (en) * | 2009-06-25 | 2010-12-29 | Nishiyama Shuhei | Database management device using key-value store with attributes, and key-value-store structure caching-device therefor |
-
2011
- 2011-10-17 CN CN201110314927A patent/CN102332030A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1588366A (en) * | 2004-08-02 | 2005-03-02 | 中国科学院计算机网络信息中心 | Ciphertext data base search technology |
| CN101136014A (en) * | 2006-09-01 | 2008-03-05 | 北大方正集团有限公司 | Method for supporting full text retrieval system, and searching numerical value categorical data domain meanwhile |
| WO2009012604A1 (en) * | 2007-07-20 | 2009-01-29 | Ferag Ag | Method and device for inserting, collecting or collating a plurality of flexible, planar products |
| CN101136035A (en) * | 2007-10-15 | 2008-03-05 | 金蝶软件（中国）有限公司 | Ordering method and system for different attribute haracter |
| WO2009126041A1 (en) * | 2008-04-11 | 2009-10-15 | Alfatroll As | Reson - dynamic search key for knowledge based information management |
| CN101477555A (en) * | 2009-01-09 | 2009-07-08 | 南京联创科技股份有限公司 | Fast retrieval and generation display method for task tree based on SQL database |
| WO2010150750A1 (en) * | 2009-06-25 | 2010-12-29 | Nishiyama Shuhei | Database management device using key-value store with attributes, and key-value-store structure caching-device therefor |
Non-Patent Citations (1)
| Title |
|---|
| NINGOO: "Cassandra存储机制", 《NINGOO"S BLOG》, 25 February 2010 (2010-02-25), pages 1 - 4 * |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750356B (en) * | 2012-06-11 | 2014-08-20 | 清华大学 | Construction and management method for secondary indexes of key value library |
| CN102750356A (en) * | 2012-06-11 | 2012-10-24 | 清华大学 | Construction and management method for secondary indexes of key value library |
| CN103838753B (en) * | 2012-11-23 | 2018-04-27 | 腾讯科技（北京）有限公司 | A kind of storage of redemption code, verification method and device |
| US9619657B2 (en) | 2012-11-23 | 2017-04-11 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for storing redeem code, and method and apparatus for verifying redeem code |
| CN103838753A (en) * | 2012-11-23 | 2014-06-04 | 腾讯科技（北京）有限公司 | Storage and verification method and device for exchange codes |
| WO2014079282A1 (en) * | 2012-11-23 | 2014-05-30 | 腾讯科技（深圳）有限公司 | Method and apparatus for storing and verifying redeem code |
| US10176304B2 (en) | 2012-11-23 | 2019-01-08 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for storing redeem code, and method and apparatus for verifying redeem code |
| CN103885721A (en) * | 2012-12-24 | 2014-06-25 | 中国移动通信集团公司 | Data storing or reading method and device for key-value system |
| CN103914483A (en) * | 2013-01-07 | 2014-07-09 | 深圳市腾讯计算机系统有限公司 | File storage method and device and file reading method and device |
| CN103914483B (en) * | 2013-01-07 | 2018-09-25 | 深圳市腾讯计算机系统有限公司 | File memory method, device and file reading, device |
| WO2014161261A1 (en) * | 2013-07-24 | 2014-10-09 | 中兴通讯股份有限公司 | Data storage method and apparatus |
| CN103744952A (en) * | 2014-01-02 | 2014-04-23 | 北京集奥聚合网络技术有限公司 | Method and system for implementing multi-table property of NoSQL database |
| CN103823850A (en) * | 2014-02-13 | 2014-05-28 | 南京梅山冶金发展有限公司 | Storage method of multilayer structural relationship database |
| CN103823850B (en) * | 2014-02-13 | 2017-02-22 | 南京梅山冶金发展有限公司 | Storage method of multilayer structural relationship database |
| CN104008134A (en) * | 2014-05-06 | 2014-08-27 | 武汉邮电科学研究院 | Efficient storage method and system based on Hbase |
| CN104008134B (en) * | 2014-05-06 | 2017-02-15 | 武汉邮电科学研究院 | Efficient storage method and system based on Hbase |
| CN104008158A (en) * | 2014-05-27 | 2014-08-27 | 青岛海信移动通信技术股份有限公司 | Data verification method, verification device and mobile terminal |
| CN104699815A (en) * | 2015-03-24 | 2015-06-10 | 北京嘀嘀无限科技发展有限公司 | Data processing method and system |
| CN105187568A (en) * | 2015-08-12 | 2015-12-23 | 广东睿江科技有限公司 | IPV4 address conversion method and device |
| CN105187568B (en) * | 2015-08-12 | 2018-09-25 | 广东睿江云计算股份有限公司 | A kind of IPV4 address conversion methods and device |
| WO2017028514A1 (en) * | 2015-08-18 | 2017-02-23 | 华为技术有限公司 | Method and device for storing and reading data |
| CN105608103A (en) * | 2015-10-16 | 2016-05-25 | 北京奇虎科技有限公司 | Transforming method and device of data storing structure |
| CN105260464A (en) * | 2015-10-16 | 2016-01-20 | 北京奇虎科技有限公司 | Data storage structure conversion method and apparatus |
| CN105260464B (en) * | 2015-10-16 | 2018-09-07 | 北京奇虎科技有限公司 | The conversion method and device of data store organisation |
| WO2017092470A1 (en) * | 2015-12-01 | 2017-06-08 | 中兴通讯股份有限公司 | Data storage method and device |
| CN107767132A (en) * | 2016-08-23 | 2018-03-06 | 百度在线网络技术（北京）有限公司 | Method and apparatus for generating prepaid card password |
| CN107767132B (en) * | 2016-08-23 | 2022-01-14 | 百度在线网络技术（北京）有限公司 | Method and device for generating rechargeable card password |
| WO2018059430A1 (en) * | 2016-09-27 | 2018-04-05 | 新华三技术有限公司 | Database searching |
| CN106407442A (en) * | 2016-09-28 | 2017-02-15 | 中国银行股份有限公司 | Massive text data processing method and apparatus |
| CN107301017A (en) * | 2017-06-06 | 2017-10-27 | 北京云知声信息技术有限公司 | A kind of date storage method and device |
| CN107301017B (en) * | 2017-06-06 | 2021-03-09 | 云知声智能科技股份有限公司 | Data storage method and device |
| CN107766494A (en) * | 2017-10-19 | 2018-03-06 | 北京科技大学 | The storage method and system of material genetic engineering data |
| CN107766494B (en) * | 2017-10-19 | 2021-11-30 | 北京科技大学 | Storage method and system of material genetic engineering data |
| CN109145081A (en) * | 2018-07-27 | 2019-01-04 | 安康市惠企财税服务有限公司 | A kind of financial data search method and system |
| WO2020041928A1 (en) * | 2018-08-27 | 2020-03-05 | 深圳市锐明技术股份有限公司 | Data storage method and system and terminal device |
| CN109542908A (en) * | 2018-11-23 | 2019-03-29 | 中科驭数（北京）科技有限公司 | Data compression method, storage method, access method and system in key-value database |
| CN109918426A (en) * | 2018-12-21 | 2019-06-21 | 航天信息股份有限公司 | Eat the building method of drug administration data management platform |
| CN110059153B (en) * | 2019-03-29 | 2023-09-26 | 中国人民解放军63791部队 | Data exchange method for real-time data processing system |
| CN110069452A (en) * | 2019-04-26 | 2019-07-30 | 北京字节跳动网络技术有限公司 | Date storage method, device and computer readable storage medium |
| CN110554878A (en) * | 2019-09-04 | 2019-12-10 | 网易（杭州）网络有限公司 | data conversion method, game data processing method and device and server |
| CN110837510B (en) * | 2019-11-13 | 2020-08-07 | 北京深演智能科技股份有限公司 | Data processing method, device, storage medium and processor |
| CN110837510A (en) * | 2019-11-13 | 2020-02-25 | 北京深演智能科技股份有限公司 | Data processing method, device, storage medium and processor |
| CN111061698A (en) * | 2019-12-30 | 2020-04-24 | 语联网（武汉）信息技术有限公司 | Storage method and device of Ether house contract data |
| CN111061698B (en) * | 2019-12-30 | 2023-09-05 | 语联网（武汉）信息技术有限公司 | Method and device for storing Ethernet contract data |
| CN112632075A (en) * | 2020-12-25 | 2021-04-09 | 创新科技术有限公司 | Storage and reading method and device of cluster metadata |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102332030A (en) | Data storing, managing and inquiring method and system for distributed key-value storage system | |
| CN106611046B (en) | Spatial data storage processing middleware system based on big data technology | |
| CN106708993B (en) | Method for realizing space data storage processing middleware framework based on big data technology | |
| CN104794123B (en) | A kind of method and device building NoSQL database indexes for semi-structured data | |
| US9805079B2 (en) | Executing constant time relational queries against structured and semi-structured data | |
| CN103020281B (en) | A kind of data storage and retrieval method based on spatial data numerical index | |
| CN100468402C (en) | Sort data storage and split catalog inquiry method based on catalog tree | |
| CN103678491A (en) | Method based on Hadoop small file optimization and reverse index establishment | |
| CN113986873B (en) | Method for processing, storing and sharing data modeling of mass Internet of things | |
| CN107368527B (en) | Multi-attribute index method based on data stream | |
| CN102456055B (en) | Method and device for retrieving interest points | |
| CN102375853A (en) | Distributed database system, method for building index therein and query method | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN100458784C (en) | Researching system and method used in digital labrary | |
| CN107357843B (en) | Massive network data searching method based on data stream structure | |
| CN111427847B (en) | Indexing and querying method and system for user-defined metadata | |
| WO2013182054A1 (en) | Memory retrieval, real time retrieval system and method, and computer storage medium | |
| CN103631909A (en) | System and method for combined processing of large-scale structured and unstructured data | |
| CN109857898A (en) | A kind of method and system of mass digital audio-frequency fingerprint storage and retrieval | |
| CN110245134B (en) | Increment synchronization method applied to search service | |
| CN105975495A (en) | Big data storage and search method and apparatus | |
| CN104268158A (en) | Structural data distributed index and retrieval method | |
| CN113656397A (en) | Index construction and query method and device for time series data | |
| CN105912696A (en) | DNS (Domain Name System) index creating method and query method based on logarithm merging | |
| CN112231351A (en) | Real-time query method and device for PB-level mass data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120125 |