CN107562804A - Data buffer service system and method, terminal - Google Patents
Data buffer service system and method, terminal Download PDFInfo
- Publication number
- CN107562804A CN107562804A CN201710670167.2A CN201710670167A CN107562804A CN 107562804 A CN107562804 A CN 107562804A CN 201710670167 A CN201710670167 A CN 201710670167A CN 107562804 A CN107562804 A CN 107562804A
- Authority
- CN
- China
- Prior art keywords
- data
- key assignments
- label value
- variety
- data buffer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
A kind of data buffer service system and method, terminal, the system include：Data loading module, suitable for a variety of data of multi-data source are loaded, each data have label value and key assignments；At least one data buffer storage, suitable for when obtaining a variety of data from the data loading module, the label value and/or key assignments of each data are encoded, to cause the label value after coding to be less than the memory space that label value takes, the memory space that key assignments after coding takes is less than the memory space that key assignments takes, and stores the key assignments after the label value and coding after the coding of a variety of data；At least one data inquiry module, matching inquiry is carried out to a variety of data at least one data buffer storage suitable for inquiry request square according to demand.Technical solution of the present invention can avoid data cached expansion and realize the quick search of data.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of data buffer service system and method, terminal.
Background technology
In the data field of circulation, in mass data dispatching, data supply system need to meet high concurrent, high-throughput,
The requirement of low delay and real-time property.
In the system of existing supply data, it will usually which the big data full dose of data source is loaded onto into database.Party in request
When there is demand data, data can be obtained from database.In database the access of data or file often rely on disk input/
Export (Input/Output, IO) operation.
But in the case of data high concurrent, there is problems with existing supply data system：1. substantial amounts of disk
I/O operation, cause database reading data efficiency low, the Millisecond low delay and high-throughput of data distribution can not be met
Requirement.2. traditional database, file and normal cache system are difficult to realize with business development and extending transversely.3. by number
After caching is loaded into, data expansion often be present, and take a large amount of internal memories.4. in hardware resource finite sum mass data premise
Under, database is difficult to keep real-time update with data source, and keeps stable external service ability in the data renewal period.
The content of the invention
Present invention solves the technical problem that it is the quick search for how avoiding data cached expansion and realizing data.
In order to solve the above technical problems, the embodiment of the present invention provides a kind of data buffer service system, data buffer service
System includes：Data loading module, suitable for a variety of data of multi-data source are loaded, each data have label value and key
Value；At least one data buffer storage, suitable for from the data loading module obtain a variety of data when, to the mark of each data
Label value and/or key assignments are encoded, to cause the label value after coding to be less than the memory space that label value takes, the key after coding
The memory space that value takes is less than the memory space that key assignments takes, and stores label value and volume after the coding of a variety of data
Key assignments after code；At least one data inquiry module, delay suitable for inquiry request square according to demand at least one data
Matching inquiry is carried out to a variety of data in depositing.
Optionally, the data buffer storage includes：Label value coding unit, suitable for being compiled to the label value of each data
Code, to form identification code corresponding with the label value, the identification code includes character and/or numeral.
Optionally, the data buffer storage includes：Key assignments processing unit, suitable for obtaining two according to the key assignments of each data
Level key-value pair, to include one-level key assignments and two level key assignments, the one-level key assignments as the key assignments after coding, the two-stage key-value pair
Number of types be less than key assignments number of types.
Optionally, the key assignments processing unit includes：Quantity determination subelement, suitable for the key assignments according to a variety of data
Number of types determine the number of types of the one-level key assignments；Numerical value conversion subunit, suitable for by the key assignments of a variety of data
Be converted to the first numerical string；One-level key assignments determination subelement, suitable for the type by first numerical string to the one-level key assignments
Quantity modulus, and the value after modulus is converted into second value string, to be used as the one-level key assignments；Two level key assignments determines that son is single
Member, the character of quantity is set in first numerical string suitable for choosing, to be used as the two level key assignments.
Optionally, the data loading module includes：Sequencing unit, the second data are carried out suitable for the size according to key assignments
Sequence, wherein, the first data sort according to the size of key assignments, and second data are the data in the data source, and described the
One data are the data in the data buffer storage；Comparing unit, described in after sequentially choosing first data and sorting
Second data, and the first data at least to selection and the key assignments of the second data are compared, to obtain comparison result；Type is true
Order member, suitable for determining the type of first data and/or the type of second data according to the comparison result；Renewal
Unit, suitable for the type of the type according to first data and/or second data at least one data buffer storage
Carry out data renewal.
Optionally, the sequencing unit is ranked up according to the order of key assignments from small to large to second data, described
First data sort according to the order of key assignments from small to large, and the type determining units include：First kind determination subelement, fit
When the key assignments for showing first data in the comparison result is less than the key assignments of second data, first number is determined
According to for data to be deleted；Second Type determination subelement, suitable for showing that the key assignments of first data is big in the comparison result
When the key assignments of second data, it is data to be increased newly to determine second data.
Optionally, the comparing unit is when the key assignments of first data is consistent with the key assignments of second data, than
The label value of label value and second data to first data, to obtain the comparison result.
Optionally, the type determining units include：3rd type determination unit, suitable for showing in the comparison result
When the label value of first data and the inconsistent label value of second data, determine second data for change number
According to.
Optionally, the updating block deletes the data to be deleted, and by the delta data and the number to be increased newly
According to being loaded onto at least one data buffer storage.
Optionally, the data loading module by the way of pipeline transmission, HTTP or cli interface by a variety of data
Key assignments and label value be stored at least one data buffer storage.
Optionally, the data inquiry module, which pre-establishes association Cheng Chi and connection pool, the association Cheng Chi, includes multiple threads,
The connection pool includes connection of multiple party in request's access interfaces to cache interface.
Optionally, the data buffer service system also includes：Reverse proxy and load balancing module, it is multiple suitable for receiving
Inquiry request, and the multiple inquiry request is uniformly distributed at least one data inquiry module.
The embodiment of the invention also discloses a kind of data buffer service method, methods described includes：By the more of multi-data source
Kind data are loaded, and each data have label value and key assignments；A variety of data are being obtained from the data loading module
When, the label value and/or key assignments of each data are encoded, to cause the label value after coding to be less than depositing for label value occupancy
Space is stored up, the memory space that the key assignments after coding takes is less than the memory space that key assignments takes, and stores a variety of data
The key assignments after label value and coding after coding；Square inquiry request carries out matching inquiry to a variety of data according to demand.
Optionally, the label value to each data, which carries out coding, includes：The label value of each data is encoded,
To form identification code corresponding with the label value, the identification code includes character and/or numeral.
Optionally, the key assignments to each data, which carries out coding, includes：Two are obtained according to the key assignments of each data
Level key-value pair, to include one-level key assignments and two level key assignments, the one-level key assignments as the key assignments after coding, the two-stage key-value pair
Number of types be less than key assignments number of types.
Optionally, the key assignments according to each data, which obtains two-stage key-value pair, includes：According to a variety of data
The number of types of key assignments determine the number of types of the one-level key assignments；The key assignments of a variety of data is converted into the first numerical value
String；Number of types modulus by first numerical string to the one-level key assignments, and the value after modulus is converted into second value
String, to be used as the one-level key assignments；The character that quantity is set in first numerical string is chosen, to be used as the two level key assignments.
Optionally, it is described by a variety of data of multi-data source be loaded including：According to the size of key assignments to the second data
Be ranked up, wherein, the first data according to key assignments size sort, second data be the data source in data, institute
The first data are stated as the data in the data buffer storage；Order chooses second data after first data and sequence,
And the first data at least to selection and the key assignments of the second data are compared, to obtain comparison result；Tied according to described compare
Fruit determines the type of first data and/or the type of second data；According to the type of first data and/or institute
The type for stating the second data carries out data renewal at least one data buffer storage.
Optionally, second data are ranked up according to the order of key assignments from small to large, first data according to
The order sequence of key assignments from small to large, the type and/or described that first data are determined according to the comparison result
The type of two data includes：If the comparison result shows that the key assignments of first data is less than the key of second data
Value, it is determined that first data are data to be deleted；If the comparison result shows that the key assignments of first data is more than
The key assignments of second data, it is determined that second data are data to be increased newly.
Optionally, the key assignments of first data at least to selection and the second data be compared including：It is if described
The key assignments of first data is consistent with the key assignments of second data, then compares the label value of first data and second number
According to label value, to obtain the comparison result.
Optionally, it is described that the type of first data and/or second data are determined according to the comparison result
Type includes：If the comparison result shows that the label value of first data and the label value of second data differ
Cause, it is determined that second data are delta data.
Optionally, it is described according to the type of first data and/or the type of second data to described at least one
Individual data buffer storage, which carries out data renewal, to be included：Delete the data to be deleted, and by the delta data and the number to be increased newly
According to being loaded onto at least one data buffer storage.
Optionally, it is described by a variety of data of multi-data source be loaded including：Transmitted using pipeline, HTTP or CLI connect
The key assignments of a variety of data and label value are loaded by the mode of mouth.
Optionally, association Cheng Chi and connection pool are pre-established, the association Cheng Chi includes multiple threads, and the connection pool includes more
Connection of the individual party in request's access interface to cache interface.
Optionally, the inquiry request of the side according to demand to a variety of data before matching inquiries also include：
Receive the multiple queries request uniformly distributed.
The embodiment of the invention also discloses a kind of computer-readable recording medium, is stored thereon with computer instruction, described
The step of computer instruction performs the data buffer service method when running.
The embodiment of the invention also discloses a kind of terminal, including memory and processor, being stored with the memory can
The computer instruction run on the processor, the processor perform data buffer service when running the computer instruction
The step of method.
Compared with prior art, the technical scheme of the embodiment of the present invention has the advantages that：
The data buffer service system of technical solution of the present invention includes data loading module, suitable for by a variety of of multi-data source
Data are loaded, and each data have label value and key assignments；At least one data buffer storage, suitable for loading mould from the data
When block obtains a variety of data, the label value and/or key assignments of each data are encoded, to cause the label value after coding
Less than the memory space of label value occupancy, the memory space that the key assignments after coding takes is less than the memory space that key assignments takes, and
Store the key assignments after the label value and coding after the coding of a variety of data；At least one data inquiry module, suitable for basis
The inquiry request of party in request carries out matching inquiry at least one data buffer storage to a variety of data.The technology of the present invention
The data buffer storage of scheme is encoded by the label value to data and/or key assignments, compared to initial data, the data after coding
Smaller memory space can be taken, so as to substantially reduce data expansion, saves the memory space of caching；In addition, pass through saving
The memory space of caching, faster data query service can also be provided, improve the performance of data buffer service system.Enter one
Step, at least one data buffer storage and at least one data inquiry module in technical solution of the present invention can with dynamic expansion, so as to
Realize the dynamic expansion of whole system.
Further, data loading module includes：Sequencing unit, the second data are arranged suitable for the size according to key assignments
Sequence, wherein, the first data according to key assignments size sort, second data be the data source in data, described first
Data are the data in the data buffer storage；Comparing unit, suitable for sequentially choosing described the after first data and sequence
Two data, and the first data at least to selection and the key assignments of the second data are compared, to obtain comparison result；Type determines
Unit, suitable for determining the type of first data and/or the type of second data according to the comparison result；Renewal is single
Member, at least one data buffer storage is entered suitable for the type of the type according to first data and/or second data
Row data update.Technical solution of the present invention is by determining data type, and the mode being updated according to data type, Ye Jitong
The data load mode of incremental update is crossed, data is greatly reduced and updates the spent time, improve the efficiency of data loading；
Further, it is also possible to avoid loading duplicate data, the memory space of caching is further saved.
Further, the data inquiry module, which pre-establishes association Cheng Chi and connection pool, the association Cheng Chi, includes multiple threads,
The connection pool includes connection of multiple party in request's access interfaces to cache interface.Technical solution of the present invention is by pre-establishing
Cheng Chi and connection pool are assisted, can be when inquiry request arrive, directly from association, Cheng Chi transfers thread, transfers party in request from connection pool and visits
Connection of the interface to cache interface is asked, it is achieved thereby that the quick search service of data, and then high concurrent, low delay can be provided
Data query access service.
Brief description of the drawings
Fig. 1 is a kind of structural representation of data buffer service system of the embodiment of the present invention；
Fig. 2 is the concrete structure schematic diagram of key assignments processing unit in the embodiment of the present invention；
Fig. 3 is the concrete structure schematic diagram of data loading module shown in Fig. 1 101；
Fig. 4 is the structural representation of another kind data buffer service system of the embodiment of the present invention；
Fig. 5 is a kind of flow chart of data buffer service method of the embodiment of the present invention；
Fig. 6 is a kind of specific implementation flow chart of step S501 shown in Fig. 5.
Embodiment
As described in the background art, in the case of data high concurrent, there is problems with existing supply data system：
1. substantial amounts of disk I/O operation, cause database reading data efficiency low, the Millisecond low delay of data distribution can not be met
And the requirement of high-throughput.2. traditional database, file and normal cache system are difficult to realize with business development and horizontal
To extension.3. after data are loaded into caching, data expansion often be present, and take a large amount of internal memories.It is 4. limited in hardware resource
Under the premise of mass data, database is difficult to keep real-time update with data source, and keeps stabilization in the data renewal period
External service ability.
To sum up, there is provided one can be with dynamic expansion, low delay, high concurrent and can be with the data buffer service of real-time update
System is the technical problem that data trade field is badly in need of solving.
The data buffer storage of technical solution of the present invention is encoded by the label value to data and/or key assignments, compared to original
Beginning data, the data after coding can take smaller memory space, so as to substantially reduce data expansion, save the storage of caching
Space；In addition, the memory space by saving caching, can also provide faster data query service, data buffer storage is improved
The performance of service system.Further, at least one data buffer storage in technical solution of the present invention and at least one data query mould
Block can be with dynamic expansion, so as to realize the dynamic expansion of whole system.
It is understandable to enable the above objects, features and advantages of the present invention to become apparent, below in conjunction with the accompanying drawings to the present invention
Specific embodiment be described in detail.
Fig. 1 is a kind of structural representation of data buffer service system of the embodiment of the present invention.
Data buffer service system 10 shown in Fig. 1 can include data loading module 101, at least one data buffer storage
102 and at least one data inquiry module 103.
Wherein, data loading module 101 is suitable to a variety of data of multi-data source being loaded, and each data have label
It is worth (value) and key assignments (key)；Data buffer storage 102 is suitable to obtaining a variety of data from the data loading module 101
When, the label value and/or key assignments of each data are encoded, to cause the label value after coding to be less than depositing for label value occupancy
Space is stored up, the memory space that the key assignments after coding takes is less than the memory space that key assignments takes, and stores a variety of data
The key assignments after label value and coding after coding；Data inquiry module 103 be suitable to inquiry request square according to demand it is described extremely
Matching inquiry is carried out to a variety of data in a few data buffer storage 102.
In specific implementation, data source can be data supplier.The number that data loading module 101 can provide supplier
According to being loaded onto data buffer storage 102.Because the speed of service ratio for caching (cache) is very fast, therefore buffer memory can be used a variety of
Data.A variety of data in data buffer storage 102 can come from multiple data suppliers.Specifically, each data can include
Key assignments (key) and label value (value).Wherein, label value can be multiple selectable value under key assignments.Specifically, key assignments can be with
Label as data, then label value is corresponding with key assignments.For example, when key assignments is the age, label value can be age value
Or the number range where the age.
The total amount for a variety of data that usual multi-data source provides is very big, if a variety of data full doses are stored in data buffer storage
In 102, it will occupy substantial amounts of memory space, and influence access speed.In the present embodiment, data buffer storage 102 fills from data
When load module 101 obtains a variety of data, the label value and/or key assignments of each data are encoded.Before coding
Data label value and/or key assignments, the memory space shared by label value and/or key assignments after coding is smaller.Data buffer storage
102 storage be coding after label value and/or key assignments.Specifically, data buffer storage 102 can be carried out only to the key assignments of data
Coding, can also only encode to the label value of data, the key assignments and label value of data all can also be encoded.
Furthermore, the mapping relations of the label value of the data after the label value and coding of the data before coding, and
The mapping relations of the key assignments of data after the key assignments and coding of data before coding can store.
It will be appreciated by persons skilled in the art that data buffer storage 102 can use redis, memcache etc. arbitrarily may be used
The technology of implementation realizes that the embodiment of the present invention is without limitation.
In specific implementation, data inquiry module 103 square inquiry request can inquire about in data buffer storage 102 according to demand
Data.Specifically, because the data in data buffer storage 102 are encoded, therefore data inquiry module 103 will inquire about
Request can also be encoded inquiry request when being matched with the data in data buffer storage 102, then with the data in caching
Matched, coded system is identical with the coded system of label value and/or key assignments；Or or closed according to above-mentioned mapping
System and inquiry request come match caching in data.
The data buffer storage of the embodiment of the present invention is encoded by the label value to data and/or key assignments, compared to original
Data, the data after coding can take smaller memory space, and so as to substantially reduce data expansion, the storage for saving caching is empty
Between；In addition, the memory space by saving caching, can also provide faster data query service, data buffer storage clothes are improved
The performance of business system.Further, at least one data buffer storage in the embodiment of the present invention and at least one data inquiry module can
With dynamic expansion, so as to realize the dynamic expansion of whole system.
In actual applications, Experimental Comparison, experiment knot have been carried out to the storage effect of prior art and the embodiment of the present invention
Fruit is as shown in table 1.
| Data volume | Prior art space-consuming | The present embodiment space-consuming |
| 500000000 datas | 18.4G | 15.4G |
| 1000000000 datas | 39.74G | 31.13G |
Table 1
As shown in table 1, for 500,000,000 datas, using prior art, it will 18.4G memory space is taken in the buffer,
And use the embodiment of the present invention, it is only necessary to take 15.4G memory space；Similarly, for 1,000,000,000 datas, using prior art,
39.74G memory space will be taken in the buffer, and use the embodiment of the present invention, it is only necessary to take 31.13G memory space.
Thus, the memory space of 20% or so caching can be reduced compared to prior art, the embodiment of the present invention.
Preferably, data buffer storage 102 can include label value coding unit (not shown), and label value coding unit is suitable to pair
The label value of each data is encoded, to form corresponding with label value identification code, the identification code including character and/or
Numeral.
Specifically, label value and identification code can be one-to-one mapping relations.Storage shared by identification code is empty
Between be less than label value shared by memory space.
In one embodiment of the invention, the identification code can be numeral.All label values are encoded, such as label value
Value1 is encoded to 0, and label value value2 is encoded to 1, and label value value3 is encoded to 2 ..., and label value valueN is encoded to N-
1.That is, if data label value is " value1 ", integer 0 is stored in, if data label value is " value2 ", is deposited
Enter integer 1.
In another embodiment of the present invention, the identification code can be character.If data label value is " value1 ", deposit
Enter character a, if data label value is " value2 ", be stored in character b.
In yet another embodiment of the invention, the identification code can be character and numeral.If data label value is
" value1 ", then character a1 is stored in, if data label value is " value2 ", is stored in character a2.
By being encoded to the label value of data, the mapping relations established between label value and identification code so that data
Caching 102 can carry out shared storage, namely the label value of same data can only store once, greatly reduce memory space
Occupancy.
Preferably, data buffer storage 102 can include mark key assignments processing unit (not shown), and key assignments processing unit is suitable to basis
The key assignments of each data obtains two-stage key-value pair, to include one-level key as the key assignments after coding, the two-stage key-value pair
Value and two level key assignments, the number of types of the one-level key assignments are less than the number of types of key assignments.
In specific implementation, by being encoded to key assignments, the mapping relations established between two-stage key-value pair and key assignments.One-level
The number of types of key assignments is less than the number of types of key assignments, then compared to the number of types of key assignments, is stored in data buffer storage 102
One-level key assignments number of types it is smaller, same key assignments need to only store once in the buffer, reduce memory space occupancy.
In addition, in the prior art, key assignments can be that each type key assignments distributes a data structure, in key when being stored in data buffer storage 102
When Value Types quantity is larger, larger memory space will be occupied.The present embodiment passes through one-level key assignments, it is possible to reduce the number distributed
According to the quantity of structure, so as to further reduce memory space.
Further, as shown in Fig. 2 key assignments processing unit can include：Quantity determination subelement 21, suitable for according to
The number of types of the key assignments of a variety of data determines the number of types of the one-level key assignments；Numerical value conversion subunit 22, suitable for by institute
The key assignments for stating a variety of data is converted to the first numerical string；One-level key assignments determination subelement 23, suitable for by first numerical string pair
The number of types modulus of the one-level key assignments, and the value after modulus is converted into second value string, to be used as the one-level key assignments；
Two level key assignments determination subelement 24, the character of quantity is set in first numerical string suitable for choosing, to be used as the weak linkage
Value.
In specific implementation, the number of types for the one-level key assignments that quantity determination subelement 21 determines is less than the number of types of key assignments
Amount.In order that the otherness between the key assignments of data is bigger, the distribution of key assignments is more scattered, can turn the key assignments of a variety of data
It is changed to the first numerical string.Such as Message Digest 5 (Message Digest Algorithm, MD) etc. can be used to realize；
More specifically, Message Digest Algorithm 5 or sixth version can be used.
One-level key assignments can be the first numerical string to the value after the number of types modulus of the one-level key assignments.Further,
One-level key assignments can also be the second value string of the value conversion after modulus.The second value string can be the numerical value of regular length
String, when to distribute memory space for one-level key assignments, the memory space of regular length to be distributed, deposited so as to further save
Store up space.Specifically, the hash algorithms such as MD5, MD6 can be used to realize conversion.
In the present embodiment, have differences, then two level key assignments is chosen in the first numerical string between multiple first numerical strings
The character of quantity is set, with being combined with one-level key assignments to realize the differentiation and mapping to key assignments.For example, the first numerical string is
During 32 characters, 16 characters among 32 characters can be chosen and be used as two level key assignments；First numerical string is 16 characters
When, 8 characters among 16 characters can be chosen and be used as two level key assignments.It will be apparent to a skilled person that institute
It can also be other any enforceable numerical value to state setting quantity, and the embodiment of the present invention is without limitation.
If it should be noted that otherness were larger between the key assignments of a variety of data, numerical value conversion subunit 22
The key assignments of a variety of data need not be converted to the first numerical string, and one-level key assignments determination subelement 23 can be directly by key assignments
To the number of types modulus of the one-level key assignments as the one-level key assignments；Two level key assignments determination subelement 24 is then chosen in key assignments
The character of quantity is set, to be used as the two level key assignments.
In a concrete application scene of the invention, the type of one-level key assignments is determined according to the number of types of the key assignments of initial data
Quantity, it is designated as num (level1_key).The key assignments of initial data is changed using MD5 algorithms, obtains the first numerical string.
Using the first numerical string to num (level1_key) modulus, the value after modulus is changed using MD5 algorithms, obtains one-level
Key assignments level1_key.Middle 16 characters are taken to the first numerical string, obtain two level key assignments level2_key.Thus two-stage is obtained
Key-value pair, the key assignments after being encoded as initial data.
Preferably, as shown in figure 3, data loading module 101 (reference picture 1) can include：Sequencing unit 31, suitable for according to
The size of key assignments is ranked up to the second data, wherein, the first data sort according to the size of key assignments, and second data are institute
The data in data source are stated, first data are the data in the data buffer storage；Comparing unit 32, institute is chosen suitable for order
Second data after the first data and sequence are stated, and the first data at least to selection and the key assignments of the second data compare
It is right, to obtain comparison result；Type determining units 33, suitable for determining the type of first data according to the comparison result
And/or the type of second data；Updating block 34, suitable for the type according to first data and/or second number
According to type data renewal is carried out at least one data buffer storage.
The embodiment of the present invention is by determining data type, and the mode being updated according to data type, namely passes through increasing
The data load mode of renewal is measured, data is greatly reduced and updates the spent time, improve the efficiency of data loading；This
Outside, it can also avoid loading duplicate data, further save the memory space of caching.
Further, the sequencing unit 31 is ranked up according to the order of key assignments from small to large to second data,
First data sort according to the order of key assignments from small to large, and the type determining units 33 can include：The first kind is true
Stator unit (not shown), suitable for showing that the key assignments of first data is less than the key of second data in the comparison result
During value, it is data to be deleted to determine first data；Second Type determination subelement (not shown), suitable for comparing knot described
When fruit shows that the key assignments of first data is more than the key assignments of second data, it is number to be increased newly to determine second data
According to.
Further, comparing unit 32 is when the key assignments of first data is consistent with the key assignments of second data, than
The label value of label value and second data to first data, to obtain the comparison result.Then type determines single
Member 33 can include：3rd type determination unit (not shown), suitable for showing first data in the comparison result
When label value and the inconsistent label value of second data, it is delta data to determine second data.
It should be noted that in an of the invention change case, the sequencing unit 31 can also be according to key assignments from big to small
Order is ranked up to second data, and first data sort according to the order of key assignments from big to small.In this feelings
Under condition, first kind determination subelement is suitable to show that the key assignments of first data is more than the described second number in the comparison result
According to key assignments when, it is data to be deleted to determine first data；Second Type determination subelement is suitable in the comparison result
When showing that the key assignments of first data is less than the key assignments of second data, it is data to be increased newly to determine second data.
Further, updating block 34 deletes the data to be deleted, and by the delta data and the number to be increased newly
According to being loaded onto at least one data buffer storage.
In an of the invention specific embodiment, data loading module 101 from data source obtain data mode can be selected from
Under type：By accessing data source application DLL (Application Programming Interface, API)
Pulling data；Number is obtained by secure file transportation protocol (Secure File Transfer Protocol, SFTP) mode
According to；Data, such as kafka are obtained from other stream data sources；Obtained from big data processing platform, such as Hadoop.
Usual data file is bigger, therefore can also carry out burst to data file.Specifically, can be according to data
Central processing unit (Central Processing Unit, CPU) the nuclear volume C of server where caching and load (load) L,
Determine file being divided into N parts；N calculation formula is：N is 1 if server is monokaryon, otherwise N=L>(C/2)((C/
4):(C/2)), wherein, N is positive integer.
Data loading module 101 is ranked up to slicing files, and directly data can also be ranked up.Then will sequence
The data file in the data file and caching of gained is contrasted afterwards, obtains data to be increased newly, delta data and number to be deleted
According to.
Preferably, the data inquiry module 103 can pre-establish association Cheng Chi and connection pool, and the association Cheng Chi includes more
Individual thread, the connection pool include connection of multiple party in request's access interfaces to cache interface.
The embodiment of the present invention by the association Cheng Chi and connection pool that pre-establish, can when inquiry request arrives, directly from
Association Cheng Chi transfers thread, connection of party in request's access interface to cache interface is transferred from connection pool, it is achieved thereby that data is fast
Service is ask in quick checking, and then can provide the data query access service of high concurrent, low delay.
Specifically, HTTP (Hyper Text Transfer Protocol, HTTP) structure can be utilized
HTTP service is built, connection pool and association Cheng Chi are created when service starts.When inquiry request occurs, data inquiry module 103 is first
From association, Cheng Chizhong takes association's journey, connection is then taken out from connection pool, to access data buffer storage 102, and backward reference result.
Service can also be built using Thrift modes, connection pool and association Cheng Chi are created when service starts.Work as inquiry request
During generation, data inquiry module 103 takes association's journey from association Cheng Chizhong first, connection is then taken out from connection pool, to access data
Caching 102, and backward reference result.
Fig. 4 is the structural representation of another kind data buffer service system of the embodiment of the present invention.
Data buffer service system 40 shown in Fig. 4 can include：Data loading module 401, data buffer storage cluster 402,
Data inquiry module 403 and reverse proxy and load balancing module 404.
In the present embodiment, the multiple queries request of reverse proxy and load balancing module 404 suitable for receiving party in request 10, and
The multiple inquiry request is uniformly distributed at least one data inquiry module 403.Enquiry module 403 accesses data and delayed
Cluster 402 is deposited, and party in request 10 is fed back to by result is accessed, completes this request.Meanwhile data loading module 401 can be real
The source data of Shi Gengxin suppliers 20 is to data buffer storage cluster 402.
Data loading module 401 by the data from supplier 20 when being loaded into data buffer storage cluster 402, Ke Yiman
Foot is efficient, the demand of real-time update, and loading process does not influence 402 external service ability of data buffer storage cluster and stability.
Fig. 5 is a kind of flow chart of data buffer service method of the embodiment of the present invention.
Data buffer service method shown in Fig. 5 may comprise steps of：
Step S501：A variety of data of multi-data source are loaded, each data have label value and key assignments；
Step S502：When obtaining a variety of data from the data loading module, to the label values of each data and/
Or key assignments is encoded, to cause the label value after coding to be less than the memory space that label value takes, the key assignments after coding takes
Memory space be less than the memory space that key assignments takes, and after storing the label value and coding after the coding of a variety of data
Key assignments；
Step S503：Square inquiry request is entered at least one data buffer storage to a variety of data according to demand
Row matching inquiry.
The embodiment of the present invention is encoded by the label value to data and/or key assignments, compared to initial data, after coding
Data can take smaller memory space, so as to substantially reduce data expansion, save the memory space of caching；It is in addition, logical
The memory space for saving caching is crossed, faster data query service can also be provided, improves the property of data buffer service system
Energy.
Preferably, step S502 may comprise steps of：The label value of each data is encoded, to be formed with being somebody's turn to do
Identification code corresponding to label value, the identification code include character and/or numeral.
In one embodiment of the invention, the identification code can be numeral.All label values are encoded, such as label value
Value1 is encoded to 0, and label value value2 is encoded to 1, and label value value3 is encoded to 2 ..., and label value valueN is encoded to N-
1.That is, if data label value is " value1 ", integer 0 is stored in, if data label value is " value2 ", is deposited
Enter integer 1.
In another embodiment of the present invention, the identification code can be character.If data label value is " value1 ", deposit
Enter character a, if data label value is " value2 ", be stored in character b.
In yet another embodiment of the invention, the identification code can be character and numeral.If data label value is
" value1 ", then character a1 is stored in, if data label value is " value2 ", is stored in character a2.
By being encoded to the label value of data, the mapping relations established between label value and identification code so that data
Caching can carry out shared storage, namely the label value of same data can only store once, greatly reduce memory space and account for
Dosage.
Preferably, step S502 may comprise steps of：Two-stage key-value pair is obtained according to the key assignments of each data,
To include one-level key assignments and two level key assignments, the number of types of the one-level key assignments as the key assignments after coding, the two-stage key-value pair
Number of types of the amount less than key assignments.
In specific implementation, by being encoded to key assignments, the mapping relations established between two-stage key-value pair and key assignments.One-level
The number of types of key assignments is less than the number of types of key assignments, then compared to the number of types of key assignments, is stored in data buffer storage
The number of types of one-level key assignments is smaller, and same key assignments need to only store once in the buffer, reduce memory space occupancy.This
Outside, in the prior art, key assignments can be that each type key assignments distributes a data structure, in key assignments class when being stored in data buffer storage
When type quantity is larger, larger memory space will be occupied.The present embodiment passes through one-level key assignments, it is possible to reduce the data knot distributed
The quantity of structure, so as to further reduce memory space.
Preferably, the key assignments according to each data, which obtains two-stage key-value pair, includes：According to a variety of data
The number of types of key assignments determine the number of types of the one-level key assignments；The key assignments of a variety of data is converted into the first numerical value
String；Number of types modulus by first numerical string to the one-level key assignments, and the value after modulus is converted into second value
String, to be used as the one-level key assignments；The character that quantity is set in first numerical string is chosen, to be used as the two level key assignments.
Preferably, step S501 may comprise steps of：The second data are ranked up according to the size of key assignments, its
In, the first data sort according to the size of key assignments, and second data are the data in the data source, and first data are
Data in the data buffer storage；Order chooses second data after first data and sequence, and at least to choosing
The first data and the key assignments of the second data be compared, to obtain comparison result；Described is determined according to the comparison result
The type of the type of one data and/or second data；According to the type of first data and/or second data
Type carries out data renewal at least one data buffer storage.
In specific implementation, in order that the otherness between the key assignments of data is bigger, the distribution of key assignments is more scattered, can incite somebody to action
The key assignments of a variety of data is converted to the first numerical string.Such as Message Digest 5 (Message Digest can be used
Algorithm, MD) etc. realize；More specifically, Message Digest Algorithm 5 or sixth version can be used.
Further, second data are ranked up according to the order of key assignments from small to large, first data are pressed
Sorted according to key assignments order from small to large, it is described that the type of first data and/or described is determined according to the comparison result
The type of second data includes：If the comparison result shows that the key assignments of first data is less than the key of second data
Value, it is determined that first data are data to be deleted；If the comparison result shows that the key assignments of first data is more than
The key assignments of second data, it is determined that second data are data to be increased newly.
Further, the key assignments of first data at least to selection and the second data be compared including：If institute
It is consistent with the key assignments of second data to state the key assignments of the first data, then compares the label value and described second of first data
The label value of data, to obtain the comparison result.
Further, it is described that the type of first data and/or second data are determined according to the comparison result
Type include：If the comparison result shows that the label value of first data and the label value of second data differ
Cause, it is determined that second data are delta data.
Further, it is described according to the type of first data and/or the type of second data to it is described at least
One data buffer storage, which carries out data renewal, to be included：Delete the data to be deleted, and by the delta data and described wait to increase newly
Data are loaded onto at least one data buffer storage.
Preferably, it is described by a variety of data of multi-data source be loaded including：Transmitted using pipeline, Hyper text transfer association
Discuss (Hyper Text Transfer Protocol, HTTP) or command line interface (command line interface, CLI)
Mode the key assignments of a variety of data and label value are stored at least one data buffer storage.
Preferably, association Cheng Chi and connection pool are pre-established, the association Cheng Chi includes multiple threads, and the connection pool includes more
Connection of the individual party in request's access interface to cache interface.
The embodiment of the present invention by the association Cheng Chi and connection pool that pre-establish, can when inquiry request arrives, directly from
Association Cheng Chi transfers thread, connection of party in request's access interface to cache interface is transferred from connection pool, it is achieved thereby that data is fast
Service is ask in quick checking, and then can provide the data query access service of high concurrent, low delay.
Preferably, also include before step S503：Receive the multiple queries request uniformly distributed.
On more contents of the working method of the data buffer service method, implementation shown in Fig. 1 to Fig. 4 is referred to
The associated description of example, is repeated no more here.
Fig. 6 is refer to, Fig. 6 is a kind of specific implementation flow chart of step S501 shown in Fig. 5.
In the present embodiment, the number in data file (namely file of data source offer) after data cached file and sequence
According to R1, R2 is designated as respectively, its key assignments is respectively key1, and key2, label value is respectively v1, v2.Number after cache file and sequence
Sorted according to file according to the order of key assignments from small to large.Data file after cache file and sequence can include a plurality of number
According to.
Step S501 may comprise steps of：
Step S601：Judge whether key assignments key1 is less than key assignments key2, if it is, into step S602, otherwise enter step
Rapid S603；
Step S602：It is data to be deleted to determine data R1, and removes a data from data cached file, and returns to step
Rapid S601；
Step S603：Judge whether key assignments key1 is more than key assignments key2, if it is, into step S604, otherwise enter step
Rapid S605；
Step S604：It is data to be increased newly to determine data R2, takes the lower a data of data file, and return to step
S601；
Step S605：Judge whether label value v1 is equal to label value v2, if it is, into step S606, otherwise enter step
Rapid S607；
Step S606：It is persistent data to determine data R1, takes the lower a data of data cached file and data file, and
Return to step S601；
Step S607：It is delta data to determine data R2, takes the lower a data of data cached file and data file, and
Return to step S601.
Further, it can also carry out following step after step S602, step S604, step S606 and step S607
Suddenly：If the data in data cached file have taken, remaining do not fetch is treated according to all in the data file after sequence
Newly-increased data, and terminate to perform；It is remaining in data cached file not take if data have taken in the data file after sequence
The all data to be deleted of data, and terminate to perform.
Further, it can also carry out following step after step S602, step S604, step S606 and step S607
Suddenly：Pair determine data to be deleted, data increase, persistent data and delta data handled in real time, that is to say, that deletion
Data to be deleted in data cached file, retain the persistent data in data cached file, by data to be increased newly and change number
According to being loaded onto data cached file.
In a change case of the embodiment of the present invention, step S602, step S604, step S606 and step S607 it
After can also carry out following steps：The data to be deleted of determination, data to be increased newly, persistent data and delta data are separately added into
File, file to be increased newly, constant file and change file to be deleted, take in the data file after data cached file and sequence
All data and then respectively to file to be deleted, file to be increased newly, constant file and change file handle.In other words,
The data to be deleted in file to be deleted are deleted, retain the persistent data in constant file, by file to be increased newly and change file
In data to be increased newly and delta data be loaded onto data cached file.
The embodiment of the present invention is by determining data type, and the mode being updated according to data type, namely passes through increasing
The data load mode of renewal is measured, data is greatly reduced and updates the spent time, improve the efficiency of data loading；This
Outside, it can also avoid loading duplicate data, further save the memory space of caching.
Further, file to be deleted, when newly-increased file, constant file and change file it is larger when, it can also be entered
Row burst, multiple less files are divided into, then carry out the loading of data.
In the present embodiment, by when newly-increased data and delta data are loaded onto data cached file, load mode can select
From following two modes：Write data into caching using cache client and data are converted into caching protocol and passed by pipeline
It is defeated.
In actual applications, Experimental Comparison has been carried out to the data of prior art and embodiment of the present invention loading effect, tried
It is as shown in table 2 to test result.
| Data volume | Prior art spends the time | The present embodiment spends the time |
| 1000000000 datas | 3 hours 20 points | 40 minutes |
| 2500000000 datas | 6 hours 32 points | 1 hour |
Table 2
As shown in table 2, for 1,000,000,000 datas, using prior art, 3 hours will be spent by being loaded to data buffer storage
20 points of time, and use the embodiment of the present invention, it is only necessary to spend the time of 40 minutes；For 2,500,000,000 datas, using existing skill
Art, 6 hours 32 points of time will be spent by being loaded to data buffer storage, and use the embodiment of the present invention, it is only necessary to spend 1 hour
Time.
Thus, test proves that, on the premise of same data volume is loaded, relative to prior art, using the present invention
The time that the data refresh mode of embodiment is spent can be reduced to original renewal and spend time taking 1/5, substantially increase
Data renewal speed.In addition, of the prior art load the time spent as the increase of data volume is linear increase, and this
The time and data volume that loading in inventive embodiments is spent are non-linear relation, further increase data renewal speed.
The embodiment of the invention also discloses a kind of computer-readable recording medium, is stored thereon with computer instruction, described
The step of computer instruction can perform method shown in Fig. 5 or Fig. 6 when running.The storage medium can include ROM, RAM,
Disk or CD etc..
The embodiment of the invention also discloses a kind of terminal, the terminal can include memory and processor, the storage
The computer instruction that can be run on the processor is stored with device.The processor can be with when running the computer instruction
The step of performing method shown in Fig. 5 or Fig. 6.The terminal can include but is not limited to mobile phone, computer, tablet personal computer etc. eventually
End equipment.
Although present disclosure is as above, the present invention is not limited to this.Any those skilled in the art, this is not being departed from
In the spirit and scope of invention, it can make various changes or modifications, therefore protection scope of the present invention should be with claim institute
The scope of restriction is defined.
Claims (21)
- A kind of 1. data buffer service system, it is characterised in that including：Data loading module, suitable for a variety of data of multi-data source are loaded, each data have label value and key assignments；At least one data buffer storage, suitable for from the data loading module obtain a variety of data when, to each data Label value and/or key assignments are encoded, to cause the label value after coding to be less than the memory space that label value takes, after coding Key assignments take memory space be less than key assignments take memory space, and store the label value after the coding of a variety of data and Key assignments after coding；At least one data inquiry module, the inquiry request suitable for side according to demand is at least one data buffer storage to institute State a variety of data and carry out matching inquiry.
- 2. data buffer service system according to claim 1, it is characterised in that the data buffer storage includes：Label value coding unit, suitable for being encoded to the label value of each data, to form mark corresponding with the label value Code, the identification code include character and/or numeral.
- 3. data buffer service system according to claim 1, it is characterised in that the data buffer storage includes：Key assignments processing unit, suitable for obtaining two-stage key-value pair according to the key assignments of each data, using as the key assignments after coding, The two-stage key-value pair includes one-level key assignments and two level key assignments, and the number of types of the one-level key assignments is less than the number of types of key assignments Amount.
- 4. data buffer service system according to claim 3, it is characterised in that the key assignments processing unit includes：Quantity determination subelement, the number of types suitable for the key assignments according to a variety of data determine the type of the one-level key assignments Quantity；Numerical value conversion subunit, suitable for the key assignments of a variety of data is converted into the first numerical string；One-level key assignments determination subelement, suitable for the number of types modulus by first numerical string to the one-level key assignments, and will Value after modulus is converted to second value string, to be used as the one-level key assignments；Two level key assignments determination subelement, the character of quantity is set in first numerical string suitable for choosing, to be used as the two level Key assignments.
- 5. data buffer service system according to claim 1, it is characterised in that the data loading module includes：Sequencing unit, the second data are ranked up suitable for the size according to key assignments, wherein, the first data according to key assignments size Sequence, second data are the data in the data source, and first data are the data in the data buffer storage；Comparing unit, suitable for sequentially choosing second data after first data and sequence, and at least to the of selection The key assignments of one data and the second data is compared, to obtain comparison result；Type determining units, suitable for the type that first data are determined according to the comparison result and/or second data Type；Updating block, suitable for the type of the type according to first data and/or second data to described at least one Data buffer storage carries out data renewal.
- 6. data buffer service system according to claim 5, it is characterised in that the sequencing unit is according to key assignments from small Second data are ranked up to big order, first data sort according to the order of key assignments from small to large, described Type determining units include：First kind determination subelement, the key assignments suitable for showing first data in the comparison result are less than the described second number According to key assignments when, it is data to be deleted to determine first data；Second Type determination subelement, the key assignments suitable for showing first data in the comparison result are more than the described second number According to key assignments when, it is data to be increased newly to determine second data.
- 7. data buffer service system according to claim 6, it is characterised in that the comparing unit is in the described first number According to key assignments it is consistent with the key assignments of second data when, compare the label value of first data and the mark of second data Label value, to obtain the comparison result；The type determining units include：3rd type determination unit, suitable for the label value for showing first data in the comparison result and second number According to label value it is inconsistent when, it is delta data to determine second data；The updating block deletes the data to be deleted, and described in the delta data and the data increase be loaded onto At least one data buffer storage.
- 8. data buffer service system according to claim 1, it is characterised in that the data loading module uses pipeline The key assignments of a variety of data and label value are stored at least one data buffer storage by the mode of transmission, HTTP or cli interface.
- 9. data buffer service system according to claim 1, it is characterised in that the data inquiry module pre-establishes Cheng Chi and connection pool are assisted, the association Cheng Chi includes multiple threads, and the connection pool includes multiple party in request's access interfaces to caching The connection of interface.
- 10. data buffer service system according to claim 1, it is characterised in that also include：Reverse proxy and load balancing module, suitable for receiving multiple queries request, and the multiple inquiry request is uniformly distributed To at least one data inquiry module.
- A kind of 11. data buffer service method, it is characterised in that including：A variety of data of multi-data source are loaded, each data have label value and key assignments；When obtaining a variety of data, the label value and/or key assignments of each data are encoded, to cause the mark after coding Label value is less than the memory space that label value takes, and it is empty that the memory space that the key assignments after coding takes is less than the storage that key assignments takes Between, and store at least one data buffer storage the key assignments after the label value and coding after the coding of a variety of data；Square inquiry request carries out matching inquiry to a variety of data according to demand.
- 12. data buffer service method according to claim 11, it is characterised in that the label value to each data Carrying out coding includes：The label value of each data is encoded, to form identification code corresponding with the label value, the identification code includes word Symbol and/or numeral.
- 13. data buffer service method according to claim 11, it is characterised in that the key assignments to each data enters Row coding includes：Two-stage key-value pair is obtained according to the key assignments of each data, to be used as the key assignments after coding, the two-stage key-value pair bag One-level key assignments and two level key assignments are included, the number of types of the one-level key assignments is less than the number of types of key assignments.
- 14. data buffer service method according to claim 13, it is characterised in that described according to each data Key assignments, which obtains two-stage key-value pair, to be included：The number of types of the one-level key assignments is determined according to the number of types of the key assignments of a variety of data；The key assignments of a variety of data is converted into the first numerical string；Number of types modulus by first numerical string to the one-level key assignments, and the value after modulus is converted into second value String, to be used as the one-level key assignments；The character that quantity is set in first numerical string is chosen, to be used as the two level key assignments.
- 15. data buffer service method according to claim 11, it is characterised in that a variety of numbers by multi-data source According to be loaded including：The second data are ranked up according to the size of key assignments, wherein, the first data sort according to the size of key assignments, and described second Data are the data in the data source, and first data are the data in the data buffer storage；Order chooses second data after first data and sequence, and at least the first data of selection and second are counted According to key assignments be compared, to obtain comparison result；The type of first data and/or the type of second data are determined according to the comparison result；Line number is entered at least one data buffer storage according to the type of the type of first data and/or second data According to renewal.
- 16. data buffer service method according to claim 15, it is characterised in that according to the order of key assignments from small to large Second data are ranked up, first data sort according to the order of key assignments from small to large, described according to the ratio Determine that the type of first data and/or the type of second data include to result：If the comparison result shows that the key assignments of first data is less than the key assignments of second data, it is determined that described the One data are data to be deleted；If the comparison result shows that the key assignments of first data is more than the key assignments of second data, it is determined that described the Two data are data to be increased newly.
- 17. data buffer service method according to claim 16, it is characterised in that described at least to the first number of selection According to the key assignments with the second data be compared including：If the key assignments of first data is consistent with the key assignments of second data, the label value of first data is compared With the label value of second data, to obtain the comparison result；It is described to determine that the type of first data and/or the type of second data include according to the comparison result：If the comparison result show first data label value and second data label value it is inconsistent, really Fixed second data are delta data；It is described that at least one data buffer storage is entered according to the type of first data and/or the type of second data The renewal of row data includes：The data to be deleted are deleted, and the delta data and the data to be increased newly are loaded onto at least one data Caching.
- 18. data buffer service method according to claim 11, it is characterised in that a variety of numbers by multi-data source According to be loaded including：Transmitted using pipeline, be loaded the key assignments of a variety of data and label value by the way of HTTP or cli interface.
- 19. data buffer service method according to claim 11, it is characterised in that pre-establish association Cheng Chi and connection Pond, the association Cheng Chi include multiple threads, and the connection pool includes connection of multiple party in request's access interfaces to cache interface.
- 20. data buffer service method according to claim 11, it is characterised in that the inquiry of the side according to demand please Ask and a variety of data also include before matching inquiry：Receive the multiple queries request uniformly distributed.
- 21. a kind of terminal, including memory and processor, the meter that can be run on the processor is stored with the memory Calculation machine instructs, it is characterised in that perform claim requirement any one of 11 to 20 when the processor runs the computer instruction The step of data buffer service method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710670167.2A CN107562804B (en) | 2017-08-08 | 2017-08-08 | Data caching service system and method and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710670167.2A CN107562804B (en) | 2017-08-08 | 2017-08-08 | Data caching service system and method and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107562804A true CN107562804A (en) | 2018-01-09 |
| CN107562804B CN107562804B (en) | 2020-09-01 |
Family
ID=60974981
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710670167.2A Active CN107562804B (en) | 2017-08-08 | 2017-08-08 | Data caching service system and method and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107562804B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609926A (en) * | 2019-09-20 | 2019-12-24 | 中国银行股份有限公司 | Data tag storage management method and device |
| CN110955682A (en) * | 2019-11-22 | 2020-04-03 | 北京金山云网络技术有限公司 | Method and device for deleting cache data, data cache and reading cache data |
| CN111259060A (en) * | 2020-02-18 | 2020-06-09 | 北京百度网讯科技有限公司 | Data query method and device |
| WO2021087990A1 (en) * | 2019-11-08 | 2021-05-14 | 深圳市欢太科技有限公司 | Tag updating method and device, electronic apparatus, and storage medium |
| CN115334158A (en) * | 2022-07-29 | 2022-11-11 | 重庆蚂蚁消费金融有限公司 | Cache management method and device, storage medium and electronic equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739424A (en) * | 2008-11-13 | 2010-06-16 | 中国科学院计算机网络信息中心 | Method and system for converting and storing keyword and resource record of keyword |
| US7788240B2 (en) * | 2004-12-29 | 2010-08-31 | Sap Ag | Hash mapping with secondary table having linear probing |
| CN103838844A (en) * | 2014-03-03 | 2014-06-04 | 珠海市君天电子科技有限公司 | Method and device for storing and transmitting key value pair data |
| US8977597B2 (en) * | 2008-05-21 | 2015-03-10 | Oracle International Corporation | Generating and applying redo records |
| CN104794162A (en) * | 2015-03-25 | 2015-07-22 | 中国人民大学 | Real-time data storage and query method |
| US20150278271A1 (en) * | 2014-03-31 | 2015-10-01 | Sandisk Enterprise Ip Llc | Compaction of Information in Tiered Data Structure |
| CN105574076A (en) * | 2015-11-27 | 2016-05-11 | 湖南大学 | Key value pair storage structure based on Bloom Filter and method |
| CN105740442A (en) * | 2016-02-02 | 2016-07-06 | 北京量科邦信息技术有限公司 | Data storage method in HBase |
-
2017
- 2017-08-08 CN CN201710670167.2A patent/CN107562804B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7788240B2 (en) * | 2004-12-29 | 2010-08-31 | Sap Ag | Hash mapping with secondary table having linear probing |
| US8977597B2 (en) * | 2008-05-21 | 2015-03-10 | Oracle International Corporation | Generating and applying redo records |
| CN101739424A (en) * | 2008-11-13 | 2010-06-16 | 中国科学院计算机网络信息中心 | Method and system for converting and storing keyword and resource record of keyword |
| CN103838844A (en) * | 2014-03-03 | 2014-06-04 | 珠海市君天电子科技有限公司 | Method and device for storing and transmitting key value pair data |
| US20150278271A1 (en) * | 2014-03-31 | 2015-10-01 | Sandisk Enterprise Ip Llc | Compaction of Information in Tiered Data Structure |
| CN104794162A (en) * | 2015-03-25 | 2015-07-22 | 中国人民大学 | Real-time data storage and query method |
| CN105574076A (en) * | 2015-11-27 | 2016-05-11 | 湖南大学 | Key value pair storage structure based on Bloom Filter and method |
| CN105740442A (en) * | 2016-02-02 | 2016-07-06 | 北京量科邦信息技术有限公司 | Data storage method in HBase |
Non-Patent Citations (2)
| Title |
|---|
| 康凯森: "HBase Rowkey 设计", 《HTTP：//BLOG.BCMENG.COM/POST/HBASE-ROWKEY.HTML》 * |
| 涂振发: "云计算环境下海量空间数据高效存储关键技术研究", 《中国博士学位论文全文数据库 基础科学辑》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609926A (en) * | 2019-09-20 | 2019-12-24 | 中国银行股份有限公司 | Data tag storage management method and device |
| WO2021087990A1 (en) * | 2019-11-08 | 2021-05-14 | 深圳市欢太科技有限公司 | Tag updating method and device, electronic apparatus, and storage medium |
| CN110955682A (en) * | 2019-11-22 | 2020-04-03 | 北京金山云网络技术有限公司 | Method and device for deleting cache data, data cache and reading cache data |
| CN111259060A (en) * | 2020-02-18 | 2020-06-09 | 北京百度网讯科技有限公司 | Data query method and device |
| CN111259060B (en) * | 2020-02-18 | 2023-08-15 | 北京百度网讯科技有限公司 | Data query method and device |
| CN115334158A (en) * | 2022-07-29 | 2022-11-11 | 重庆蚂蚁消费金融有限公司 | Cache management method and device, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107562804B (en) | 2020-09-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107562804A (en) | Data buffer service system and method, terminal | |
| US10572475B2 (en) | Leveraging columnar encoding for query operations | |
| CN110222048B (en) | Sequence generation method, device, computer equipment and storage medium | |
| US9213732B2 (en) | Hash table and radix sort based aggregation | |
| US11151126B2 (en) | Hybrid column store providing both paged and memory-resident configurations | |
| US9292560B2 (en) | Reducing collisions within a hash table | |
| US8862625B2 (en) | Accessing data in a column store database based on hardware compatible indexing and replicated reordered columns | |
| CN104202423B (en) | A kind of system by software architecture expansion buffer memory | |
| CN105117351B (en) | To the method and device of buffering write data | |
| US20140379985A1 (en) | Multi-level aggregation techniques for memory hierarchies | |
| US10417257B2 (en) | Non-blocking database table alteration | |
| US20180239808A1 (en) | Efficient partitioning of relational data | |
| US20120290615A1 (en) | Switching algorithms during a run time computation | |
| US11294816B2 (en) | Evaluating SQL expressions on dictionary encoded vectors | |
| CN108628898A (en) | The method, apparatus and equipment of data loading | |
| CN106155934A (en) | Based on the caching method repeating data under a kind of cloud environment | |
| US20080168071A1 (en) | Storing Data in Predicted Formats | |
| US9836513B2 (en) | Page feed for efficient dataflow between distributed query engines | |
| US10997175B2 (en) | Method for predicate evaluation in relational database systems | |
| CN110401681A (en) | For data transmission, the method for data receiver and electronic equipment | |
| CN116860798A (en) | Data query method, electronic device and computer readable storage medium | |
| CN116760661A (en) | Data storage method, apparatus, computer device, storage medium, and program product | |
| US9449046B1 (en) | Constant-vector computation system and method that exploits constant-value sequences during data processing | |
| CN113157692B (en) | Relational memory database system | |
| CN113127717A (en) | Key retrieval method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |