CN104050250B - A kind of distributed key-value querying method and query engine system - Google Patents
A kind of distributed key-value querying method and query engine system Download PDFInfo
- Publication number
- CN104050250B CN104050250B CN201410256925.2A CN201410256925A CN104050250B CN 104050250 B CN104050250 B CN 104050250B CN 201410256925 A CN201410256925 A CN 201410256925A CN 104050250 B CN104050250 B CN 104050250B
- Authority
- CN
- China
- Prior art keywords
- node
- routing table
- table information
- memory node
- global routing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/563—Data redirection of data network streams
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application provides a kind of distributed key assignments query engine systems and querying method, the system to include：Meta data server, at least one agent node and at least one memory node；Wherein, the meta data server, for safeguarding the global routing table information for being directed toward memory node；The agent node includes：Forwarding module is inquired about, for the request for including target KEY that customer in response end is sent, based on the global routing table information, client request is transmitted to corresponding memory node；Data forwarding module, the response bag including target Value for respective stored node to be returned send client to；The memory node for based on the client request received, from data needed for middle acquisition are locally stored, and send and includes the response of target Value and wrap to agent node.The present invention can realize distributed storage and the inquiry of key assignments data, and can accommodate magnanimity key assignments data, and the adaptability accessed high concurrent is stronger.
Description
Present patent application is the applying date on December 31st, 2011, Application No. 201110460494.8, entitled
A kind of divisional application of the Chinese invention patent application of " distributed key-value querying method and query engine system ".
Technical field
This application involves Distributed Storages and inquiring technology field, are inquired about more particularly to a kind of distributed key-value
Method and query engine system.
Background technology
For the support of large-scale internet application, cloud computing, it can generally use relational data library storage related
Data.
Traditional relevant database, the usually a certain category information (such as website user information) by some application are all deposited
Storage provides application program and reads and the communication interface of write operation in the single library of Database Systems individual table.In realization,
Typical relevant database such as MySQL, the inquiry and write operation of single table have certain upper limit, more than this upper limit, SQL query
It will become very slowly with write operation, disk input/output (IO) module is also present with bottleneck.In internet mass data
Today, traditional Relational DataBase will carry mass data, and data are exactly divided into the multiple tables in multiple storehouses and deposited by common practice
Storage, such as website user information data can calculate Hash (hash) by modulus algorithm to user identifier (user_id) field
Value by different user data burst to disparate databases, is solved the storage of single storehouse list table and is read and write stress problems with this.
That is, traditional relevant database, to support Large Volume Data, it is necessary to data level is split to multiple
On multiple tables of storehouse, it is ensured that the record number of individual table is limited.It is such to be advantageous in that the size for reducing index file, so as to
Promote query performance.
But traditional relevant database, it is intended to solve the problems, such as memory capacity using single machine.To expand
Hold more machines, it is necessary to which doing data in application layer splits strategy.Here there are problems that two, one be for each application,
Application layer is required for do data fractionation, and each application fractionation strategy can be different, efficiency is low；Another problem is increase machine
Data after device split that cost is very high, it is necessary to which manual stoppage's attended operation, does not support automatic elastic to extend again.
Secondly, traditional relevant database, the concurrent capability of single machine have certain upper limit, and the access of high concurrent can incite somebody to action
Machine resources exhaust.One of the reason for high concurrent cannot being supported to access is that relevant database needs to do SQL parsings in retrieval
Operation is big to the expense of cpu resource.
In short, needing the technical problem that those skilled in the art urgently solve to be at present：How one kind is provided
Mass data can be accommodated, adapt to more optimal storage/query scheme that high concurrent accesses.
The content of the invention
Technical problems to be solved in this application are to provide a kind of distributed key-value querying method and query engine system,
Magnanimity key-value data can be accommodated, the adaptability accessed high concurrent is stronger.
To solve the above-mentioned problems, this application discloses a kind of distributed key-value query engine system, can specifically wrap
It includes：
Meta data server, for safeguarding the global routing table information for being directed toward memory node；
At least one agent node, including：
Forwarding module is inquired about, for the request for including target KEY that customer in response end is sent, based on the global routing table
Client request is transmitted to corresponding memory node by information；
Data forwarding module, the response bag including target Value for respective stored node to be returned send client to
End；
At least one memory node, for based on the client request received, from data needed for middle acquisition are locally stored,
And it sends and includes the response of target Value and wrap to agent node.
Preferably, the agent node can also include：Routing table synchronization module, for by the overall situation of meta data server
Routing table information is synchronized to local；Wherein, the inquiry forwarding module is inquired about based on local global routing table information.
Preferably, the agent node can also include：Routing table update module receives meta data server for working as
When routing table update notifies, new global routing table information is obtained from meta data server.
Preferably, the inquiry forwarding module specifically includes：Hash submodule, for including target based on what is received
The client request of KEY calculates the data block where target KEY by hash algorithm；Submodule is positioned, for according to calculating
The mark of data block where obtained target KEY inquires about the global routing table information, positions corresponding memory node；
Submodule is forwarded, for client request to be forwarded to respective stored node.
Preferably, the meta data server can also include：Data simultaneous module, for notifying the newly-increased storage
Node synchronous N number of data block from selected memory node；The N is calculated by the property parameters of the newly-increased memory node
It obtains；Routing table changes module, after working as the newly-increased memory node completion data synchronization, generates new global routing table
Information；First update notification module, for sending routing table update notice to agent node.
Preferably, the meta data server can also include：Monitoring module, for monitoring the survival shape of each memory node
State when monitoring a memory node failure, changes global routing table information；The modification includes：For with the failure
Data block of the memory node as host node, its host node is remapped in another redundant node；Second update is logical
Module is known, for sending routing table update notice to agent node.
Preferably, the memory node can include：For the high speed storing node for storing hot spot data and for storing
The other types memory node of non-thermal point data；Wherein, the high speed storing node includes memory node or solid state disk section
Point, the other types memory node include SAS/SATA hard disk nodes.
Another embodiment according to the invention also discloses a kind of distributed key-value querying method, can specifically include：
Receive the request for including target KEY that client is directed toward current agent node；Current agent node is by inquiring about global routing table
Client request is transmitted to corresponding memory node by information；Current agent node includes mesh by what respective stored node returned
The response bag of mark Value sends client to.
Preferably, the distributed key-value querying method can also include：Current agent node will be tieed up by asynchronous thread
The global routing table information of shield on the server is synchronized to local, is locally carrying out the inquiry to global routing table information.
Preferably, the distributed key-value querying method can also include：When receiving routing table update notice, obtain
New global routing table information.
Preferably, client request is transmitted to accordingly by the current agent node by inquiring about global routing table information
The process of memory node specifically include：Based on the client request for including target KEY received, calculated by hash algorithm
Data block where target KEY；According to the data block mark where the target KEY being calculated, the global road is inquired about
By table information, corresponding memory node is positioned；Client request is transmitted to corresponding memory node.
Another embodiment according to the invention also discloses a kind of distributed key-value (Key-Value) query engine system
Failover method, including：Monitor whether each memory node survives；Wherein, same data block is stored in a storage
On host node and at least one storage redundant node；When monitoring a memory node failure, global routing table information is changed；
The modification includes：For the data block using the memory node of the failure as host node, its host node is remapped to
In another redundant node；New global routing table information is updated to each agent node；The agent node is used to respond visitor
The request for including target KEY that family end is sent based on the global routing table information, client request is transmitted to and is deposited accordingly
Store up node；And the response bag including target Value for returning to respective stored node sends client to.
Another embodiment according to the invention also discloses a kind of distributed key-value (Key-Value) query engine system
Increase physical store nodal method, including：Receive the registration request of newly-increased memory node；Attribute according to newly-increased memory node
Parameter notifies the newly-increased memory node synchronous N number of data block from selected memory node；The N is by described new
The property parameters for increasing memory node are calculated；After the newly-increased memory node completes data synchronization, the new overall situation is generated
Routing table information；New global routing table information is updated to agent node；The agent node sends for customer in response end
The request for including target KEY, based on the global routing table information, client request is transmitted to corresponding memory node；
And the response bag including target Value for returning to respective stored node sends client to.
Preferably, this method after the registration request of newly-increased memory node is received, further includes：If described increase newly is deposited
Storage node is legal working node, then into next step, otherwise, refuses the addition of the newly-increased memory node.
Preferably, this method after new global routing table information is updated to agent node, further includes：Selected
The free time of fixed memory node progressively deletes its N number of number for being synchronized to the newly-increased memory node stored
According to block.
Another embodiment according to the invention, also discloses a kind of meta data server, including：
Memory module, for storing the global routing table information of reflection target KEY and memory node mapping relations；
Global routing table information synchronization module saves for the global routing table information to be synchronized at least one agency
Point；The agent node is used for the request for including target KEY that sends of customer in response end, based on the overall situation routing table information,
Client request is transmitted to corresponding memory node；And the response for including target Value for returning to respective stored node
Bag sends client to.
Preferably, meta data server can also include：Data simultaneous module, for the attribute according to newly-increased memory node
Parameter notifies the newly-increased memory node synchronous N number of data block from selected memory node；The N is by described new
The property parameters for increasing memory node are calculated；Routing table changes module, and data are completed for working as the newly-increased memory node
After synchronization, new global routing table information is generated；First update notification module is led to for sending routing table update to agent node
Know.
Preferably, meta data server can also include：Monitoring module, for monitoring the existing state of each memory node,
When monitoring a memory node failure, global routing table information is changed；The modification includes：For with the storage of the failure
Data block of the node as host node, its host node is remapped in another redundant node；Second update notification mould
Block, for sending routing table update notice to agent node.
Another embodiment according to the invention, also discloses a kind of agent node, can include：
Forwarding module is inquired about, for the request for including target KEY that customer in response end is sent, by inquiring about global routing table
Client request is transmitted to corresponding memory node by information；
Data forwarding module, the response bag including target Value for respective stored node to be returned send client to
End.
Preferably, the agent node can also include：Routing table synchronization module, for by the global road on server
By table synchronizing information to local；Wherein, the inquiry forwarding module is inquired about based on local global routing table information.
Preferably, the agent node can also include：Routing table update module is led to for that ought receive routing table update
When knowing, new global routing table information is obtained.
Preferably, the inquiry forwarding module specifically includes：
Hash submodule, for based on the client request for including target KEY received, mesh to be calculated by hash algorithm
Mark the data block where KEY；
Submodule is positioned, for according to the data block mark where the target KEY being calculated, inquiring about the global road
By table information, corresponding memory node is positioned；
Submodule is forwarded, for client request to be forwarded to respective stored node.
Compared with prior art, the application has the following advantages：
The present invention proposes a kind of new distributed key-value query engine framework, using the query engine system of the framework
And querying method, it can realize distributed storage and the inquiry of key-value data, and magnanimity key-value data can be accommodated, to height
The adaptability concurrently accessed is stronger.
Wherein, the inquiry overall situation can be passed through since present invention employs the request of data of global routing table information, client
Routing table and then find corresponding storage physical node, then obtain needed for data.Therefore, when need increase physical store
During node, it is only necessary to the increased information of physical store node are updated to global routing table, can realize smooth increase
Or the number of physical store node is reduced, it is serviced without stopping.In comparison, the prior art is directed to memory node not
Global routing table, but do data in application layer and split strategy, increasing the data after memory node, to split cost again very high, needs
Artificial maintenance shut-downs is wanted to operate, automatic elastic are not supported to extend.
Further, since storage and inquiry framework are optimized for distributed key-value (Key-Value) structure by the present invention, it can be with
The complex query mode using SQL database is avoided, improves inquiry velocity, so the intensity concurrently accessed is substantially increased, especially
The information storage and inquiry whether being harmful to it is suitable for determining some application program or some network address；For example, it is particularly suitable for
The occasions such as the killing of wooden horse cloud, the killing of network address cloud and cloud security centre data index.Furthermore due to the present invention be not directly by
Meta data server receives user's request to handle global routing table by agent node, and is acted on behalf of to being synchronized to
The global routing table of node local carries out query processing, in this manner it is possible to which user's request of high concurrent is distributed to multiple agencies
On node, query processing is carried out at the same time by it, so as to also substantially increase the processing capacity of high concurrent access.
Description of the drawings
Fig. 1 is a kind of module diagram of distributed key-value query engine system embodiment of the present invention；
Fig. 2 is a kind of step flow chart of distributed key-value querying method embodiment of the present invention；
Fig. 3 is a kind of step flow chart of failover method of the present invention；
Fig. 4 is a kind of step flow chart for the method for increasing physical store node of the present invention；
Fig. 5 is a kind of structure diagram of preferred distribution KEY-VALUE query engine systems of the present invention；
Fig. 6 is the mapping relations schematic diagram between VBucket and physical store node between a kind of KEY and VBucket；
Fig. 7 is a kind of module diagram of meta data server embodiment of the present invention；
Fig. 8 is a kind of module diagram of agent node embodiment of the present invention.
Specific embodiment
It is below in conjunction with the accompanying drawings and specific real to enable the above-mentioned purpose of the application, feature and advantage more obvious understandable
Mode is applied to be described in further detail the application.
With reference to Fig. 1, show a kind of distributed key-value query engine system embodiment of the present invention, can specifically include：
Meta data server 101, for safeguarding the global routing table information for being directed toward memory node；The maintenance can include
Storage and corresponding modification update etc.；
At least one agent node 102, is shown in FIG. 1 X agent node, and each agent node can include：
Forwarding module is inquired about, for the request for including target KEY that customer in response end is sent, based on the global routing table
Client request is transmitted to corresponding memory node by information；
Data forwarding module, the response bag including target Value for respective stored node to be returned send client to
End；
At least one memory node 103, for based on the client request received, being counted from being locally stored needed for middle acquisition
According to, and send and include the response of target Value and wrap to agent node.Y memory node is shown in FIG. 1.
Using query engine system above, the key-value data of more magnanimity can be accommodated, the adaptation accessed high concurrent
Property is stronger.Wherein, the global routing table information in meta data server 101 is stored with the mapping pass of target KEY → memory node
System.Certainly, which can be direct mapping or indirect mappers, in the Examples hereinafter description of the present invention,
The preferred implementation of an indirect mappers can be provided.
In the preferred embodiments of the present invention, the agent node 102 can also include：Routing table synchronization mould
Block, for the global routing table information of meta data server to be synchronized to local；Wherein, the inquiry forwarding module is based on local
Global routing table information inquired about.This way it is possible to avoid inquiry proxy node all goes to ask to meta data server every time
Inquiry.Simultaneously as agent node can set it is multiple, so as to realize the concurrently execution of routing inquiry, improve to it is high simultaneously
Send out the adaptability accessed.Certainly, the specific synchronous time need not be limited, for example, when agent node is started shooting synchronously
It can；It is also possible to it is synchronized in fixed time, to prevent from not receiving the situation of update notification.
In the preferred embodiments of the present invention, the agent node 102 can also include：Routing table update mould
Block, for when the routing table update for receiving meta data server notifies, new global routing table to be obtained from meta data server
Information.When there is some memory node failure or newly-increased memory node, the global routing table safeguarded on meta data server
Information can all change, thus, global routing table information can be updated by routing table update module at this time.
It should be noted that various storage mediums may be employed in memory node.Such as require the memory of low latency high concurrent
Hash table storage engines or capacity is higher but operating lag is fine, concurrent capability general solid state disk (SSD, Solid
State Disk) storage engines or large capacity but operating lag is high, SAS (the Serial attached of concurrent capability difference
SCSI)/SATA (Serial Advanced Technology Attachment) storage engines.The memory node of the present invention is made
For " engine ", it can increase newly, search, more it is, of course, also possible to have with the executive capability for receiving request and returned data
Newly, the executive capability of (CRUD) data etc. is deleted.
In the preferred implementation of the present invention, it is proposed that using multistage mixing storage engines.It is specifically, described at least one
Memory node includes：It is deposited for the high speed storing node for storing hot spot data and for storing the other types of non-thermal point data
Store up node；Wherein, the high speed storing node includes memory node or solid state disk node, the other types memory node
Including SAS/SATA hard disk nodes.That is the situation based on different data is formed multistage mixing using different storage mediums and deposited
Engine is stored up, to be optimal effect.
The present invention provides a kind of specific realization for client request inquiry, positioning and forwarding below.Wherein, it is described
Inquiry forwarding module can specifically include：Hash submodule, for based on the client request for including target KEY received,
The data block where target KEY is calculated by hash algorithm；Submodule is positioned, for according to the target KEY institutes being calculated
Data block mark, inquire about the global routing table information, position corresponding memory node；Submodule is forwarded, is used for
Client request is forwarded to respective stored node.
Compared with by the way of directly map-table look-up, the method for fixed Hash can effectively reduce the size of routing table,
Improve search efficiency.Specifically, the value range of usually KEY is very big, it is assumed that the composition of KEY is 128 MD5 cryptographic Hash, if
Using direct mapping-look-up method, routing table at most needs to be made of 2^128 items " KEY- memory nodes " mapping relations, this
Be realize in be unpractical.Using hash algorithm calculation data area block, then pass through this inter-species of data block positioning memory node
The method for connecing mapping can effectively reduce the size of routing table.
In some cases, memory node may need to increase.It is described then for preferably management routing in this case
Meta data server can also include：Data simultaneous module, for notifying the newly-increased memory node from selected storage
Synchronous N number of data block in node；The N is calculated by the property parameters of the newly-increased memory node；Routing table changes mould
Block after working as the newly-increased memory node completion data synchronization, generates new global routing table information；First update notification
Module, for sending routing table update notice to agent node.
In practical applications, it is also possible to some memory node occur and break down the situation of failure, then in order to tackle in time
Such case, the meta data server further include：Monitoring module for monitoring the existing state of each memory node, works as monitoring
When failing to a memory node, global routing table information is changed；The modification includes：For the memory node work with the failure
For the data block of host node, its host node is remapped in another redundant node；Second update notification module, is used for
Routing table update notice is sent to agent node.
With reference to Fig. 2, show a kind of distributed key-value querying method embodiment of the present invention, can specifically include：
Step 201 receives the request for including target KEY that client is directed toward current agent node；
Client request is transmitted to and deposits accordingly by inquiring about global routing table information by step 202, current agent node
Store up node；
The response bag including target Value that step 203, current agent node return to respective stored node sends visitor to
Family end.
In the method, global routing table information can be stored on other servers, can also be stored directly in generation
It manages on node, for example, be not the situation of special magnanimity for data volume, using the situation of an a small amount of even agent node, directly
It connects by agent node to safeguard that global routing table information is also feasible；Maintenance is synchronized between each agent node i.e.
It can.
It is, of course, preferable to, in order to improve the ability that agent node processing concurrently accesses, also it is utilized in other server
On separately maintain global routing table information and be preferred.At this point, current agent node can will be maintained in server by asynchronous thread
On global routing table information be synchronized to local, locally carrying out inquiry to global routing table information.
Preferably, in memory node itself or institute's storage location change, above method embodiment can also include：
When receiving routing table update notice, new global routing table information is obtained.
In the preferred implementation of the present invention, global routing table information and indirect storage target KEY → memory node
Direct mapping relations, but store an indirect mappers relation " data block → memory node where target KEY ",
But it so can preferably improve global road directly as the mark of the data block where it is calculated in target KEY
By the inquiry velocity of table.Because in most cases, mark inquiry is much faster than the inquiry of " target KEY ".
In the implementation, the current agent node is turned client request by inquiring about global routing table information
The process for issuing corresponding memory node specifically includes：Based on the client request for including target KEY received, pass through Hash
Algorithm calculates the data block where target KEY；According to the data block mark where the target KEY being calculated, institute is inquired about
Global routing table information is stated, positions corresponding memory node；Client request is transmitted to corresponding memory node.
Correspondingly, with reference to Fig. 3, the present invention also provides a kind of distributed key-value (Key-Value) query engine systems
Failover method is exclusively used in solving the situation that memory node breaks down, including：
Whether step 301, each memory node of monitoring survive；Wherein, same data block is stored in a main section of storage
On point and at least one storage redundant node；
Step 302, when monitoring the failure of memory node, change global routing table information；The modification includes：Pin
To the data block using the memory node of the failure as host node, its host node is remapped to another redundant node
On；
New global routing table information is updated to each agent node by step 303；The agent node is used for customer in response
The request for including target KEY sent is held, based on the global routing table information, client request is transmitted to corresponding storage
Node；And the response bag including target Value for returning to respective stored node sends client to.
Traditional relevant database, such as MySQL if database server breaks down, are typically necessary database
Administrative staff will be switched on backup server the access request of database.And the present invention can realize that the smooth of failure turns
It moves.
In addition, with reference to Fig. 4, the present invention also provides a kind of increases of distributed key-value (Key-Value) query engine system
Physical store nodal method can include：
Step 401, the registration request for receiving newly-increased memory node；
Step 402, the property parameters according to newly-increased memory node, notify the newly-increased memory node to be deposited from selected
Store up synchronous N number of data block in node；The N is calculated by the property parameters of the newly-increased memory node；
Step 403, after the newly-increased memory node completes data synchronization, generate new global routing table information；
New global routing table information is updated to agent node by step 404；The agent node is used for customer in response end
What is sent includes the request of target KEY, and based on the global routing table information, client request is transmitted to corresponding storage section
Point；And the response bag including target Value for returning to respective stored node sends client to.
Certainly, in order to ensure the management and control to memory node, the registration examination ＆ verification to memory node can also be increased.Example
Such as, after the registration request of newly-increased memory node is received, further include：If the newly-increased memory node is legal work section
Point then into next step, otherwise, refuses the addition of the newly-increased memory node.
If it should be noted that some data blocks are transferred on newly-increased memory node, original storage
These data blocks on node can then be used as its redundancy backup to exist, and update in global routing table information.Certainly, it is former
These data blocks on memory node can also be used as invalid data and delete in the case where redundancy backup quantity is enough,
Reduce the load of former memory node.
For example, after new global routing table information is updated to agent node, further include：In selected storage section
The free time of point, progressively delete its N number of data block for being synchronized to the newly-increased memory node stored.
In short, for Fig. 3 and Fig. 4 failover method provided and increasing physical store nodal method, can help
The present invention smoothly increases in distributed key-value (Key-Value) query engine system or reduces machine number, without stopping
Service.After increasing and decreasing machine, data can reach smooth point (such as by host node and redundancy section again between more machines automatically
The configuration of point is realized smooth).
With reference to Fig. 5, a kind of structural representation of preferred distribution KEY-VALUE query engine systems of the present invention is shown
Figure.
Wherein, the client of multiple concurrent requests is shown.Client (Client) is mainly used for initiating request.Specifically
, the present invention can support the client (C/C++/Python/PHP etc.) of multilingual.
Meta data server (Config Server)：It is responsible for safeguarding global routing table information, and monitors all memory nodes
Existing state, fail in memory node and when increasing memory node newly, meta data server plays crucial coordinative role.
Agent node (Storage Proxy)：It is responsible for the request at customer in response end and the forwarding of request.It is taken according to metadata
The routing table information being engaged on device, the request of client can be transmitted to the memory node in downstream by agent node, and storage is saved
The response bag of point passs client.
Memory node (Storage Node)：It is responsible for the actual storage of data, can be selective there are many storage engines, example
Such as require the memory Hash table storage engines of low latency high concurrent or capacity is higher but operating lag still, concurrent capability one
As solid state disk (SSD) storage engines or large capacity but operating lag are high, the SAS/SATA storages of concurrent capability difference are drawn
The multistage mixing storage engines held up or customized.
In Figure 5, between client and memory node, including Nginx/UDP servers and agent node, certainly,
Nginx/UDP servers are generally only a required transmission equipment, people in the art in distributing communication system
Member can also use other equipment, not use even.Also it is a pair of that one is not necessarily between Nginx/UDP servers and agent node
The relation answered.
This need not be subject to it should be noted that there may be multiple agent nodes, the present invention in an entity hardware
Limitation.And the situation of data block redundancy backup storage is also shown in Figure 5.
Specifically, Fig. 5 gives 4 memory nodes, wherein,
Memory node a is the host node of data block _ 0, the host node of data block _ 1, the redundancy section of data block _ 6
Point, the redundant node of data block _ 7；
Memory node b is the redundant node of data block _ 0, the redundant node of data block _ 1, the main section of data block _ 2
Point, the host node of data block _ 3；
Memory node c is the redundant node of data block _ 2, the redundant node of data block _ 3, the main section of data block _ 4
Point, the host node of data block _ 5；
Memory node d is the redundant node of data block _ 4, the redundant node of data block _ 5, the main section of data block _ 6
Point, the host node of data block _ 7；
Such storage mode, including ensureing in the case where being uniformly distributed, each data block may be stored in one
On a host node, also it is stored in simultaneously in a redundant node.
That is, each physical store node can be certain several VBucket (Virtual Bucket；Mathematical logic point
Trivial, i.e. one kind of data block) host node, and do the redundant node of other several VBucket.Have on memory node solely
Vertical thread, from host node the responsible redundancy VBucket of synchronous institute it is newly-increased/change data.
In Figure 5, meta data server can be configured with the server of a backup.
Below for system shown in Figure 5, description one is initiated request to from client receives the complete of query engine response
The example of flow.
1) by taking GET request as an example, client initiates the GET request of single KEY to agent node (Storage Proxy),
It can also be the GET request of batch KEY.
2) after agent node receives the request of upstream client, KEY places are calculated by fixed hash algorithm (such as MD5)
VBucket sequence numbers, i.e. data block area code.
3) agent node searches global routing table, positions physical store node by the VBucket sequence numbers belonging to KEY
(Storage Node).In realization, meta data server (Config need not be asked every time by searching the process of global routing table
Server), but routing iinformation is synchronized to local by agent node by an asynchronous thread.
4) what proxy server was navigated to by VBucket sequence numbers is one group of physical store node, wherein first node
It is host node, is responsible for all readings and the write request of the VBucket, is subsequently redundancy backup node.Proxy server is by visitor
The GET request at family end is transmitted to host node.
5) after physical store node receives GET request, the corresponding data of KEY is taken out from being locally stored in engine, are sent
Response bag is to agent node.
6) after agent node receives the response bag of physical store node, response bag is returned into client.
With reference to Fig. 6, show between KEY and VBucket, the mapping relations between VBucket and physical store node are shown
It is intended to.Between KEY and VBucket, mapping relations are obtained by way of fixed Hash, in VBucket and physical store section
Between point, mapping relations are obtained by way of tabling look-up.
The example of the failure transfer flow of one physical store node failure is described below.
1) A node failures.
2) there is the real time monitoring process to all physics memory node existing states on meta data server.When discovery A sections
During point failure, i.e., routing table can be changed, using A nodes as the VBucket of host node, host node is remapped to next superfluous
On remaining node.
3) all agent nodes, all can be from meta data server synchronization routing table information.After routing table change is found, after
Continuous request will be route according to new routing table.
The example of the flow of one increase, one physical store node is described below.
1) registration request can be sent to meta data server by increasing physical store node B, B newly.
2) after meta data server receives registration request, confirm whether B is working node, if not legal working node is then
Refusal, if then entering the flow of following new node data preparation.
3) there is m physical store node before assuming, and VBucket sums are vt, meta data server calculates new add in
The VBucket number that node can carry is number=abs (vt/m+1).Meta data server initiate instruction notification B from work as
Synchronization number (N number of) VBucket in preceding active memory node.
4) it is synchronous to complete VBucket data by new memory node B, its is ready for notice meta data server.
5) meta data server generates a new routing table.
6) agent node receives the routing table update notice of meta data server, by subsequent request according to new routing table road
By so far, new physics node B is formally added among cluster and services.
Because newly-increased memory node has shared VBucket, power system capacity is added.During idle time, current active is deposited
Storage node can be phased out being not belonging to the VBucket of itself.
The present invention realizes one and is made of more machines, can accommodate mass data, and can be by increasing clustered machine
Quantity expand company-data scale, reliable and stable key-value storage system and storage on adaptations high concurrent access net
Network model.The present invention has some following innovative points：
The present invention realizes the data distribution strategy based on uniformity hash algorithm.Cluster internal has the metadata of lightweight
Server only stores global routing table information, i.e. data block area code and the mapping relations of storage physical node, the access to key
Request first can calculate data block area code according to hash algorithm, then be responsible for storage physics is found by global routing table
Node.
The cluster internal of the present invention can smoothly increase or reduce machine number, be serviced without stopping.After increasing and decreasing machine,
Data can reach smooth point again between more machines automatically.
The present invention supports data redundancy storage.All back end can simultaneously as other nodes redundant node.
The network access interface for supporting high concurrent externally may be employed in the cluster of the present invention, and can be by increasing agency's section
The quantity of point and memory node improves the concurrent capability of entire cluster.Agent node externally communicates, agent node and memory node
Between internal communication can preferably use the TCP network models based on I O multiplexing.
The present invention also supports the customization of storage engines.It can select memory storage engine according to application demand or consolidate
The storage engines of state hard-disc storage engine or SAS/SATA hard disks；Or mixed multistage storage engines can be selected, by heat
Data cache on high-speed equipment (memory, solid state disk).
In short, the present invention can realize the cluster dilatation of automation, by the number for increasing cluster internal physical store node
Amount, internal data can be automatically transferred on new memory node, so as to reach the memory capacity of expansion cluster and concurrently access energy
The purpose of power.The present invention can carry the storage of mass data and the high concurrent of daily hundred billion rank accesses.By it is suitable for wood
The associated safeties products such as horse cloud killing service, network address cloud killing service, cloud security centre data index center and background application clothes
In business.
A kind of meta data server embodiment of the present invention is shown with reference to Fig. 7, as the critical component in the present invention program,
It can include：
Memory module 701, for storing the global routing table information of reflection target KEY and memory node mapping relations；
Global routing table information synchronization module 702, for the global routing table information to be synchronized at least one agency
Node；The agent node is used for the request for including target KEY that customer in response end is sent, and is believed based on the global routing table
Breath, corresponding memory node is transmitted to by client request；And the returning including target Value for returning to respective stored node
It should wrap and send client to.
In the preferred implementation, meta data server can also include：Data simultaneous module 703, for according to newly-increased storage
The property parameters of node notify the newly-increased memory node synchronous N number of data block from selected memory node；It is described
N is calculated by the property parameters of the newly-increased memory node；Routing table changes module 704, for working as the newly-increased storage
After node completes data synchronization, new global routing table information is generated；First update notification module 705, for agent node
Send routing table update notice.
For the meta data server with failover functionality, it can also include：Monitoring module 706, is used for
The existing state of each memory node is monitored, when monitoring a memory node failure, changes global routing table information；It is described to repair
Change including：For the data block using the memory node of the failure as host node, its host node is remapped to another
In redundant node；Second update notification module 707, for sending routing table update notice to agent node.
With reference to Fig. 8, a kind of agent node embodiment of the present invention is shown, as the critical component in the present invention program,
It can include：
Forwarding module 801 is inquired about, for the request for including target KEY that customer in response end is sent, by inquiring about global road
By table information, client request is transmitted to corresponding memory node；
Data forwarding module 802, the response bag including target Value for respective stored node to be returned send visitor to
Family end.
In the preferred implementation, agent node can also include：Routing table synchronization module 803, for will be complete on server
Office's routing table information is synchronized to local；Wherein, the inquiry forwarding module is inquired about based on local global routing table information.
In the preferred implementation, agent node can also include：Routing table update module 804 receives routing table more for working as
During new notice, new global routing table information is obtained.It certainly, in the concrete realization, can also be directly by routing table update module
804 function is integrated into routing table synchronization module 803.
In the preferred implementation scheme of the present invention, the inquiry forwarding module 801 can specifically include：
Hash submodule 8011, for based on the client request for including target KEY received, passing through hash algorithm meter
Calculate the data block where target KEY；
Submodule 8012 is positioned, for according to the data block mark where the target KEY being calculated, inquiring about described complete
Office's routing table information, positions corresponding memory node；
Submodule 8013 is forwarded, for client request to be forwarded to respective stored node.
Each embodiment in this specification is described by the way of progressive, the highlights of each of the examples are with
The difference of other embodiment, just to refer each other for identical similar part between each embodiment.For device embodiment
For, since it is basicly similar to embodiment of the method, so description is fairly simple, referring to the portion of embodiment of the method in place of correlation
It defends oneself bright.
Above to a kind of distributed key-value query engine system provided herein, distributed key-value querying method,
Agent node, meta data server, the method for increasing physical store node and a kind of failover method are described in detail,
The principle and implementation of this application are described for specific case used herein, and the explanation of above example is simply used
Understand the present processes and its core concept in help；Meanwhile for those of ordinary skill in the art, according to the application's
Thought, there will be changes in specific embodiments and applications, in conclusion this specification content should not be construed as
Limitation to the application.
Claims (16)
1. a kind of distributed key-value query engine system, which is characterized in that including：
Meta data server, at least one agent node and at least one memory node；
Meta data server, for safeguarding the global routing table information for being directed toward memory node；
At least one agent node, including：
Routing table synchronization module, inquiry forwarding module and data forwarding module；
Routing table synchronization module, for the global routing table information of meta data server to be synchronized to the sheet of the agent node
Ground specifically includes：Current agent node will safeguard that global routing table information on the server is synchronized to institute by asynchronous thread
The local of agent node is stated, the inquiry to global routing table information is carried out in the local of the agent node；
Forwarding module is inquired about, for the request for including target KEY that customer in response end is sent, is believed based on the global routing table
Client request is transmitted to corresponding memory node by breath, the local of the inquiry forwarding module based on the agent node
Global routing table information is inquired about；
The inquiry forwarding module includes：Hash submodule, positioning submodule and forwarding submodule；
Hash submodule, for based on the client request for including target KEY received, target to be calculated by hash algorithm
Data block where KEY；
Submodule is positioned, for the mark according to the data block where the target KEY being calculated, inquires about the global routing
Table information positions corresponding memory node；
Submodule is forwarded, for client request to be forwarded to respective stored node；
Data forwarding module, the response bag including target Value for respective stored node to be returned send client to；
At least one memory node, for based on the client request received, from data needed for middle acquisition are locally stored, concurrently
The response including target Value is sent to wrap to agent node.
2. system according to claim 1, which is characterized in that the agent node further includes：
Routing table update module, for when the routing table update for receiving meta data server notifies, being obtained from meta data server
Take new global routing table information.
3. system according to claim 1, which is characterized in that the meta data server further includes：
Data simultaneous module, for notifying newly-increased memory node synchronous N number of data block from selected memory node；Institute
N is stated to be calculated by the property parameters of the newly-increased memory node；
Routing table changes module, after working as the newly-increased memory node completion data synchronization, generates new global routing table
Information；
First update notification module, for sending routing table update notice to agent node.
4. the system according to claim 1 or 3, which is characterized in that the meta data server further includes：
Monitoring module, for monitoring the existing state of each memory node, when monitoring a memory node failure, modification is global
Routing table information；The modification includes：For the data block using the memory node of the failure as host node, by its host node
It is remapped in another redundant node；
Second update notification module, for sending routing table update notice to agent node.
5. system according to claim 1, which is characterized in that at least one memory node includes：
Other types memory node for the high speed storing node for storing hot spot data and for storing non-thermal point data；
Wherein, the high speed storing node includes memory node or solid state disk node, the other types memory node bag
Include SAS/SATA hard disk nodes.
6. a kind of distributed key-value querying method, which is characterized in that including：
Receive the request for including target KEY that client is directed toward current agent node；
Current agent node will safeguard that global routing table information on the server is synchronized to agency's section by asynchronous thread
The local of point, the inquiry to global routing table information is carried out in the local of the agent node；
Client request is transmitted to corresponding memory node by inquiring about global routing table information by current agent node, described
Client request is transmitted to corresponding memory node and specifically included by current agent node by inquiring about global routing table information：
Based on the client request for including target KEY received, the data block where target KEY is calculated by hash algorithm；According to
According to the mark of the data block where the target KEY being calculated, the global routing table information is inquired about, positions corresponding storage
Node；Client request is forwarded to respective stored node；
The response bag including target Value that current agent node returns to respective stored node sends client to.
7. method as claimed in claim 6, which is characterized in that further include：
When receiving routing table update notice, new global routing table information is obtained.
8. a kind of failover method of distributed key-value (Key-Value) query engine system, which is characterized in that including：
Current agent node will safeguard that global routing table information on the server is synchronized to agency's section by asynchronous thread
The local of point, the inquiry to global routing table information is carried out in the local of the agent node；
Monitor whether each memory node survives；Wherein, same data block is stored in a storage host node and at least one
It stores in redundant node；
When monitoring a memory node failure, global routing table information is changed；The modification includes：For with the failure
Data block of the memory node as host node, its host node is remapped in another redundant node；
New global routing table information is updated to each agent node；The agent node includes for what customer in response end was sent
Client request based on the global routing table information, is transmitted to corresponding memory node, the base by the request of target KEY
In the new global routing table information, client request is transmitted to corresponding memory node and is specifically included：Based on being received
The client request for including target KEY, pass through hash algorithm and calculate data block where target KEY；Foundation is calculated
Target KEY where data block mark, inquire about the new global routing table information, position corresponding memory node；
Client request is forwarded to respective stored node；And the response bag for including target Value for returning to respective stored node
Send client to.
9. a kind of increase physical store nodal method of distributed key-value (Key-Value) query engine system, feature exist
In, including：
Current agent node will safeguard that global routing table information on the server is synchronized to agency's section by asynchronous thread
The local of point, the inquiry to global routing table information is carried out in the local of the agent node；
Receive the registration request of newly-increased memory node；
According to the property parameters of newly-increased memory node, notify that the newly-increased memory node is synchronous from selected memory node
N number of data block；The N is calculated by the property parameters of the newly-increased memory node；
After the newly-increased memory node completes data synchronization, new global routing table information is generated；
New global routing table information is updated to agent node；The agent node includes mesh for what customer in response end was sent
The request of KEY is marked, based on the global routing table information, client request is transmitted to corresponding memory node, it is described to be based on
The new global routing table information, is transmitted to corresponding memory node by client request and specifically includes：Based on what is received
Include the client request of target KEY, the data block where target KEY is calculated by hash algorithm；According to what is be calculated
The mark of data block where target KEY inquires about the new global routing table information, positions corresponding memory node；It will
Client request is forwarded to respective stored node；And the response bag including target Value for returning to respective stored node passes
Give client.
10. method as claimed in claim 9, which is characterized in that after the registration request of newly-increased memory node is received, also wrap
It includes：
If the newly-increased memory node is legal working node, into next step, otherwise, refuse the newly-increased memory node
Addition.
11. method as claimed in claim 9 after new global routing table information is updated to agent node, further includes：
In the free time of selected memory node, that progressively deletes that it is stored has been synchronized to the newly-increased memory node
N number of data block.
12. a kind of meta data server, which is characterized in that including：
Memory module and global routing table information synchronization module；
Memory module, for storing the global routing table information of reflection target KEY and memory node mapping relations；
Global routing table information synchronization module, for by the global routing table information be synchronized at least one agent node and,
The global routing table information of meta data server is synchronized to the local of the agent node,
Global routing table information synchronization module includes：Current agent node passes through the overall situation of the asynchronous thread by maintenance on the server
Routing table information is synchronized to the local of the agent node, is carried out in the local of the agent node to global routing table information
Inquiry；The agent node is used for the request for including target KEY that customer in response end is sent, and is believed based on the global routing table
Client request is transmitted to corresponding memory node by breath, described based on the global routing table information, and client request is turned
Corresponding memory node is issued to specifically include：Based on the client request for including target KEY received, pass through hash algorithm meter
Calculate the data block where target KEY；According to the mark of the data block where the target KEY being calculated, inquire about described complete
Office's routing table information, positions corresponding memory node；Client request is forwarded to respective stored node；And by respective stored
The response bag including target Value that node returns sends client to.
13. meta data server according to claim 12, which is characterized in that further include：
Data simultaneous module for the property parameters according to newly-increased memory node, notifies the newly-increased memory node from selected
Synchronous N number of data block in fixed memory node；The N is calculated by the property parameters of the newly-increased memory node；
Routing table changes module, after working as the newly-increased memory node completion data synchronization, generates new global routing table
Information；
First update notification module, for sending routing table update notice to agent node.
14. the meta data server according to claim 12 or 13, which is characterized in that further include：
Monitoring module, for monitoring the existing state of each memory node, when monitoring a memory node failure, modification is global
Routing table information；The modification includes：For the data block using the memory node of the failure as host node, by its host node
It is remapped in another redundant node；
Second update notification module, for sending routing table update notice to agent node.
15. a kind of agent node, which is characterized in that including：
Routing table information synchronization module inquires about forwarding module and data forwarding module；
Routing table information synchronization module, for the global routing table information of meta data server to be synchronized to the agent node
It is local, it specifically includes：Current agent node will safeguard that global routing table information on the server is synchronized to by asynchronous thread
The local of the agent node carries out the inquiry to global routing table information in the local of the agent node；
Forwarding module is inquired about, for the request for including target KEY that customer in response end is sent, is believed by inquiring about global routing table
Client request is transmitted to corresponding memory node by breath, the local of the inquiry forwarding module based on the agent node
Global routing table information is inquired about；
The inquiry forwarding module includes：
Hash submodule, positioning submodule and forwarding submodule；
Hash submodule, for based on the client request for including target KEY received, target to be calculated by hash algorithm
Data block where KEY；
Submodule is positioned, for the mark according to the data block where the target KEY being calculated, inquires about the global routing
Table information positions corresponding memory node；
Submodule is forwarded, for client request to be forwarded to respective stored node；
Data forwarding module, the response bag including target Value for respective stored node to be returned send client to.
16. agent node according to claim 15, which is characterized in that further include：
Routing table update module, for when receiving routing table update notice, obtaining new global routing table information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410256925.2A CN104050250B (en) | 2011-12-31 | 2011-12-31 | A kind of distributed key-value querying method and query engine system |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410256925.2A CN104050250B (en) | 2011-12-31 | 2011-12-31 | A kind of distributed key-value querying method and query engine system |
| CN201110460494.8A CN102591970B (en) | 2011-12-31 | 2011-12-31 | Distributed key-value query method and query engine system |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110460494.8A Division CN102591970B (en) | 2011-12-31 | 2011-12-31 | Distributed key-value query method and query engine system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104050250A CN104050250A (en) | 2014-09-17 |
| CN104050250B true CN104050250B (en) | 2018-06-05 |
Family
ID=51503082
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410256925.2A Expired - Fee Related CN104050250B (en) | 2011-12-31 | 2011-12-31 | A kind of distributed key-value querying method and query engine system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104050250B (en) |
Families Citing this family (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105530162A (en) * | 2014-10-21 | 2016-04-27 | 中兴通讯股份有限公司 | Offline message processing method and device |
| CN104461817B (en) * | 2014-11-26 | 2017-08-11 | 华为技术有限公司 | A kind of method and server for detecting key |
| CN104601720A (en) * | 2015-01-30 | 2015-05-06 | 乐视网信息技术（北京）股份有限公司 | Cache access control method and device |
| CN106453449A (en) * | 2015-08-06 | 2017-02-22 | 泰兴市智瀚科技有限公司 | Information instant pushing method and distribution type system server |
| CN106254240B (en) * | 2016-09-18 | 2019-07-05 | 腾讯科技（深圳）有限公司 | A kind of data processing method and routing layer equipment and system |
| CN106817402B (en) * | 2016-11-29 | 2018-02-27 | 上海壹账通金融科技有限公司 | Data cached processing method and processing device |
| CN108206779B (en) * | 2016-12-16 | 2020-08-14 | 北京金山云网络技术有限公司 | Cluster access system, method and device |
| CN106878414B (en) * | 2017-02-14 | 2019-06-07 | 北京奇虎科技有限公司 | Data write request processing method, device and distributed data-storage system |
| CN109582213B (en) * | 2017-09-29 | 2020-10-30 | 杭州海康威视系统技术有限公司 | Data reconstruction method and device and data storage system |
| CN108197498A (en) * | 2017-12-22 | 2018-06-22 | 中国联合网络通信集团有限公司 | Obtain the method and device of data |
| CN108494835B (en) * | 2018-03-08 | 2020-11-03 | 浪潮云信息技术股份公司 | Method and system for realizing distributed dynamic routing based on Raft algorithm |
| CN112995285B (en) * | 2018-03-29 | 2023-04-18 | 北京忆芯科技有限公司 | Distributed KV storage system based on block technology |
| CN113542382B (en) * | 2018-03-30 | 2024-04-26 | 北京忆芯科技有限公司 | KV storage device in cloud computing and fog computing system |
| CN110198269B (en) * | 2018-04-03 | 2021-10-08 | 腾讯科技（深圳）有限公司 | Route synchronization system, method and related device for distributed cluster |
| CN108829545B (en) * | 2018-07-02 | 2021-08-10 | 上海浪潮云计算服务有限公司 | Method for realizing distributed database backup |
| CN109933568A (en) * | 2019-03-13 | 2019-06-25 | 安徽海螺集团有限责任公司 | A kind of industry big data platform system and its querying method |
| CN111581451B (en) * | 2019-03-28 | 2021-08-06 | 北京忆芯科技有限公司 | Update and query of distributed KV storage system |
| CN110334543B (en) * | 2019-06-20 | 2023-01-17 | 广东技术师范大学天河学院 | Calculation power-based block chain knowledge system and use method thereof |
| CN110275681A (en) * | 2019-06-27 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of date storage method and data-storage system |
| CN112448985B (en) * | 2019-09-02 | 2022-07-15 | 阿里巴巴集团控股有限公司 | Distributed system, network processing method and device and electronic equipment |
| CN113065054B (en) * | 2021-03-31 | 2024-03-01 | 北京达佳互联信息技术有限公司 | Request processing method, request processing device, electronic equipment and storage medium |
| CN115934006B (en) * | 2023-03-14 | 2023-05-12 | 北京志凌海纳科技有限公司 | IO access point and data processing task management method, device, equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170416A (en) * | 2006-10-26 | 2008-04-30 | 阿里巴巴公司 | Network data storage system and data access method |
| CN101232422A (en) * | 2008-01-18 | 2008-07-30 | 北京交通大学 | Network storage system based on graticule technique |
| CN102201010A (en) * | 2011-06-23 | 2011-09-28 | 清华大学 | Distributed database system without sharing structure and realizing method thereof |
-
2011
- 2011-12-31 CN CN201410256925.2A patent/CN104050250B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170416A (en) * | 2006-10-26 | 2008-04-30 | 阿里巴巴公司 | Network data storage system and data access method |
| CN101232422A (en) * | 2008-01-18 | 2008-07-30 | 北京交通大学 | Network storage system based on graticule technique |
| CN102201010A (en) * | 2011-06-23 | 2011-09-28 | 清华大学 | Distributed database system without sharing structure and realizing method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104050250A (en) | 2014-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104050250B (en) | A kind of distributed key-value querying method and query engine system | |
| CN104050249B (en) | Distributed query engine system and method and meta data server | |
| CN102591970B (en) | Distributed key-value query method and query engine system | |
| US11294890B2 (en) | Batch data ingestion in database systems | |
| Jiang et al. | THE optimization of HDFS based on small files | |
| AU2015240916B2 (en) | File storage using variable stripe sizes | |
| JP6371858B2 (en) | Atomic writing for multiple extent operations | |
| Liao et al. | Multi-dimensional index on hadoop distributed file system | |
| EP2169909B1 (en) | System and method to maintain coherence of cache contents in a multi-tier software system aimed at interfacing large databases | |
| JP4116413B2 (en) | Prefetch appliance server | |
| CN109871367A (en) | A kind of distributed cold and heat data separation method based on Redis and HBase | |
| JP2017510004A (en) | Session management in distributed storage systems | |
| JP2005196602A (en) | System configuration changing method in unshared type database management system | |
| CN104184812B (en) | A kind of multipoint data transmission method based on private clound | |
| Fu et al. | Performance optimization for managing massive numbers of small files in distributed file systems | |
| US20170351620A1 (en) | Caching Framework for Big-Data Engines in the Cloud | |
| CN101404649A (en) | Data processing system based on CACHE and its method | |
| JP5331050B2 (en) | Data synchronization system, data synchronization method, information processing apparatus, information processing method, and program | |
| Le et al. | Namenode and datanode coupling for a power-proportional hadoop distributed file system | |
| Yan et al. | Data caching techniques in web application | |
| Le et al. | NDCouplingHDFS: A coupling architecture for a power-proportional Hadoop distributed file system | |
| Alshamrani et al. | An efficient approach for storage of big data streams in distributed stream processing systems | |
| CN117539915B (en) | Data processing method and related device | |
| Wang et al. | Provably Good Randomized Strategies for Data Placement in Distributed Key-Value Stores | |
| Xu et al. | ALIRS: A High Scalability and High Cache Hit Ratio Replacement Algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180605 Termination date: 20211231 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |