CN103810209B - A kind of method and system saving data - Google Patents
A kind of method and system saving data Download PDFInfo
- Publication number
- CN103810209B CN103810209B CN201210455442.6A CN201210455442A CN103810209B CN 103810209 B CN103810209 B CN 103810209B CN 201210455442 A CN201210455442 A CN 201210455442A CN 103810209 B CN103810209 B CN 103810209B
- Authority
- CN
- China
- Prior art keywords
- data
- backup point
- newly
- increased
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The present invention proposes a kind of date storage method and system, and wherein method includes: that multiple backup points are arranged for a data, and each backup point is ranked up and saves to this part of data according to respective ordering rule；Wherein, at least one backup point is full backup point, saves the full content of this part of data；Other backup points are incremental backup point, save the partial content of this part of data.The present invention can be improved the access speed of data.
Description
Technical field
The present invention relates to data processing and technical field of memory more particularly to a kind of method and system for saving data.
Background technique
With the development of internet technology, the data for storing and applying in internet are increasingly longer, for example, UGC class application
The mainstream of current internet is increasingly becomed, microblogging, social networks, blog are all that UGC class is mainly applied.These applications are often gathered around
There are mass users, stores the UGC data of magnanimity.In such applications, for certain special users, certain class data may be non-
Chang Duo, it grows very much, for example in microblogging application, the bean vermicelli list (listening to the other users list of the user) of any active ues can be non-
Often long, the number of fans of some any active ues is even more than 1 hundred million.
The storage scheme of existing data is to store in the database mostly, and every table safeguards one group of long data, such as will
Someone bean vermicelli information establishes a table, and table name is distinguished with the Key of user.When the external world reads these data, backstage is needed to deposit
Storage services type according to different requests, is ranked up data different field (such as according to time sequence, by place sequence, by word
Female size sequence etc.), the data that extraneous request is read are searched, result is returned again to；Calculation amount therein is bigger, to request
Have biggish time delay.
More complete scheme can build multiple hot standby databases, and a master writes a little, multiple standby read points, realization pair
The shunting of request.This mode is when reading data, it is also desirable to storing data is ranked up, equally exist it is computationally intensive and
Be delayed larger problem, therefore extraneous slower to the access speed of data.
Summary of the invention
The present invention provides a kind of date storage methods, can be improved the access speed of data.
The present invention also provides a kind of data-storage systems, can be improved the access speed of data.
The technical scheme of the present invention is realized as follows:
A kind of date storage method, comprising:
Multiple backup points are set for a data, each backup point carries out this part of data according to respective ordering rule
It sorts and saves；Wherein, at least one backup point is full backup point, saves the full content of this part of data；Other backup points
For incremental backup point, the partial content of this part of data is saved.
In the above method, when access layer receives the newly-increased request for this part of data, modification request or removal request,
The newly-increased request, modification request or removal request are sent to each backup point；Each backup point is according to respective row
Sequence rule, increases, modifies or deletes this part of data.
It wherein, will be newly-increased according to the keyword (Key) of newly-increased data when full backup point increases this part of data newly
Data are added to suitable position and save, or update original data；
When the incremental backup point increases this part of data newly, it is standby to judge whether the Key of newly-increased data belongs to the part
The range of the Key of part point stored data, if it is, the suitable position of newly-increased data is searched according to the Key of newly-increased data, it will
Newly-increased data are added to suitable position and save, or update original data.
When each backup point is modified or deleted to this part of data, according to respective ordering rule and modification or deletion
The data are modified or are deleted by the data that the Key of data finds request modification or deletes；
For incremental backup point, if the data volume deleted is more than preset threshold value, from the full backup
Point extracts new data and preservation.
In the above method, when access layer receives the read requests for this part of data, according to the read requests
The read requests are forwarded to corresponding backup point by ordering rule；The corresponding backup point is by the data feedback of reading to connecing
Enter layer.
A kind of data-storage system, system include access layer and multiple backup points；Wherein,
For a data, each backup point is ranked up and protects to this part of data according to respective ordering rule
It deposits；Wherein, at least one backup point is full backup point, saves the full content of this part of data；Other backup points are that part is standby
Part point, saves the partial content of this part of data.
The access layer is used for, will when receiving the newly-increased request for a data, modification request or removal request
The newly-increased request, modification request or removal request are sent to each backup point for saving this part of data；
Each backup point increases, modifies or deletes this part of data according to respective ordering rule.
Wherein, when full backup point increases this part of data newly, newly-increased data are added according to the Key of newly-increased data
It to suitable position and saves, or updates original data；
When the incremental backup point increases this part of data newly, it is standby to judge whether the Key of newly-increased data belongs to the part
The range of the Key of part point stored data, if it is, the suitable position of newly-increased data is searched according to the Key of newly-increased data, it will
Newly-increased data are added to suitable position and save, or update original data.
When each backup point is modified or deleted to this part of data, according to respective ordering rule and modification or deletion
The data are modified or are deleted by the data that the Key of data finds request modification or deletes；
For incremental backup point, if the data volume deleted is more than preset threshold value, from the full backup
Point extracts new data and preservation.
Access layer is also used to, when receiving the read requests for a data, according to the sequence of the read requests
The read requests are forwarded to corresponding backup point by rule；
The corresponding backup point is by the data feedback of reading to access layer
As it can be seen that date storage method proposed by the present invention and system, same part data are stored in multiple backup points, each
Backup point is ranked up data according to respective ordering rule, and this avoid the sequence calculating when reading data, realizes
The quick access of the same multiple dimensions of part data, improves the flexibility and responding ability of the long data storage service of magnanimity.
Detailed description of the invention
Fig. 1 is data storage system architecture schematic diagram proposed by the present invention；
Fig. 2 is that the message of the embodiment of the present invention one flows to schematic diagram；
Fig. 3 is that the message of the embodiment of the present invention two flows to schematic diagram.
Specific embodiment
The present invention proposes a kind of date storage method and system, if Fig. 1 is data storage system architecture proposed by the present invention
Schematic diagram, system proposed by the present invention are divided into upper layer and lower layer, accumulation layer and access layer.For same part data, set in accumulation layer
Multiple backup points are set, each backup point is ranked up and saves to this part of data according to respective ordering rule；Wherein, at least one
A backup point is full backup point, saves the full content of this part of data；Other backup points are incremental backup point, save the number
According to partial content.
The read and write access of extraneous all pairs of accumulation layers is both needed to be forwarded to accumulation layer by access layer, and access layer perceives accumulation layer
Backup point number and each backup point ordering rule.The ordering rule of each backup point is different.Each backup point storage
Data be not necessarily required to full dose, can be a part of data therein, but at least one backup data be full dose.
For example, temporally sorting from the near to the distant in backup point 1 for same part data, 1000 datas are stored；In backup point 2
It sorts by user's name, stores full dose data；It sorts in backup point 3 by user integral, stores 10,000 datas.Except full dose is standby
Other than part point, the maximum amount of data of other backup points storage is determined by the object of its service.It is needed when the later period extends new sequence
When asking, the corresponding configuration of access layer only need to be changed, and in addition build a set of new backup point.
In the manner described above after storing data, for extraneous different types of request, the present invention uses different processing
Mode.Below referring to attached drawing, lifts specific embodiment and be discussed in detail.
Embodiment one:
In the present embodiment, handle from extraneous newly-increased request, modification request or removal request.If Fig. 2 is this
The message of inventive embodiments one flows to schematic diagram.Access layer receives the newly-increased request for a data, modification request or deletes
When except request, the newly-increased request, modification request or removal request are sent to each backup point of this part of message；Each backup
After point receives request, according to respective ordering rule, this part of data are increased, modified or deleted.
Wherein, for newly-increased request, the newly-increased data of request are added to by each backup point according to respective ordering rule
Suitable position simultaneously saves, or updates original data.In full backup point, suitable position is inserted data into according to its Key
It sets.On incremental backup point, if the Key of newly-increased data belongs to the range of the Key of the incremental backup point stored data, press
Newly-increased data are added to suitable position and saved by the suitable position that newly-increased data are searched according to the Key of newly-increased data, or
Update original data；If the sequence Key of newly-increased data is more than the maximum or minimum k ey of existing sorting data, it is not necessarily to
Insertion is (if original data are compared according to permutation with positive order (from small to large) with maximum key and the key of newly-increased data；If
Data originally are then compared with the key of minimum k ey and newly-increased data according to reversing (from big to small)).
It is requested for modification, each backup point finds request according to the Key of respective ordering rule and the data of modification and repairs
The data changed modify to the data.
For removal request, each backup point finds request according to the Key of respective ordering rule and the data of deletion and deletes
The data removed delete the data.On incremental backup point, if deleting data volume is more than that preset threshold value (is such as more than
50%) it, then needs to pull new data and preservation from full backup point, fills the incremental backup point.
It,, can be with so using general binary chop since data volume is less for incremental backup point when searching data
It realizes and quickly searches.It, can be using the mechanism for allocating data segment in advance since data volume is huger for full backup point.
For example, splitting data into N number of section as segmentation basis using user id.Using id%N as segment number.Again by user's in each section
Id sorts.When queried, segment number is determined first with user id%N, then do binary chop in section again, find user.
Embodiment two:
In the present embodiment, handle from extraneous read requests.Access layer receives a variety of external read requests, this
A little read passes through access layer, is uniformly converted to the order that data are read from accumulation layer.These external read requests are according to accumulation layer
The various ordering rules that are capable of providing determine.Access layer is by these external read requests and corresponding accumulation layer backup point pair
It answers, therefore access layer is capable of determining that the ordering rule of read requests, and read requests are sent to the standby of identical ordering rule
Part point；After the backup point receives request, access layer is fed data back to, requestor is returned data to by access layer.
Message if Fig. 3 is the embodiment of the present invention two flows to schematic diagram.When access layer receives the reading for a data
When taking request, which is forwarded to by corresponding backup point according to the ordering rule of read requests；Corresponding backup point will
The data feedback of reading is to access layer.
The present invention also proposes a kind of data-storage system, including access layer and multiple backup points；Wherein,
For a data, each backup point is ranked up and protects to this part of data according to respective ordering rule
It deposits；Wherein, at least one backup point is full backup point, saves the full content of this part of data；Other backup points are that part is standby
Part point, saves the partial content of this part of data.
Above-mentioned access layer is used for, will when receiving the newly-increased request for a data, modification request or removal request
The newly-increased request, modification request or removal request are sent to each backup point for saving this part of data；
Each backup point increases, modifies or deletes this part of data according to respective ordering rule.
Wherein, when full backup point increases this part of data newly, number that will be newly-increased according to the keyword Key of newly-increased data
According to being added to suitable position and save, or update original data；
When the incremental backup point increases this part of data newly, it is standby to judge whether the Key of newly-increased data belongs to the part
The range of the Key of part point stored data, if it is, the suitable position of newly-increased data is searched according to the Key of newly-increased data, it will
Newly-increased data are added to suitable position and save, or update original data.
When each backup point is modified or deleted to this part of data, according to respective ordering rule and modification or deletion
The data are modified or are deleted by the data that the Key of data finds request modification or deletes；
For incremental backup point, if the data volume deleted is more than preset threshold value, from the full backup
Point extracts new data and preservation.
Above-mentioned access layer is also used to, when receiving the read requests for a data, according to the read requests
The read requests are forwarded to corresponding backup point by ordering rule；
The corresponding backup point is by the data feedback of reading to access layer.
To sum up, the present invention saves same part data in multiple backup points by the way of space for time,
Each backup point is ranked up data according to respective ordering rule, and the mistake of data is write due to having advanceed to the pressure of sequence
Cheng Zhong, therefore the reading performance of system is largely improved, improve the response speed of system.Also, the present invention proposes
System can with fast construction rise a set of new sorting data, to meet the needs of extraneous.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the present invention.
Claims (8)
1. a kind of date storage method, which is characterized in that the method be used for from extraneous newly-increased request, modification request,
Removal request or read requests are handled, comprising:
Multiple backup points are set for same part data, each backup point arranges this part of data according to respective ordering rule
Sequence simultaneously saves；Wherein, at least one backup point is full backup point, saves the full content of this part of data；Other backup points are
Incremental backup point, saves the partial content of this part of data, wherein the ordering rule of each backup point is different；
It, will be described when access layer receives the newly-increased request for this part of data, modification request, removal request or read requests
Newly-increased request, modification request or removal request are sent to each backup point for saving this part of data；
After each backup point receives request, according to respective ordering rule, this part of data is increased newly, modified or deleted
It removes；
Wherein, for the newly-increased request, each backup point adds the newly-increased data of request according to respective ordering rule
Enter to suitable position and save, or updates original data；In full backup point, conjunction is inserted data into according to its Key
It sets right position；In backup point, if the Key of newly-increased data belongs to the range of the Key of the incremental backup point stored data, press
Newly-increased data are added to suitable position and saved by the suitable position that newly-increased data are searched according to the Key of newly-increased data, or
Update original data；
It is requested for the modification, each backup point is found according to the Key of respective ordering rule and the data of modification asks
The data for asking modification modify to the data；
For the removal request, each backup point is found according to the Key of respective ordering rule and the data of deletion is asked
The data for asking deletion delete the data；On incremental backup point, if deleting data volume is more than preset threshold value
50%, then it needs to pull new data and preservation from full backup point, fills the incremental backup point；
When receiving read requests for data, the ordering rule of the read requests is determined, according to the read requests
Ordering rule the read requests are forwarded to the backup point of identical ordering rule；The backup point is by the data feedback of reading
To access layer.
2. the method according to claim 1, wherein in described the step of being handled for newly-increased request,
If the sequence Key of newly-increased data is more than the maximum or minimum k ey of existing sorting data, without insertion；If originally number
According to according to permutation with positive order, then compared with maximum key and the key of newly-increased data；If data originally according to reversing,
It is compared with minimum k ey and the key of newly-increased data.
3. method according to claim 1 or 2, which is characterized in that when searching data, for the incremental backup point, by
It is less in data volume, general binary chop can be used, is quickly searched with realizing；For the full backup point, due to
Data volume is huger, can be using the mechanism for allocating data segment in advance.
4. according to the method described in claim 3, it is characterized in that, the mechanism for allocating data segment in advance includes: using use
Family id splits data into N number of section as segmentation basis；Using id%N as segment number, sort again by the id of user in each section,
When queried, segment number is determined first with user id%N, then do binary chop in section again, find user.
5. a kind of data-storage system, which is characterized in that the system be used for from extraneous newly-increased request, modification request,
Removal request or read requests are handled, and the system comprises access layers and multiple backup points；Wherein,
For same part data, each backup point is ranked up and saves to this part of data according to respective ordering rule；Wherein,
At least one backup point is full backup point, saves the full content of this part of data；Other backup points are incremental backup point, are saved
The partial content of this part of data, wherein the ordering rule of each backup point is different；
The access layer is also used to, and is asked when receiving the newly-increased request for this part of data, modification request, removal request or reading
When asking,
The newly-increased request, modification request or removal request are sent to each backup point for saving this part of data；
After each backup point receives request, according to respective ordering rule, this part of data is increased newly, modified or deleted
It removes；
Wherein, for the newly-increased request, each backup point adds the newly-increased data of request according to respective ordering rule
Enter to suitable position and save, or updates original data；In full backup point, conjunction is inserted data into according to its Key
It sets right position；In backup point, if the Key of newly-increased data belongs to the range of the Key of the incremental backup point stored data, press
Newly-increased data are added to suitable position and saved by the suitable position that newly-increased data are searched according to the Key of newly-increased data, or
Update original data；
It is requested for the modification, each backup point is found according to the Key of respective ordering rule and the data of modification asks
The data for asking modification modify to the data；
For the removal request, each backup point is found according to the Key of respective ordering rule and the data of deletion is asked
The data for asking deletion delete the data；On incremental backup point, if deleting data volume is more than preset threshold value
50%, then it needs to pull new data and preservation from full backup point, fills the incremental backup point；
When receiving read requests for data, the ordering rule of the read requests is determined, according to the read requests
Ordering rule the read requests are forwarded to the backup point of identical ordering rule；
The backup point is by the data feedback of reading to access layer.
6. system according to claim 5, which is characterized in that each backup point is executing described be directed at newly-increased request
In the step of reason, if the sequence Key of newly-increased data is more than the maximum or minimum k ey of existing sorting data, without insertion；
If originally data are compared according to permutation with positive order with maximum key and the key of newly-increased data；If data originally according to
It reverses, is then compared with minimum k ey and the key of newly-increased data.
7. system according to claim 5 or 6, which is characterized in that when the incremental backup point searches data, due to data
Measure it is less, can use general binary chop, with realize quickly search；The full backup point, since data volume is huger
It greatly, can be using the mechanism for allocating data segment in advance.
8. system according to claim 7, which is characterized in that the mechanism for allocating data segment in advance includes: using use
Family id splits data into N number of section as segmentation basis；Using id%N as segment number, sort again by the id of user in each section,
When queried, segment number is determined first with user id%N, then do binary chop in section again, find user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210455442.6A CN103810209B (en) | 2012-11-14 | 2012-11-14 | A kind of method and system saving data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210455442.6A CN103810209B (en) | 2012-11-14 | 2012-11-14 | A kind of method and system saving data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103810209A CN103810209A (en) | 2014-05-21 |
| CN103810209B true CN103810209B (en) | 2019-05-10 |
Family
ID=50706992
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210455442.6A Active CN103810209B (en) | 2012-11-14 | 2012-11-14 | A kind of method and system saving data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103810209B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021161B (en) * | 2014-05-27 | 2018-06-15 | 华为技术有限公司 | A kind of clustering storage method and device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158540A (en) * | 2011-02-18 | 2011-08-17 | 广州从兴电子开发有限公司 | System and method for realizing distributed database |
-
2012
- 2012-11-14 CN CN201210455442.6A patent/CN103810209B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158540A (en) * | 2011-02-18 | 2011-08-17 | 广州从兴电子开发有限公司 | System and method for realizing distributed database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103810209A (en) | 2014-05-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11288282B2 (en) | Distributed database systems and methods with pluggable storage engines | |
| CN104794123B (en) | A kind of method and device building NoSQL database indexes for semi-structured data | |
| CN110168523B (en) | Change monitoring cross-graph query | |
| EP2924594A1 (en) | Data encoding and corresponding data structure in a column-store database | |
| US20150143065A1 (en) | Data Processing Method and Apparatus, and Shared Storage Device | |
| JP2005267600A5 (en) | ||
| CN107704202B (en) | Method and device for quickly reading and writing data | |
| US20180144061A1 (en) | Edge store designs for graph databases | |
| CN106599091B (en) | RDF graph structure storage and index method based on key value storage | |
| CN110888837B (en) | Object storage small file merging method and device | |
| JP6642651B2 (en) | Storage method using user access preference model | |
| CN108509437A (en) | A kind of ElasticSearch inquiries accelerated method | |
| US20140229496A1 (en) | Information processing device, information processing method, and computer program product | |
| CN105468644B (en) | Method and equipment for querying in database | |
| US11868328B2 (en) | Multi-record index structure for key-value stores | |
| CN107092667A (en) | Group's lookup method and device based on social networks | |
| CN107305570B (en) | Data retrieval method and system | |
| KR101640733B1 (en) | System for Managing data based In-Memory DataBase and method thereof | |
| CN111026709A (en) | Data processing method and device based on cluster access | |
| US9275091B2 (en) | Database management device and database management method | |
| CN110362590A (en) | Data managing method, device, system, electronic equipment and computer-readable medium | |
| CN103810209B (en) | A kind of method and system saving data | |
| CN107291875B (en) | Metadata organization management method and system based on metadata graph | |
| CN108509438A (en) | A kind of ElasticSearch fragments extended method | |
| Li et al. | Cluster-based distributed dynamic cuckoo filter system for Redis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20190808 Address after: 518057 Nanshan District science and technology zone, Guangdong, Zhejiang Province, science and technology in the Tencent Building on the 1st floor of the 35 layer Co-patentee after: Tencent cloud computing (Beijing) limited liability company Patentee after: Tencent Technology (Shenzhen) Co., Ltd. Address before: Shenzhen Futian District City, Guangdong province 518044 Zhenxing Road, SEG Science Park 2 East Room 403 Patentee before: Tencent Technology (Shenzhen) Co., Ltd. |