CN108363813A - Date storage method, device and system - Google Patents
Date storage method, device and system Download PDFInfo
- Publication number
- CN108363813A CN108363813A CN201810214998.3A CN201810214998A CN108363813A CN 108363813 A CN108363813 A CN 108363813A CN 201810214998 A CN201810214998 A CN 201810214998A CN 108363813 A CN108363813 A CN 108363813A
- Authority
- CN
- China
- Prior art keywords
- data
- data record
- record
- distributed
- base system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
This application provides a kind of date storage method, device and system, this method includes：It is stored in distributed data base system and records corresponding timestamp per data；The data record of cold data is determined to be according to current time and timestamp；The data record being determined as cold data is migrated from distributed data base system to database middleware cluster and is stored.The dilatation cost and maintenance cost for reducing distributed data base system, prevent distributed data base system framework more and more huger and complicated.
Description
Technical field
This application involves database technologys, and in particular to a kind of date storage method, device and system.
Background technology
Distributed data base system (Distributed DataBase System, DDBS) includes distributed data depositary management
Reason system (Distributed Database Management System, DDBMS) and distributed data base
(Distributed DataBase, DDB).
In DDBS, an application program can carry out DDB transparent operation, and the data in DDB are respectively in different offices
Store in portion's database, be managed by different DDBMS, being run on different machines, being supported by different operating system,
It is linked together by different communication networks, writing traffic distribution to multiple fragment data libraries cluster, for solving reading and writing flow
The performance issue that excessively high single machine cluster can not carry, while the function of reading and writing flow separation being provided and reads flow automatic load
Balanced function.
DBProxy is a set of middleware software for providing relevant database read and write abruption function, is handed over using MySQL
Mutual agreement, to upstream layer transparent.The SQL statement read and write abruption of offer relevant database level, load balancing, connection
Pond, high concurrent connection, session status holding, security control, node survival detection, the high functions such as fault-tolerant.
If pressing 1 to cold data and dsc data with DDBS:1 scheme detaches, and is required for increasing again when splitting dilatation every time
The equipment to double so that the framework of DDBS becomes increasingly complex, and the cost of each dilatation also can be higher and higher.
Invention content
The embodiment of the present application provides a kind of date storage method, device and system, is expanded with solving DDBS in the prior art
The excessively high technical problem of Rongcheng sheet.
According to the one side of the embodiment of the present application, a kind of date storage method is provided, the method includes：It is being distributed
It is stored in formula Database Systems and records corresponding timestamp per data；It is determined to be according to current time and the timestamp cold
The data record of data；Described be determined is migrated from distributed data base system to database as the data record of cold data
Middleware cluster is stored.
In one embodiment, described be determined is migrated as the data record of cold data from distributed data base system
Carrying out storage to database middleware cluster includes：
Inquiry is determined the data record as cold data from the distributed data base system；Call the distribution
The data record inquired is sent to the database middleware cluster by Database Systems, by the database middleware cluster
Compression storage is carried out to the data record inquired.
In one embodiment, data-recording status table is also stored in the distributed data base system, for preserving number
According to the type and state of record, the starting type of data record is dsc data, and original state is not migrate；It is judged to described
Surely it migrates to database middleware cluster store from distributed data base system as the data record of cold data and further includes：
Call the distributed data base system that the type of the data record is revised as cold data, status modifier is to move
In shifting.
Preferably, the method further includes：In the data record migration is completed to the database middleware cluster
Afterwards, call the distributed data base system by the status modifier of the data record be have moved.
In one embodiment, call the distributed data base system that the data record inquired is sent to the number
According to library middleware cluster, carrying out compression storage to the data record inquired by the database middleware cluster includes：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by described
DBProxy clusters carry out compression storage to the data record inquired by TokuDB storage engines.
In one embodiment, the method further includes：
The operation requests for target data record are received, the operation requests include the target data record corresponding time
Stamp；According to the corresponding timestamp of the target data record, the target data is inquired from the distributed data base system
The type and state of record；If the type of the target data record is cold data, and state is to have moved, then will be described
Operation requests are distributed to the database middleware cluster and are handled；If the type of the target data record is cold number
According to, and state is that the operation requests are then distributed to the distributed data base system and are handled, are pressed simultaneously in migration
According to receiving operation requests described in the sequence of event of operation requests；It is modified in the state of the target data record
After migration, the operation requests of record are distributed to the database middleware cluster and are handled according to the time sequencing；
If the type of the target data record is dsc data, the operation requests are distributed to the distributed data base system
It is handled.
Preferably, the operation requests are distributed to the database middleware cluster and handle and include：
It will be distributed to the database middleware cluster for the update of target data record request；From the data
The data after the decompression of the target data record are obtained in the middleware cluster of library；Target data record after decompression is delayed
It deposits to the distributed data base system；The distributed data base system is called to be updated the target data record,
And the updated target data record is sent to the database middleware cluster and is stored.
Preferably, the operation requests are distributed to the database middleware cluster and handle and include：
It will be distributed to the database middleware cluster for the inquiry request of the target data record；Call the number
According to processing is unziped it to the target data record in the middleware cluster of library, from the database between part cluster obtain solution
Compressed target data record.
In one embodiment, the timestamp is creation time or the renewal time of data record.
Preferably, when the timestamp is the renewal time of data record, the method further includes：In the database
After the data record preserved in middleware cluster is updated, the data record is determined to be dsc data and repaiies timestamp
It is changed to the time that update is completed；Call the database middleware cluster that the data record is sent to the distributed data
Library system is stored.
According to the other side of the embodiment of the present application, a kind of data storage device is provided, including：
Timestamp memory module records corresponding timestamp for being stored in distributed data base system per data；
Cold data determination module, the data record for being determined to be cold data according to current time and the timestamp；Cold data is moved
Shifting formwork block, for being migrated described be determined to database from distributed data base system as the data record of cold data
Part cluster is stored.
In one embodiment, the cold data transferring module includes：
Query unit, for inquiring the data record being determined as cold data from the distributed data base system；
First call unit, for calling the distributed data base system that the data record inquired is sent in the database
Between part cluster, compression storage is carried out to the data record inquired by the database middleware cluster.
In one embodiment, data-recording status table is also stored in the distributed data base system, for preserving number
According to the type and state of record, the starting type of data record is dsc data, and original state is not migrate；The cold data
Transferring module is further used for calling the distributed data base system that the type of the data record is revised as cold data, shape
State is revised as in migration.
Preferably, after the data record completes migration to the database middleware cluster, the cold data is moved
It by the status modifier of the data record is to have moved that shifting formwork block, which is further used for calling the distributed data base system,.
In one embodiment, first call unit is further used for：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by described
DBProxy clusters carry out compression storage to the data record inquired by TokuDB storage engines.
In one embodiment, described device further includes：
Operation requests receiving module, for receiving the operation requests for target data record, the operation requests include
The corresponding timestamp of target data record；State-query module, for according to the corresponding timestamp of the target data record, from
The type and state of the target data record are inquired in the distributed data base system；First distribution module, if for
The type of the target data record is cold data, and state is to have moved, then the operation requests is distributed to the number
It is handled according to library middleware cluster；Second distribution module, if the type for the target data record is cold data, and
And state is that the operation requests are then distributed to the distributed data base system and are handled, while according to connecing in migration
Receive operation requests described in the sequence of event of operation requests；It is modified to have moved in the state of the target data record
Later, the operation requests of record the database middleware cluster is distributed to handle according to the time sequencing；Third
The operation requests are distributed to described point by distribution module if the type for the target data record is dsc data
Cloth Database Systems are handled.
Preferably, first distribution module or the second distribution module include：
First Dispatching Unit, for the update request for being directed to the target data record to be distributed among the database
Part cluster；Data capture unit, the decompression for obtaining the target data record in part cluster between from the database
Data afterwards；Data buffer storage unit, for caching the target data record after decompression to the distributed data base system；
Second call unit calls the distributed data base system to be updated the target data record, and by the update
Target data record afterwards is sent to the database middleware cluster and is stored.
Preferably, first distribution module or the second distribution module include：
Second Dispatching Unit, for the inquiry request for being directed to the target data record to be distributed among the database
Part cluster；Third call unit decompresses the target data record for calling in the database middleware cluster
Contracting handle, from the database between part cluster obtain decompression after target data record.
In one embodiment, the timestamp is creation time or the renewal time of data record.
Preferably, when the timestamp is the renewal time of data record, described device further includes：
Timestamp modified module, after the data record for being preserved in part cluster between in the database is updated,
The data record is determined to be dsc data and timestamp is revised as the time that update is completed；Dsc data transferring module is used
Be sent to after calling the database middleware cluster to decompress the data record distributed data base system into
Row storage.
According to the other side of the embodiment of the present application, a kind of data-storage system is provided, including：Distributed data base
System records timestamp corresponding with data record for storing data；Server, for according to current time and the time
Become the data record of cold data in stamp judgement distributed data base system；Database middleware cluster, for storing from described
The data record of distributed data base system migration being determined as cold data.
According to the other side of the embodiment of the present application, a kind of electronic equipment, including memory and processor are provided；It deposits
Reservoir is for storing one or more computer instruction, wherein one or more computer is executed by the processor to realize
Above-mentioned date storage method.
According to the other side of the embodiment of the present application, a kind of computer readable storage medium is provided, is stored thereon with
Above-mentioned date storage method is realized in computer instruction, the instruction when being executed by processor.
The advantageous effect of the embodiment of the present application includes：The data record for being determined as cold data in DDBS is migrated to number
Compression storage is carried out according to library middleware cluster, the dilatation cost and maintenance cost of DDBS can be effectively reduced, prevent DDBS frameworks from getting over
Come huger and complicated.
Description of the drawings
By referring to the drawings to the description of the embodiment of the present application, the above-mentioned and other purpose of the application, feature and
Advantage will be apparent from, in the accompanying drawings：
Fig. 1 is the configuration diagram of the embodiment of the present application data-storage system；
Fig. 2 a are server apparatus quantity schematic diagrames when current DDBS carries out dilatation；
Fig. 2 b are the server apparatus quantity schematic diagrames of the embodiment of the present application；
Fig. 3 is the flow diagram of date storage method provided by the embodiments of the present application；
Fig. 4 is the block diagram of data storage device provided by the embodiments of the present application
Fig. 5 is the structural schematic diagram of electronic equipment provided by the embodiments of the present application.
Specific implementation mode
The application is described below based on embodiment, but the application is not restricted to these embodiments.Under
Text is detailed to describe some specific detail sections in the datail description of the application.Do not have for a person skilled in the art
The description of these detail sections can also understand the application completely.In order to avoid obscuring the essence of the application, well known method, mistake
There is no narrations in detail for journey, flow, element and circuit.
In addition, it should be understood by one skilled in the art that provided herein attached drawing be provided to explanation purpose, and
What attached drawing was not necessarily drawn to scale.
Unless the context clearly requires otherwise, "include", "comprise" otherwise throughout the specification and claims etc. are similar
Word should be construed as the meaning for including rather than exclusive or exhaustive meaning；That is, being containing for " including but not limited to "
Justice.
In the description of the present application, it is to be understood that term " first ", " second " etc. are used for description purposes only, without
It can be interpreted as indicating or implying relative importance.In addition, in the description of the present application, unless otherwise indicated, the meaning of " multiple "
It is two or more.
Data record and its timestamp are corresponded to and are preserved by the DDBS of the embodiment of the present application, so as to according to current time and time
Stamp, which determines the data record as cold data and migrated to database middleware cluster, carries out compression storage, can effectively subtract
The dilatation cost of few DDBS, prevents DDBS frameworks more and more huger and complicated.
Fig. 1 is the configuration diagram of the data-storage system of the embodiment of the present application, which includes in DDBS 1, database
Between part cluster 2 and server 3, wherein server 3 is used to handle the global data request of DDBS1 and database middleware cluster 2.
DDBS 1 can be system R, for example, MySQL, PostpreSQL, Oracle etc..
DDBS 1 includes host node 11 and memory node 12, and host node 11 and memory node 12 can also configure standby node
(not shown), tables of data is distributed to be stored in each memory node 12, and host node 11 records the storage of each tables of data
Path.Tables of data includes a plurality of data record (record), and per data, record includes multiple fields.
The timestamp of the also corresponding storage records per data of DDBS 1.One field of data record can be exclusively used in remembering
Its corresponding timestamp is recorded, can also be per the corresponding timestamp of data record storage, the number by data-recording status table
It can be stored in host node 11 according to recording status table, distributed can also be stored in each memory node 12.In data record
The type for recording corresponding timestamp per data and being recorded per data is stored in state table.
Timestamp can be creation time or the renewal time of data record the last time of data record.Data
The type of record includes cold data and dsc data, and the differentiation between cold data and dsc data with the timestamp of data record and is worked as
The preceding time, the starting type of all data records was dsc data as decision condition.If current time range data records
Timestamp be more than pre-determined threshold (for example, be more than 72 hours), then judge that the data record becomes cold data, then by its type from
Dsc data is revised as cold data.Judge data record whether become cold data, be over time and it is ongoing.
For example, data record a is determined at the t1 moment as cold data, its type is revised as cold data, is then migrated to data
Library middleware cluster 2；T2 moment after two hours, data record b also comply with the decision condition of cold data, then resulting number
Become cold data according to record b, its type is revised as cold data, then migrates data record b to database middleware cluster
2。
Server 3 moves to database middleware cluster 2 by the data record for becoming cold data is determined from DDBS 1.Number
Can be a kind of single machine data-base cluster, memory node 22 of the middleware 21 as server 3 and cluster according to library middleware cluster 2
Between Proxy program, be responsible for detection and analysis and filter request and memory node 22 load balancing and read and write abruption
Etc. functions, avoid delay machine caused by relevant database single point failure and pressure abruptly increase from causing database service is not available to ask
Topic.The function that single machine data-base cluster can also provide single node write-in, multinode is read.
Database middleware cluster 2 reduces the memory space of cold data occupancy using compression storage engines, to reduce
The cost of dilatation is carried out to DDBS 1.By taking DDBS 1 is MySQL database as an example, database middleware cluster 2 can be used
The data storage engines such as InnoDB, TokuDB.In the embodiment of the present application, it is preferred to use TokuDB storage engines can provide
Highest 25:1 compression ratio, and can create, be inserted into online, deleting index, " row " are added online, are extended, are deleted,
Renaming, is more suitable for filing the not high data of access frequency or historical data and (is determined in the embodiment of the present application
For the data record of cold data).TokuDB storage engines can use TokuDB_ZLIB to the data compression algorithm of data record
Compression algorithm, can provide higher compression ratio, the suitable cpu resource of consumption, and TokuDB_ZLIB compression algorithms can provide
16:1 compression ratio, i.e. space occupancy rate are reduced to original (i.e. MySQL database) 1/16th.
From the perspective of saving system architecture cost, as shown in Figure 2 a, if by way of carrying out dilatation to DDBS
Storage is judged as the data record of cold data, needs more server resources.By taking 8 current fragment DDBS clusters as an example,
Each fragment cluster is 10 servers, and 8 fragments just need 8 × 10=80 platform server resources, if using cold data and heat
Data fragmentation 1:1 scheme detaches, it is necessary to configure 80 server resources to store cold data.When DDBS clusters continue to expand
Rong Shi is calculated according to the number of servers for increasing by 1 times, and cold data, the director server quantity of dsc data fragment cluster need after dilatation
It to be further added by one times from 80+80=160 platforms and rises to 320.Dilatation is split every time to DDBS, machine quantity can all rise to original
2 times come.
But if using the TokuDB_ZLIB algorithms of TokuDB storage engines to being judged as the data record of cold data
According to 16:After 1 compression ratio is compressed, the space occupied resource becomes original 1/16.Relative to complicated DDBS
Distributed structure/architecture, for cold data, DBProxy frameworks may be used in database middleware cluster, and 8 fragments are corresponding cold
Data focus on inside 1 cluster.As shown in Figure 2 b, it is assumed that originally DDBS is used to store the storage of 80 servers of cold data
Resource is all occupied, then for 10 servers of each fragment, using TokuDB storage engines according to 16:1 pressure
Contracting is than after being compressed, the server resource of occupancy is 10/16, for all 8 fragments, total clothes for being occupied after compression
Business device resource is 8 × 10/16=5 platforms.As it can be seen that the number of servers that uses of storage cold data is from original 8 fragments (totally 80
Server) it is reduced to 1 fragment (totally 10 servers actually only take up 5 therein).Simultaneously as DBProxy clusters are not
It needs to be SQL and splits parsing, SQL fragments router-level topology and SQL fragment query results and summarize, CPU and the memory source ratio of consumption
DDBS is much smaller, and the response performance of SQL request can also be promoted than DDBS, due to less, the lighter frame using number of servers
Structure realizes that corresponding O＆M cost also reduced and saved.
It, can will when will be determined the data record as cold data and move to database middleware cluster 2 from DDBS 1
The data record migrates immediately, can also will be judged as cold data within the period every predetermined period (such as 24 hours)
Data record is migrated.The process of migration is：(select) is inquired first from DDBS 1 is determined the number as cold data
According to record, the data record that type is modified to cold data can be inquired by data-recording status table, call DDBS 1 will be right
The data record answered is sent to database middleware cluster 2.Then, call database middleware cluster 2 by these data records
Compression processing is carried out, and compressed data record is written to the memory node 22 of (insert) cluster.After being written successfully,
The data record that Delete query arrives in DDBS 1, to discharge its occupied storage resource.
In the embodiment of the present application, all SQL for increasing data record, deleting, update, inquiring that server 3 receives are asked
It asks and is required for including corresponding timestamp, corresponding data record is determined by the title of timestamp and library, table and row together.
It, can be with other than storing and recording corresponding timestamp and type per data in data-recording status table
The further state of storage record per data.The state of data record includes not migrating, migrating to neutralize and have moved.Remember in data
It records in state table, the original state of all data records is not migrate.It is further that data are remembered in data-recording status table
Record setting state parameter, is that will be distributed to DDBS 1 for the SQL request of cold data for the ease of judgement or be distributed to data
Library middleware cluster 2.It is above-mentioned the data record for being determined to be cold data is migrated during, inquired in DDBS 1
When being modified to the data record of cold data to type, the state of the data record is never migrated and is revised as in migration；It should
Data record is compressed and is written after database middleware cluster 2, and the state of the data record is revised as having moved from migration
It moves, represents the data record to migrate to database middleware cluster 2.
After server 3 receives the operation requests for data record, according in operation requests timestamp and library and
Title or library, the title of table and row of table, determine the target data record of the operation requests, the data note stored from DDBS 1
The type and state of the target data record are inquired in record state table.If the type of target data record is cold data, and
State is to have moved, then the operation requests are distributed to database middleware cluster 2 and handled by server 3.If number of targets
Type according to record is cold data, and state is in migration, then the operation requests are distributed at DDBS 1 by server 3
Reason, while according to each operation requests of the sequence of event for receiving operation requests；It is repaiied in the state of target data record
It is changed to after having moved, the operation requests of record are distributed to database middleware cluster 2 by server 3, by database middleware
Cluster 2 handles the operation requests of record successively sequentially in time, to make the target data record in transition process
It is updated in part cluster 2 between the variation in the database of middle generation.If the type of target data record is dsc data, take
Operation requests are distributed to DDBS 1 and handled by business device 3.
For being determined for the data record for becoming cold data, it is migrated among database for the purpose of backup
Part cluster 2, therefore database middleware cluster 2 is mainly that inquiry request and update are asked to the processing of these data records, it is real
It is not in the removal request for these data records in border.Database middleware cluster 2 is looked into for cold data below
The processing procedure for asking request and update request illustrates respectively.
After server 3 receives the inquiry request for cold data, which is distributed to database middleware collection
Group 2.Database middleware cluster 2 is according to the title or library of library and table, the name query data library middleware cluster of table and row
2 index lists determine storage location, read the data record compressed.Database middleware cluster 2 remembers the data compressed
Record unzips it processing, and the data record after decompression is back to server 3.
After server 3 receives the update request for cold data, update request is distributed to database middleware cluster
2.Database middleware cluster 2 is according to the title or library of library and table, the name query data library middleware cluster 2 of table and row
Index list determines storage location, reads the data record compressed.Database middleware cluster 2 remembers the data compressed
Record unzips it processing, and the data record after decompression is back to server 3.Server 3 remembers the data after decompression
Record and update request caching are updated the data record by DDBS 1 to DDBS 1.Server 3 calls DDBS 1 will more
Data record after new is migrated to database middleware cluster 2, by database middleware cluster 2 to updated data record into
Row compression storage.
When the timestamp of the embodiment of the present application is that data are recorded in the creation time in DDBS 1, the record per data
Timestamp is constant.Over time, it is cold data that per data, the type of record can all change from dsc data, and
It is migrated to database middleware cluster 2.For example, when current time differs 72 hours or more with the creation time of data record,
Then the data record is migrated to database middleware cluster 2.
In another embodiment, timestamp can be additionally configured to the renewal time of data record, at this time per data
The timestamp of record is that there may be variations.Type is after the data record of dsc data is updated, in data-recording status
Also to complete the newer time, type will continue to remain dsc data timestamp in table for variation therewith.Type is cold data
Data record be updated after, the timestamp in data-recording status table changes to complete newer time, type therewith
It is revised as dsc data from cold data.At this point, server 3 migrates back updated cold data from database middleware cluster 2
DDBS 1 is stored.Database middleware cluster 2 unzips it processing to updated cold data, by the data record of decompression
It is sent to the host node 11 of DDBS 1, the data record of decompression is written to memory node 12 by host node 11.By the data
After record migrates back DDBS 1, in the data-recording status table that host node 11 is safeguarded, by the state of the data record from
Migration is revised as not migrating.So far, the type of the data record is dsc data, and state is not migrate, and timestamp is in database
The newer time is completed in middleware cluster 2.When it meets the decision condition of cold data, can be migrated again into database
Between part cluster 2 carry out compression storage.
Based on described above, the embodiment of the present application provides a kind of date storage method, is suitable for server 3, such as Fig. 3 institutes
Show, this approach includes the following steps.
S31 is stored in DDBS and is recorded corresponding timestamp per data.
S32 is determined to be the data record of cold data according to current time and the timestamp；
S33, will be determined the data record as cold data and be migrated from DDBS to database middleware cluster and store.
In one embodiment, S33 includes：
S301 inquires the data record being determined as cold data from DDBS；
S302 calls DDBS that the data record inquired is sent to database middleware cluster, by database middleware
Cluster carries out compression storage to the data record inquired.
In one embodiment, also store data-recording status table in the DDBS, for preserve data record type and
The starting type of state, data record is dsc data, and original state is not migrate；S33 further includes：Call DDBS by data
The type of record is revised as cold data, and status modifier is in migration.
Preferably, this method further includes：After data record completes migration to the database middleware cluster, call
The status modifier of the data record is to have moved by the DDBS.
In one embodiment, S302 includes：Call DDBS that the data record inquired is sent to DBProxy clusters,
Compression storage is carried out by TokuDB storage engines to the data record inquired by DBProxy clusters.
In one embodiment, this method further includes：
S34 receives the operation requests for target data record, and operation requests include the target data record corresponding time
Stamp；
S35 inquires the type and shape of target data record according to the corresponding timestamp of target data record from DDBS
State；
S361, if the type of target data record is cold data, and state is to have moved, then by the operation requests
The database middleware cluster is distributed to be handled；
S362, if the type of target data record is cold data, and state is then to distribute operation requests in migration
To DDBS processing, while according to the sequence of event operation requests for receiving operation requests；In target data record
State is modified to after having moved, and the operation requests of record are distributed to database middleware cluster to be carried out sequentially in time
Processing；
Operation requests are distributed to DDBS processing by S363 if the type of target data record is dsc data.
Preferably, operation requests are distributed to database middleware cluster and handle and include：
It will be distributed to database middleware cluster for the update of target data record request；From database middleware cluster
Data after the middle decompression for obtaining target data record；Target data record after decompression is cached to DDBS；It calls
DDBS is updated target data record, and updated target data record is sent to database middleware cluster and is carried out
Storage.
Preferably, operation requests are distributed to database middleware cluster and handle and include：
It will be distributed to database middleware cluster for the inquiry request of target data record；Call database middleware collection
Processing is unziped it to target data record in group, the target data note after decompression is obtained from database middleware cluster
Record.
In one embodiment, timestamp is creation time or the renewal time of data record.
Preferably, when timestamp is the renewal time of data record, this method further includes：
S37, in the database between after the data record that preserves in part cluster is updated, which is determined to be
Timestamp is simultaneously revised as the time that update is completed by dsc data；
S38 calls database middleware cluster that data record is sent to DDBS and stores.
The embodiment of the present application also provides a kind of data storage devices, as shown in figure 4, the device includes：
Timestamp memory module 41 records corresponding timestamp for being stored in DDBS per data；
Cold data determination module 42, the data for being determined to be cold data according to current time and the timestamp are remembered
Record；
Cold data transferring module 43, for being migrated the data record for being determined as cold data into database from DDBS
Between part cluster stored.
In one embodiment, cold data transferring module 43 includes：
Query unit, for inquiring the data record being determined as cold data from DDBS；
First call unit, for calling DDBS that the data record inquired is sent to database middleware cluster, by
Database middleware cluster carries out compression storage to the data record inquired.
In one embodiment, data-recording status table is also stored in DDBS, the type for preserving data record and shape
The starting type of state, data record is dsc data, and original state is not migrate；Cold data transferring module 43 is further used for
Call DDBS that the type of data record is revised as cold data, status modifier is in migration.
Preferably, after data record completes migration to the database middleware cluster, cold data transferring module 43
It is to have moved to be further used for calling DDBS by the status modifier of data record.
In one embodiment, the first call unit is further used for：DDBS is called to send the data record inquired
To DBProxy clusters, compression storage is carried out by TokuDB storage engines to the data record inquired by DBProxy clusters.
In one embodiment, which further includes：
Operation requests receiving module, for receiving the operation requests for target data record, operation requests include target
The corresponding timestamp of data record；
State-query module, for according to the corresponding timestamp of target data record, target data note to be inquired from DDBS
The type and state of record；
First distribution module, if the type for target data record is cold data, and state is to have moved, then will
Operation requests are distributed to database middleware cluster and are handled；
Second distribution module, if the type for target data record is cold data, and state is then will in migration
Operation requests are distributed to DDBS processing, while according to the sequence of event operation requests for receiving operation requests；In mesh
Mark data record state be modified to after having moved, by the operation requests of record be distributed to database middleware cluster according to
Time sequencing is handled；
Operation requests are distributed to by third distribution module if the type for target data record is dsc data
DDBS processing.
Preferably, the first distribution module or the second distribution module include：
First Dispatching Unit, for the update request for being directed to target data record to be distributed to database middleware cluster；
Data capture unit is used for from the number after the decompression for obtaining target data record in database middleware cluster
According to；
Data buffer storage unit, for caching the target data record after decompression to DDBS；
Second call unit calls DDBS to be updated target data record, and by updated target data record
Database middleware cluster is sent to be stored.
Preferably, the first distribution module or the second distribution module include：
Second Dispatching Unit, for the inquiry request for being directed to target data record to be distributed to database middleware cluster；
Third call unit unzips it processing for calling in database middleware cluster to target data record,
The target data record after decompression is obtained from database middleware cluster.
In one embodiment, timestamp is creation time or the renewal time of data record.
Preferably, when timestamp is the renewal time of data record, which further includes：
Timestamp modified module, after the data record for being preserved in part cluster between in the database is updated, by this
Data record is determined to be dsc data and timestamp is revised as the time that update is completed；
Dsc data transferring module, for be sent to after calling database middleware cluster to decompress data record DDBS into
Row storage.
In addition, the data storage device can also be above-mentioned to realize by hardware processor (hardware processor)
Each program step.Below with reference to Fig. 5, it illustrates the departments of computer science suitable for the electronic equipment for realizing the embodiment of the present application
The structural schematic diagram of system 600.Electronic equipment shown in Fig. 5 is only an example, should not to the function of the embodiment of the present application and
Use scope brings any restrictions.
As shown in figure 5, computer system 600 includes central processing unit (CPU) 601, it can be read-only according to being stored in
Program in memory (ROM) 602 or be loaded into the program in random access storage device (RAM) 603 from storage section 608 and
Execute various actions appropriate and processing.In RAM 603, also it is stored with system 600 and operates required various programs and data.
CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always
Line 604.
It is connected to I/O interfaces 605 with lower component：Importation 606 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 608 including hard disk etc.；
And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net executes communication process.Driver 610 is also according to needing to be connected to I/O interfaces 606.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 610, as needed in order to be read from thereon
Computer program be mounted into storage section 608 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed by communications portion 609 from network, and/or from detachable media
611 are mounted.When the computer program is executed by central processing unit (CPU) 601, limited in execution the present processes
Above-mentioned function.It should be noted that computer-readable medium described herein can be computer-readable signal media or
Computer readable storage medium either the two arbitrarily combines.Computer readable storage medium for example can be --- but
Be not limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or arbitrary above combination.
The more specific example of computer readable storage medium can include but is not limited to：Electrical connection with one or more conducting wires,
Portable computer diskette, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only deposit
Reservoir (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory
Part or above-mentioned any appropriate combination.In this application, computer readable storage medium can any be included or store
The tangible medium of program, the program can be commanded the either device use or in connection of execution system, device.And
In the application, computer-readable signal media may include the data letter propagated in a base band or as a carrier wave part
Number, wherein carrying computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including but not
It is limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer
Any computer-readable medium other than readable storage medium storing program for executing, the computer-readable medium can send, propagate or transmit use
In by instruction execution system, device either device use or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to：Wirelessly, electric wire, optical cable, RF etc., Huo Zheshang
Any appropriate combination stated.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part for a part for one module, program segment, or code of table, the module, program segment, or code includes one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it wants
It is noted that the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart, Ke Yiyong
The dedicated hardware based system of defined functions or operations is executed to realize, or can be referred to specialized hardware and computer
The combination of order is realized.
Being described in module involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be arranged in the processor, for example, can be described as：A kind of processor packet
Include timestamp memory module, cold data determination module and cold data transferring module.
As on the other hand, present invention also provides a kind of nonvolatile computer storage media, the non-volatile calculating
Machine storage medium can be nonvolatile computer storage media included in device described in above-described embodiment；Can also be
Individualism, without the nonvolatile computer storage media in supplying electronic equipment.Above-mentioned non-volatile computer storage is situated between
Matter is stored with one or more program, when one or more program stored is executed by an equipment so that described
Equipment：It is stored in DDBS and records corresponding timestamp per data；It is determined to be cold data according to current time and timestamp
Data record；It the data record as cold data will be determined will be migrated from DDBS to database middleware cluster and stored.
The foregoing is merely the preferred embodiments of the application, are not intended to limit this application, for those skilled in the art
For, the application can have various modifications and changes.It is all within spirit herein and principle made by any modification, equivalent
Replace, improve etc., it should be included within the protection domain of the application.
A1, a kind of date storage method, the method includes：
It is stored in distributed data base system and records corresponding timestamp per data；According to current time and it is described when
Between stamp be determined to be the data record of cold data；By the data record being determined as cold data from distributed data base system
System, which is migrated to database middleware cluster, to be stored.
A2, the method according to a1, by the data record being determined as cold data from distributed data base system
System migrates to database middleware cluster store：
Inquiry is determined the data record as cold data from the distributed data base system；Call the distribution
The data record inquired is sent to the database middleware cluster by Database Systems, by the database middleware cluster
Compression storage is carried out to the data record inquired.
A3, the method according to a1 or a2 also store data-recording status table in the distributed data base system, use
In the type and state that preserve data record, the starting type of data record is dsc data, and original state is not migrate；
Described be determined is migrated from distributed data base system to database middleware as the data record of cold data
Cluster carries out storage：
Call the distributed data base system that the type of the data record is revised as cold data, status modifier is to move
In shifting.
A4, the method according to a3, the method further include：In the data record to the database middleware collection
Group complete migration after, call the distributed data base system by the status modifier of the data record be have moved.
A5, the method according to a2, call the distributed data base system to be sent to the data record inquired
The database middleware cluster carries out compression storage packet by the database middleware cluster to the data record inquired
It includes：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by described
DBProxy clusters carry out compression storage to the data record inquired by TokuDB storage engines.
A6, the method according to a1, the method further include：
The operation requests for target data record are received, the operation requests include the target data record corresponding time
Stamp；According to the corresponding timestamp of the target data record, the target data is inquired from the distributed data base system
The type and state of record；If the type of the target data record is cold data, and state is to have moved, then will be described
Operation requests are distributed to the database middleware cluster and are handled；If the type of the target data record is cold number
According to, and state is that the operation requests are then distributed to the distributed data base system and are handled, are pressed simultaneously in migration
According to receiving operation requests described in the sequence of event of operation requests；It is modified in the state of the target data record
After migration, the operation requests of record are distributed to the database middleware cluster and are handled according to the time sequencing；
If the type of the target data record is dsc data, the operation requests are distributed to the distributed data base system
It is handled.
The operation requests are distributed to the database middleware cluster and handled by a7, the method according to a6
Including：
It will be distributed to the database middleware cluster for the update of target data record request；From the data
The data after the decompression of the target data record are obtained in the middleware cluster of library；Target data record after decompression is delayed
It deposits to the distributed data base system；The distributed data base system is called to be updated the target data record,
And the updated target data record is sent to the database middleware cluster and is stored.
The operation requests are distributed to the database middleware cluster and handled by a8, the method according to a6
Including：
It will be distributed to the database middleware cluster for the inquiry request of the target data record；Call the number
According to processing is unziped it to the target data record in the middleware cluster of library, from the database between part cluster obtain solution
Compressed target data record.
A9, the method according to a1, the timestamp are creation time or the renewal time of data record.
A10, the method according to a9, when the timestamp is the renewal time of data record, the method is also wrapped
It includes：
After the data record preserved in part cluster between in the database is updated, by the data record judgement at
It is revised as the time that update is completed for dsc data and by timestamp；Call the database middleware cluster by the data record
The distributed data base system is sent to be stored.
B11, a kind of data storage device, including：
Timestamp memory module records corresponding timestamp for being stored in distributed data base system per data；
Cold data determination module, the data record for being determined to be cold data according to current time and the timestamp；Cold data is moved
Shifting formwork block, for being migrated described be determined to database from distributed data base system as the data record of cold data
Part cluster is stored.
B12, the device according to b11, the cold data transferring module include：
Query unit, for inquiring the data record being determined as cold data from the distributed data base system；
First call unit, for calling the distributed data base system that the data record inquired is sent in the database
Between part cluster, compression storage is carried out to the data record inquired by the database middleware cluster.
B13, the device according to b11 or b12 also store data-recording status in the distributed data base system
The starting type of table, type and state for preserving data record, data record is dsc data, and original state is not move
It moves；
The cold data transferring module is further used for calling the distributed data base system by the data record
Type is revised as cold data, and status modifier is in migration.
B14, the device according to b13 complete migration in the data record to the database middleware cluster
Afterwards, the cold data transferring module is further used for the status maintenance for calling the distributed data base system by the data record
It is changed to have moved.
B15, the device according to b12, first call unit are further used for：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by described
DBProxy clusters carry out compression storage to the data record inquired by TokuDB storage engines.
B16, the device according to b11, described device further include：
Operation requests receiving module, for receiving the operation requests for target data record, the operation requests include
The corresponding timestamp of target data record；State-query module, for according to the corresponding timestamp of the target data record, from
The type and state of the target data record are inquired in the distributed data base system；First distribution module, if for
The type of the target data record is cold data, and state is to have moved, then the operation requests is distributed to the number
It is handled according to library middleware cluster；Second distribution module, if the type for the target data record is cold data, and
And state is that the operation requests are then distributed to the distributed data base system and are handled, while according to connecing in migration
Receive operation requests described in the sequence of event of operation requests；It is modified to have moved in the state of the target data record
Later, the operation requests of record the database middleware cluster is distributed to handle according to the time sequencing；Third
The operation requests are distributed to described point by distribution module if the type for the target data record is dsc data
Cloth Database Systems are handled.
B17, the device according to b16, first distribution module or the second distribution module include：
First Dispatching Unit, for the update request for being directed to the target data record to be distributed among the database
Part cluster；Data capture unit, the decompression for obtaining the target data record in part cluster between from the database
Data afterwards；Data buffer storage unit, for caching the target data record after decompression to the distributed data base system；
Second call unit calls the distributed data base system to be updated the target data record, and by the update
Target data record afterwards is sent to the database middleware cluster and is stored.
B18, the device according to b16, first distribution module or the second distribution module include：
Second Dispatching Unit, for the inquiry request for being directed to the target data record to be distributed among the database
Part cluster；Third call unit decompresses the target data record for calling in the database middleware cluster
Contracting handle, from the database between part cluster obtain decompression after target data record.
B19, the device according to b11, the timestamp are creation time or the renewal time of data record.
B20, the device according to b19, when the timestamp is the renewal time of data record, described device is also wrapped
It includes：
Timestamp modified module, after the data record for being preserved in part cluster between in the database is updated,
The data record is determined to be dsc data and timestamp is revised as the time that update is completed；Dsc data transferring module is used
Be sent to after calling the database middleware cluster to decompress the data record distributed data base system into
Row storage.
C21, a kind of data-storage system, including：
Distributed data base system records timestamp corresponding with data record for storing data；Server is used for root
According to the data record for becoming cold data in current time and timestamp judgement distributed data base system；Database middleware
Cluster, for storing the data record being determined as cold data migrated from the distributed data base system.
D22, a kind of electronic equipment, including memory and processor；The memory is calculated for storing one or more
Machine instructs, wherein one or more computer is executed by the processor to realize following steps：
It is stored in distributed data base system and records corresponding timestamp per data；According to current time and it is described when
Between stamp be determined to be the data record of cold data；By the data record being determined as cold data from distributed data base system
System, which is migrated to database middleware cluster, to be stored.
D23, the electronic equipment according to d22, by the data record being determined as cold data from distributed number
Carrying out storage according to library system migration to database middleware cluster includes：
Inquiry is determined the data record as cold data from the distributed data base system；Call the distribution
The data record inquired is sent to the database middleware cluster by Database Systems, by the database middleware cluster
Compression storage is carried out to the data record inquired.
D24, the electronic equipment according to d22 or d23 also store data record shape in the distributed data base system
The starting type of state table, type and state for preserving data record, data record is dsc data, and original state is not
Migration；
Described be determined is migrated from distributed data base system to database middleware as the data record of cold data
Cluster carries out storage：Call the distributed data base system that the type of the data record is revised as cold data,
Status modifier is in migration.
D25, the electronic equipment according to d24, the processor are further configured execution：The data record to
After the database middleware cluster completes migration, call the distributed data base system by the state of the data record
It is revised as having moved.
D26, the electronic equipment according to d23, the data record for calling the distributed data base system that will inquire
It is sent to the database middleware cluster, carrying out compression to the data record inquired by the database middleware cluster deposits
Storage includes：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by described
DBProxy clusters carry out compression storage to the data record inquired by TokuDB storage engines.
D27, the electronic equipment according to d22, the processor are further configured execution：It receives and is directed to target data
The operation requests of record, the operation requests include the corresponding timestamp of target data record；According to the target data record
Corresponding timestamp inquires the type and state of the target data record from the distributed data base system；If institute
The type for stating target data record is cold data, and state is to have moved, then the operation requests is distributed to the data
Library middleware cluster is handled；If the type of the target data record is cold data, and state is then will in migration
The operation requests are distributed to the distributed data base system and are handled, while suitable according to the time for receiving operation requests
Sequence records the operation requests；After the state of the target data record is modified to have moved, the operation of record is asked
It asks and is distributed to the database middleware cluster and is handled according to the time sequencing；If the class of the target data record
Type is dsc data, then the operation requests are distributed to the distributed data base system is handled.
D28, the electronic equipment according to d27, by the operation requests be distributed to the database middleware cluster into
Row is handled：
It will be distributed to the database middleware cluster for the update of target data record request；From the data
The data after the decompression of the target data record are obtained in the middleware cluster of library；Target data record after decompression is delayed
It deposits to the distributed data base system；The distributed data base system is called to be updated the target data record,
And the updated target data record is sent to the database middleware cluster and is stored.
D29, the electronic equipment according to d27, by the operation requests be distributed to the database middleware cluster into
Row is handled：
It will be distributed to the database middleware cluster for the inquiry request of the target data record；Call the number
According to processing is unziped it to the target data record in the middleware cluster of library, from the database between part cluster obtain solution
Compressed target data record.
D30, the electronic equipment according to d22, the timestamp are creation time or the renewal time of data record.
D31, the electronic equipment according to d30, when the timestamp is the renewal time of data record, the processing
Device is further configured execution：
After the data record preserved in part cluster between in the database is updated, by the data record judgement at
It is revised as the time that update is completed for dsc data and by timestamp；Call the database middleware cluster by the data record
The distributed data base system is sent to be stored.
E32, a kind of computer readable storage medium, are stored thereon with computer instruction, when which is executed by processor
Realize following steps：
It is stored in distributed data base system and records corresponding timestamp per data；According to current time and it is described when
Between stamp be determined to be the data record of cold data；By the data record being determined as cold data from distributed data base system
System, which is migrated to database middleware cluster, to be stored.
E33, the computer readable storage medium according to e32, by the data record being determined as cold data
It migrates to database middleware cluster store from distributed data base system and includes：
Inquiry is determined the data record as cold data from the distributed data base system；Call the distribution
The data record inquired is sent to the database middleware cluster by Database Systems, by the database middleware cluster
Compression storage is carried out to the data record inquired.
E34, the computer readable storage medium according to e32 or e33 also store in the distributed data base system
The starting type of data-recording status table, type and state for preserving data record, data record is dsc data, initially
State is not migrate；
Described be determined is migrated from distributed data base system to database middleware as the data record of cold data
Cluster carries out storage：
Call the distributed data base system that the type of the data record is revised as cold data, status modifier is to move
In shifting.
E35, the computer readable storage medium according to e34, the processor are further configured execution：Described
After data record completes migration to the database middleware cluster, call the distributed data base system by the data
The status modifier of record is to have moved.
E36, the computer readable storage medium according to e33, call the distributed data base system that will inquire
Data record be sent to the database middleware cluster, by the database middleware cluster to the data record that inquires
Carrying out compression storage includes：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by described
DBProxy clusters carry out compression storage to the data record inquired by TokuDB storage engines.
E37, the computer readable storage medium according to e32, the processor are further configured execution：
The operation requests for target data record are received, the operation requests include the target data record corresponding time
Stamp；According to the corresponding timestamp of the target data record, the target data is inquired from the distributed data base system
The type and state of record；If the type of the target data record is cold data, and state is to have moved, then will be described
Operation requests are distributed to the database middleware cluster and are handled；If the type of the target data record is cold number
According to, and state is that the operation requests are then distributed to the distributed data base system and are handled, are pressed simultaneously in migration
According to receiving operation requests described in the sequence of event of operation requests；It is modified in the state of the target data record
After migration, the operation requests of record are distributed to the database middleware cluster and are handled according to the time sequencing；
If the type of the target data record is dsc data, the operation requests are distributed to the distributed data base system
It is handled.
E38, the computer readable storage medium according to e37, the operation requests are distributed in the database
Between part cluster carry out processing include：
It will be distributed to the database middleware cluster for the update of target data record request；From the data
The data after the decompression of the target data record are obtained in the middleware cluster of library；Target data record after decompression is delayed
It deposits to the distributed data base system；The distributed data base system is called to be updated the target data record,
And the updated target data record is sent to the database middleware cluster and is stored.
E39, the computer readable storage medium according to e37, the operation requests are distributed in the database
Between part cluster carry out processing include：
It will be distributed to the database middleware cluster for the inquiry request of the target data record；Call the number
According to processing is unziped it to the target data record in the middleware cluster of library, from the database between part cluster obtain solution
Compressed target data record.
E40, the computer readable storage medium according to e32, the timestamp be data record creation time or
Person's renewal time.
E41, the computer readable storage medium according to e40, when the renewal time that the timestamp is data record
When, the processor is further configured execution：
After the data record preserved in part cluster between in the database is updated, by the data record judgement at
It is revised as the time that update is completed for dsc data and by timestamp；Call the database middleware cluster by the data record
The distributed data base system is sent to be stored.
Claims (10)
1. a kind of date storage method, which is characterized in that the method includes：
It is stored in distributed data base system and records corresponding timestamp per data；
The data record of cold data is determined to be according to current time and the timestamp；
Described be determined is migrated from distributed data base system to database middleware cluster as the data record of cold data
It is stored.
2. according to the method described in claim 1, it is characterized in that, by the data record being determined as cold data from point
Cloth Database Systems migrate to database middleware cluster store：
Inquiry is determined the data record as cold data from the distributed data base system；
Call the distributed data base system that the data record inquired is sent to the database middleware cluster, by institute
It states database middleware cluster and compression storage is carried out to the data record inquired.
3. method according to claim 1 or 2, which is characterized in that also store data in the distributed data base system
The starting type of recording status table, type and state for preserving data record, data record is dsc data, original state
It is not migrate；
Described be determined is migrated from distributed data base system to database middleware cluster as the data record of cold data
Carrying out storage further includes：
Call the distributed data base system that the type of the data record is revised as cold data, status modifier is migration
In.
4. according to the method described in claim 3, it is characterized in that, the method further includes：In the data record to described
After database middleware cluster completes migration, call the distributed data base system by the status modifier of the data record
To have moved.
5. according to the method described in claim 2, it is characterized in that, calling the number that the distributed data base system will inquire
It is sent to the database middleware cluster according to record, the data record inquired is carried out by the database middleware cluster
Compression stores：
Call the distributed data base system that the data record inquired is sent to DBProxy clusters, by the DBProxy
Cluster carries out compression storage to the data record inquired by TokuDB storage engines.
6. according to the method described in claim 1, it is characterized in that, the method further includes：
The operation requests for target data record are received, the operation requests include the corresponding timestamp of target data record；
According to the corresponding timestamp of the target data record, the target data is inquired from the distributed data base system
The type and state of record；
If the type of the target data record is cold data, and state is to have moved, then distributes the operation requests
It is handled to the database middleware cluster；
If the type of the target data record is cold data, and state is then to distribute the operation requests in migration
It is handled to the distributed data base system, while being asked according to operation described in the sequence of event of operation requests is received
It asks；After the state of the target data record is modified to have moved, the operation requests of record are distributed to the data
Library middleware cluster is handled according to the time sequencing；
If the type of the target data record is dsc data, the operation requests are distributed to the distributed data base
System is handled.
7. a kind of data storage device, which is characterized in that including：
Timestamp memory module records corresponding timestamp for being stored in distributed data base system per data；
Cold data determination module, the data record for being determined to be cold data according to current time and the timestamp；
Cold data transferring module, for migrating described be determined from distributed data base system as the data record of cold data
It is stored to database middleware cluster.
8. a kind of data-storage system, which is characterized in that including：
Distributed data base system records timestamp corresponding with data record for storing data；
Server, for according to the data for becoming cold data in current time and timestamp judgement distributed data base system
Record；
Database middleware cluster, for storing the number being determined as cold data migrated from the distributed data base system
According to record.
9. a kind of electronic equipment, which is characterized in that including memory and processor；The memory is for storing one or more
Computer instruction, wherein one or more computer is executed by the processor to realize following steps：
It is stored in distributed data base system and records corresponding timestamp per data；
The data record of cold data is determined to be according to current time and the timestamp；
Described be determined is migrated from distributed data base system to database middleware cluster as the data record of cold data
It is stored.
10. a kind of computer readable storage medium, is stored thereon with computer instruction, which is characterized in that the instruction is by processor
Following steps are realized when execution：
It is stored in distributed data base system and records corresponding timestamp per data；
The data record of cold data is determined to be according to current time and the timestamp；
Described be determined is migrated from distributed data base system to database middleware cluster as the data record of cold data
It is stored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810214998.3A CN108363813B (en) | 2018-03-15 | 2018-03-15 | Data storage method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810214998.3A CN108363813B (en) | 2018-03-15 | 2018-03-15 | Data storage method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108363813A true CN108363813A (en) | 2018-08-03 |
| CN108363813B CN108363813B (en) | 2020-06-02 |
Family
ID=63000345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810214998.3A Active CN108363813B (en) | 2018-03-15 | 2018-03-15 | Data storage method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108363813B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376141A (en) * | 2018-08-31 | 2019-02-22 | 阿里巴巴集团控股有限公司 | A kind of data migration method and device |
| CN110134723A (en) * | 2019-05-22 | 2019-08-16 | 网易（杭州）网络有限公司 | A kind of method and database of storing data |
| CN110377585A (en) * | 2019-07-01 | 2019-10-25 | 北京奇艺世纪科技有限公司 | Distributive type calculates the on-line rapid estimation method, apparatus of application, computer equipment |
| CN110531938A (en) * | 2019-09-02 | 2019-12-03 | 广东紫晶信息存储技术股份有限公司 | A kind of cold and hot data migration method and system based on various dimensions |
| CN110532234A (en) * | 2019-08-30 | 2019-12-03 | 杭州迪普科技股份有限公司 | A kind of storage method and device of vulnerability scanning data |
| CN110798222A (en) * | 2019-09-27 | 2020-02-14 | 北京浪潮数据技术有限公司 | Data compression method and device |
| CN111061357A (en) * | 2019-12-13 | 2020-04-24 | 北京奇艺世纪科技有限公司 | Energy saving method and device, electronic equipment and storage medium |
| CN111339105A (en) * | 2020-03-27 | 2020-06-26 | 泰康保险集团股份有限公司 | Data storage method and device and server |
| CN111414138A (en) * | 2020-03-19 | 2020-07-14 | 青岛镕铭半导体有限公司 | Wear leveling method and device for solid state disk |
| CN111597160A (en) * | 2020-04-21 | 2020-08-28 | 中国人民财产保险股份有限公司 | Distributed database system, distributed data processing method and device |
| CN112988040A (en) * | 2019-12-18 | 2021-06-18 | 深圳大普微电子科技有限公司 | Data storage method, device and equipment and readable storage medium |
| CN113452726A (en) * | 2020-03-24 | 2021-09-28 | 中兴通讯股份有限公司 | Data transmission method, terminal and computer readable storage medium |
| CN113590586A (en) * | 2021-07-29 | 2021-11-02 | 东方微银科技股份有限公司 | Method and device for migrating fragmented data among nodes of distributed graph database system |
| CN117216068A (en) * | 2023-11-07 | 2023-12-12 | 本原数据(北京)信息技术有限公司 | Data storage method, device and storage medium supporting database mixed load |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102053962A (en) * | 2009-11-02 | 2011-05-11 | 清华大学深圳研究生院 | Lightweight middleware-based networking RFID system and data interaction method |
| CN105069109A (en) * | 2015-08-07 | 2015-11-18 | 北京思特奇信息技术股份有限公司 | Dilatation method and system of distributed database |
| US20170011074A1 (en) * | 2015-07-10 | 2017-01-12 | Ab Initio Technology Llc | Distributed database system |
| CN106341454A (en) * | 2016-08-23 | 2017-01-18 | 世纪龙信息网络有限责任公司 | Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method |
| CN106503079A (en) * | 2016-10-10 | 2017-03-15 | 语联网（武汉）信息技术有限公司 | A kind of blog management method and system |
| CN106570074A (en) * | 2016-10-14 | 2017-04-19 | 深圳前海微众银行股份有限公司 | Distributed database system and implementation method thereof |
| CN106934001A (en) * | 2017-03-03 | 2017-07-07 | 广州天源迪科信息技术有限公司 | Distributed quick inventory inquiry system and method |
| CN107193500A (en) * | 2017-05-26 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of distributed file system Bedding storage method and system |
| CN108052664A (en) * | 2017-12-29 | 2018-05-18 | 北京小度信息科技有限公司 | The data migration method and device of database purchase cluster |
-
2018
- 2018-03-15 CN CN201810214998.3A patent/CN108363813B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102053962A (en) * | 2009-11-02 | 2011-05-11 | 清华大学深圳研究生院 | Lightweight middleware-based networking RFID system and data interaction method |
| US20170011074A1 (en) * | 2015-07-10 | 2017-01-12 | Ab Initio Technology Llc | Distributed database system |
| CN105069109A (en) * | 2015-08-07 | 2015-11-18 | 北京思特奇信息技术股份有限公司 | Dilatation method and system of distributed database |
| CN106341454A (en) * | 2016-08-23 | 2017-01-18 | 世纪龙信息网络有限责任公司 | Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method |
| CN106503079A (en) * | 2016-10-10 | 2017-03-15 | 语联网（武汉）信息技术有限公司 | A kind of blog management method and system |
| CN106570074A (en) * | 2016-10-14 | 2017-04-19 | 深圳前海微众银行股份有限公司 | Distributed database system and implementation method thereof |
| CN106934001A (en) * | 2017-03-03 | 2017-07-07 | 广州天源迪科信息技术有限公司 | Distributed quick inventory inquiry system and method |
| CN107193500A (en) * | 2017-05-26 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of distributed file system Bedding storage method and system |
| CN108052664A (en) * | 2017-12-29 | 2018-05-18 | 北京小度信息科技有限公司 | The data migration method and device of database purchase cluster |
Non-Patent Citations (1)
| Title |
|---|
| 张耀华: "基于MYSQL的分布式数据中间层", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376141A (en) * | 2018-08-31 | 2019-02-22 | 阿里巴巴集团控股有限公司 | A kind of data migration method and device |
| CN110134723A (en) * | 2019-05-22 | 2019-08-16 | 网易（杭州）网络有限公司 | A kind of method and database of storing data |
| CN110377585B (en) * | 2019-07-01 | 2022-02-11 | 北京奇艺世纪科技有限公司 | Online capacity expansion method and device for distributed streaming computing application and computer equipment |
| CN110377585A (en) * | 2019-07-01 | 2019-10-25 | 北京奇艺世纪科技有限公司 | Distributive type calculates the on-line rapid estimation method, apparatus of application, computer equipment |
| CN110532234A (en) * | 2019-08-30 | 2019-12-03 | 杭州迪普科技股份有限公司 | A kind of storage method and device of vulnerability scanning data |
| CN110531938A (en) * | 2019-09-02 | 2019-12-03 | 广东紫晶信息存储技术股份有限公司 | A kind of cold and hot data migration method and system based on various dimensions |
| CN110798222B (en) * | 2019-09-27 | 2022-04-22 | 北京浪潮数据技术有限公司 | Data compression method and device |
| CN110798222A (en) * | 2019-09-27 | 2020-02-14 | 北京浪潮数据技术有限公司 | Data compression method and device |
| CN111061357A (en) * | 2019-12-13 | 2020-04-24 | 北京奇艺世纪科技有限公司 | Energy saving method and device, electronic equipment and storage medium |
| CN112988040A (en) * | 2019-12-18 | 2021-06-18 | 深圳大普微电子科技有限公司 | Data storage method, device and equipment and readable storage medium |
| CN112988040B (en) * | 2019-12-18 | 2023-02-24 | 深圳大普微电子科技有限公司 | Data storage method, device and equipment and readable storage medium |
| CN111414138A (en) * | 2020-03-19 | 2020-07-14 | 青岛镕铭半导体有限公司 | Wear leveling method and device for solid state disk |
| CN111414138B (en) * | 2020-03-19 | 2024-02-13 | 镕铭微电子(济南)有限公司 | Wear balancing method and device for solid state disk |
| CN113452726A (en) * | 2020-03-24 | 2021-09-28 | 中兴通讯股份有限公司 | Data transmission method, terminal and computer readable storage medium |
| CN113452726B (en) * | 2020-03-24 | 2024-02-20 | 中兴通讯股份有限公司 | Data transmission method, terminal and computer readable storage medium |
| CN111339105A (en) * | 2020-03-27 | 2020-06-26 | 泰康保险集团股份有限公司 | Data storage method and device and server |
| CN111339105B (en) * | 2020-03-27 | 2023-08-15 | 泰康保险集团股份有限公司 | Data storage method, device and server |
| CN111597160A (en) * | 2020-04-21 | 2020-08-28 | 中国人民财产保险股份有限公司 | Distributed database system, distributed data processing method and device |
| CN113590586A (en) * | 2021-07-29 | 2021-11-02 | 东方微银科技股份有限公司 | Method and device for migrating fragmented data among nodes of distributed graph database system |
| CN113590586B (en) * | 2021-07-29 | 2022-03-22 | 东方微银科技股份有限公司 | Method and device for migrating fragmented data among nodes of distributed graph database system |
| CN117216068A (en) * | 2023-11-07 | 2023-12-12 | 本原数据(北京)信息技术有限公司 | Data storage method, device and storage medium supporting database mixed load |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108363813B (en) | 2020-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108363813A (en) | Date storage method, device and system | |
| US11288267B2 (en) | Pluggable storage system for distributed file systems | |
| CN109254733B (en) | Method, device and system for storing data | |
| US20230090977A1 (en) | Synchronized content library | |
| US9218141B2 (en) | Managing write operations to an extent of tracks migrated between storage devices | |
| US7043479B2 (en) | Remote-directed management of media content | |
| CN105677250B (en) | The update method and updating device of object data in object storage system | |
| US11294958B2 (en) | Managing a distributed knowledge graph | |
| CN108629029B (en) | Data processing method and device applied to data warehouse | |
| US9348517B2 (en) | Using a migration threshold and a candidate list for cache management of sequential write storage | |
| CN104808952B (en) | data cache method and device | |
| US8131671B2 (en) | Uninterrupted data access during the migration of data between physical file systems | |
| CN111104063A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN110109868A (en) | Method, apparatus and computer program product for index file | |
| US20170161050A1 (en) | Methods for Downloading and Installing Computer Software Applications on Demand | |
| JP2011191835A (en) | Computer system and method of executing application program | |
| CN109086462A (en) | The management method of metadata in a kind of distributed file system | |
| US10789234B2 (en) | Method and apparatus for storing data | |
| US11429311B1 (en) | Method and system for managing requests in a distributed system | |
| CN105051731A (en) | Grouping files for optimized file operations | |
| CN110196683A (en) | A kind of method, apparatus, terminal device and the medium of operation control | |
| EP4030309A1 (en) | Management computer and data management method by management computer | |
| US20170161056A1 (en) | Methods for Managing the Writing of Datasets by Computer-Implemented Processes | |
| CN113127438B (en) | Method, apparatus, server and medium for storing data | |
| JP2013088920A (en) | Computer system and data management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100085 Beijing, Haidian District on the road to the information on the ground floor of the 1 to the 3 floor of the 2 floor, room 11, 202 Applicant after: Beijing Xingxuan Technology Co., Ltd. Address before: 100085 Beijing, Haidian District on the road to the information on the ground floor of the 1 to the 3 floor of the 2 floor, room 11, 202 Applicant before: BEIJING XIAODU INFORMATION TECHNOLOGY CO., LTD. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |