WO2021057253A1

WO2021057253A1 - Data separation and storage method and apparatus, computer device and storage medium

Info

Publication number: WO2021057253A1
Application number: PCT/CN2020/105962
Authority: WO
Inventors: 杨念磊; 叶国华; 司孝波; 杨盈; 李钥珑
Original assignee: 苏宁云计算有限公司
Priority date: 2019-09-27
Filing date: 2020-07-30
Publication date: 2021-04-01
Also published as: CN110795427A; CA3156068A1; CN110795427B

Abstract

A data separation and storage method and apparatus, a computer device and a storage medium. The method comprises: obtaining each piece of user data of a target user (S202); respectively determining a period in which the generation time of each piece of user data of the target user is (S204); and separating and storing these pieces of user data of the target user into a plurality of databases according to the period of each generation time (S206). By using the method, cold and hot data can be separated, the calculation approach can be simplified, and the hysteresis of separation and storage of newly generated data can be avoided.

Description

Data separation storage method, device, computer equipment and storage medium

Technical field

This application relates to the field of data storage technology, and in particular, to a data separation storage method, device, computer equipment, and storage medium.

Background technique

With the rapid development of computer technology and Internet technology, users can easily access the Internet and interact with servers on the Internet to obtain various services provided by the servers. The server can provide services for multiple different users, and for each user, save the user data of the user. For some users of some servers, the number of pieces of user data tends to be large, making the total amount of data too large, and data with a large amount of data often needs to be split.

The traditional data splitting scheme of user data is to split the database and tables according to the user dimension, and split the user data of all users into multiple databases. When the data grows to the limited capacity of the database, the existing database needs to be split into two. For example, the A database is split into the A1 database and the A2 database, and then the redundant data in the A1 and A2 databases are deleted. However, all user data of the same user still exists together, and the existence of a large amount of cold data will cause the storage and query of hot data to increase time-consuming, and affect the use of hot data. Therefore, effective processing and storage of hot and cold data is becoming more and more important.

The traditional method of separating hot and cold data is based on the heat value of the data. The calculation of the heat value needs to consider many factors such as the visit start time, the visit end time, the amount of visits, and the length of unit time. Not only is the calculation method more complicated, but also due to The amount of visits needed is only suitable for the separation of historical data with visits. It is not suitable for the direct separation of newly generated data. The separation and storage of newly generated data has a lag.

Summary of the invention

Based on this, it is necessary to address the above technical problems to provide a data separation storage method, device, computer equipment, and storage medium that can realize cold and hot data separation, simplify calculation methods, and avoid the hysteresis of newly generated data separation and storage.

A data separation storage method, the method includes: acquiring each piece of user data of a target user; respectively determining the generation time period of each piece of user data of the target user; according to the period of each generation time, dividing each piece of the target user Pieces of user data are stored separately in multiple databases.

In one embodiment, the above method further includes: requesting to allocate a current period database, a historical database, a first database, and a second database to the target user; Separately stored in multiple databases, including: according to the period of each generation time, separate and store each piece of user data of the target user into the current period database, the historical database, the first database and the second database; among them, the current period database It is used to store the user data whose generation time of the target user is in the current cycle, the historical database is used to store the user data whose generation time of the target user is in the historical period, and the first database is used to store the generation time of the target user. The processing period is the user data of the current period and the previous period, and the second database is used to store the user data of the current period and the next period in which the generation time of the target user is located.

In one embodiment, the aforementioned pieces of user data of the target user include newly generated data of the target user and pieces of historical data; the aforementioned pieces of user data of the target user are stored separately and stored in the current period according to the period of each generation time. The period database, the historical database, the first database and the second database include: storing the newly generated data of the target user in the current period database, the first database and the second database respectively; according to the generation of each piece of historical data of the target user When the time is in the cycle, each piece of historical data of the target user is stored separately in the current cycle database, the historical database and the first database.

In one embodiment, the above-mentioned storing the newly generated data of the target user in the current period database, the first database and the second database respectively includes: writing the newly generated data of the target user into the current period database; The newly generated data of the user is written into the operation log generated by the current periodic database, and the newly generated data of the target user is written into the first database and the second database in an asynchronous manner.

In one embodiment, the above method further includes: before entering the next period, requesting to allocate a new current period database and a new second database for the target user; after entering the next period, assigning the target user corresponding to the current period The data in the current cycle database of the target user is migrated to the historical database of the target user, and the second database corresponding to the current cycle of the target user is used as the new first database of the target user; after the data migration is completed, the current cycle corresponding to the target user is destroyed The current period database of the target user, and the first database corresponding to the current period of the target user.

In an embodiment, the above method further includes: segmenting the user data according to the user dimension, and the user data of each segmented user after the segmentation processing is used as each piece of user data of the target user.

In an embodiment, the above method further includes:

When receiving the user data modification request of the target user, query the generation period information of the user data to be modified; when the period determined according to the queried generation period information is the current period, route the user data modification request to the current period corresponding to the target user The database performs data modification; when the period determined according to the queried generation period information is a historical period, the user data modification request is routed to the historical database corresponding to the target user for data modification;

Or/and

When a user data query request from a target user is received, the query type is determined according to the query conditions carried in the query request; when the query type is a periodical query, if the query period determined according to the query conditions is the current period, the user data query request Route to the current period database corresponding to the target user for data query. If the query period determined according to the query conditions is a historical period, then the user data query request is routed to the historical database corresponding to the target user for data query; when the query type is query by time period At the time, the start time and end time corresponding to the query period determined according to the query conditions; according to the start time and end time, determine the target database of the target user that matches the start time and end time, and route the user data query request to the target The database performs data query.

A data separation storage device, which includes:

The data acquisition module is used to acquire various pieces of user data of the target user;

The period determination module is used to respectively determine the period in which the generation time of each piece of user data of the target user is located;

The separate storage module is used to separate and store each piece of user data of the target user into multiple databases according to the cycle of each generation time.

A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor implements the following steps when the processor executes the computer program:

Obtain various pieces of user data of the target user;

Respectively determine the period in which each piece of user data of the target user is generated;

According to the cycle of each generation time, each piece of user data of the target user is stored separately in multiple databases.

A computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

Obtain various pieces of user data of the target user;

The above-mentioned data separation storage method, device, computer equipment and storage medium are to obtain each piece of user data of the target user, respectively determine the period in which each piece of user data of the target user is generated, and according to the period in which the generation time is located, Each piece of user data of the target user is stored separately in multiple databases. In the embodiment of the present application, since each piece of user data of the target user is stored separately based on the period of the generation time, it is only necessary to determine the corresponding period according to the generation time of each piece of user data, which simplifies the calculation method. At the same time, the newly generated data can also be separated and stored directly according to the embodiment of the application to avoid the hysteresis of the separation and storage of the newly generated data. Moreover, because each piece of user data of the target user is stored separately according to the period of each generation time In multiple different databases, the physical separation of hot and cold data can be achieved.

Description of the drawings

Figure 1 is an application environment diagram of a data separation storage method in an embodiment;

FIG. 2 is a schematic flowchart of a method for data separation and storage in an embodiment;

FIG. 3 is a detailed flowchart of the data separation and storage step in an embodiment;

FIG. 4 is a detailed flow diagram of a step of storing newly generated data in an embodiment;

FIG. 5 is a schematic flowchart of a data separation and storage step in another embodiment;

6 is an example diagram of data separation and storage for two consecutive weeks in an embodiment;

FIG. 7 is a structural block diagram of a data separation storage device in an embodiment;

Fig. 8 is an internal structure diagram of a computer device in an embodiment.

detailed description

In order to make the purpose, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

It can be understood that the terms "first", "second", etc. used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first client may be referred to as the second client, and similarly, the second client may be referred to as the first client. Both the first client and the second client are clients, but they are not the same client. In addition, it can be understood that the term "or/and" used in this application describes the association relationship of associated objects, indicating that there can be three types of relationships, for example, A or/and B can mean that A alone exists, and both A and A exist at the same time. B, there are three cases of B alone. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

The following takes order data storage as an example to illustrate the problems existing in the prior art. The traditional order data splitting scheme is that the order data is divided into database and table according to the member dimension, and the order data of all members is split into multiple (for example, 32) databases. When the data grows to the limited capacity of the database, it is required Split the existing database into two. However, the data for all time periods of the same member still exists together. However, 90% of the user's access to the data basically falls in the hot data of the most recent period of time (such as the last 1-2 years), and the existence of a large amount of cold data will cause As a result, the storage and query time of hotspot data increases, which affects the use of hotspot data.

The data separation storage method provided in this application can be applied to the application environment as shown in FIG. 1. The application environment includes a terminal 102, a network 104, a server 106, and a database device 108. The terminal 102, the server 106 and the database device 108 can all be connected through the network 104 for communication. The network system composed of the terminal 102, the network 104, the server 106, and the database device 108 may be based on the Internet, or a local area network, or a combined network of the Internet and a local area network, which will not be repeated here.

The terminal 102 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. At the same time, FIG. 1 shows a case where the number of terminals 102 is two, but this application does not limit the number of terminals 102, and it may also include more or less terminals. An application program runs on the terminal 102, and the application program may be a shopping application, a communication application, or other applications. The database device 108 includes multiple databases or database clusters.

The network 104 is used to implement the network connection between the terminal 102 and the server 106, the server 106 and the database device 108, etc., and may include multiple types of wired or wireless networks. The network 104 may include the Internet, a local area network ("LAN"), a wide area network ("WAN"), an intranet, a mobile phone network, a virtual private network (VPN), a cellular or other mobile communication network, Bluetooth, NFC, or any combination thereof . The data transmission on the network 104 is also based on the corresponding communication protocol. For example, when the web browser receives the corresponding service code of the webpage, it will be based on the HTTP communication protocol, and the service code received by the mobile application may be based on the Websocket communication protocol.

The server 106 may be implemented as an independent server or a server cluster composed of multiple servers. During the running process, the server 106 can obtain each piece of user data of the target user, respectively determine the period of the generation time of each piece of user data of the target user, and according to the period of each generation time, the target user Each piece of user data is stored separately in multiple databases. In this way, the cold and hot data can be separated.

In one embodiment, as shown in FIG. 2, a data separation storage method is provided. Taking the method applied to the server in FIG. 1 as an example for description, the method includes the following steps:

Step 202: Obtain various pieces of user data of the target user.

Here, the target users may be users of the sales platform, specifically all users of the sales platform, or some users of the sales platform, for example, a part of users selected from all users of the sales platform. The target user can also be other types of platform users. For sales platform users, user data may include order data. The number of target users can be one or more.

Specifically, the server may obtain various pieces of user data of the target user from historical data and current data. Here, the current data may be newly generated data or data generated in the current cycle.

Step 204, respectively determine the period in which the generation time of each piece of user data of the target user is located;

Here, the period in which the generation time is located refers to the period in which the generation time is located. The division method of the cycle can be set according to the actual situation, for example, divided by year, divided by month, or divided by day, etc. For the division by year, it can be a cycle per year, or it can be a cycle per multiple years, by month Division or division by day is equivalent.

Specifically, the server may query the generation time of each piece of user data of the target user, and determine the period of the generation time of each piece of user data of the target user according to the generation time of each piece of user data that is queried. Wherein, the accuracy of the generation time that is queried can be determined according to the period division method, for example, if it is divided by year, only the generation year information can be queried.

Step 206: Separately store each piece of user data of the target user into multiple databases according to the period of each generation time.

Here, the number of databases can be determined according to actual needs.

Specifically, the server can separate and store each piece of user data of the target user into multiple databases according to preset data separation rules according to the period of each generation time obtained in step 204, where the data separation rules can be based on actual conditions. It needs to be set. The same piece of user data can be stored in only one database or in multiple databases as required.

In the above data separation and storage method, each piece of user data of the target user is obtained, and the period of the generation time of each piece of user data of the target user is determined respectively, and the period of each piece of user data of the target user is determined according to the period of the generation time. Separately stored in multiple databases. In this embodiment, since each piece of user data of the target user is stored separately based on the period of the generation time, it is only necessary to determine the corresponding period according to the generation time of each piece of user data, which simplifies the calculation method, and at the same time For newly generated data, it is also possible to separate and store the newly generated data directly according to the scheme of this embodiment to avoid the hysteresis of the separate storage of newly generated data. Moreover, because each piece of user data of the target user is stored separately according to the period of each generation time In multiple different databases, the physical separation of hot and cold data can be achieved, which can reduce the time-consuming storage and query of hot data, and avoid or reduce the impact of cold data on the use of hot data.

In one embodiment, the data separation storage method of the present application may further include the steps of: requesting to allocate the current period database, historical database, first database, and second database to the target user; the above-mentioned target users are allocated according to the period of each generation time. The step of storing each piece of user data of the user separately into multiple databases may include the step of: according to the period of each generation time, each piece of user data of the target user is separately stored in the current period database, the historical database, the first database and The second database; where the current period database is used to store user data whose target user’s generation time is in the current period, and the historical database is used to store user data whose target user’s generation time is in the historical period. The first The database is used to store the user data of the current period and the previous period in which the generation time of the target user is in the period, and the second database is used to store the user data in the current period and the next period in which the generation time of the target user is in the period.

Among them, the historical period is the period corresponding to the generation time of the first user data of the target user to the previous period. The previous period, the current period and the next period are three periods immediately adjacent in time. Over time, the expression of the same period can also be different. For example, in 2019, 2018, 2019, and 2020 are the previous cycle, the current cycle, and the next cycle, respectively, and in 2020, 2019 is the previous cycle.

Taking the period divided by year and user data as orders as an example, the order data of the same user is distributed in 4 databases: historical order database, current year order database, previous year and current year order database, and next year and The current year order database. Among them, the previous year is the year minus 1 from the current year, and the next year is the year plus 1 from the current year. The historical order database, the current year order database, the previous year and current year order database, and the next year and current year order database are respectively equivalent to the aforementioned historical database, current period database, first database and second database.

In the solution of this embodiment, each piece of user data of the target user is separately stored in the current period database, the historical database, the first database, and the second database, which realizes the isolated storage of hot and cold user data. In addition, since the current cycle database, the historical database and the first database respectively store user data whose generation time is in the current cycle, user data whose generation time is in the historical cycle, and the generation time is in the current cycle. As well as the user data of the previous period, operations on hot data (such as query or storage) can be routed to the current period database or the first database for operations, which can improve the processing efficiency of hot data. At the same time, the current period database, the first database and the second database all store the user data of the current period, and the first database and the second database are also used to store the user data of the previous period and the user data of the next period, respectively. When a cycle change occurs (for example, entering the next cycle), the data in the current cycle database is migrated to the historical database and the identity of the second database is changed, that is, the second database is changed to the first database without discarding this Two databases, that is, when a periodic change occurs, the number of abandoned databases is small, which can improve the efficiency of the change and improve the time continuity of the solution.

In one embodiment, the aforementioned pieces of user data of the target user may include the newly generated data of the target user and various pieces of historical data; as shown in FIG. 3, the aforementioned pieces of user data of the target user are changed according to the period of each generation time. The step of storing the pieces of user data separately in the current period database, the historical database, the first database and the second database may include the following steps:

Step 302: Store the newly generated data of the target user in the current period database, the first database and the second database respectively;

Step 304: Separately store each piece of historical data of the target user in the current period database, the historical database and the first database according to the period in which each piece of historical data of the target user is generated.

In this embodiment, the newly generated data can be directly stored, that is, there is no need to determine the period of the generation time, and the efficiency of data separation and storage can be improved.

In one embodiment, as shown in FIG. 4, the step of storing the newly generated data of the target user in the current period database, the first database, and the second database, respectively, may include the following steps:

Step 402: Write the newly generated data of the target user into the current cycle database;

Step 404: Write the newly generated data of the target user into the operation log generated by the current periodic database, and write the newly generated data of the target user into the first database and the second database in an asynchronous manner.

Specifically, the server can write the newly generated data of the target user into the current cycle database. When writing the newly generated data of the target user into the current cycle database, an operation log is generated. The operation log is generally a binary log (binlog). The operation log can be parsed in an asynchronous manner, and the newly generated data of the target user will be inserted into the first database and the second database according to the result of the analytic processing, so that it does not affect the data processing flow based on the newly generated data, such as user orders The submission process.

In one embodiment, as shown in FIG. 5, a data separation storage method is provided. The method is applied to the server in FIG. 1 as an example for description. As shown in FIG. 5, the data separation storage method in this embodiment The method includes the following steps in addition to the above-mentioned step 202, step 204, and step 206:

Step 502, before entering the next cycle, request to allocate a new current cycle database and a new second database to the target user;

Step 504: After entering the next cycle, migrate the data in the current cycle database of the target user corresponding to the current cycle to the historical database of the target user, and use the second database corresponding to the current cycle of the target user as the new first database of the target user. A database;

Step 506: After the data migration is completed, destroy the current period database corresponding to the current period of the target user and the first database corresponding to the current period of the target user.

The solution of this embodiment has very good time scalability for a database with a rapidly growing data volume, and can be implemented infinitely extended.

The data separation and storage method in an embodiment may further include the step of fragmenting user data according to the user dimension, and the user data of each fragmented user after the fragmentation processing is used as each piece of user data of the target user.

Specifically, the order data of all members is divided into n segments according to the member dimension, where n is a positive integer, and the size can be set according to actual needs. Multiple pieces of user data of the same user are generally in the same shard, that is to say, the order data of all members is divided into multiple groups according to the members, and the user data of each group of users can be used as the pieces of user data of the target user, according to the above implementation In this way, each shard user can correspond to the above-mentioned multiple (four) databases, and the user data of different shard users can be separated and stored in parallel, which can further reduce the storage and storage of hot data. The query is time consuming.

The data separation storage method in one embodiment may further include the steps of: when a user data modification request of the target user is received, query the generation period information of the user data to be modified; when the period determined according to the queried generation period information is the current period When the user data modification request is routed to the current period database corresponding to the target user for data modification; when the period determined according to the queried generation period information is a historical period, the user data modification request is routed to the historical database corresponding to the target user for data modification. Data modification.

Here, the generation period information refers to the period information of the generation time, that is, the period in which the generation time is located. Among them, the user data to be modified can be determined according to the user data modification request. In this embodiment, different user data modification requests are routed to the current period database or historical database for data modification, and the modification of cold and hot data can be processed separately, which can improve processing efficiency.

The data separation storage method in one embodiment may further include the step of: when a user data query request from a target user is received, the query type is determined according to the query conditions carried in the query request; when the query type is a periodic query, if the query is based on the query If the query period determined by the condition is the current period, the user data query request is routed to the current period database corresponding to the target user for data query. If the query period determined according to the query condition is the historical period, the user data query request is routed to the target user Corresponding historical database for data query; when the query type is query by time period, the start time and end time corresponding to the query period determined according to the query conditions; according to the start time and end time, it is determined to match the start time and end time The target database of the target user, the user data query request is routed to the target database for data query.

Wherein, the step of determining the target database of the target user matching the start time and the end time according to the start time and the end time may specifically include the step of detecting whether the start time and the end time both fall within the current target user's Period database or historical database corresponding to the time period; if yes, determine the current period database or historical database as the target database; if not, check whether the start time and end time both fall in the first database of the target user, if yes, Then the first database is determined as the target database; if not, the historical database and the current database are used as the target database. That is to say, query in a single database is preferred, and cross-database query is performed when a single database cannot match the start time and end time, so that cross-database queries can be avoided as much as possible, and the response efficiency to user data query requests can be improved as much as possible . For scenarios that require cross-database queries, the query period can be split into two sub-query periods that match the historical database and the current database. Based on these two sub-query periods, data queries are performed on the corresponding historical database and the current database respectively, and you will get The two query results are aggregated to obtain the final data query result.

In order to facilitate the understanding of the solution of the present invention, the following is an example of applying the data separation storage method in the foregoing embodiment to the separate storage of member order data. The data separation storage method in this embodiment includes the following steps:

Step 1: Divide into n segments according to the member dimension, and split the full amount of order data according to the members of each segment.

Among them, the number of members in each shard can be equal or unequal.

Specifically, all members are first divided into n segments according to the member dimension, and then the full amount of order data is split according to the members of each segment, so that multiple pieces of order data of the same member are in the same segment.

Step 2: For each shard, apply for four databases: history database, now database, before&now database and now&after database.

Among them, the history database refers to the historical order database, the now database refers to the database of orders in the current year, the before&now database refers to the database of current year-1 (that is, the previous year) and the current year's orders, and the now&after database refers to the current year&current year+1 ( That is, the database of orders for the next year.

The history database, now database, before&now database, and now&after database here are respectively equivalent to the history database, current period database, first database, and second database in the foregoing embodiment.

Step 3: Define data storage routing rules, which are used to instruct the service program in the server to read and write the corresponding database;

Among them, the data storage routing rules may include new order storage rules, historical order modification rules, and order query rules.

New order storage rule: the year when the order is placed is the current year, routed to the now database for storage;

Historical order modification rules: query the year of placing the historical order, if it is the current year, route to the now database for modification; if it is a historical year, route to the history database for modification;

Order query rules: If query by year, query the current year, route to the now database query, query the history year, route to the history database query; if the query condition is a time period, that is, the start time to the end time, if the time period corresponds to the now database The time period is routed to the now database for query. If the time period is in the period corresponding to the before&now database, it is routed to the before&now database for query. If the time period is in the period corresponding to the history database, it is routed to the history database for query.

The solution of this embodiment is to consider the externally provided query service, which generally involves requests in the last three months/six months/a certain year, and therefore does not involve a scenario where results are aggregated across two databases. However, cross-database queries can also be performed according to actual needs.

Step 4: Establish an asynchronous association writing mechanism among the now database, before&now database and now&after database;

Specifically, the database binary log analysis component RDRS can be used to write all operations of the now database into the before&now database and the now&after database to ensure the consistency of the current data in the three databases.

For example, as shown in Figure 6, when the current year is 2019, a historical order database, 18+19 year order database, 19 year database, and 19+20 year order data are applied for a sharded user. Historical orders The database is used to store the order data of the corresponding user from the start time to the end of 2018, the 18+19 order database is used to store the order data of the corresponding user in 2018 and 2019, and the 19+20 order database It is used to store the order data of the corresponding user in 2019 and 2020. The order data generated on the terminal application is stored in these four databases according to the year of production. Specifically, the historical order data is stored in the historical order database, the 18+19 year order database or the 19 year database according to the year of production, 2019 The newly generated data of the year is stored in the 19 year database, and is stored in the 18+19 year order database and 19 year order database in an asynchronous manner according to the asynchronous association write mechanism of the 18+19 year order database, the 19 year database and the 19+20 year order database. +20 years of order database.

Step 5: Preparation before the new switching cycle

Before entering the next year, apply for a new now database and a new now&after database in advance, and formulate a new year in the database routing rules to route to the new now database and the new now&after database;

Step 6: File;

Specifically, after entering the next year, archive the now database of the previous year into the history database.

Step 7: Clean up redundant data;

Specifically, the now database and the before&now database of the previous year will be destroyed to complete the cutting.

For example, as shown in Figure 6, before entering 2020, apply for a new now database and a new now&after database for the corresponding user in advance, that is, 20-year order data and 20+21-year order database. After entering 2020, will The data of the 19-year database is archived to the historical order database, and the 18+19-year order database and the 19-year database are destroyed.

It should be understood that although the various steps in the flowcharts of FIGS. 2-5 are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in Figures 2-5 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. These sub-steps or stages The execution order of is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 7, a data separation storage device is provided, including: a data acquisition module 702, a period determination module 704, and a separation storage module 706, wherein:

The data acquisition module 702 is used to acquire various pieces of user data of the target user;

The period determining module 704 is used to determine the period in which the generation time of each piece of user data of the target user is located;

The separate storage module 706 is used to separate and store each piece of user data of the target user into multiple databases according to the period of each generation time.

In an embodiment, the data separation storage device of the present application may further include: a database allocation request module (not shown in the figure), the database allocation request module is used to request the target user to allocate the current period database, historical database, first Database and the second database; the separate storage module 706 can separate and store each piece of user data of the target user into the current period database, the historical database, the first database and the second database according to the period of each generation time; among them, the current period The database is used to store the user data whose generation time of the target user is in the current period, the historical database is used to store the user data whose generation time is the target user in the historical period, and the first database is used to store the generation time of the target user The period is the user data of the current period and the previous period, and the second database is used to store the user data of the period in which the generation time of the target user is in the current period and the next period.

In one embodiment, the aforementioned pieces of user data of the target user may include newly generated data of the target user and various pieces of historical data; the separate storage module 706 may store the newly generated data of the target user in the current period database and the first In the database and the second database, each piece of historical data of the target user is separately stored in the current cycle database, the historical database and the first database according to the period in which each piece of historical data of the target user is generated.

In one embodiment, the separate storage module 706 can write the newly generated data of the target user into the current cycle database, write the newly generated data of the target user into the operation log generated by the current cycle database, and send the target user asynchronously. The newly generated data of is written into the first database and the second database.

In one embodiment, the aforementioned database allocation request module may also be used to request the target user to allocate a new current period database and a new second database before entering the next period; the data separation storage device of the present application may also include Data migration module (not shown in the figure), the data migration module is used to migrate the data in the current period database of the target user corresponding to the current period to the historical database of the target user after entering the next period. The second database corresponding to the current cycle of the target user is used as the new first database of the target user. After the data migration is completed, the current cycle database corresponding to the current cycle of the target user and the first database corresponding to the current cycle of the target user are destroyed.

In an embodiment, the data separation storage device of the present application may further include a user division module (not shown in the figure), and the user division module is used to perform fragmentation processing on user data according to the user dimension, and each segment after the fragmentation processing The user data of the segmented users are respectively used as the pieces of user data of the target user.

In an embodiment, the data separation storage device of the present application may further include a first request routing module (not shown in the figure), and the first request routing module is configured to query the user data modification request of the target user. The generation period information of the user data to be modified. When the period determined according to the queried generation period information is the current period, the user data modification request is routed to the current period database corresponding to the target user for data modification. When the period determined by the information is a historical period, the user data modification request is routed to the historical database corresponding to the target user for data modification.

In one embodiment, the data separation storage device of the present application may further include a second request routing module (not shown in the figure), and the second request routing module is configured to receive a user data query request from a target user according to the query The query condition carried in the request determines the query type; when the query type is a cycle query, if the query cycle determined according to the query condition is the current cycle, the user data query request is routed to the current cycle database corresponding to the target user for data query, If the query period determined according to the query conditions is a historical period, the user data query request is routed to the historical database corresponding to the target user for data query; when the query type is query by period, the start of the query period determined according to the query conditions Time and end time: According to the start time and end time, determine the target database of the target user that matches the start time and end time, and route the user data query request to the target database for data query.

In an embodiment, the data separation storage device of the present application may further include the above-mentioned first request routing module and second request routing module.

For the specific definition of the data separation storage device, please refer to the above definition of the data separation storage method, which will not be repeated here. Each module in the above-mentioned data separation storage device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 8. The computer equipment includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to realize a data separation storage method.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.

In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the following steps are implemented: Obtain various items of a target user User data; respectively determine the period in which each piece of user data of the target user is generated; according to the period of each generation time, separate and store each piece of user data of the target user into multiple databases.

In one embodiment, the processor further implements the following steps when executing the computer program: requesting to allocate the current cycle database, historical database, first database, and second database to the target user; When the user data of the target user is separated and stored in multiple databases, the following steps are specifically implemented: according to the period of each generation time, the user data of the target user is separated and stored in the current period database, In the historical database, the first database and the second database; among them, the current period database is used to store the user data of the target user's generation time period as the current period, and the historical database is used to store the target user's generation time period as the history Periodic user data. The first database is used to store the target user’s generation time in the current period and user data in the previous period. The second database is used to store the target user’s generation time in the current period and the next period. Periodic user data.

In one embodiment, each piece of user data of the target user mentioned above includes newly generated data of the target user and various pieces of historical data; the processor executes the computer program to realize the above-mentioned pieces of user data of the target user according to the period of each generation time. When user data is stored separately in the current period database, historical database, the first database and the second database, the following steps are specifically implemented: store the newly generated data of the target user in the current period database, the first database and the second database, respectively Middle; According to the period in which each piece of historical data of the target user is generated, separate and store each piece of historical data of the target user into the current period database, the historical database and the first database.

In one embodiment, when the processor executes the computer program to implement the above-mentioned step of storing the newly generated data of the target user in the current period database, the first database and the second database respectively, the following steps are specifically implemented: The generated data is written into the current cycle database;

Write the newly generated data of the target user into the operation log generated by the current cycle database, and write the newly generated data of the target user into the first database and the second database in an asynchronous manner.

In one embodiment, the processor further implements the following steps when executing the computer program: before entering the next cycle, requesting the target user to be allocated a new current cycle database and a new second database; after entering the next cycle, the current The data in the current period database of the target user corresponding to the period is migrated to the historical database of the target user, and the second database corresponding to the current period of the target user is taken as the new first database of the target user; after the data migration is completed, the target is destroyed The current period database corresponding to the current period of the user, and the first database corresponding to the current period of the target user.

In an embodiment, the processor further implements the following steps when executing the computer program: user data is segmented according to the user dimension, and the user data of each segmented user after the segmentation processing is used as each piece of user data of the target user.

In one embodiment, the processor further implements the following steps when executing the computer program: when receiving the user data modification request of the target user, query the generation period information of the user data to be modified; and the period determined according to the queried generation period information is In the current cycle, the user data modification request is routed to the current cycle database corresponding to the target user for data modification; when the cycle determined according to the queried generation cycle information is a historical cycle, the user data modification request is routed to the history corresponding to the target user The database performs data modification.

In one embodiment, the processor further implements the following steps when executing the computer program: when receiving the user data query request of the target user, the query type is determined according to the query conditions carried in the query request; when the query type is a periodic query, if If the query period determined according to the query conditions is the current period, the user data query request is routed to the current period database corresponding to the target user for data query. If the query period determined according to the query conditions is the historical period, the user data query request is routed to The historical database corresponding to the target user performs data query; when the query type is query by time period, the start time and end time corresponding to the query period determined according to the query conditions; according to the start time and end time, determine the start time and end time The target database of the target user matching the time, the user data query request is routed to the target database for data query.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are realized: each piece of user data of the target user is obtained; and each piece of the target user is determined respectively. The period of the generation time of the user data; according to the period of each generation time, each piece of user data of the target user is stored separately in multiple databases.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: request to allocate the current cycle database, historical database, first database, and second database to the target user; the computer program is executed by the processor to realize the above-mentioned generation according to each When the time is in the cycle, when the user data of the target user is separated and stored in multiple databases, the following steps are specifically implemented: According to the cycle of each generation time, the user data of the target user is separately stored in the current cycle Database, historical database, first database and second database; among them, the current period database is used to store the user data of the target user's generation time period, and the historical database is used to store the target user's generation time period User data of the historical period. The first database is used to store the target user’s generation time in the current period and user data in the previous period. The second database is used to store the target user’s generation time in the current period and the previous period. User data for the next cycle.

In one embodiment, each piece of user data of the target user mentioned above includes newly generated data of the target user and various pieces of historical data; the computer program is executed by the processor to realize the above-mentioned period according to the generation time of each target user. When the user data is stored separately in the current period database, the historical database, the first database, and the second database, the following steps are specifically implemented: store the newly generated data of the target user in the current period database, the first database and the second database, respectively. In the database; according to the cycle of the generation time of each piece of historical data of the target user, each piece of historical data of the target user is separately stored in the current cycle database, the historical database and the first database.

In one embodiment, when the computer program is executed by the processor to realize the above-mentioned steps of storing the newly generated data of the target user in the current period database, the first database and the second database, the following steps are specifically implemented: The newly generated data is written into the current cycle database;

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: before entering the next cycle, requesting the target user to be allocated a new current cycle database and a new second database; after entering the next cycle, The data in the current cycle database of the target user corresponding to the current cycle is migrated to the historical database of the target user, and the second database corresponding to the current cycle of the target user is taken as the new first database of the target user; after the data migration is completed, it is destroyed The current period database corresponding to the current period of the target user, and the first database corresponding to the current period of the target user.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: user data is segmented according to the user dimension, and the user data of each segmented user after the segmentation processing is used as each piece of user data of the target user. .

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: upon receiving the user data modification request of the target user, query the generation period information of the user data to be modified; in the period determined according to the queried generation period information When it is the current period, the user data modification request is routed to the current period database corresponding to the target user for data modification; when the period determined according to the queried generation period information is the historical period, the user data modification request is routed to the target user corresponding Historical database for data modification.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when a user data query request from a target user is received, the query type is determined according to the query conditions carried in the query request; when the query type is a periodic query, If the query period determined according to the query conditions is the current period, the user data query request is routed to the current period database corresponding to the target user for data query, and if the query period determined according to the query conditions is the historical period, the user data query request is routed Query data in the historical database corresponding to the target user; when the query type is query by time period, the start time and end time corresponding to the query period determined according to the query conditions; according to the start time and end time, determine the start time and the end time The target database of the target user whose end time matches, the user data query request is routed to the target database for data query.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the procedures of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered as the range described in this specification.

The above-mentioned embodiments only express several implementation manners of the present application, and their description is relatively specific and detailed, but they should not be understood as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

A data separation storage method, the method includes:

Obtain various pieces of user data of the target user;

Respectively determine the period in which the generation time of each piece of user data of the target user is in;

According to the period of each generation time, each piece of user data of the target user is separately stored in multiple databases.
The method according to claim 1, wherein the method further comprises: requesting to allocate a current period database, a historical database, a first database, and a second database to the target user;

Said separating and storing each piece of user data of the target user into multiple databases according to the period of each generation time includes: storing each piece of the target user according to the period of each generation time. User data is stored separately in the current period database, the historical database, the first database and the second database;

Wherein, the current period database is used to store user data in which the period of the generation time of the target user is the current period, and the history database is used to store the period in which the generation time of the target user is in the history Periodic user data, the first database is used to store user data in the current period and the previous period of the generation time of the target user, and the second database is used to store all user data of the target user The period of the generation time is the user data of the current period and the next period.
The method according to claim 2, wherein each piece of user data of the target user includes newly generated data of the target user and each piece of historical data;

Said storing each piece of user data of said target user separately in said current period database, said historical database, said first database and said second database according to the period in which each said generation time is located, including :

Storing the newly generated data of the target user in the current period database, the first database and the second database respectively;

Separately store each piece of historical data of the target user in the current period database, the historical database, and the first database according to the period in which each piece of historical data of the target user is generated.
The method according to claim 3, wherein said storing the new birth data of the target user in the current period database, the first database and the second database, respectively, comprises:

Write the newly generated data of the target user into the current cycle database;

According to the operation log generated by writing the newly generated data of the target user into the current periodic database, and writing the newly generated data of the target user into the first database and the second database in an asynchronous manner.
The method according to any one of claims 2 to 4, wherein the method further comprises:

Before entering the next cycle, request to allocate a new current cycle database and a new second database to the target user;

After entering the next cycle, the data in the current cycle database of the target user corresponding to the current cycle is migrated to the historical database of the target user, and the second database corresponding to the current cycle of the target user is used as the The new first database of target users;

After the data migration is completed, the current period database corresponding to the current period of the target user and the first database corresponding to the current period of the target user are destroyed.
The method according to any one of claims 2 to 4, characterized in that the method further comprises: fragmenting user data according to the user dimension, and the user data of each fragmented user after the fragmentation processing is used as the user data. Describe each piece of user data of the target user.
The method according to claim 2 or 3, wherein the method further comprises:

When receiving the user data modification request of the target user, query the generation period information of the user data to be modified;

When the period determined according to the queried generation period information is the current period, routing the user data modification request to the current period database corresponding to the target user for data modification;

When the period determined according to the queried generation period information is a historical period, routing the user data modification request to the historical database corresponding to the target user for data modification;

Or/and

When receiving the user data query request of the target user, determine the query type according to the query condition carried in the query request;

When the query type is a periodical query, if the query period determined according to the query conditions is the current period, the user data query request is routed to the current period database corresponding to the target user for data query, if If the query period determined by the query condition is a historical period, the user data query request is routed to the historical database corresponding to the target user for data query;

When the query type is query by time period, the start time and end time corresponding to the query time period determined according to the query conditions;

According to the start time and the end time, determine the target database of the target user that matches the start time and the end time, and route the user data query request to the target database for data query .
A data separation storage device, characterized in that the device comprises:

The data acquisition module is used to acquire various pieces of user data of the target user;

A cycle determining module, configured to separately determine the cycle in which the generation time of each piece of user data of the target user is located;

The separate storage module is used to separate and store each piece of user data of the target user into multiple databases according to the period of each generation time.
A computer device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor implements any one of claims 1 to 7 when the computer program is executed The steps of the method.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed by a processor.