CN114385657A

CN114385657A - Data storage method, device and storage medium

Info

Publication number: CN114385657A
Application number: CN202210039175.8A
Authority: CN
Inventors: 方飞; 黄跃文; 吕昊航; 黎勇
Original assignee: Zhejiang Geely Holding Group Co Ltd; Hangzhou Youxing Technology Co Ltd
Current assignee: Zhejiang Geely Holding Group Co Ltd; Hangzhou Youxing Technology Co Ltd
Priority date: 2022-01-13
Filing date: 2022-01-13
Publication date: 2022-04-22

Abstract

The method comprises the steps of establishing a query concurrent thread for data tables to be processed in a hot store, further determining the relevant configuration of each data table in the data tables to be processed based on the query concurrent thread, copying the relevant data in the hot store to a refrigeration house according to the relevant configuration, and deleting the data meeting preset deleting conditions in the hot store. The cold and hot storage separation can be completed only by investing little labor cost without a great deal of modification of business system codes. In addition, the embodiment of the application can be accessed by a plurality of services, has universal capability, further improves the efficiency and reduces the labor cost. In addition, the embodiment of the application can adjust the related data at any time in the data migration process so as to ensure that the data migration does not influence the normal service capability of the service system and can achieve smooth migration.

Description

Data storage method, device and storage medium

Technical Field

The present application relates to the field of data storage technologies, and in particular, to a data storage method, an apparatus, and a storage medium.

Background

The storage requirement is continuously increased due to the rapid increase of the data volume, so that the performance bottleneck of a single database occurs. In order to solve the problem, a conventional database is separated into a cold storage and a hot storage to optimize a storage space, hot data generated in a near term and accessed at a high frequency is stored in the hot storage, and cold data generated in the past and accessed infrequently is stored in the cold storage.

In the related art, when a cold storage and a hot storage are separated, data migration is generally performed based on double writing, database log synchronization, and the like. Taking double writing as an example, the steps of double writing are: the service system backups the full data of the original database to the refrigeration storage; double-writing the generated data to two banks; and deleting the historical data of the original database.

However, when the existing cold storage and hot storage are separated, data migration needs to modify service system codes, and each service needs to be customized, so that the engineering period is long, and the labor cost is high.

Disclosure of Invention

In order to solve the problems in the prior art, the application provides a data storage method, a data storage device and a storage medium.

In a first aspect, an embodiment of the present application provides a data storage method, where the method includes:

establishing query concurrent threads according to data tables to be processed in a hot bank, wherein each data table in the data tables to be processed corresponds to one thread in the query concurrent threads;

determining the relevant configuration of each data table in the data tables to be processed based on the query concurrent thread;

acquiring a target data table meeting a preset migration condition from the data tables to be processed according to the relevant configuration of each data table in the data tables to be processed;

and copying the related data in the hot storage to a cold storage based on the target data table, and deleting the data meeting preset deleting conditions in the hot storage according to the related data copied to the hot storage of the cold storage.

In one possible implementation, the copying, based on the target data table, related data in the hot store to a cold store includes:

acquiring each migration reference field in the target data table, wherein each migration reference field in each migration reference field is used for uniquely identifying one piece of data in the target data table;

inquiring data corresponding to each migration reference field in the refrigeration house;

and copying the related data in the hot store to the cold store according to the query result.

In a possible implementation manner, the copying, according to the query result, the relevant data in the hot store to the cold store includes:

if the refrigerator does not have data corresponding to a migration reference field i, copying the data corresponding to the migration reference field i to the refrigerator, wherein the migration reference field i is any one of the migration reference fields, i is 1, …, n is equal to the number of the migration reference fields;

if the data corresponding to the migration reference field i exist in the cold storage, comparing the data corresponding to the migration reference field i in the cold storage with the data corresponding to the migration reference field i in the hot storage, and if the comparison is different, updating the data corresponding to the migration reference field i in the cold storage based on the data corresponding to the migration reference field i in the hot storage.

In a possible implementation manner, the creating a query concurrent thread according to a to-be-processed data table in a hot bank includes:

creating a query lock and a cursor value for each data table in the data tables to be processed according to the data tables to be processed, wherein the query lock comprises a lock identifier and a lock state, the cursor value comprises a mapping table consisting of key-values (values), the key represents a cursor field, and the value represents a current cursor value;

and creating the query concurrent thread according to the query lock and the vernier value created by each data table in the to-be-processed data tables, the preset fragment size and the preset fragment vernier field, wherein the preset fragment size represents the number of data read each time, and the preset fragment vernier field is used for identifying the data read by each thread in the query concurrent thread.

In a possible implementation manner, after the copying, based on the target data table, related data in the hot store to a cold store, and deleting, in the hot store, data that meets a preset deletion condition according to the related data copied to the hot store of the cold store, the method further includes:

acquiring a data query request, and querying data from the hot store based on the data query request;

determining whether compensation query is carried out on the refrigeration house or not according to the type of the value returned by the hot house;

and if the compensation query is carried out on the refrigeration house, determining target data corresponding to the data query request according to a compensation query result of the refrigeration house.

In a possible implementation manner, the determining whether to perform a compensation query on the cold storage according to the type of the value returned by the hot storage includes:

if the type of the value returned by the hot store is single data, judging whether the value returned by the hot store is empty, and if the value returned by the hot store is empty, determining to perform compensation query on the cold store;

and if the types of the values returned by the hot store are multiple pieces of data, determining to perform compensation query on the cold store.

In a possible implementation manner, before the determining, according to the compensation query result from the cold storage, target data corresponding to the data query request, the method further includes:

if the types of the values returned by the hot store are multiple pieces of data and the values returned by the hot store are not empty, comparing the values returned by the hot store with the compensation query result of the cold store;

the determining the target data corresponding to the data query request according to the compensation query result from the refrigeration house comprises the following steps:

if the comparison result is that the value returned by the hot store and the compensation query result of the cold store are repeated, performing duplicate removal treatment on the value returned by the hot store and the compensation query result of the cold store;

and obtaining target data corresponding to the data query request based on the value returned by the hot storage and the compensation query result of the cold storage after the duplication removal processing.

In a second aspect, an embodiment of the present application provides a data storage device, including:

the thread creating module is used for creating query concurrent threads according to-be-processed data tables in the hot bank, wherein each data table in the to-be-processed data tables corresponds to one thread in the query concurrent threads respectively;

the configuration determining module is used for determining the relevant configuration of each data table in the data tables to be processed based on the query concurrent thread;

the data acquisition module is used for acquiring a target data table meeting a preset migration condition from the data tables to be processed according to the relevant configuration of each data table in the data tables to be processed;

and the storage processing module is used for copying the related data in the hot storage to a cold storage based on the target data table, and deleting the data meeting the preset deleting condition in the hot storage according to the related data copied to the hot storage of the cold storage.

In a possible implementation manner, the storage processing module is specifically configured to:

In a possible implementation manner, the thread creating module is specifically configured to:

creating a query lock and a cursor value for each data table in the data tables to be processed according to the data tables to be processed, wherein the query lock comprises a lock identifier and a lock state, the cursor value comprises a mapping table consisting of key-values, the key represents a cursor field, and the value represents a current cursor value;

In a possible implementation manner, the data query module is further configured to copy, by the storage processing module, related data in the hot store to a cold store based on the target data table, delete data meeting a preset deletion condition in the hot store according to the related data copied to the hot store of the cold store, acquire a data query request, and query, based on the data query request, the data from the hot store;

In a possible implementation manner, the data query module is specifically configured to:

In a third aspect, an embodiment of the present application provides a data storage system, including:

a processor;

a memory; and

a computer program;

wherein the computer program is stored in the memory and configured to be executed by the processor, the computer program comprising instructions for performing the method of the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored, and the computer program causes a server to execute the method according to the first aspect.

In a fifth aspect, the present application provides a computer program product, which includes computer instructions for executing the method of the first aspect by a processor.

According to the data storage method, the data storage device and the data storage medium, a query concurrent thread is established for data tables to be processed in a hot store, further, based on the query concurrent thread, the relevant configuration of each data table in the data tables to be processed is determined, and according to the relevant configuration, a target data table meeting a preset migration condition is obtained from the data tables to be processed, so that based on the target data table, relevant data in the hot store is copied to the cold store, and according to the relevant data copied to the hot store of the cold store, data meeting a preset deletion condition is deleted in the hot store. The cold and hot storage separation can be completed only by investing little labor cost without a great deal of modification of business system codes. In addition, the embodiment of the application can be accessed by a plurality of services, has universal capability, further improves the efficiency and reduces the labor cost. In addition, the embodiment of the application can adjust the related data at any time in the data migration process so as to ensure that the data migration does not influence the normal service capability of the service system and can achieve smooth migration.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.

FIG. 1 is a schematic diagram of a data storage system according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a data storage method according to an embodiment of the present application;

FIG. 3 is a schematic flow chart illustrating another data storage method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a data storage according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a data storage device according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of another data storage device according to an embodiment of the present application;

FIG. 7A is a diagram illustrating a basic hardware architecture of a data storage system according to an embodiment of the present application;

fig. 7B is a schematic diagram of a basic hardware architecture of a data storage system according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terms "first," "second," "third," and "fourth," if any, in the description and claims of this application and the above-described figures are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Many existing internet enterprises often adopt single database design when performing system design at the initial stage of business, and with the development of business, more and more business data are generated, so that the database reaches the performance bottleneck, and the risk that data query is slow or even the system is rushed exists. However, most services have the characteristics of hot data and cold data, wherein the hot data is data generated recently and has a large query amount, and the cold data is data generated in the past and has a small query amount. Therefore, the separation of the cold storage and the hot storage of the database is an effective solution for solving the performance bottleneck of a single database.

In the related art, when a cold storage and a hot storage are separated, data migration is generally performed based on double writing, database log synchronization, and the like. Taking double writing as an example, the steps of double writing are: the service system backups the full data of the original database to the refrigeration storage; double-writing the generated data to two banks; and deleting the historical data of the original database. However, when the cold storage and the hot storage are separated, data migration requires modification of service system codes, and each service needs to be customized, which results in long engineering period and high labor cost.

In order to solve the above problems, an embodiment of the present application provides a data storage method, where a query concurrent thread is created for a to-be-processed data table in a hot store, and then, based on the query concurrent thread, a relevant configuration of each data table in the to-be-processed data table is determined, and according to the relevant configuration, relevant data in the hot store is copied to a cold store, and data meeting a preset deletion condition is deleted in the hot store, so as to solve the problems that when a cold store is separated from a hot store, data migration requires modification of service system codes, each service needs to be customized, and therefore, an engineering period is long, and labor cost is high.

Optionally, the data storage method provided by the embodiment of the present application may be applied to the data storage system shown in fig. 1. In fig. 1, the system may include a receiving device 101, a processing device 102, and a display device 103.

In a specific implementation process, the receiving device 101 may be an input/output interface or a communication interface, and may be used to receive information related to a thermal storage and a refrigeration storage.

The processing device 102 may obtain the to-be-processed data tables in the hot store through the receiving device 101, further create an inquiry concurrent thread for the to-be-processed data tables, determine the relevant configuration of each data table in the to-be-processed data tables based on the inquiry concurrent thread, copy the relevant data in the hot store to the cold store according to the relevant configuration, and delete the data meeting a preset deletion condition in the hot store. The cold and hot storage separation can be completed only by investing little labor cost without a great deal of modification of business system codes. Moreover, the processing device 102 can be accessed by a plurality of services, and has general-purpose capability, thereby further improving efficiency and reducing labor cost. In addition, the processing device 102 may adjust the relevant data at any time during the data migration process, so as to ensure that the data migration does not affect the normal service capability of the service system, and may achieve smooth migration.

The display device 103 may be used to display the above-described cold storage and hot storage separation process.

The display device may also be a touch display screen for receiving user instructions while displaying the above-mentioned content to enable interaction with a user.

It should be understood that the processing device may be implemented by reading instructions in the memory and executing the instructions, or may be implemented by a chip circuit.

The system is only an exemplary system, and when the system is implemented, the system can be set according to application requirements.

It is to be understood that the system architecture described in the embodiment of the present application is for more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation to the technical solution provided in the embodiment of the present application, and as a person having ordinary skill in the art knows that along with the evolution of the system architecture and the appearance of a new service scenario, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

The technical solutions of the present application are described below with several embodiments as examples, and the same or similar concepts or processes may not be described in detail in some embodiments.

Fig. 2 is a schematic flow diagram of a data storage method according to an embodiment of the present application, where an execution subject of the embodiment may be the processing device in fig. 1, and a specific execution subject may be determined according to an actual application scenario, which is not limited in the embodiment of the present application. As shown in fig. 2, the data storage method provided in the embodiment of the present application may include the following steps:

s201: and establishing query concurrent threads according to the to-be-processed data tables in the hot bank, wherein each data table in the to-be-processed data tables corresponds to one thread in the query concurrent threads respectively.

Here, before creating the query concurrent thread according to the to-be-processed data table in the hot bank, the processing device may obtain corresponding configuration information, for example, obtain a timing task configuration table, a base configuration table, and a rule configuration table.

Wherein, the timing task configuration table is configured to execute the data migration task in a specified time period, as shown in table 1 below, fields in the table include an Identity document (id) (unique identifier of the task), an id of the basic configuration, a start state (a global switch controlling the data migration task, if the state is true (true), indicating that the migration task is on, otherwise, it is off), an execution time (the execution time period of the migration task may be specified, which may be specifically determined according to an actual situation, for example, specifying that the time period for executing the migration task is in a low valley period of a service traffic), the number of concurrent threads (which may be adjusted according to the actual situation), a size of a slice (number of data read each time), a slice cursor field (used in conjunction with the size of a slice, if the number of concurrent threads is configured to be greater than 1, a slice cursor field needs to be set for identifying data read by each thread in the concurrent threads, to ensure that each thread does not repeat reads when reading hot bank data).

TABLE 1 timed task configuration Table

id

Base configuration id

Starting state

Execution time

Number of concurrent threads

Size of the slice

Sliced cursor field

1

true

22:00-06:00

4

100

id

2

true

00:00-00:00

4

200

id

The basic configuration table may be as shown in table 2 below, and includes id (unique identifier), name of the hot store to be migrated, login account, and login password; migrating the data to a cold store name, a login account and a login password; association rule configuration id list (since there are multiple tables in a database, a base configuration associates multiple table rules).

TABLE 2 basic configuration Table

The rule configuration table can be as shown in the following table 3, id (unique identifier), table name (for example, a data table to be processed in a hot store), switch (setting a switch for each table in the store, so that different migration requirements of different tables can be flexibly controlled), a migration reference field (a field for identifying only one piece of data), a migration condition (which can be set according to actual conditions, such as the table, for example, that "now-create _ time >30 days" indicates that data can be copied from the hot storage to the cold storage before the creation time of the data is 30 days, and "id < N" indicates that data can be copied from the hot storage to the cold storage when the value of the field id in the data table is less than N), and a deletion condition (for deleting the operation of the hot storage data after the data is copied to the cold storage, which can also be set according to actual conditions, such as the table, for example, "bill _ status in (10, 20)" indicates that data can be deleted only if the field bill _ status in (10,20) "is 10 or 20).

Table 3 rule configuration table

For example, the processing device may perform subsequent processing according to the configuration information, for example, the timing task configuration table, the basic configuration table, and the rule configuration table, for example, create a query concurrent thread according to the to-be-processed data table in the hot library. Specifically, the processing device may register the hot store and the cold store according to the information in the basic configuration table, determine a data table to be processed in the hot store according to the information in the rule configuration table, and execute subsequent processing according to the start state and the execution time in the timed task configuration table.

In this embodiment of the present application, when creating a query concurrent thread according to a to-be-processed data table in a hot repository, the processing apparatus also considers the timing task configuration table, the basic configuration table, and the rule configuration table, for example, a query lock and a cursor value are created for each of the to-be-processed data tables according to the to-be-processed data table, where the query lock includes a lock identifier and a lock state, the cursor value includes a mapping table composed of key-values, key represents a cursor field, and value represents a current cursor value; and then, creating the query concurrent thread according to the query lock and the vernier value created by each data table in the to-be-processed data tables, the fragment size and the fragment vernier field, wherein the preset fragment size represents the number of data read each time, and the preset fragment vernier field is used for identifying the data read by each thread in the query concurrent thread.

Wherein, the lock identification (lock _ table name), lock state (0-represents unlocked, 1-represents locked).

S202: and determining the relevant configuration of each data table in the data tables to be processed based on the query concurrent thread.

Here, the processing device starts the query concurrent thread, so that all relevant configurations of each data table in the to-be-processed data table are determined based on the query concurrent thread.

For example, the processing device starts the query concurrent thread, which may be as shown in the specific example of the data table tableA in table 4. The lock identifier of tableA is lock _ tableA, the state is 0, the cursor value of tableA, the key is tableA _ id, and the value is 0.

Specifically, the starting of the query concurrent thread by the processing device may include: 1. starting a thread to read the relevant configuration of a data table to be processed; if the switch is turned off in the rule configuration of the table, the thread exits without executing the subsequent steps, and if the switch is turned on, the next step is carried out; 2. reading the lock state of 'lock _ table name' in the query lock, if the lock state is 0, indicating that the lock is idle, setting the lock state to be 1, setting the occupied thread to be the thread name, and entering the next step; if the lock state is 1, indicating that the lock is occupied, repeating step 2 to continuously acquire the lock; 3. inquiring a cursor value according to the table name-cursor field; 4. the data of the next fragment is queried according to the query cursor, for example: when the current cursor is 100 and the fragment size is 50, the query data is 50 pieces of data starting from 101, and mysql contains: select from tablemame where id >100limit 50; 5. and taking out the maximum value corresponding to the segment cursor field from the searched data, updating the cursor value corresponding to the key in the cache, releasing the corresponding query lock after the updating is successful, and entering the subsequent processing. Other threads may then obtain the query lock and repeat steps 1-5 above.

TABLE 4

S203: and acquiring a target data table meeting a preset migration condition from the data tables to be processed according to the relevant configuration of each data table in the data tables to be processed.

The preset migration condition may be determined according to actual conditions, for example, the migration condition "now-create _ time >30 days" in the rule configuration table indicates that data can be copied from the hot storage to the cold storage before the creation time of the data is 30 days, and "id < N" indicates that data can be copied from the hot storage to the cold storage if the value of field id in the data table is less than N.

The processing device determines a target data table from the data tables to be processed based on the relevant configuration of each data table in the data tables to be processed and the migration condition, wherein the target data table comprises data meeting the migration condition.

S204: and copying the related data in the hot storage to a cold storage based on the target data table, and deleting the data meeting preset deleting conditions in the hot storage according to the related data copied to the hot storage of the cold storage.

Here, the processing device may acquire each migration reference field in the target data table, where each migration reference field in each migration reference field is used to uniquely identify one piece of data in the target data table, and then query the cold storage for data corresponding to each migration reference field, so as to copy the relevant data in the hot storage to the cold storage according to the query result.

For example, if the refrigerator does not have data corresponding to a transition reference field i, which is any one of the transition reference fields, i is 1, …, n, n is equal to the number of the transition reference fields, the processing device may copy the data corresponding to the transition reference field i to the refrigerator. The processing device may compare data corresponding to the migration reference field i in the cold storage with data corresponding to the migration reference field i in the hot storage if the data corresponding to the migration reference field i exists in the cold storage, and update the data corresponding to the migration reference field i in the cold storage based on the data corresponding to the migration reference field i in the hot storage if the data corresponding to the migration reference field i in the hot storage is different from the data corresponding to the migration reference field i in the cold storage.

For example, taking the preset migration condition "now-create _ time >30 days" as an example, the processing device filters out a target data table containing data whose creation time is 30 days before from the to-be-processed data table, then obtains the migration reference fields, and queries the cold storage through the migration reference fields. Examples are: assuming that the migration reference field is "id", the value of the field in the hot bank is 1 to 100; then the statement for querying the freezer is: "select from cold _ DB _ name. tablename where in (1,2, 3.,. 100); ". If the hot storage is not provided with the cold storage, the processing device can add the hot storage data into the insertList in the list of the cold storage. If the hot store has a cold store, the processing device can compare the value of each field, if the difference exists, the hot store data is added to the updateList in the list to be updated, and the next step is carried out after all data are processed. Finally, the processing device inserts the data in the insetlist into the freezer and inserts the data in the updateList into the freezer.

In addition, after the related data in the hot storage is copied to a cold storage, the data satisfying a preset deletion condition may be deleted in the hot storage according to the related data in the hot storage copied to the cold storage. The preset deletion condition may be determined according to actual conditions, for example, the "now-create _ time >60 days and the order _ status ═ 10" in the rule configuration table indicate that data whose creation time is 60 days ago needs to be filtered, and the value of the "order _ status" field is equal to 10.

For example, the processing device may acquire each transition reference field of the related data copied to the hot storage of the cold storage, and delete the related data in the hot storage according to the transition reference field and the preset deletion condition.

The cold storage and the hot storage may be determined according to actual conditions, and for example, the processing device may copy data in the hot storage a to the cold storage a, and delete data satisfying a predetermined deletion condition in the hot storage a based on the data copied to the hot storage a of the cold storage a.

According to the method and the device for processing the data, a query concurrent thread is established for the data tables to be processed in the hot storage, and further, the related configuration of each data table in the data tables to be processed is determined based on the query concurrent thread, and the target data table meeting the preset migration condition is obtained from the data tables to be processed according to the related configuration, so that the related data in the hot storage are copied to the cold storage based on the target data table, and the data meeting the preset deletion condition are deleted in the hot storage according to the related data copied to the hot storage of the cold storage. The cold and hot storage separation can be completed only by investing little labor cost without a great deal of modification of business system codes. In addition, the embodiment of the application can be accessed by a plurality of services, has universal capability, further improves the efficiency and reduces the labor cost. In addition, the embodiment of the application can adjust the related data at any time in the data migration process so as to ensure that the data migration does not influence the normal service capability of the service system and can achieve smooth migration.

In addition, the processing device copies the related data in the hot storage to a refrigeration storage based on the target data table, deletes the data meeting the preset deletion condition in the hot storage according to the related data copied to the hot storage of the refrigeration storage, and provides a data query function, such as accessing a refrigeration storage query assembly to obtain a data query request, splicing cold and hot data based on the data query request, determining the target data corresponding to the data query request, and completing the query of the refrigeration storage in a way without service system code intrusion, thereby providing a general solution for the query of the refrigeration storage. Fig. 3 is a schematic flowchart of another data storage method according to an embodiment of the present application. As shown in fig. 3, the method includes:

s301: and establishing query concurrent threads according to the to-be-processed data tables in the hot bank, wherein each data table in the to-be-processed data tables corresponds to one thread in the query concurrent threads respectively.

S302: and determining the relevant configuration of each data table in the data tables to be processed based on the query concurrent thread.

S303: and acquiring a target data table meeting a preset migration condition from the data tables to be processed according to the relevant configuration of each data table in the data tables to be processed.

S304: and copying the related data in the hot storage to a cold storage based on the target data table, and deleting the data meeting preset deleting conditions in the hot storage according to the related data copied to the hot storage of the cold storage.

S305: and acquiring a data query request, and querying data from the hot store based on the data query request.

S306: and determining whether to perform compensation inquiry in the refrigeration house according to the type of the value returned by the hot house.

Here, if the type of the value returned by the hot store is single data, the processing device may determine whether the value returned by the hot store is empty, and if the value returned by the hot store is empty, determine that a compensation query is performed in the cold store. If the type of the value returned by the hot store is a plurality of pieces of data (such as a list), the processing device may determine to perform a compensation query in the cold store.

In this embodiment of the application, if the type of the value returned by the hot store is a single piece of data, the processing device may determine whether the returned value is empty, if not, the returned value is directly returned based on query data in the hot store, if so, it indicates that no data is found from the hot store, a cold store needs to be removed for compensation query, join references of a cold store data query interface according to the data query request, and after calling the cold store data query interface to obtain a result, determine target data corresponding to the data query request based on the result.

If the type of the value returned by the hot store is a plurality of pieces of data, the processing device can judge whether the returned value is empty, if so, the processing device splices the access parameters of the data query interface of the cold store according to the data query request, and determines target data corresponding to the data query request based on the data query result of the cold store; and if the data query request is not empty, calling a cold storage query interface to query the cold storage data, and determining target data corresponding to the data query request based on the queried cold storage data and the value returned by the hot storage.

S307: and if the compensation inquiry is carried out in the refrigeration house, determining target data corresponding to the data inquiry request according to a compensation inquiry result from the refrigeration house.

For example, if the type of the value returned by the hot store is a plurality of pieces of data and the value returned by the hot store is not empty, the processing device may compare the value returned by the hot store with the compensation query result of the cold store, perform deduplication processing on the value returned by the hot store and the compensation query result of the cold store if the comparison result is that the value returned by the hot store and the compensation query result of the cold store are duplicated, and obtain target data corresponding to the data query request based on the value returned by the hot store and the compensation query result of the cold store after the deduplication processing. For example, the value returned by the hot store is hotList, wherein the data has five data items with id being 3, 4 and 5, and the compensation query result of the cold store is that the cold store has four data items with id being 1,2,3 and 4, the data is repeated for 3 and 4, 3 and 4 in the cold list are removed, and 1 and 2 are spliced to the hotList, so as to obtain the target data corresponding to the data query request.

In the embodiment of the application, the processing device further provides a data query function, for example, the processing device is connected to a cold storage query assembly to obtain a data query request, and based on the data query request, cold and hot data are spliced to determine target data corresponding to the data query request, so that query of the cold storage can be completed in a non-service system code intrusion mode, and a general solution for query of the cold storage is provided. The processing device can also establish a query concurrent thread for the data tables to be processed in the hot store, further determine the relevant configuration of each data table in the data tables to be processed based on the query concurrent thread, copy the relevant data in the hot store to the cold store according to the relevant configuration, and delete the data meeting the preset deletion condition in the hot store, thereby solving the problems that when the cold store and the hot store are separated, the data migration needs to reform service system codes, each service needs to be customized respectively, and the project period is long, and the labor cost is high.

Here, as shown in fig. 4, the embodiment of the present application takes a hot storage a, a cold storage a, a hot storage B, and a cold storage B as examples, and provides a schematic diagram of data storage, and the processing device provides functions such as data migration, data deletion, data query, and the like, and further has functions of timing task configuration, basic configuration, and rule configuration. Specifically, the processing device provides a timing task configuration function based on the timing task configuration table, a basic configuration function based on the basic configuration table, and a rule configuration function based on the rule configuration table.

The processing device first performs information configuration based on the timed task configuration, the basic configuration and the rule configuration function, further creates a query concurrent thread for the to-be-processed data tables in the hot store a (or the hot store B) based on the configuration information, then determines the relevant configuration of each data table in the to-be-processed data tables based on the query concurrent thread, and provides the data migration function according to the relevant configuration, for example, copies the relevant data in the hot store a (or the hot store B) to the cold store a (or the cold store B), and provides the data deletion function based on the data copied to the cold store a (or the cold store B), for example, deletes the data meeting a preset deletion condition in the hot store a (or the hot store B).

The processing device can be connected to the refrigeration house query assembly to obtain a data query request (for example, data needs to be queried in the system A or the system B), and based on the data query request, cold and hot data are spliced to determine target data corresponding to the data query request, so that a data query function is provided.

The cold and hot storage separation can be completed only by investing little labor cost without a great deal of modification of business system codes. In addition, the embodiment of the application can be accessed by a plurality of services, has universal capability, further improves the efficiency and reduces the labor cost. In addition, the embodiment of the application can adjust the related data at any time in the data migration process so as to ensure that the data migration does not influence the normal service capability of the service system and can achieve smooth migration. The embodiment of the application also provides a data query function, can complete the query of the refrigeration house in a non-service system code intrusion mode, and provides a general solution for the query of the refrigeration house.

Fig. 5 is a schematic structural diagram of a data storage device according to an embodiment of the present application, corresponding to the data storage method according to the foregoing embodiment. For convenience of explanation, only portions related to the embodiments of the present application are shown. Fig. 5 is a schematic structural diagram of a data storage device according to an embodiment of the present application, where the data storage device 50 includes: a thread creation module 501, a configuration determination module 502, a data acquisition module 503, and a storage processing module 504. The data storage means here may be the processing means itself as described above, or a chip or an integrated circuit implementing the functionality of the processing means. It should be noted here that the division of the thread creating module, the configuration determining module, the data acquiring module, and the storage processing module is only a division of logical functions, and the two may be integrated or independent physically.

The thread creating module 501 is configured to create query concurrent threads according to-be-processed data tables in the hot store, where each data table in the to-be-processed data tables corresponds to one thread in the query concurrent threads.

A configuration determining module 502, configured to determine, based on the query concurrent thread, a relevant configuration of each data table in the to-be-processed data tables.

A data obtaining module 503, configured to obtain, according to the relevant configuration of each data table in the to-be-processed data tables, a target data table that meets a preset migration condition from the to-be-processed data tables.

And the storage processing module 504 is configured to copy the relevant data in the hot store to a cold store based on the target data table, and delete the data meeting a preset deletion condition in the hot store according to the relevant data copied to the hot store of the cold store.

In a possible implementation manner, the storage processing module 504 is specifically configured to:

In a possible implementation manner, the thread creating module 501 is specifically configured to:

The apparatus provided in the embodiment of the present application may be configured to implement the technical solution of the method embodiment shown in fig. 2, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 6 is a schematic structural diagram of another data storage device according to an embodiment of the present application, and based on the embodiment shown in fig. 5, the data storage device 50 further includes: a data query module 505.

In a possible implementation manner, the data query module 505 is configured to copy, by the storage processing module 504, the relevant data in the hot store to a cold store based on the target data table, delete, in the hot store, data meeting a preset deletion condition according to the relevant data copied to the hot store of the cold store, acquire a data query request, and query, based on the data query request, the data from the hot store;

In a possible implementation manner, the data query module 505 is specifically configured to:

The apparatus provided in the embodiment of the present application may be used to implement the technical solution of the method embodiment shown in fig. 3, which has similar implementation principles and technical effects, and is not described herein again in the embodiment of the present application.

Alternatively, FIGS. 7A and 7B schematically provide a schematic diagram of one possible basic hardware architecture of the data storage system described herein.

Referring to fig. 7A and 7B, a data storage system 700 includes at least one processor 701 and a communication interface 703. Further optionally, a memory 702 and a bus 704 may also be included.

Among them, in the data storage system 700, the number of the processors 701 may be one or more, and fig. 7A and 7B illustrate only one of the processors 701. Alternatively, the processor 701 may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or a Digital Signal Processor (DSP). If the data storage system 700 has multiple processors 701, the types of the multiple processors 701 may be different, or may be the same. Optionally, the processors 701 of the data storage system 700 may also be integrated as a multi-core processor.

Memory 702 stores computer instructions and data; the memory 702 may store computer instructions and data needed to implement the above-described data storage methods provided herein, e.g., the memory 702 stores instructions for implementing the steps of the above-described data storage methods. Memory 702 can be any one or any combination of the following storage media: nonvolatile memory (e.g., Read Only Memory (ROM), Solid State Disk (SSD), hard disk (HDD), optical disk), volatile memory.

The communication interface 703 may provide information input/output for the at least one processor. Any one or any combination of the following devices may also be included: a network interface (e.g., an ethernet interface), a wireless network card, etc. having a network access function.

Optionally, the communication interface 703 may also be used for data communication between the data storage system 700 and other computing devices or terminals.

Further alternatively, fig. 7A and 7B show the bus 704 by a thick line. The bus 704 may connect the processor 701 with the memory 702 and the communication interface 703. Thus, via bus 704, processor 701 may access memory 702 and may also interact with other computing devices or terminals using communication interface 703.

In the present application, the data storage system 700 executes computer instructions in the memory 702, so that the data storage system 700 implements the data storage method provided in the present application, or so that the data storage system 700 deploys the data storage device.

From the viewpoint of logical functional division, exemplarily, as shown in fig. 7A, a thread creating module 501, a configuration determining module 502, a data obtaining module 503, and a storage processing module 504 may be included in the memory 702. The inclusion herein merely refers to that the instructions stored in the memory may, when executed, implement the functions of the thread creating module, the configuration determining module, the data acquiring module, and the storage processing module, respectively, without limitation to physical structures.

Illustratively, as shown in FIG. 7B, a data query module 505 may be included in memory 702. The inclusion herein merely refers to that instructions stored in the memory may, when executed, implement the functionality of the data query module and is not limited to a physical structure.

In addition, the data storage system may be implemented by software as shown in fig. 7A and 7B, or may be implemented by hardware as a hardware module or a circuit unit.

The present application provides a computer readable storage medium, the computer program product comprising computer instructions that instruct a computing device to perform the above-mentioned data storage method provided herein.

The present application provides a computer program product comprising computer instructions for execution by a processor of the above-described data storage method.

The present application provides a chip comprising at least one processor and a communication interface providing information input and/or output for the at least one processor. Further, the chip may also include at least one memory for storing computer instructions. The at least one processor is used for calling and executing the computer instructions to execute the data storage method provided by the application.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

Claims

1. A method of storing data, comprising:

2. The method of claim 1, wherein the copying the relevant data in the hot store to a cold store based on the target data table comprises:

3. The method according to claim 2, wherein the copying the relevant data in the hot store to the cold store according to the query result comprises:

4. The method according to any one of claims 1 to 3, wherein the creating a query concurrent thread according to the to-be-processed data table in the hot bank comprises:

creating a query lock and a cursor value for each data table in the data tables to be processed according to the data tables to be processed, wherein the query lock comprises a lock identifier and a lock state, the cursor value comprises a mapping table consisting of keywords and values, the keywords represent cursor fields, and the values represent current cursor values;

5. The method according to any one of claims 1 to 3, wherein after the copying the relevant data in the hot store to a cold store based on the target data table and deleting the data meeting a preset deletion condition in the hot store according to the relevant data copied to the hot store of the cold store, the method further comprises:

6. The method of claim 5, wherein said determining whether to make a compensation query at said cold store based on the type of value returned by said hot store comprises:

7. The method according to claim 6, before the determining the target data corresponding to the data query request according to the compensation query result from the refrigerator, further comprising:

8. A data storage device, comprising:

9. A data storage system, comprising:

a processor;

a memory; and

a computer program;

wherein the computer program is stored in the memory and configured to be executed by the processor, the computer program comprising instructions for performing the method of any of claims 1-7.

10. A computer-readable storage medium, characterized in that it stores a computer program that causes a server to execute the method of any one of claims 1-7.