CN113127443A - Method and device for updating cache data - Google Patents
Method and device for updating cache data Download PDFInfo
- Publication number
- CN113127443A CN113127443A CN202010037735.7A CN202010037735A CN113127443A CN 113127443 A CN113127443 A CN 113127443A CN 202010037735 A CN202010037735 A CN 202010037735A CN 113127443 A CN113127443 A CN 113127443A
- Authority
- CN
- China
- Prior art keywords
- task
- updating
- cache data
- abnormal
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000002159 abnormal effect Effects 0.000 claims description 106
- 238000011084 recovery Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 17
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 238000012986 modification Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000013502 data validation Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 239000003607 modifier Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 235000010627 Phaseolus vulgaris Nutrition 0.000 description 1
- 244000046052 Phaseolus vulgaris Species 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a method and a device for updating cache data, and relates to the technical field of computers. One embodiment of the method comprises: receiving an updating request, and generating a parent task for updating cache data to each cluster according to the updating request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks; obtaining the cache data, and respectively updating the cache data to the clusters corresponding to the subtasks; and respectively sending a version switching request to each cluster, wherein the version switching request carries the version information of the cached data. The implementation method can solve the technical problem of data consistency among the cache clusters.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for updating cache data.
Background
In order to carry massive order processing requests, caching technology (such as Redis) has become a technical solution for standard allocation. However, as the service scale is expanded, a situation that a single cache cluster cannot bear more pressure often occurs. For example, the client application expands the capacity of the machine without limit, resulting in a corresponding increase in the number of connections of the cache clients; the buffer cluster has too many slave instances mounted, the data copy amount between the master and the slave is large, and the cluster pressure is increased due to the factors of too long copy link and the like. To solve this problem, a feasible solution is: and copying the whole cache cluster into a plurality of cache clusters, wherein the client applications are distributed in a vertical grouping manner, and different client applications are connected with different cache clusters, so that the pressure is shared. However, this solution again leads to a problem: if these client applications require strong consistency of data across multiple cache clusters and require strict consistency of data validation time. Otherwise, different client applications may cause inconsistent calculation results due to inconsistent data of multiple cache clusters, which causes online problems and loss of customer experience. At present, there are two main ways to update the cached data: the single-thread serial updating cache data and the multi-thread parallel updating cache data.
When the same cache data needs to be written to a plurality of cache clusters, the following problems can occur no matter the cache data is updated serially by adopting a single thread or the cache data is updated in parallel by adopting multiple threads:
1) data consistency among multiple clusters cannot be guaranteed: due to cluster failure or network failure and other reasons, the refreshing of partial cluster data is successful, and the refreshing of partial cluster data is failed.
2) The data validation time among a plurality of clusters cannot achieve synchronous validation: the cache data is updated in a single-thread serial mode, and the scheme cannot achieve synchronous data validation; and the multithread parallel updating of the cache data is adopted, and due to various factors such as uncontrollable thread scheduling of the bottom layer of the operating system, JVM GC (garbage collection mechanism) pause, not strictly consistent network request response time and the like, the synchronous effectiveness of the data among a plurality of clusters can not be ensured.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for updating cache data, so as to solve the technical problem of data consistency among cache clusters.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method for updating cache data, including:
receiving an updating request, and generating a parent task for updating cache data to each cluster according to the updating request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks;
obtaining the cache data, and respectively updating the cache data to the clusters corresponding to the subtasks;
and respectively sending a version switching request to each cluster, wherein the version switching request carries the version information of the cached data.
Optionally, generating a parent task for updating the cache data to each cluster according to the update request includes:
generating a task registry according to the updating request and updating the cache data to the father task of each cluster;
registering the parent task and the corresponding child tasks thereof in the task registry;
the number of the subtasks is the same as the number of the clusters, and the task registry stores information of the parent task, information of each subtask corresponding to the parent task, task state of each subtask, and version information of the cache data.
Optionally, the obtaining the cache data and updating the cache data to the clusters corresponding to the respective subtasks respectively includes:
and for each subtask, obtaining the cache data, updating the cache data to a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Optionally, the sending the version switching request to each cluster respectively includes:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
if yes, respectively sending version switching requests to the clusters;
if not, constructing an abnormal task according to the subtask with the task state being failure, and storing the abnormal task into an abnormal task table; and the abnormal task table stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Optionally, after saving the exception task in the exception task table, the method further includes:
scanning the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
Optionally, the task metadata includes a task name, a reusable component name for performing the task, and a method for performing the task;
scanning the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy, wherein the method comprises the following steps:
scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism so as to call a method for executing the task, thereby executing the abnormal task.
Optionally, the version information of the cached data includes a service identifier and timestamp information of the cached data.
In addition, according to another aspect of the embodiments of the present invention, there is provided an apparatus for updating cache data, including:
the registration module is used for receiving an updating request and generating a parent task for updating the cache data to each cluster according to the updating request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks;
the updating module is used for acquiring the cache data and respectively updating the cache data to the clusters corresponding to the subtasks;
and the switching module is used for respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data.
Optionally, the registration module is further configured to:
generating a task registry according to the updating request and updating the cache data to the father task of each cluster;
registering the parent task and the corresponding child tasks thereof in the task registry;
the number of the subtasks is the same as the number of the clusters, and the task registry stores information of the parent task, information of each subtask corresponding to the parent task, task state of each subtask, and version information of the cache data.
Optionally, the update module is further configured to:
and for each subtask, obtaining the cache data, updating the cache data to a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Optionally, the switching module is further configured to:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
if yes, respectively sending version switching requests to the clusters;
if not, constructing an abnormal task according to the subtask with the task state being failure, and storing the abnormal task into an abnormal task table; and the abnormal task table stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Optionally, the method further comprises:
the recovery module is used for scanning the abnormal task table after the abnormal task is stored in the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
Optionally, the task metadata includes a task name, a reusable component name for performing the task, and a method for performing the task;
the recovery module is further to:
scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism so as to call a method for executing the task, thereby executing the abnormal task.
Optionally, the version information of the cached data includes a service identifier and timestamp information of the cached data.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any of the embodiments described above.
According to another aspect of the embodiments of the present invention, there is also provided a computer readable medium, on which a computer program is stored, which when executed by a processor implements the method of any of the above embodiments.
One embodiment of the above invention has the following advantages or benefits: because the technical means of respectively updating the cache data into the clusters corresponding to the subtasks and respectively sending the version switching requests to the clusters is adopted, the technical problem of data consistency among the cache clusters in the prior art is solved. In the embodiment of the invention, because the cache cluster can simultaneously keep the historical effective version data, the online service can be quickly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem is quickly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization effectiveness among a plurality of cache clusters, avoid the generation of on-line problems and bring better user experience. Meanwhile, the problem that the capacity of the online service system cannot be expanded due to the bottleneck that the capacity of the cache cluster cannot be expanded any more is solved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a method of updating cache data according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a main flow of a method for updating cache data according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a main flow of a method for updating cache data according to another embodiment of the present invention;
FIG. 4 is a block diagram of a method for updating cached data according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating major blocks of an apparatus for updating cache data according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a method for updating cache data according to an embodiment of the present invention. As an embodiment of the present invention, as shown in fig. 1, the method for updating cache data may include:
step 101, receiving an update request, and generating a parent task for updating cache data to each cluster according to the update request.
When the data in the cache cluster needs to be updated, an update request can be initiated by a user or a timing task. After receiving an update request of the cache data, generating a parent task for updating the cache data to each cache cluster (such as a Redis cluster) according to the update request. The update request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks. Therefore, the service identifier of the cached data and the information (such as the cluster ID) of each cluster can be obtained by parsing the update request. Version information (such as version number) of the cache data can be further generated according to the service identifier and the current timestamp of the cache data, so that different versions of cache data can be distinguished conveniently, and the cache data of each version can be switched and managed conveniently.
Alternatively, the version number of the cached data may be generated by using "service identifier _ server system time", for example, route _2018.08.01-11:00:00, and a version number is generated every time the data is updated. The service identifier is a unique service identifier of the registration task table and is used for distinguishing different services. The server system time is the server system time when the update request is received, and the date format is for convenient preview and for quick switching of the validation version if necessary.
Optionally, generating a parent task for updating the cache data to each cluster according to the update request includes: generating a task registry according to the updating request and updating the cache data to the father task of each cluster; and registering the parent task and the corresponding child tasks thereof in the task registry. The number of the subtasks is the same as that of the clusters, and the task content of each subtask updates the cache data to the cache cluster corresponding to the subtask. For example, the task content of the subtask 1 is to update the cache data to the cache cluster 1, the task content of the subtask 2 is to update the cache data to the cache cluster 2, and the task content of the subtask 3 is to update the cache data to the cache cluster 3, so that each cache cluster can be guaranteed to have the cache data of the version, and subsequent synchronous version switching is facilitated.
Optionally, the task registry stores information of the parent task, information of each child task corresponding to the parent task, a task state of each child task, and version information of the cached data. Specifically, the task registry may store a parent task code, a child task code, a service identifier, a currently effective version number, a modified version number, a task state, and the like, and may also store information such as a creator, creation time, a modifier, modification time, and the like. When generating a parent task, the same number of subtasks can be automatically generated according to the number of clusters, and subtask codes can be generated by adopting a snowfall algorithm.
And 102, acquiring the cache data, and respectively updating the cache data to the clusters corresponding to the sub-tasks.
The corresponding cache data may be obtained from the database according to the changed version number recorded in the task registry, and then the cache data is updated to the cache clusters (e.g., cache cluster 1, cache cluster 2, cache cluster 3, etc.) corresponding to each subtask. During the updating process, the updated cache data is not effective, so that the data consistency is not influenced by serial updating or parallel updating.
Optionally, step 102 includes, for each subtask, obtaining the cache data, updating the cache data to a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful. After data updating is completed on a certain cache cluster, the task state of the subtask corresponding to the cache cluster is updated to be successful in the registered task table, so that the task state of each subtask can be conveniently inquired, and therefore data consistency among clusters is guaranteed during version switching.
And 103, respectively sending version switching requests to the clusters.
And after the data updating of each cache cluster is finished, respectively sending a version switching request to each cache cluster, wherein the version switching request carries the version information of the cache data. After receiving the version switching request, the cache cluster analyzes the version switching request, obtains the version information of the cache data, and switches the currently effective version into the version number carried in the version switching request, so that the cache data of the version can synchronously take effect in each cluster, and the data consistency among the clusters is ensured. It is noted that multiple versions of cache data may be stored in a cache cluster, such as the last few available versions remaining on a line: 1. 2, 3, …, 10, then the switch between these several versions can be made arbitrarily fast, so that the switch to any version can be made fast when there is a problem with the cached data on the line.
Optionally, the sending the version switching request to each cluster respectively includes: scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful; if yes, respectively sending version switching requests to the clusters; if not, constructing an abnormal task according to the subtask with the task state being failure, and storing the abnormal task into an abnormal task table. And the abnormal task table stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Detecting the task state of each subtask in the registered task table is a fundamental guarantee for realizing data consistency among clusters, and if the task states of all subtasks are successful, the latest effective version number is distributed to each cache cluster; however, when one subtask fails to be executed, an abnormal task corresponding to the subtask is generated, and the abnormal task is stored in the abnormal task table. The exception task may be completed by initiating a task recovery step.
According to the various embodiments described above, it can be seen that the technical problem of data consistency among cache clusters in the prior art is solved by the technical means of updating the cache data into the clusters corresponding to the respective subtasks and sending version switching requests to the respective clusters. In the embodiment of the invention, because the cache cluster can simultaneously keep the historical effective version data, the online service can be quickly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem is quickly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization effectiveness among a plurality of cache clusters, avoid the generation of on-line problems and bring better user experience. Meanwhile, the problem that the capacity of the online service system cannot be expanded due to the bottleneck that the capacity of the cache cluster cannot be expanded any more is solved.
Fig. 2 is a schematic diagram illustrating a main flow of a method for updating cache data according to a referential embodiment of the present invention. As another embodiment of the present invention, the method for updating the cache data includes the following steps:
step 201, receiving and analyzing the update request to obtain the information of each cluster and the service identifier of the cache data.
When the data in the cache cluster needs to be updated, an update request can be initiated by a user or a timing task. And after receiving the updating request, analyzing the updating request to obtain the information of each cluster, the version information of the cache data and the service identifier.
Step 202, judging whether the service has a task in execution or not according to the service identifier; if yes, ending; if not, go to step 203.
Specifically, whether the service has the task in execution can be judged by inquiring whether the identification of the service exists in the generated task registry. If the task which is being executed exists, the cache data of the service is indicated to be updated currently, so that the version conflict occurs, the version conflict can be solved through a task version conflict solution, and the method can also be finished directly.
Step 203, generating a task registry and updating the cache data to the parent task of each cluster.
And if the task which is being executed does not exist, which indicates that no version conflict exists, generating a task registry according to the analyzed information and updating the cache data to the parent task of each cluster. The parent task comprises a plurality of subtasks, the number of the subtasks is the same as that of the clusters, and each cluster corresponds to one subtask. Meanwhile, the version information of the cache data can be generated according to the service identifier and the current timestamp of the cache data.
And step 204, registering the parent task and the corresponding child task thereof in the task registry.
And the task registry stores the information of the parent task, the information of each subtask corresponding to the parent task, the task state of each subtask and the version information of the cache data so as to detect the task state of each subtask.
Step 205, obtaining the cache data, and updating the cache data to the clusters corresponding to the respective subtasks respectively.
The corresponding cache data may be obtained from the database according to the changed version number recorded in the task registry, and then the cache data is updated to the cache clusters corresponding to the respective subtasks, respectively. And for each subtask, obtaining the cache data, updating the cache data to a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Step 206, scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful; if yes, go to step 207; if not, go to step 208.
Step 207, sending version switching requests to each cluster respectively.
And if the task state of each subtask is successful, sending a version switching request to each cluster, wherein the version switching request carries the version information of the cached data. After receiving the version switching request, the cache cluster analyzes the version switching request, obtains the version information of the cache data, and switches the currently effective version into the version number carried in the version switching request, so that the cache data of the version can synchronously take effect in each cluster, and the data consistency among the clusters is ensured.
And step 208, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table.
However, when one subtask fails to be executed, an abnormal task corresponding to the subtask is generated, and the abnormal task is stored in the abnormal task table. The exception task may be completed by initiating a task recovery step. And the abnormal task table stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
In addition, in a reference embodiment of the present invention, the detailed implementation of the method for updating the cache data is described in detail in the above-mentioned method for updating the cache data, and therefore the repeated content is not described herein.
Fig. 3 is a schematic diagram illustrating a main flow of a method for updating cache data according to another embodiment of the present invention. As another embodiment of the present invention, the method for updating the cache data may further include the following steps:
step 301, scanning the task state in the abnormal task table as the abnormal task to be recovered.
The abnormal task table can be scanned at regular time through the regular task, so that the task state is the abnormal task to be recovered. The abnormal task table stores information (such as task codes) of the subtasks, retry times, retry strategies, task states and task metadata. The task metadata includes a task Name, a reusable component Name (Bean Name) for performing the task, and a method for performing the task. The abnormal task table can also store information such as creator, creation time, modifier, modification time and the like.
Step 302, analyzing the task metadata of the abnormal task.
And reading the metadata of the abnormal task from an abnormal task table, and analyzing the task metadata of the abnormal task to obtain the task name of the abnormal task, the reusable component name of the execution task and the method for executing the task.
Step 303, generating a reusable component for task execution through a reflection mechanism to call a method for executing a task, so as to execute the abnormal task.
Step 304, judging whether the abnormal task is executed successfully; if yes, go to step 305; if not, go to step 306.
Step 305, updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful, and updating the task state of the subtask corresponding to the abnormal task in the abnormal task table to be recovered.
In addition, in another embodiment of the present invention, the detailed implementation of the method for updating the cache data is described in detail in the above-mentioned method for updating the cache data, and therefore the repeated content is not described herein.
Fig. 4 is a schematic diagram of a framework for implementing a method for updating cache data according to an embodiment of the present invention. As shown in fig. 4, the version control module is responsible for providing functions of task registration, version number generation, and version conflict detection. Specifically, the version control module receives and analyzes the update request to obtain information of each cluster, version information of the cache data and a service identifier; then, judging whether the service has a task in execution or not according to the service identifier; if yes, ending; if not, generating a task registry and updating the cache data to the parent task of each cluster, and registering the parent task and the corresponding sub tasks (such as 831 sub task, 832 sub task and 833 sub task) in the task registry. Wherein the registered task table may be stored in a change control repository.
The cache updating module may obtain corresponding cache data from the database according to the changed version number recorded in the task registry, and then update the cache data to the cache clusters (such as cache cluster 1, cache cluster 2, cache cluster 3, and the like) corresponding to the respective subtasks respectively; and after the updating of the cache data is finished, updating the task state of the subtask in the task registry to be successful.
The task state detection module scans the task state of each subtask in the task registry and judges whether the task state of each subtask is successful or not; if yes, respectively sending version switching requests to the clusters; if not, constructing an abnormal task according to the subtask with the task state being failure, and storing the abnormal task into an abnormal task table.
And the abnormal task recovery module is responsible for regularly scanning the abnormal task table and re-executing the abnormal task according to the recovery strategy. Specifically, the abnormal task recovery module scans the abnormal task table at regular time, so as to obtain the abnormal task with the task state to be recovered; analyzing task metadata of the abnormal task; generating a reusable component for task execution through a reflection mechanism to call a method for executing a task so as to execute the abnormal task; if the execution is successful, updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful, and updating the task state of the subtask corresponding to the abnormal task in the abnormal task table to be recovered; and if the execution is unsuccessful, adding 1 to the retry times of the subtasks corresponding to the abnormal tasks in the abnormal task table.
The cache version management module is a WEB end of the whole framework and can provide functions of cache version query, cache version manual rollback, historical version removal strategy configuration and the like.
The framework provided by the embodiment of the invention enables each application to be accessed with smaller modification cost through abstraction of a bottom data model and mutual cooperation among modules, and infinite replication of a cache cluster is replaced, so that theoretically infinite expansibility is brought, and cache cluster decoupling among the applications is used.
Since a plurality of versions of cache data are stored in the cache cluster, the interface for reading the cache by the client needs to be adapted. In order to make the use of the service transparent to the upper layer, a proxy class of the read cache interface can be generated by adopting an interceptor method (bottom layer principle: bytecode modification), and the latest effective version number is woven in the proxy class.
Fig. 5 is a schematic diagram of main blocks of an apparatus for updating cache data according to an embodiment of the present invention, and as shown in fig. 5, the apparatus 500 for updating cache data includes a registration module 501, an update module 502, and a switching module 503. The registration module 501 is configured to receive an update request, and generate a parent task for updating cache data to each cluster according to the update request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks; the updating module 502 is configured to obtain the cache data, and update the cache data to the clusters corresponding to the respective subtasks respectively; the switching module 503 is configured to send a version switching request to each cluster, where the version switching request carries version information of the cached data.
Optionally, the registration module 501 is further configured to:
generating a task registry according to the updating request and updating the cache data to the father task of each cluster;
registering the parent task and the corresponding child tasks thereof in the task registry;
the number of the subtasks is the same as the number of the clusters, and the task registry stores information of the parent task, information of each subtask corresponding to the parent task, task state of each subtask, and version information of the cache data.
Optionally, the updating module 502 is further configured to:
and for each subtask, obtaining the cache data, updating the cache data to a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Optionally, the switching module 503 is further configured to:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
if yes, respectively sending version switching requests to the clusters;
if not, constructing an abnormal task according to the subtask with the task state being failure, and storing the abnormal task into an abnormal task table; and the abnormal task table stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Optionally, the method further comprises:
the recovery module is used for scanning the abnormal task table after the abnormal task is stored in the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
Optionally, the task metadata includes a task name, a reusable component name for performing the task, and a method for performing the task;
the recovery module is further to:
scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism so as to call a method for executing the task, thereby executing the abnormal task.
Optionally, the version information of the cached data includes a service identifier and timestamp information of the cached data.
According to the various embodiments described above, it can be seen that the technical problem of data consistency among cache clusters in the prior art is solved by the technical means of updating the cache data into the clusters corresponding to the respective subtasks and sending version switching requests to the respective clusters. In the embodiment of the invention, because the cache cluster can simultaneously keep the historical effective version data, the online service can be quickly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem is quickly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization effectiveness among a plurality of cache clusters, avoid the generation of on-line problems and bring better user experience. Meanwhile, the problem that the capacity of the online service system cannot be expanded due to the bottleneck that the capacity of the cache cluster cannot be expanded any more is solved.
It should be noted that, in the implementation of the apparatus for updating cache data according to the present invention, the above method for updating cache data has been described in detail, and therefore, the repeated content is not described herein.
Fig. 6 illustrates an exemplary system architecture 600 to which the method of updating cached data or the apparatus for updating cached data of embodiments of the invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. The terminal devices 601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 601, 602, 603. The background management server may analyze and otherwise process the received data such as the item information query request, and feed back a processing result (for example, target push information, item information — just an example) to the terminal device.
It should be noted that the method for updating the cache data provided by the embodiment of the present invention is generally performed by the server 605, and accordingly, the apparatus for updating the cache data is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program article comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program articles according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a registration module, an update module, and a switch module, where the names of the modules do not in some cases constitute a limitation on the modules themselves.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving an updating request, and generating a parent task for updating cache data to each cluster according to the updating request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks; obtaining the cache data, and respectively updating the cache data to the clusters corresponding to the subtasks; and respectively sending a version switching request to each cluster, wherein the version switching request carries the version information of the cached data.
According to the technical scheme of the embodiment of the invention, because the technical means of respectively updating the cache data into the clusters corresponding to the subtasks and respectively sending the version switching requests to the clusters is adopted, the technical problem of data consistency among the cache clusters in the prior art is solved. In the embodiment of the invention, because the cache cluster can simultaneously keep the historical effective version data, the online service can be quickly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem is quickly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization effectiveness among a plurality of cache clusters, avoid the generation of on-line problems and bring better user experience. Meanwhile, the problem that the capacity of the online service system cannot be expanded due to the bottleneck that the capacity of the cache cluster cannot be expanded any more is solved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for updating cached data, comprising:
receiving an updating request, and generating a parent task for updating cache data to each cluster according to the updating request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks;
obtaining the cache data, and respectively updating the cache data to the clusters corresponding to the subtasks;
and respectively sending a version switching request to each cluster, wherein the version switching request carries the version information of the cached data.
2. The method of claim 1, wherein generating a parent task for updating the cache data to each cluster according to the update request comprises:
generating a task registry according to the updating request and updating the cache data to the father task of each cluster;
registering the parent task and the corresponding child tasks thereof in the task registry;
the number of the subtasks is the same as the number of the clusters, and the task registry stores information of the parent task, information of each subtask corresponding to the parent task, task state of each subtask, and version information of the cache data.
3. The method according to claim 2, wherein the obtaining the cache data and updating the cache data to the clusters corresponding to the respective subtasks respectively comprises:
and for each subtask, obtaining the cache data, updating the cache data to a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
4. The method of claim 3, wherein sending version switch requests to the respective clusters comprises:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
if yes, respectively sending version switching requests to the clusters;
if not, constructing an abnormal task according to the subtask with the task state being failure, and storing the abnormal task into an abnormal task table; and the abnormal task table stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
5. The method of claim 4, wherein after saving the exception task to an exception task table, further comprising:
scanning the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
6. The method of claim 5, wherein the task metadata includes a task name, a name of a reusable component to perform the task, and a method to perform the task;
scanning the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy, wherein the method comprises the following steps:
scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism so as to call a method for executing the task, thereby executing the abnormal task.
7. The method of claim 1, wherein the version information of the cached data comprises a service identifier and timestamp information of the cached data.
8. An apparatus for updating cached data, comprising:
the registration module is used for receiving an updating request and generating a parent task for updating the cache data to each cluster according to the updating request; the updating request carries information of each cluster and a service identifier of the cache data, and the parent task comprises a plurality of subtasks;
the updating module is used for acquiring the cache data and respectively updating the cache data to the clusters corresponding to the subtasks;
and the switching module is used for respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010037735.7A CN113127443A (en) | 2020-01-14 | 2020-01-14 | Method and device for updating cache data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010037735.7A CN113127443A (en) | 2020-01-14 | 2020-01-14 | Method and device for updating cache data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127443A true CN113127443A (en) | 2021-07-16 |
Family
ID=76771137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010037735.7A Pending CN113127443A (en) | 2020-01-14 | 2020-01-14 | Method and device for updating cache data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127443A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114745438A (en) * | 2022-03-22 | 2022-07-12 | 北京乐我无限科技有限责任公司 | Cache data processing method, device and system for multiple data centers and electronic equipment |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102790788A (en) * | 2011-05-23 | 2012-11-21 | 同济大学 | Grid resource management system |
| US20130125097A1 (en) * | 2011-11-15 | 2013-05-16 | Global Supercomputing Corporation | Method and system for converting a single-threaded software program into an application-specific supercomputer |
| CN104331255A (en) * | 2014-11-17 | 2015-02-04 | 中国科学院声学研究所 | Embedded file system-based reading method for streaming data |
| WO2017041486A1 (en) * | 2015-09-10 | 2017-03-16 | 中兴通讯股份有限公司 | Cluster deployment implementation method and apparatus |
| US9600500B1 (en) * | 2013-06-21 | 2017-03-21 | Amazon Technologies, Inc. | Single phase transaction commits for distributed database transactions |
| CN107133234A (en) * | 2016-02-29 | 2017-09-05 | 阿里巴巴集团控股有限公司 | The method of data cached renewal, apparatus and system |
| CN107315629A (en) * | 2017-06-14 | 2017-11-03 | 北京小米移动软件有限公司 | Task processing method, device and storage medium |
| CN109451065A (en) * | 2018-12-26 | 2019-03-08 | 中电福富信息科技有限公司 | A kind of soft load balancing shunts automated system and its operation method |
| CN109582684A (en) * | 2018-11-30 | 2019-04-05 | 深圳市盟天科技有限公司 | A kind of update method of local cache data, device, server and storage medium |
| CN110175159A (en) * | 2019-05-29 | 2019-08-27 | 京东数字科技控股有限公司 | Method of data synchronization and system for object storage cluster |
| CN110442605A (en) * | 2019-07-11 | 2019-11-12 | 阿里巴巴集团控股有限公司 | The buffer memory management method and device of server |
-
2020
- 2020-01-14 CN CN202010037735.7A patent/CN113127443A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102790788A (en) * | 2011-05-23 | 2012-11-21 | 同济大学 | Grid resource management system |
| US20130125097A1 (en) * | 2011-11-15 | 2013-05-16 | Global Supercomputing Corporation | Method and system for converting a single-threaded software program into an application-specific supercomputer |
| US9600500B1 (en) * | 2013-06-21 | 2017-03-21 | Amazon Technologies, Inc. | Single phase transaction commits for distributed database transactions |
| CN104331255A (en) * | 2014-11-17 | 2015-02-04 | 中国科学院声学研究所 | Embedded file system-based reading method for streaming data |
| WO2017041486A1 (en) * | 2015-09-10 | 2017-03-16 | 中兴通讯股份有限公司 | Cluster deployment implementation method and apparatus |
| CN107133234A (en) * | 2016-02-29 | 2017-09-05 | 阿里巴巴集团控股有限公司 | The method of data cached renewal, apparatus and system |
| CN107315629A (en) * | 2017-06-14 | 2017-11-03 | 北京小米移动软件有限公司 | Task processing method, device and storage medium |
| CN109582684A (en) * | 2018-11-30 | 2019-04-05 | 深圳市盟天科技有限公司 | A kind of update method of local cache data, device, server and storage medium |
| CN109451065A (en) * | 2018-12-26 | 2019-03-08 | 中电福富信息科技有限公司 | A kind of soft load balancing shunts automated system and its operation method |
| CN110175159A (en) * | 2019-05-29 | 2019-08-27 | 京东数字科技控股有限公司 | Method of data synchronization and system for object storage cluster |
| CN110442605A (en) * | 2019-07-11 | 2019-11-12 | 阿里巴巴集团控股有限公司 | The buffer memory management method and device of server |
Non-Patent Citations (2)
| Title |
|---|
| 孟宪福;刘伟伟;: "基于选择性复制前驱任务的DAG调度算法", 计算机辅助设计与图形学学报, no. 06 * |
| 肖熠;鲁永泉;谢思烨;: "一种针对GPU资源的深度学习容器云研究", 中国传媒大学学报(自然科学版), no. 06 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114745438A (en) * | 2022-03-22 | 2022-07-12 | 北京乐我无限科技有限责任公司 | Cache data processing method, device and system for multiple data centers and electronic equipment |
| CN114745438B (en) * | 2022-03-22 | 2024-02-09 | 北京乐我无限科技有限责任公司 | Cache data processing method, device and system of multi-data center and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019350B (en) | Data query method and device based on configuration information | |
| CN107783975B (en) | Method and device for synchronous processing of distributed databases | |
| US9721246B2 (en) | Synchronization patterns for mobile applications | |
| US10621211B2 (en) | Language tag management on international data storage | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| CN110727468A (en) | Method and apparatus for managing algorithm models | |
| CN110765187A (en) | Data source route management method and device | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| US10474696B2 (en) | Replication groups for content libraries | |
| CN111198892A (en) | Information processing method, information processing device, electronic equipment and storage medium | |
| CN109960212B (en) | Task sending method and device | |
| CN111338834B (en) | Data storage method and device | |
| US8549007B1 (en) | System and method for indexing meta-data in a computer storage system | |
| CN107526838B (en) | Method and device for database cluster capacity expansion | |
| CN112115113B (en) | Data storage system, method, device, equipment and storage medium | |
| CN113127443A (en) | Method and device for updating cache data | |
| CN110580216B (en) | Application lifting method and device | |
| US10728323B2 (en) | Method and apparatus for operating infrastructure layer in cloud computing architecture | |
| CN112825525A (en) | Method and apparatus for processing transactions | |
| CN109218338B (en) | Information processing system, method and device | |
| CN113076343B (en) | Data query method, device, equipment and storage medium | |
| CN115168384A (en) | Data consistency processing method, device, server and storage medium | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN113760469A (en) | Distributed computing method and device | |
| CN108833147B (en) | Configuration information updating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |