CN114217986A

CN114217986A - Data processing method, device, equipment, storage medium and product

Info

Publication number: CN114217986A
Application number: CN202111472505.4A
Authority: CN
Inventors: 吴盛圳; 李斌; 朱标亦; 曾楚伟; 利炳键
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-12-03
Filing date: 2021-12-03
Publication date: 2022-03-22

Abstract

The embodiment of the application discloses a data processing method, a data processing device, data processing equipment, a storage medium and a product. The method is applied to the micro service, the micro service comprises a storage service and a shared cache service, and the method comprises the following steps: receiving data processing requests respectively initiated by a plurality of service parties, wherein the data processing requests comprise read operation requests or write operation requests; responding to the received at least one read operation request, and if the fact that the atom variable of the latest version is cached in the common caching service is determined, executing read operation on the atom variable of the latest version cached in the common caching service according to the at least one read operation request; and in response to the received at least one write operation request, sequentially executing write operations on the atomic variables stored in the storage service, and executing synchronous processing on the same atomic variable cached in the common caching service. The technical scheme of the embodiment of the application greatly optimizes the data processing scheme, improves the accuracy of data processing and saves the storage space.

Description

Data processing method, device, equipment, storage medium and product

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

Background

At present, the query for mass data usually adopts an indexing technology, and it can be understood that the indexing technology is to use a storage form of key-value pairs to quickly find data. The bit-based mapping (bitmap) is a storage form of key-value pairs, and a value element value is shared by a plurality of key elements, so that the purpose of saving storage space is achieved.

In the related art, a process (usually including multiple threads) is run on a stand-alone device, and the process running on the stand-alone device creates a bitmap instance in the process, that is, a process has a bitmap instance.

Therefore, how to improve the accuracy of data processing is an urgent problem to be solved.

Disclosure of Invention

In order to solve the foregoing technical problems, embodiments of the present application provide a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, so that the accuracy of data processing can be improved at least to a certain extent.

According to an aspect of an embodiment of the present application, there is provided a data processing method applied to a microservice, where the microservice includes a storage service and a shared cache service, the method including: receiving data processing requests respectively initiated by a plurality of service parties, wherein the data processing requests comprise read operation requests or write operation requests; responding to at least one received read operation request, and if the fact that the atom variable of the latest version is cached in the common cache service is determined, executing read operation on the atom variable of the latest version cached in the common cache service according to the at least one read operation request; and responding to at least one received write operation request, sequentially executing write operation on the atomic variables stored in the storage service, and executing synchronous processing on the same atomic variable cached in the common cache service.

According to an aspect of the embodiments of the present application, there is provided a data processing apparatus applied to a microservice, the microservice including a storage service and a shared cache service, the apparatus including: the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is configured to receive data processing requests respectively initiated by a plurality of service parties, and the data processing requests comprise read operation requests or write operation requests; the reading module is configured to respond to at least one received reading operation request, and if the atom variable of the latest version cached in the common caching service is determined to be cached in the common caching service, the reading operation is executed on the atom variable of the latest version cached in the common caching service according to the at least one reading operation request; and the writing and synchronizing module is configured to respond to at least one received writing operation request, sequentially execute writing operation on the atomic variables stored in the storage service, and execute synchronous processing on the same atomic variable cached in the common caching service.

According to an aspect of an embodiment of the present application, there is provided an electronic device including one or more processors; storage means for storing one or more programs that, when executed by the electronic device, cause the electronic device to implement the data processing method as described above.

According to an aspect of embodiments of the present application, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data processing method as described above.

According to an aspect of embodiments of the present application, there is provided a computer program product comprising computer instructions which, when executed by a processor, implement the data processing method as described above.

In the technical scheme provided by the embodiment of the application, on one hand, as a common cache service is established in the micro-service, each single machine device does not need to establish a cache instance independently, and thus the storage space is saved to the great extent; on the other hand, when a write operation request initiated by a service party is received, after the write operation is performed on the atomic variables stored in the storage service according to the write operation request, the synchronous processing is performed on the same atomic variable cached in the shared cache service, therefore, the atomic variables cached in the common cache service are consistent with the atomic variables stored in the storage service, when a business side initiates a read operation request, it is sufficient to perform a read operation on the atomic variable of the latest version cached in the common caching service, i.e. each business party is the latest version of the atomic variable cached in the common caching service read, therefore, the atomic variables read by all service parties are consistent, and the phenomenon of low data processing accuracy caused by inconsistency of the atomic variables acquired from the cache instances established by all single-machine equipment is avoided.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:

fig. 1 is a schematic diagram of data processing in the related art;

FIG. 2 is a schematic diagram of an exemplary implementation environment in which aspects of embodiments of the present application may be applied;

FIG. 3 is an architectural diagram of an exemplary microservice to which aspects of embodiments of the present application may be applied;

FIG. 4 is a flow chart illustrating a data processing method according to an exemplary embodiment of the present application;

FIG. 5 is a flow chart of a data processing method shown in an exemplary embodiment of the present application;

FIG. 6 is a flow chart illustrating a data processing method according to an exemplary embodiment of the present application;

FIG. 7 is a schematic diagram of read data shown in an exemplary embodiment of the present application;

FIG. 8 is a flowchart of step S403 in the embodiment shown in FIG. 4 in an exemplary embodiment;

FIG. 9 is a flow chart of step S802 in the embodiment shown in FIG. 8 in an exemplary embodiment;

FIG. 10 is a flowchart of step S403 in the embodiment shown in FIG. 4 in an exemplary embodiment;

FIG. 11 is a schematic diagram of write data shown in an exemplary embodiment of the present application;

FIG. 12 is a flow chart diagram illustrating a data processing method in accordance with an exemplary embodiment of the present application;

FIG. 13 is a schematic diagram illustrating read and write data in accordance with an exemplary embodiment of the present application;

FIG. 14 is a block diagram of a data processing device shown in an exemplary embodiment of the present application;

FIG. 15 is a block diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to the present application. Rather, they are merely examples of the same apparatus and methods as some aspects of the present application, as detailed in the appended claims.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

In the present application, the term "plurality" means two or more. "and/or" describe the association relationship of the associated objects, meaning that there may be three relationships, e.g., A and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Before the technical solutions of the embodiments of the present application are described, terms and expressions referred to in the embodiments of the present application are explained, and the terms and expressions referred to in the embodiments of the present application are applied to the following explanations.

Micro-service: dividing an application program into a plurality of independent small services, wherein each service is dedicated to an independent function of the service and runs in a respective independent process, a relatively clear boundary exists between the services, and the services communicate with each other through a lightweight communication mechanism, for example, a Remote Procedure Call (RPC) communication mechanism; the micro-service is very suitable for the Internet application with frequent change of the current requirements, high iteration speed and short delivery cycle.

A bitmap: the value corresponding to a certain element is marked by a bit, and the key is the element. For example, there is a requirement to find out whether a certain number m exists in 20 hundred million random integers, and there is a 32-bit operating system, 4G memory, where in Java, int occupies 4 bytes, and 1 byte is 8 bits; therefore, if each digit is stored by int, that is 20 hundred million int, and thus the occupied space is about (2000000000 × 4/1024/1024/1024) ≈ 7.45G, while if the digit is stored, 20 hundred million is 20 hundred million bits, and the occupied space is about (2000000000/8/1024/1024/1024) ≈ 0.233G, it can be seen that bitmap can save the storage space to a great extent.

Atomic variables: the shared data is called as an atomic variable, which corresponds to a mechanism for solving the security problem caused by the operation of multiple threads to share data under the concurrent condition, namely, one thread modifies the atomic variable, and other threads can see the atomic variable immediately, and the action is similar to a locking mechanism.

Paxos: is a message-passing based consistency algorithm that solves the problem of how a distributed system agrees on a certain value (resolution). In a typical scenario, in a distributed database system, if the initial states of the nodes are consistent and each node executes the same operation sequence, they can finally obtain a consistent state, and in order to ensure that each node executes the same command sequence, a "consistency algorithm" needs to be executed on each instruction to ensure that the instructions seen by each node are consistent. The Paxos algorithm applies to several cases: multiple processes/threads in the single machine equipment reach data consistency; a plurality of clients in a distributed file system or a distributed database concurrently read and write data; consistency of multiple copies in the distributed storage in response to read and write requests.

And (3) Raft: is a consensus algorithm that solves the problem of how a distributed system agrees on a certain value (resolution) as well. The Raft algorithm is a relatively independent sub-problem that breaks down a large number of computational problems into simpler ones relative to the Paxos algorithm.

In the related art, a process (usually including multiple threads) is run on a stand-alone device, and the process running on the stand-alone device creates a bitmap instance in the process, where the bitmap instance is used to copy and cache data in a storage service in the bitmap instance, so that when there is a service demand, the data in the bitmap instance is used for processing. Because a process has a bitmap example, namely each single machine equipment corresponds to one bitmap example, the storage space is wasted to the maximum extent; moreover, because inter-process communication does not support cross-machine communication, data cached in the bitmap instance on each single machine device faces the phenomenon of inconsistency/synchronization, so that a series of problems such as data processing errors and the like are caused, and the accuracy rate of data processing is low.

Referring to fig. 1, a bitmap _1 instance is created by a process 1 corresponding to a stand-alone device 1, wherein when a service requirement 1 exists, cached data (i.e., data in a buffer shown in the figure, wherein the buffer represents storage, the same applies below) is acquired from the bitmap _1 instance and processed; and a bitmap _ n instance is created by the process n corresponding to the stand-alone device n, wherein when the service requirement n exists, the cached data is acquired from the bitmap _ n instance for processing. Therefore, each single machine device corresponds to one bitmap instance, and storage space is wasted to the maximum extent.

Referring back to fig. 1, process 1 modifies the data in the storage service, and the data copied in bitmap _1 is the latest, but since cross-machine communication is not supported, the data copied in bitmap _ n by process n is still the data before modification, i.e. the data is not the latest data, and therefore, when there is a service demand n, the data is processed by using the non-latest data. Therefore, each single machine device corresponds to one bitmap example, and the cached data in different bitmap examples have the phenomenon of inconsistency/synchronization, so that a series of problems such as data processing errors are easily caused, and the accuracy rate of data processing is low.

Based on this, embodiments of the present application provide a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which optimize a data processing scheme, improve accuracy of data processing to a great extent, and save a storage space.

Referring to FIG. 2, FIG. 2 is a schematic diagram of an exemplary implementation environment of the present application; the implementation environment comprises a terminal device 201 and a server 202, wherein the terminal device 201 and the server 202 communicate with each other through a wired or wireless network.

It should be noted that the server 202 is deployed with a micro service, and the micro service includes a storage service and a shared cache service. The storage service is used for storing data, is deployed in a database and can be simply understood as equivalent to a disk; the common caching service is used for caching data acquired from the storage service and is deployed in the memory.

In one embodiment of the present application, please refer to fig. 3, which is a schematic diagram illustrating an exemplary architecture of a microservice. The creation request can be received, and then a bitmap common cache service is created according to the received creation request, the bitmap common cache service and the storage service can be interacted, and specifically, the bitmap common cache service can acquire data from the storage service for caching. Therefore, when a data processing request sent by a business party according to a requirement is received, the cached data is directly acquired from the bitmap common cache service for processing, and therefore, compared with the aspect of acquiring the stored data from the storage service for processing, the efficiency of data processing is improved.

In one embodiment of the present application, the storage service may be a Paxos-based storage service.

In one embodiment of the present application, the storage service may be a Raft-based storage service.

The terminal device 201 may be an electronic device such as a smart phone, a tablet, a notebook computer, a computer, and a vehicle-mounted terminal, which is not limited herein. Meanwhile, the number of the terminal devices 201 may be one or more.

The server 202 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, which is not limited herein. Meanwhile, the number of the servers 202 may be one or more.

In an embodiment of the present application, an application program is installed in the terminal device 201, and the application program can be used by a user, that is, the terminal device 201 serves as a service side, and can send a data processing request to the server 202 according to a requirement, where the data processing request includes a read operation request or a write operation request; accordingly, if the server 202 receives at least one read operation request, the server 202 responds to the received at least one read operation request, and when it is determined that the atomic variable of the latest version is cached in the common caching service, performs a read operation on the atomic variable of the latest version cached in the common caching service according to the at least one read operation request, and if the server 202 receives at least one write operation request, the server 202 sequentially performs write operations on the atomic variables stored in the storage service and performs synchronization processing on the same atomic variable cached in the common caching service in response to the received at least one write operation request.

Based on the application scenarios shown in fig. 2-3, after the technical scheme of the embodiment of the application is adopted, the accuracy of data processing is improved, and the storage space is saved.

Various implementation details of the technical solution of the embodiments of the present application are set forth in detail below:

referring to fig. 4, fig. 4 is a flowchart illustrating a data processing method according to an embodiment of the present application. The data processing method may be executed by the server 202 shown in fig. 2, and as shown in fig. 4, the data processing method at least includes steps S401 to S403, which are described in detail as follows:

step S401, receiving data processing requests respectively initiated by a plurality of service parties, where the data processing requests include a read operation request or a write operation request.

In the embodiment of the application, the read operation request is an instruction initiated by a service party and used for instructing to execute read operation on data in cache service; optionally, the initiating of the read operation request by the service party may specifically be by triggering a thread for reading data and running the thread to perform read operation on data in the cache service, so as to implement reading of the data.

The write operation request in the embodiment of the application is an instruction initiated by a service party and used for indicating to execute write operation on data in the storage service; wherein the write operation request corresponds to a write operation procedure, the write operation procedure including but not limited to a process and a thread. Optionally, the service side may specifically initiate the write operation request by triggering a thread for writing data, and running the thread to perform write operation on the data in the storage service, so as to implement data write.

In an embodiment of the present application, the received data processing requests respectively initiated by the multiple service parties may all be read operation requests.

In an embodiment of the present application, the received data processing requests respectively initiated by the multiple service parties may all be write operation requests.

In an embodiment of the present application, the received data processing requests respectively initiated by the multiple service parties may be partly read operation requests and partly write operation requests.

Step S402, in response to the received at least one read operation request, if it is determined that the atomic variable of the latest version is cached in the common caching service, performing a read operation on the atomic variable of the latest version cached in the common caching service according to the at least one read operation request.

In this embodiment of the present application, if at least one received read operation request is received, it is required to respond to the at least one received read operation request, and it is determined that the atomic variable of the latest version is cached in the common caching service, and at this time, a read operation may be performed on the atomic variable of the latest version cached in the common caching service according to the at least one read operation request.

In the embodiment of the present application, since multiple business parties may concurrently initiate a write operation request, in order to ensure the security of data stored in the storage service, an atomic variable mechanism may be adopted, and data stored in the storage service is referred to as an atomic variable, and correspondingly, data cached in the common cache service is also referred to as an atomic variable.

In an embodiment of the present application, referring to fig. 5, before the process of performing a read operation on the atomic variable cached in the common caching service in the latest version according to at least one read operation request in step S402, the following steps may be further included:

step S501, comparing the version corresponding to the atomic variable stored in the storage service with the version corresponding to the same atomic variable cached in the common caching service;

step S502, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common caching service, replacing the same atomic variable cached in the common caching service with the atomic variable stored in the storage service, and updating the version corresponding to the replaced atomic variable so that the atomic variable cached in the common caching service is the latest version.

That is, in an optional embodiment, it may be determined first whether the atomic variable cached in the common caching service is the latest version of the atomic variable, and specifically, the version corresponding to the atomic variable stored in the storage service may be compared with the version corresponding to the same atomic variable cached in the common caching service; wherein:

in the first case, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common caching service, it represents that the atomic variable cached in the common caching service at this time is not the atomic variable of the latest version; therefore, the same atomic variable cached in the common caching service needs to be replaced by the atomic variable stored in the storage service, so that the atomic variable cached in the common caching service is the latest version, and the version corresponding to the replaced atomic variable is updated at the same time, so as to identify the version to which the replaced atomic variable belongs, and thus, comparison can be performed according to the version corresponding to the atomic variable when necessary.

In the second case, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is equal to the version corresponding to the same atomic variable cached in the common caching service, the atomic variable cached in the common caching service at this time is represented as the atomic variable of the latest version; therefore, no operation is required.

In the third situation, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is lower than the version corresponding to the same atomic variable cached in the common caching service, the atomic variable cached in the common caching service at the moment is represented as the atomic variable of the latest version; therefore, the atomic variable cached in the common caching service needs to be used to replace the same atomic variable stored in the storage service, and the version corresponding to the replaced atomic variable is updated at the same time to identify the version to which the replaced atomic variable belongs, so that comparison can be performed according to the version corresponding to the atomic variable when necessary.

It will be appreciated that since the request to perform a write operation on an atomic variable stored in the storage service in response to the write operation, the atomic variable stored in the storage service is typically the most recent version, i.e., typically in the first case or the second case, and the third case is less likely to occur.

In one embodiment of the present application, the number of received read operation requests is at least two; referring to fig. 6, before the step S501 of comparing the version corresponding to the atomic variable stored in the storage service with the version corresponding to the same atomic variable cached in the common caching service, the method may further include the following steps:

step S601, a read operation request is forwarded to the storage service, and a version corresponding to the stored atomic variable is obtained from the storage service;

step S602, forward all other read operation requests to the common cache service, and obtain the cached versions corresponding to the same atomic variable from the common cache service.

That is, in an optional embodiment, when at least two received read operation requests are received, one read operation request may be forwarded to the storage service, and the version corresponding to the stored atomic variable is obtained from the storage service, where the one read operation request forwarded to the storage service is used to perform a read operation on the atomic variable stored in the storage service; and forwarding all other read operation requests to the common cache service, and acquiring the version corresponding to the cached atomic variable from the common cache service, wherein all other read operation requests forwarded to the common cache service are used for executing read operation on the atomic variable cached in the common cache service.

For example, if there are 100 received read operation requests, at this time, 1 read operation request is forwarded to the storage service, and the version corresponding to the stored atomic variable is obtained from the storage service; and forwarding 99 read operation requests to the common cache service, and acquiring the version corresponding to the cached atomic variable from the common cache service.

In this way, the read operation programs corresponding to a plurality of read operation requests concurrently access the atomic variables cached in the common cache service, while only one read operation program corresponding to one read operation request accesses the atomic variables stored in the storage service, because the rate of performing the read operation on the atomic variables cached in the common cache service is much higher than the rate of performing the read operation on the atomic variables stored in the storage service; therefore, high concurrent reading is realized, namely, the reading efficiency of the data is improved, and the data processing efficiency is further improved.

It should be noted that step S601 and step S602 may be executed in any order, or may be executed in parallel, and in actual application, flexible adjustment may be performed according to a specific application scenario.

For a better understanding of the data reading process, an example is illustrated here, in which:

FIG. 7 is a diagram of exemplary read data. Specifically, firstly, a version corresponding to a stored atomic variable is obtained from a storage service, and a version corresponding to the same cached atomic variable is obtained from a common caching service; then, comparing the version corresponding to the atomic variable stored in the storage service with the version corresponding to the same atomic variable cached in the common cache service, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common cache service, replacing the same atomic variable cached in the common cache service with the atomic variable stored in the storage service, and updating the version corresponding to the replaced atomic variable so as to enable the atomic variable cached in the common cache service to be the latest version; and then, returning the atom variable cached in the common caching service in the latest version to the corresponding business party initiating the read operation request.

Step S403, in response to the received at least one write operation request, sequentially performing write operations on the atomic variables stored in the storage service, and performing synchronization processing on the same atomic variable cached in the common caching service.

In the embodiment of the present application, if at least one received write operation request is received, the received at least one write operation request needs to be responded, and write operations are sequentially performed on atomic variables stored in a storage service, and synchronization processing is performed on the same atomic variable cached in a common cache service.

In one embodiment of the present application, the number of received write operation requests is at least two; referring to fig. 8, the process of sequentially performing write operations on the atomic variables stored in the storage service in step S403 may include the following steps:

step S801, storing the write operation programs corresponding to the write operation requests into a queue;

step S802, sequentially fetching the write operation program from the queue to perform write operation on the atomic variables stored in the storage service.

That is, in an optional embodiment, when the number of the received write operation requests is at least two, the write operation programs corresponding to the respective write operation requests may be stored in the queue, that is, enqueue storage is performed, and then the write operation programs are sequentially taken out from the queue to perform write operation on the atomic variables stored in the storage service.

It can be understood that, since the queue is a first-in first-out mechanism, the write operation programs in the first-in queue will first fetch to perform the write operation on the atomic variables stored in the storage service, and the write operation programs in the later queue will then fetch to perform the write operation on the atomic variables stored in the storage service.

Therefore, through the queue mechanism, the safety of the data stored in the storage service is ensured, and meanwhile, all the write operation programs can be executed on the basis of the sequence, namely the write operation program corresponding to the received write operation request is executed firstly, and then the write operation program corresponding to the received write operation request is executed later, so that a series of problems of data processing errors caused by errors in the write operation sequence executed by the write operation programs are avoided.

In one embodiment of the present application, each write operator in the queue is set to a blocking state; referring to fig. 9, the process of sequentially fetching the write handler from the queue in step S802 to perform a write operation on the atomic variable stored in the storage service may include the following steps:

step S901, taking out the write operation program located at the head of the queue from the queue as a target write operation program, and switching the state of the target write operation program from a blocked state to an operating state;

step S902, performing a write operation on the atomic variable stored in the storage service through the target write operation program in the running state, and after the target write operation program in the running state completes the write operation on the atomic variable stored in the storage service, fetching the write operation program located at the head position of the queue from the queue as the target write operation program to perform the write operation on the atomic variable stored in the storage service until all the write operation programs in the queue are fetched.

That is, in the optional embodiment, the write operation program at the head position of the queue is taken out from the queue as the target write operation program, the state of the target write operation program is switched from the blocking state to the running state, the target write operation program performs the write operation on the atomic variable stored in the storage service, and after the target write operation program completes the write operation on the atomic variable stored in the storage service, the write operation program at the head position of the queue can be taken out from the queue as the target write operation program to perform the write operation on the atomic variable stored in the storage service until all the write operation programs in the queue are taken out, so that the write operation programs corresponding to all the write operation requests perform the write operation on the atomic variable stored in the storage service.

In an optional embodiment, the target write operation program is a write operation program to be currently subjected to a write operation, and therefore, the state of the target write operation program needs to be switched from a blocking state to a running state.

For example, let the write operation programs from head to tail of the queue be a1, a2, a3, a4, a 5; specifically, the write operation program a1 is taken out from the queue as a target write operation program, the state of the write operation program a1 is switched from the blocking state to the running state, the write operation program a1 performs the write operation on the atomic variable stored in the storage service, after the write operation program a1 performs the write operation on the atomic variable stored in the storage service, the write operation program a2 is taken out from the queue as the target write operation program, and the process is repeated until the write operation programs a1, a2, a3, a4 and a5 are taken out, and the write operation programs a1, a2, a3, a4 and a5 perform the write operation on the atomic variable stored in the storage service.

It can be understood that, in the embodiment of the present application, after each write operation is performed on an atomic variable stored in a storage service, the atomic variable after the write operation can be obtained, and at this time, a version corresponding to the atomic variable after the write operation needs to be updated, so that comparison is performed according to the version corresponding to the atomic variable when necessary.

In one embodiment of the present application, each write operator in the queue is set to a blocking state; referring to fig. 10, the process of performing synchronization processing on the same atomic variable cached in the common caching service in step S403 may include the following steps:

step S1001, acquiring an atomic variable after write operation in a storage service and a version corresponding to the atomic variable after write operation;

step S1002, replace the same atomic variable cached in the common caching service with the atomic variable after the write operation in the storage service, and update the version corresponding to the atomic variable after the replacement to the version corresponding to the atomic variable after the write operation.

That is, in the optional embodiment, the atomic variable after the write operation in the storage service and the version corresponding to the atomic variable after the write operation may be acquired, so that the same atomic variable cached in the common cache service is replaced by the atomic variable after the write operation in the storage service, and the version corresponding to the atomic variable after the replacement is updated to the version corresponding to the atomic variable after the write operation.

In an optional embodiment, the obtained atomic variable after the write operation in the storage service may be an atomic variable after the last write operation program in the queue performs the write operation, and correspondingly, the version corresponding to the obtained atomic variable after the write operation may be a version corresponding to an atomic variable after the last write operation program in the queue performs the write operation. Therefore, after all the write operation programs finish the write operation, the same atomic variable cached in the common caching service is replaced according to the atomic variable of the latest version stored in the storage service, and the version corresponding to the replaced atomic variable is updated to the version corresponding to the atomic variable of the latest version, so that the efficiency of synchronous processing is improved.

In an optional embodiment, the obtained atomic variable after the write operation in the storage service may be an atomic variable after each write operation program in the queue performs the write operation, and correspondingly, the version corresponding to the obtained atomic variable after the write operation may be a version corresponding to an atomic variable after each write operation program in the queue performs the write operation. Therefore, after each write operation program finishes executing the write operation, the same atomic variable cached in the common caching service is replaced according to the atomic variable of the latest version stored in the storage service, and the version corresponding to the replaced atomic variable is updated to the version corresponding to the atomic variable of the latest version, so that the accuracy of synchronous processing is improved.

For a better understanding of the writing process of data, an example is illustrated here, in which:

FIG. 11 is a diagram illustrating exemplary write data. Specifically, firstly, write operation is performed on an atomic variable stored in a storage service to obtain the atomic variable after the write operation, and meanwhile, a version corresponding to the atomic variable after the write operation is updated; and then, acquiring the atomic variable after the write operation in the storage service and the version corresponding to the atomic variable after the write operation, at this time, replacing the same atomic variable cached in the common cache service by using the atomic variable after the write operation in the storage service, and updating the version corresponding to the replaced atomic variable to the version corresponding to the atomic variable after the write operation, at this time, the same atomic variable cached in the common cache service is the latest version.

The micro-service in the embodiment of the application comprises a storage service and a common cache service, so that each single machine does not need to independently establish a cache instance, and the storage space is saved; and after the atomic variables stored in the storage service are written according to the writing operation request, because the same atomic variable cached in the common caching service is synchronously processed, the atomic variable cached in the common caching service is consistent with the atomic variable stored in the storage service, when a business party initiates a reading operation request, the business party only needs to execute the reading operation on the atomic variable cached in the common caching service in the latest version, thereby ensuring that the atomic variables read by each business party are consistent, and further improving the accuracy of data processing.

A specific application scenario of the embodiment of the present application is described in detail below:

referring to FIG. 12, a flow chart of an exemplary data processing method is shown; the data processing method may include the following steps, which are described in detail as follows:

step S1201, receiving data processing requests respectively initiated by a plurality of service parties, where the data processing requests include a read operation request or a write operation request.

Optionally, a part of the plurality of service parties initiates a read operation request, and a part of the plurality of service parties initiates a write operation request.

First, a reading process of data is described, in which:

step S1202, in response to the received multiple read operation requests, compares the version corresponding to the atomic variable stored in the storage service with the version corresponding to the same atomic variable cached in the common caching service.

Optionally, in response to receiving multiple read operation requests, at this time, the version corresponding to the stored atomic variable may be obtained from the storage service, and the version corresponding to the cached same atomic variable may be obtained from the common caching service, and then the version corresponding to the atomic variable stored in the storage service may be compared with the version corresponding to the same atomic variable cached in the common caching service, so as to determine whether the same atomic variable cached in the common caching service is the latest version.

In step S1203, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common caching service, the same atomic variable cached in the common caching service is replaced with the atomic variable stored in the storage service, and the version corresponding to the replaced atomic variable is updated so that the atomic variable cached in the common caching service is the latest version.

Optionally, if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common caching service, that is, the atomic variable cached in the common caching service at this time is not the atomic variable of the latest version; therefore, the same atomic variable cached in the common caching service needs to be replaced by the atomic variable stored in the storage service, so that the atomic variable cached in the common caching service is the latest version, and the version corresponding to the replaced atomic variable is updated at the same time, so as to identify the version to which the replaced atomic variable belongs, and thus, comparison can be performed according to the version corresponding to the atomic variable when necessary.

Step S1204, according to a plurality of read operation requests, executing read operation to the atom variable of the latest version cached in the common caching service.

Optionally, the read operation program corresponding to each read operation request may simultaneously perform a read operation on the atom variable cached in the common caching service in the latest version.

Next, a reading process of data is introduced, in which:

step S1205, in response to the received multiple write operation requests, stores the write operation programs corresponding to the respective write operation requests in the queue.

Optionally, in consideration of security of the atomic variables stored in the storage service, the write operation programs corresponding to the respective write operation requests cannot simultaneously perform write operations on the atomic variables stored in the storage service, and therefore, the write operation programs corresponding to the respective write operation requests may be stored in the queue at this time.

In step S1206, the write operation program is sequentially taken out from the queue to perform write operation on the atomic variables stored in the storage service.

Optionally, each write operation program in the queue is set to be in a blocking state, the write operation program at the head position of the queue is taken out from the queue to be used as a target write operation program, and the state of the target write operation program is switched from the blocking state to a running state; and then, writing the atomic variables stored in the storage service through the target write operation program in the running state, and after the target write operation program in the running state finishes the execution of the write operation on the atomic variables stored in the storage service, taking out the write operation program at the head position of the queue from the queue as the target write operation program to write the atomic variables stored in the storage service until all the write operation programs in the queue are taken out.

After each write operation is performed on the atomic variable stored in the storage service, the atomic variable after the write operation is obtained, and the version corresponding to the atomic variable after the write operation is updated at the same time.

Step S1207, acquiring the atomic variable after the write operation in the storage service and the version corresponding to the atomic variable after the write operation.

Optionally, obtaining an atomic variable after the last write operation program in the queue performs the write operation, that is, the atomic variable of the latest version; and acquiring the version corresponding to the atomic variable after the last write operation program in the queue executes the write operation, namely the version corresponding to the atomic variable of the latest version.

Step S1208, replacing the same atomic variable cached in the common caching service with the atomic variable after the write operation in the storage service, and updating the version corresponding to the replaced atomic variable to the version corresponding to the atomic variable after the write operation.

Optionally, the same atomic variable cached in the common caching service is replaced by the atomic variable of the latest version in the storage service, and the version corresponding to the replaced atomic variable is updated to the version corresponding to the atomic variable of the latest version.

It should be noted that, according to the service logic of different application scenarios, if the read operation request is prior to the write operation request, the read operation program of the read operation request needs to be executed first, and then the write operation program of the write operation request needs to be executed, that is, step S1202-step S1204 is executed first, and step S1205-step S1208 are executed later; if the write operation request precedes the read operation request, the write operation procedure of the write operation request needs to be executed first, and then the read operation procedure of the read operation request needs to be executed, that is, the steps S1205 to S1208 are executed first, and then the steps S1202 to S1204 are executed later.

Fig. 13 is a schematic diagram of an exemplary data processing method. For example, it is assumed that the number of received data processing requests is 3, where 1 write operation request corresponds to a write operation program (i.e., thread 1), 2 read operation requests correspond to read operation programs (i.e., thread 2 and thread 3), and thread 1, thread 2 and thread 3 may all call a function compare exchange to perform a compare exchange operation on an atomic variable; meanwhile, according to the service logic, the thread 1 is executed before the thread 2 and the thread 3, so that the thread 1 keeps the running state at this time, and the thread 2 and the thread 3 are set to be in the blocking state, and specifically, the thread 2 and the thread 3 can be blocked on a condition variable (only when the condition is met, the blocking state is switched to the running state); wherein:

the thread 1 in the running state executes write operation on the atomic variables stored in the storage service, and executes synchronous processing on the same atomic variable cached in the common cache service after the write operation is successful, namely, updates the same atomic variable cached in the common cache service;

after the same atomic variable cached in the common caching service is updated, the thread 2 and the thread 3 are waken up, so that the thread 2 and the thread 3 are switched from the blocking state to the running state, and it can be understood that the condition for wakening up the thread 2 and the thread 3 can be used when the same atomic variable cached in the common caching service is updated; and the thread 2 and the thread 3 in the running state respectively execute read operation on the same atomic variable of the latest version cached in the shared caching service.

The micro service comprises a storage service and a common cache service, and the common cache service is updated by comparing a version corresponding to an atomic variable stored in the storage service with a version corresponding to the same atomic variable cached in the common cache service, so that consistency of the atomic variables is ensured; meanwhile, a plurality of concurrent threads access common cache service, only one thread can access storage service, and high concurrent reading is achieved.

Fig. 14 is a block diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 14, the data processing apparatus is applied to a microservice including a storage service and a common cache service, and includes:

a receiving module 1401, configured to receive data processing requests respectively initiated by a plurality of service parties, where the data processing requests include a read operation request or a write operation request;

a read module 1402, configured to respond to the received at least one read operation request, and if it is determined that the atomic variable of the latest version is cached in the common caching service, perform a read operation on the atomic variable of the latest version cached in the common caching service according to the at least one read operation request;

the write and synchronization module 1403 is configured to, in response to the received at least one write operation request, sequentially perform write operations on the atomic variables stored in the storage service, and perform synchronization processing on the same atomic variable cached in the common caching service.

In one embodiment of the present application, the data processing apparatus further includes:

a comparison module configured to compare a version corresponding to an atomic variable stored in the storage service with a version corresponding to the same atomic variable cached in the common caching service;

and the updating module is configured to replace the same atomic variable cached in the common cache service by the atomic variable stored in the storage service and update the version corresponding to the replaced atomic variable so that the atomic variable cached in the common cache service is the latest version if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common cache service.

In one embodiment of the present application, the number of received read operation requests is at least two; the data processing apparatus further includes:

the first forwarding and acquiring module is configured to forward a read operation request to the storage service and acquire a version corresponding to the stored atomic variable from the storage service;

and the second forwarding and acquiring module is configured to forward all other read operation requests to the common cache service and acquire the cached versions corresponding to the same atomic variable from the common cache service.

In an embodiment of the present application, after performing a write operation on an atomic variable stored in a storage service, an atomic variable after the write operation is obtained, and a version corresponding to the atomic variable after the write operation is updated; the write and sync module 1403 includes:

the acquisition unit is configured to acquire the atomic variable after the write operation in the storage service and the version corresponding to the atomic variable after the write operation;

and the updating unit is configured to replace the same atomic variable cached in the common caching service by the atomic variable after the write operation in the storage service, and update the version corresponding to the replaced atomic variable to the version corresponding to the atomic variable after the write operation.

In one embodiment of the present application, the number of received write operation requests is at least two; the write and sync module 1403 includes:

the storage unit is configured to store the write operation programs corresponding to the write operation requests into the queue;

and the fetching unit is configured to sequentially fetch the write operation programs from the queue and execute write operation on the atomic variables stored in the storage service.

In one embodiment of the present application, each write operator in the queue is set to a blocking state; the extraction unit includes:

the switching unit is configured to take out the write operation program at the head position of the queue from the queue as a target write operation program and switch the state of the target write operation program from a blocking state to a running state;

and the writing unit is configured to perform writing operation on the atomic variable stored in the storage service through the target writing operation program in the running state, and after the target writing operation program in the running state completes the writing operation on the atomic variable stored in the storage service, the writing operation program at the head position of the queue is taken out from the queue as the target writing operation program so as to perform writing operation on the atomic variable stored in the storage service until all the writing operation programs in the queue are taken out.

a creation module configured to create a common cache service according to the received creation request; wherein, the common cache service is used for caching the atomic variables acquired from the storage service.

It should be noted that the apparatus provided in the foregoing embodiment and the method provided in the foregoing embodiment belong to the same concept, and the specific manner in which each module and unit execute operations has been described in detail in the method embodiment, and is not described again here.

Embodiments of the present application also provide an electronic device comprising one or more processors and storage for storing one or more programs which, when executed by the electronic device, implement the data processing method as before.

FIG. 15 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

It should be noted that the computer system of the electronic device shown in fig. 15 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 15, the computer system includes a Central Processing Unit (CPU)1501 which can perform various appropriate actions and processes, such as performing the method in the above-described embodiment, according to a program stored in a Read-Only Memory (ROM) 1502 or a program loaded from a storage section 1508 into a Random Access Memory (RAM) 1503. In the RAM 1503, various programs and data necessary for system operation are also stored. The CPU 1501, the ROM 1502, and the RAM 1503 are connected to each other by a bus 1504. An Input/Output (I/O) interface 1505 is also connected to bus 1504.

The following components are connected to the I/O interface 1505: an input portion 1506 including a keyboard, a mouse, and the like; an output section 1507 including a Display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 1508 including a hard disk and the like; and a communication section 1509 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 1509 performs communication processing via a network such as the internet. A drive 1510 is also connected to the I/O interface 1505 as needed. A removable medium 1511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1510 as necessary, so that a computer program read out therefrom is mounted into the storage section 1508 as necessary.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 1509, and/or installed from the removable medium 1511. When the computer program is executed by a Central Processing Unit (CPU)1501, various functions defined in the system of the present application are executed.

It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

Another aspect of the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data processing method as before. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist separately without being incorporated in the electronic device.

Another aspect of the application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method provided in the above embodiments.

The above description is only a preferred exemplary embodiment of the present application, and is not intended to limit the embodiments of the present application, and those skilled in the art can easily make various changes and modifications according to the main concept and spirit of the present application, so that the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method is applied to a micro service, wherein the micro service comprises a storage service and a shared cache service, and the method comprises the following steps:

receiving data processing requests respectively initiated by a plurality of service parties, wherein the data processing requests comprise read operation requests or write operation requests;

responding to at least one received read operation request, and if the fact that the atom variable of the latest version is cached in the common cache service is determined, executing read operation on the atom variable of the latest version cached in the common cache service according to the at least one read operation request;

and responding to at least one received write operation request, sequentially executing write operation on the atomic variables stored in the storage service, and executing synchronous processing on the same atomic variable cached in the common cache service.

2. The method of claim 1, wherein prior to the performing a read operation on a most recent version of an atomic variable cached in the common caching service in accordance with the at least one read operation request, the method further comprises:

comparing the version corresponding to the atomic variable stored in the storage service with the version corresponding to the same atomic variable cached in the common caching service;

and if the comparison result indicates that the version corresponding to the atomic variable stored in the storage service is higher than the version corresponding to the same atomic variable cached in the common cache service, replacing the same atomic variable cached in the common cache service with the atomic variable stored in the storage service, and updating the version corresponding to the replaced atomic variable so as to enable the atomic variable cached in the common cache service to be the latest version.

3. The method of claim 2, wherein the received read operation requests are at least two; before the comparing the version corresponding to the atomic variable stored in the storage service with the version corresponding to the same atomic variable cached in the common caching service, the method further includes:

forwarding a read operation request to the storage service, and acquiring a version corresponding to the stored atomic variable from the storage service;

and forwarding all other read operation requests to the common cache service, and acquiring the cached versions corresponding to the same atomic variable from the common cache service.

4. The method according to claim 1, wherein after performing a write operation on an atomic variable stored in the storage service, obtaining the atomic variable after the write operation, and updating a version corresponding to the atomic variable after the write operation; the executing the synchronous processing on the same atomic variable cached in the common caching service includes:

acquiring an atomic variable after write operation in the storage service and a version corresponding to the atomic variable after write operation;

and replacing the same atomic variable cached in the common caching service by the atomic variable after the write operation in the storage service, and updating the version corresponding to the replaced atomic variable into the version corresponding to the atomic variable after the write operation.

5. The method of claim 1, wherein the received write operation requests are at least two; the sequentially executing write operations on the atomic variables stored in the storage service includes:

storing the write operation program corresponding to each write operation request into a queue;

and sequentially taking out the write operation programs from the queue to execute write operation on the atomic variables stored in the storage service.

6. The method of claim 5, wherein each write operator in the queue is set to a blocking state; the sequentially taking out the write operation programs from the queue to execute write operation on the atomic variables stored in the storage service includes:

taking out the write operation program at the head position of the queue from the queue as a target write operation program, and switching the state of the target write operation program from a blocking state to a running state;

and after the target write operation program in the running state finishes the write operation of the atomic variable stored in the storage service, taking out the write operation program at the head position of the queue from the queue as the target write operation program to perform the write operation on the atomic variable stored in the storage service until all the write operation programs in the queue are taken out.

7. The method of any of claims 1 to 6, further comprising:

creating one common cache service according to the received creation request; wherein the common caching service is used for caching the atomic variables acquired from the storage service.

8. A data processing apparatus, applied to a microservice, the microservice including a storage service and a common cache service, the apparatus comprising:

the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is configured to receive data processing requests respectively initiated by a plurality of service parties, and the data processing requests comprise read operation requests or write operation requests;

the reading module is configured to respond to at least one received reading operation request, and if the atom variable of the latest version cached in the common caching service is determined to be cached in the common caching service, the reading operation is executed on the atom variable of the latest version cached in the common caching service according to the at least one reading operation request;

and the writing and synchronizing module is configured to respond to at least one received writing operation request, sequentially execute writing operation on the atomic variables stored in the storage service, and execute synchronous processing on the same atomic variable cached in the common caching service.

9. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs that, when executed by the electronic device, cause the electronic device to implement the data processing method of any one of claims 1 to 7.

10. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that the computer instructions, when executed by a processor, implement the data processing method of any one of claims 1 to 7.