CN113779083A - Data processing method, device, server and storage medium - Google Patents

Data processing method, device, server and storage medium Download PDF

Info

Publication number
CN113779083A
CN113779083A CN202110128043.8A CN202110128043A CN113779083A CN 113779083 A CN113779083 A CN 113779083A CN 202110128043 A CN202110128043 A CN 202110128043A CN 113779083 A CN113779083 A CN 113779083A
Authority
CN
China
Prior art keywords
data
field
technical
request
snapshot
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110128043.8A
Other languages
Chinese (zh)
Inventor
赵亚飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Tuoxian Technology Co Ltd
Original Assignee
Beijing Jingdong Tuoxian Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Tuoxian Technology Co Ltd filed Critical Beijing Jingdong Tuoxian Technology Co Ltd
Priority to CN202110128043.8A priority Critical patent/CN113779083A/en
Publication of CN113779083A publication Critical patent/CN113779083A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations
    • G06F16/24556—Aggregation; Duplicate elimination
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2291—User-Defined Types; Storage management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data processing method, a device, a server and a storage medium, in the scheme, in the server responding to data read-write operation requests, when the server receives a plurality of data write requests sent by a plurality of clients, each data write request comprises operation information, the technical field of each data write request is determined according to the operation information, at least one data write request belonging to the same technical field is aggregated according to the technical field of each data write request, and a data snapshot is generated. The data snapshot records the data information of the technical field at the current moment, the current data information can be obtained by accumulating a plurality of operations in the same field, each operation event is not required to be recorded one by one, the data consistency in a high-concurrency scene can be ensured, and the processing performance is better.

Description

Data processing method, device, server and storage medium
Technical Field
The present invention relates to the field of internet technologies, and in particular, to a data processing method, an apparatus, a server, and a storage medium.
Background
In recent years, with the development of internet technology, the increasing number of internet users is both an opportunity and a challenge for each internet company, and only a strong application program can meet the challenge, and the transaction processing under a high concurrency scene is the most important part of the application program.
At present, in order to ensure the consistency of transactions in a high concurrency scenario, there are the following ways: the lock can be used in a single machine, the transaction of the database is realized, and the optimistic lock is used.
However, in the case of a low request amount, the transaction is easy to satisfy, but as the number of requests increases, the requirements on the consistency and performance of the transaction become higher, and the performance of the transaction processing is greatly influenced by the locking or database transaction although the consistency requirement can be satisfied. In summary, no technical scheme capable of ensuring transaction consistency in a high concurrency scene and having high performance exists at present.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, an apparatus, a server, and a storage medium, and provide a technical solution that can ensure transaction consistency in a high concurrency scenario and has a higher performance.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
receiving a plurality of data writing requests sent by a plurality of clients, wherein each data writing request comprises operation information;
determining the technical field of each write data request according to the operation information;
according to the technical field to which each data writing request belongs, at least one data writing request belonging to the same technical field is aggregated, and a data snapshot is generated and used for recording data information of the technical field at the current moment.
In one embodiment, the method further comprises:
and sending the data snapshot corresponding to each technical field to a database for storage.
In a specific embodiment, the aggregating, according to the technical field to which each write data request belongs, at least one write data request belonging to the same technical field, and generating a data snapshot includes:
acquiring at least one write data request of each technical field according to the technical field to which each write data request belongs;
and for each technical field, generating the data snapshot after the execution of at least one write data request corresponding to the technical field is completed, wherein the data snapshot is used for recording data information after the execution of the at least one write data request in the technical field is completed.
In one embodiment, the method further comprises:
receiving a data reading request sent by any client;
determining the technical field of data to be read according to the data reading request;
and acquiring data information from the data snapshot of the technical field at the current moment, and returning the data information to the client.
In a specific embodiment, the obtaining data information from the data snapshot of the technical field at the current time includes:
according to the technical field, acquiring a data snapshot of the current moment corresponding to the technical field from a database;
and acquiring the data information from the data snapshot.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, including:
the system comprises a receiving module, a sending module and a receiving module, wherein the receiving module is used for receiving a plurality of data writing requests sent by a plurality of clients, and each data writing request comprises operation information;
the processing module is used for determining the technical field of each write data request according to the operation information;
the processing module is further configured to aggregate at least one data writing request belonging to the same technical field according to the technical field to which each data writing request belongs, and generate a data snapshot, where the data snapshot is used to record data information of the technical field at the current time.
In one embodiment, the apparatus further comprises:
and the sending module is used for sending the data snapshot corresponding to each technical field to the database for storage.
In a specific embodiment, the processing module is specifically configured to:
acquiring at least one writing data request of each technical field according to the technical field to which each writing data request belongs;
and for each technical field, generating the data snapshot after the execution of at least one write data request corresponding to the technical field is completed, wherein the data snapshot is used for recording data information after the execution of the at least one write data request in the technical field is completed.
In a specific embodiment, the receiving module is further configured to receive a read data request sent by any client;
the processing module is further configured to:
determining the technical field of data to be read according to the data reading request;
and acquiring data information from the data snapshot of the technical field at the current moment, and returning the data information to the client.
In a specific embodiment, the processing module is specifically configured to:
according to the technical field, acquiring a data snapshot of the current moment corresponding to the technical field from a database;
and acquiring the data information from the data snapshot.
In a third aspect, an embodiment of the present invention provides a server, including:
a receiver, a processor, a transmitter; and the number of the first and second groups,
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the data processing method of any of the first aspect via execution of the executable instructions.
In a fourth aspect, the present invention provides a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for processing data according to any one of the first aspect.
In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program is used to implement the data processing method according to any one of the first aspect.
In the scheme, in a server responding to data read-write operation requests, when the server receives a plurality of data write requests sent by a plurality of clients, each data write request comprises operation information, the technical field of each data write request is determined according to the operation information, at least one data write request belonging to the same technical field is aggregated according to the technical field of each data write request, and a data snapshot is generated. The data snapshot records the data information of the technical field at the current moment, the current data information can be obtained by accumulating a plurality of operations in the same field, each operation event is not required to be recorded one by one, the data consistency in a high-concurrency scene can be ensured, and the processing performance is better.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic diagram of an application scenario of a data processing method provided by the present invention;
FIG. 2 is a flowchart illustrating a first embodiment of a data processing method according to the present invention;
FIG. 3 is a flowchart illustrating a second embodiment of a data processing method according to the present invention;
FIG. 4 is a flowchart illustrating a third embodiment of a data processing method according to the present invention;
FIG. 5 is a schematic diagram of a data processing method according to the present invention;
FIG. 6 is a conceptual diagram of a data processing method provided by the present invention;
FIG. 7 is a schematic structural diagram of an embodiment of a data processing apparatus according to the present invention;
fig. 8 is a schematic structural diagram of an embodiment of a server provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments based on the embodiments in the present invention, which can be made by those skilled in the art in light of the present disclosure, are within the scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In the prior art, under the condition of low request quantity, transactions are easy to meet, but with the increase of requests, the requirements on the consistency and performance of the transactions are higher and higher, and the performance of transaction processing is greatly influenced although locking or database transactions can meet the requirement on consistency. Based on the above needs, the problems of the prior art mainly include:
1. locking prevents concurrent modification: the method can prevent the influence of concurrent transactions to a certain extent by a program locking mode, but the method is only suitable for being used under a single machine condition, and the consistency of the transactions cannot be ensured under a cluster environment.
2. The method comprises the following steps of realizing through database transaction: the transaction of the database is the premise that the database guarantees the consistency of the transaction, when the operation is operated in a transaction environment, the database can guarantee that only one record can be modified, but if the number of modified records is large, the locking of the whole table of the database can be caused when the influence of the database cannot be evaluated, all the operations cannot be used, and the requirement of the realization of the database transaction on developers is high.
3. The transaction is implemented using optimistic locks: except for database transactions, optimistic locks can be used to ensure the consistency of the transactions in most cases, and under the optimistic locks, only one of simultaneous modification requests can be successfully modified, and the follow-up modification failure can be caused by unsatisfied conditions.
4. Focusing only on data, the operational behavior cannot be reflected: the traditional realization is a blood loss model, data and behaviors are separated, the data are stored in a database, loaded to an application server when in use and then stored in the database after being processed.
In view of the foregoing problems, embodiments of the present invention provide a data processing method, which is used to implement data operation in a cluster environment, reduce operation failures, improve operation performance, and ensure data and behavior are integrated, and ensure data and operation behavior are integrated.
FIG. 1 is a schematic diagram of an application scenario of a data processing method provided by the present invention; as shown in fig. 1, the scenario includes a plurality of clients running on a device, where the device may be a computer, a personal computer, a smart phone, or other devices capable of sending a data read/write request to a server through a client such as an application program or a browser, the server is configured to process and respond to a write data request and a read data request sent by the client, and the database is configured to store data.
Aiming at the technical problems and scenes, the core thought of the data processing method provided by the invention is as follows: for a concurrent scenario, when a server receives a plurality of write data requests, the server needs to process one by one according to the existing method, resulting in less processing, and changes of data need to be recorded after each request is processed, however, sometimes operations of the plurality of data requests interfere with each other, different operations are performed at the same time, resulting in inconsistent recorded data results, possibly resulting in inaccurate query results in data query, so the inventor finds that, based on the idea of domain driving, recording can be performed after write data operations of the same domain type are completed, that is, the plurality of write data operations are not transaction-dependent, a data snapshot is generated after the concurrent write data operations are completed, data information at the time after the operations is recorded, that is, by recording a change record of behavior event correcting data, and a snapshot is generated at a specific event, so that the data consistency and the query efficiency are ensured.
FIG. 2 is a flowchart illustrating a first embodiment of a data processing method according to the present invention; as shown in fig. 2, in the above scenario, the data processing method is mainly applied to the server side, and specifically includes the following steps:
s101: receiving a plurality of write data requests sent by a plurality of clients, wherein each write data request comprises operation information.
In this step, for the server, the server may receive multiple write data requests sent by multiple clients simultaneously, each write data request includes specific operation information, the operation information may be specifically performed operations, data of the operations, etc., for example, a server receiving a plurality of write data requests for account data of a certain user includes that the user is transferred by several persons at the same time, or the operation information of the roll-in operation is carried out to other people at the same time of the roll-in, the operation information included in the different data writing requests can include the roll-in operation and the roll-in amount, or comprises a roll-out operation, a roll-out amount and the like, or the received data writing request is the operation after the order is placed by the network shopping, the data writing request may include ordering operation, ordering commodity information, ordering commodity quantity, etc.
In this scheme, it should be understood that the received multiple data writing requests sent by multiple clients may be received simultaneously or within a period of time, and the length of the period of time may be set according to an actual scheme.
S102: and determining the technical field of each write data request according to the operation information.
In this step, the server determines a technical field of each write data request according to the operation information in the write data request, where the technical field is used to indicate a type of the write data request, for example: transfer accounts, purchase goods, etc.
S103: according to the technical field to which each data writing request belongs, at least one data writing request belonging to the same technical field is aggregated, and a data snapshot is generated and used for recording data information of the technical field at the current moment.
One specific implementation of this step is: the server obtains at least one write data request (which may be two or more write data) in each technical field according to the technical field to which each write data request belongs, and generates the data snapshot after the execution of the at least one write data request corresponding to the technical field is completed for each technical field, where the data snapshot is used to record data information after the execution of the at least one write data request in the technical field is completed.
In this step, the server may aggregate a plurality of write data requests belonging to the same technical field, for example, if all the transfer operations are transfer operations for a certain account or a certain user, the transfer operations may be aggregated and executed, and then the reddest account or user account data is obtained, so as to generate a data snapshot in the technical field at this moment. That is, only one data snapshot is generated after the parallel multiple data writing requests in the same field are executed, and the data snapshot records the data information at the moment.
If a plurality of data writing requests of a plurality of fields are received at the same time, the requests of each field can be aggregated respectively to obtain the data snapshot corresponding to each field at the moment.
In the data processing method provided in this embodiment, in a server that responds to data read-write operation requests, when the server receives multiple write data requests sent by multiple clients, where each write data request includes operation information, a technical field to which each write data request belongs is determined according to the operation information, and at least one write data request belonging to the same technical field is aggregated according to the technical field to which each write data request belongs, and a data snapshot is generated. The data snapshot records the data information of the technical field at the current moment, the current data information can be obtained by accumulating a plurality of operations in the same field, each operation event is not required to be recorded one by one, the data consistency in a high-concurrency scene can be ensured, and the processing performance is better.
FIG. 3 is a flowchart illustrating a second embodiment of a data processing method according to the present invention; as shown in fig. 3, on the basis of the above embodiment, the data processing method further includes:
s104: and sending the data snapshot corresponding to each technical field to a database for storage.
In this step, the server obtains the data snapshot of each technical field at the current time in the scheme according to the foregoing embodiment, and sends the data snapshot at the current time to the database for storage, so that the server can perform query based on the data snapshot subsequently when reading data.
FIG. 4 is a flowchart illustrating a third embodiment of a data processing method according to the present invention; as shown in fig. 4, on the basis of any of the above embodiments, the data processing method further includes:
s201: and receiving a read data request sent by any client.
In this step, when the user needs to query data, a read data request may be sent through the client, and the same information in the read data request may determine a specific technical field, for example: the data reading request may be to query account amount, or may also be to query inventory, or order information, etc., which is not limited in this embodiment.
S202: and determining the technical field needing to read the data according to the data reading request.
S203: and acquiring data information from the data snapshot at the current moment in the technical field, and returning the data information to the client.
In the above steps, after receiving the data reading request, the server determines the technical field of the data to be read based on the information carried in the data reading request, and then obtains the data snapshot of the current time corresponding to the technical field from the database according to the technical field. Specifically, the server may send a snapshot obtaining request to the database based on the technical field, and the database returns the data snapshot to the server based on the technical field. And the server continues to acquire the data information from the data snapshot.
And after the server acquires the data information, the data information is returned to the client side, and the process of reading the data is completed.
Based on any of the above embodiments, the data processing method of the present invention will be described below by way of example.
Fig. 5 is a schematic diagram of a data processing method provided by the present invention, and fig. 6 is a conceptual schematic diagram of a data processing method provided by the present invention, as shown in fig. 5 and 6, when a server receives a plurality of write data requests, one for data add (500), another for data add (100), and another for data sub (100), then the three write data requests are all modifications to data in the same domain, in the specific operation process, belonging to the input 500, the input 100 and the output 100, when the data snapshot is generated, the specific processes of the several data operations are respectively recorded and the final execution result at this time, namely the total (500) in the figure is obtained, and after the operations corresponding to the three data writing requests are executed, recording the final data information in the snapshot.
Receiving two data writing requests sub (200) next time, wherein the operations corresponding to the two requests add (100) are output 200, inputting 100, recording the specific processes of the data operations when generating a data snapshot and obtaining the final execution result at the time, namely total (400) in the figure, and recording the final data information in the snapshot after executing the operations corresponding to the two data writing requests.
The core of the data processing method provided by the invention is a system for tracing the source of the event based on the field-driven design. The core content is an event, event tracing is generally used together with a command query responsibility isolation mode (CQRS), the event tracing mode calculates the current state of data not by recording the current state of the data but by accumulating events occurring in the data (for example, the example shown in fig. 5), the event tracing can reflect the change record of the data relative to the operation of the data, more data and operations also provide considerable convenience for only data mining, the event tracing records the events one by one, the data does not need to be locked for modification, and the whole service has higher write data request processing capacity and throughput. The CQRS mode separates the behavior of modification (Command, add, delete, change, which modifies the system state) and Query (Query, which does not modify the system state) from the traffic. Therefore, the logic is clearer, and the targeted optimization of different parts is facilitated.
It should be appreciated that event tracing employs a domain event-based concept to implement persistence of aggregations, each of which is persisted as a series of events in a database. The application retrieves and replays events from the event store to load the aggregation. The core of event tracing is to record events of each operation, change conditions of data before and after the events occur, and even record only the occurrence of the events and the influence of the occurrence of the events, the occurrence of a plurality of events cannot influence each other, and the consistency of the transactions can be ensured only by ensuring that each event operation is accurately and unmistakably recorded, and the consistency of the transactions is important. In this scheme, a transaction is a sequence of database operations that access and potentially manipulate various data items, either all or none of which are an integral unit of work. A transaction consists of all database operations performed between the beginning of the transaction and the end of the transaction.
The event tracing and the CQRS are closely related and complementary, the C (command) records of the data can be ensured not to interfere with each other through the records of the events, the snapshot of the data can be generated through the aggregation of the domain model, the Query (Query) performance of the data is ensured, the reading and the writing of the data are separated, and the reading and the writing of the data are in the most suitable model.
The method is characterized in that under the field-driven design idea, the characteristics of the model are combined, the read model and the write model of the design data are respectively read and modified, the write model ensures that the operation on the data is not lost through recording the events, the writing performance is ensured without transaction dependence and sequence of multiple operations, and the query efficiency is ensured for reading snapshots of multiple events produced according to the field model.
In a true object-oriented design, data and behavior are grouped together (e.g., the Actor model (an Actor refers to the most basic unit of computation) that can receive a message and perform computations based on it. this philosophy is much like the object-oriented language, where an object receives a message (a method call) and then operates on the received message (which method is called)), i.e., the service layer, i.e., the execution logic, holds the data. Each operation request for the data is an event once, the change record of the data is reflected by recording the behavior event, and the snapshot is generated at a specific time, so that the query efficiency is ensured.
FIG. 7 is a schematic structural diagram of an embodiment of a data processing apparatus according to the present invention; as shown in fig. 7, the data processing apparatus 10 includes:
the receiving module 11 is configured to receive multiple write data requests sent by multiple clients, where each write data request includes operation information;
the processing module 12 is configured to determine a technical field to which each write data request belongs according to the operation information;
the processing module 12 is further configured to aggregate at least one write data request belonging to the same technical field according to the technical field to which each write data request belongs, and generate a data snapshot, where the data snapshot is used to record data information of the technical field at the current time.
In a specific embodiment, the data processing apparatus 10 further includes:
and a sending module 13, configured to send the data snapshot corresponding to each technical field to a database for storage.
In a specific embodiment, the processing module 12 is specifically configured to:
acquiring at least one write data request of each technical field according to the technical field to which each write data request belongs;
and for each technical field, generating the data snapshot after the execution of at least one write data request corresponding to the technical field is completed, wherein the data snapshot is used for recording data information after the execution of the at least one write data request in the technical field is completed.
In a specific embodiment, the receiving module 11 is further configured to receive a read data request sent by any client;
the processing module 12 is further configured to:
determining the technical field of data to be read according to the data reading request;
and acquiring data information from the data snapshot of the technical field at the current moment, and returning the data information to the client.
Optionally, the processing module 12 is specifically configured to:
according to the technical field, acquiring a data snapshot of the current moment corresponding to the technical field from a database;
and acquiring the data information from the data snapshot.
The data processing apparatus provided in any of the above embodiments is configured to execute the technical solution in any of the above method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
Fig. 8 is a schematic structural diagram of an embodiment of a server provided in the present invention. As shown in fig. 8, the server 20 includes:
a receiver 21, a processor 22, a transmitter 23; and the number of the first and second groups,
a memory 24 for storing executable instructions of the processor;
wherein the processor 22 is configured to execute the data processing method provided by any of the foregoing method embodiments via execution of the executable instructions.
Alternatively, the memory 24 may be separate or integrated with the processor 22.
When the memory 24 is a separate device from the processor 22, the server may further include:
and a bus 25 for connecting the above devices.
The server is configured to execute the data processing method provided by any of the foregoing method embodiments, and the implementation principle and technical effect of the server are similar, which are not described herein again.
The embodiment of the invention also provides a readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the data processing method provided by any one of the method embodiments.
An embodiment of the present invention further provides a computer program product, which includes a computer program, and the computer program is used for implementing the data processing method provided in any of the foregoing method embodiments when being executed by a processor.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (13)

1. A data processing method, comprising:
receiving a plurality of data writing requests sent by a plurality of clients, wherein each data writing request comprises operation information;
determining the technical field of each write data request according to the operation information;
according to the technical field to which each data writing request belongs, at least one data writing request belonging to the same technical field is aggregated, and a data snapshot is generated and used for recording data information of the technical field at the current moment.
2. The method of claim 1, further comprising:
and sending the data snapshot corresponding to each technical field to a database for storage.
3. The method of claim 1 or 2, wherein the aggregating at least one write data request belonging to the same technical field according to the technical field to which each write data request belongs, and generating the data snapshot comprises:
acquiring at least one write data request of each technical field according to the technical field to which each write data request belongs;
and for each technical field, generating the data snapshot after the execution of at least one write data request corresponding to the technical field is completed, wherein the data snapshot is used for recording data information after the execution of the at least one write data request in the technical field is completed.
4. The method according to claim 1 or 2, characterized in that the method further comprises:
receiving a data reading request sent by any client;
determining the technical field of data to be read according to the data reading request;
and acquiring data information from the data snapshot of the technical field at the current moment, and returning the data information to the client.
5. The method of claim 4, wherein the obtaining data information from the data snapshot of the technical field at the current time comprises:
according to the technical field, acquiring a data snapshot of the current moment corresponding to the technical field from a database;
and acquiring the data information from the data snapshot.
6. A data processing apparatus, comprising:
the system comprises a receiving module, a sending module and a receiving module, wherein the receiving module is used for receiving a plurality of data writing requests sent by a plurality of clients, and each data writing request comprises operation information;
the processing module is used for determining the technical field of each write data request according to the operation information;
the processing module is further configured to aggregate at least one data writing request belonging to the same technical field according to the technical field to which each data writing request belongs, and generate a data snapshot, where the data snapshot is used to record data information of the technical field at the current time.
7. The apparatus of claim 6, further comprising:
and the sending module is used for sending the data snapshot corresponding to each technical field to the database for storage.
8. The apparatus according to claim 6 or 7, wherein the processing module is specifically configured to:
acquiring at least one write data request of each technical field according to the technical field to which each write data request belongs;
and for each technical field, generating the data snapshot after the execution of at least one write data request corresponding to the technical field is completed, wherein the data snapshot is used for recording data information after the execution of the at least one write data request in the technical field is completed.
9. The device according to claim 6 or 7, wherein the receiving module is further configured to receive a read data request sent by any client;
the processing module is further configured to:
determining the technical field of data to be read according to the data reading request;
and acquiring data information from the data snapshot of the technical field at the current moment, and returning the data information to the client.
10. The apparatus of claim 9, wherein the processing module is specifically configured to:
according to the technical field, acquiring a data snapshot of the current moment corresponding to the technical field from a database;
and acquiring the data information from the data snapshot.
11. A server, comprising:
a receiver, a processor, a transmitter; and the number of the first and second groups,
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the data processing method of any of claims 1 to 5 via execution of the executable instructions.
12. A readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing a method for processing data according to any one of claims 1 to 5.
13. A computer program product, comprising a computer program which, when executed by a processor, is adapted to carry out the data processing method of any one of claims 1 to 5.
CN202110128043.8A 2021-01-29 2021-01-29 Data processing method, device, server and storage medium Pending CN113779083A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110128043.8A CN113779083A (en) 2021-01-29 2021-01-29 Data processing method, device, server and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110128043.8A CN113779083A (en) 2021-01-29 2021-01-29 Data processing method, device, server and storage medium

Publications (1)

Publication Number Publication Date
CN113779083A true CN113779083A (en) 2021-12-10

Family

ID=78835460

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110128043.8A Pending CN113779083A (en) 2021-01-29 2021-01-29 Data processing method, device, server and storage medium

Country Status (1)

Country Link
CN (1) CN113779083A (en)

Similar Documents

Publication Publication Date Title
Chandra BASE analysis of NoSQL database
US11068501B2 (en) Single phase transaction commits for distributed database transactions
CN110992038B (en) Transaction processing method, device and equipment
CN104937556A (en) Recovering pages of database
CN111522631A (en) Distributed transaction processing method, device, server and medium
US10922307B2 (en) Automated transaction engine
CN110784498B (en) Personalized data disaster tolerance method and device
US20180300147A1 (en) Database Operating Method and Apparatus
CN104423982A (en) Request processing method and device
CN107391544A (en) Processing method, device, equipment and the computer storage media of column data storage
CN110121694B (en) Log management method, server and database system
CN115114370B (en) Master-slave database synchronization method and device, electronic equipment and storage medium
CN112685391A (en) Service data migration method and device, computer equipment and storage medium
CN111581227A (en) Event pushing method and device, computer equipment and storage medium
CN113779083A (en) Data processing method, device, server and storage medium
Pankowski Consistency and availability of Data in replicated NoSQL databases
US20230066110A1 (en) Creating virtualized data assets using existing definitions of etl/elt jobs
Dhanda Big data storage and analysis
CN113568892A (en) Method and equipment for carrying out data query on data source based on memory calculation
CN113902415A (en) Financial data checking method and device, computer equipment and storage medium
CN113391933A (en) Method for processing fund
CN106484379B (en) A kind of processing method and processing device of application
US20240152520A1 (en) Data query and data storage methods and apparatuses for relation network
US20230014029A1 (en) Local indexing for metadata repository objects
US20240078221A1 (en) Systems and methods of modeling and querying dynamic temporal graph on massive parallel graph processing and storage engine

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination