CN113407324A

CN113407324A - Method and device for processing database operation data

Info

Publication number: CN113407324A
Application number: CN202110720059.8A
Authority: CN
Inventors: 万帮喜; 周俊辉; 杨涛; 陆一鸣; 吴鸣震; 袁首
Original assignee: Zhejiang Taimei Medical Technology Co Ltd
Current assignee: Zhejiang Taimei Medical Technology Co Ltd
Priority date: 2021-06-28
Filing date: 2021-06-28
Publication date: 2021-09-17
Anticipated expiration: 2041-06-28
Also published as: CN113407324B

Abstract

The invention provides a method and a device for processing database operation data, wherein the method comprises the following steps: setting a plurality of processing operations and the sequence of the processing operations of each piece of running data in a plurality of pieces of running data, wherein the running data has a preset processing sequence; dividing and/or merging the plurality of processing operations according to the physical resources consumed by each processing operation in the plurality of processing operations to form N processing stages; adjusting thread resources of each processing stage in the N processing stages when the processing stage runs in the memory to enable the difference value of the running time of each processing stage to be within a preset range; setting a memory barrier at the foremost end of a memory address where data processed in each processing stage is located, wherein the memory barrier of the processing stage in the later order does not exceed the memory barrier of the processing stage in the earlier order; each memory barrier is moved with a corresponding processing of each run data in each processing stage.

Description

Method and device for processing database operation data

Technical Field

The present invention relates to the field of information technologies, and in particular, to a method and an apparatus for processing database operation data.

Background

In the operation of a large data system, a large amount of operation data is generated, and due to the requirement of a specific service specification or system function, when the operation data is processed, a certain sequence is required, at this time, one mode is to process the data in sequence in a serial mode, but when the data volume is large, the processing mode becomes extremely time-consuming, and the efficiency of data processing and system operation is greatly influenced.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a method for processing database running data, which improves the efficiency of data processing while realizing the sequential processing of data, and is beneficial to the efficient running of a system.

In order to solve the technical problem, the invention provides a method for processing database running data, which comprises the following steps: setting a plurality of processing operations of each piece of running data in a plurality of pieces of running data and the sequence of the processing operations, wherein the running data has a preset processing sequence; according to the physical resources consumed by each processing operation in the plurality of processing operations, dividing and/or merging the plurality of processing operations to form N processing stages, wherein the sequence of the N processing stages corresponds to the sequence of the plurality of processing operations; n is a positive integer greater than or equal to 2; adjusting thread resources of each processing stage in the N processing stages when the processing stage runs in a memory, so that the difference value of the running time of each processing stage is in a preset range; setting a memory barrier at the foremost end of a memory address where data processed in each processing stage is located, wherein the memory barrier of the processing stage in the later order does not exceed the memory barrier of the processing stage in the earlier order; each memory barrier is moved with a corresponding processing of each run data in each processing stage.

In an embodiment of the invention, the step of moving each memory barrier with the corresponding processing of each piece of operation data in each processing stage comprises: circularly moving the memory barriers in a defined number of memory address spaces, wherein memory barriers sequentially in a first processing phase do not exceed memory barriers sequentially in a last processing phase.

In one embodiment of the present invention, the memory barrier points to a specific memory logical address.

In an embodiment of the present invention, the predetermined range includes 10 milliseconds or less.

In an embodiment of the present invention, adjusting the thread resource of each of the N processing stages when the processing stage runs in the memory so that the difference of the running time of each processing stage is within a predetermined range includes:

and determining the thread resource of each processing stage when the processing stage runs in the memory according to the task amount required to be processed by each processing stage in the N processing stages, so that the difference value of the running time of each processing stage is in a preset range.

In an embodiment of the present invention, the database operation data includes Binlog data of a MySQL database.

In an embodiment of the invention, the processing operation includes data reception, data parsing, data conversion, data filtering, and/or data transmission.

In an embodiment of the present invention, the processing stage includes a data receiving stage, a data parsing stage, a data processing stage and a data sending stage.

The invention also provides a processing device of the database operation data, which comprises: a memory for storing instructions executable by the processor; and a processor for executing the instructions to implement the method of any preceding claim.

The invention also provides a computer readable medium having stored thereon computer program code which, when executed by a processor, implements a method as in any one of the preceding claims.

Compared with the prior art, the invention has the following advantages: according to the technical scheme, the data processing stages are divided and adjusted, and the memory barriers of the data processing stages are arranged, so that the data are processed in sequence and in parallel, the data processing sequence requirement is met, and the data processing efficiency is greatly improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principle of the invention. In the drawings:

fig. 1 is a flowchart of a processing method of database operation data according to an embodiment of the present application.

Fig. 2A-2B are schematic diagrams illustrating a principle of moving a memory barrier of a method for processing database operation data according to an embodiment of the present application.

Fig. 3A to fig. 3D are schematic process diagrams of a processing method of database operation data according to an embodiment of the present application.

Fig. 4 is a schematic diagram illustrating sequential parallel processing of data by setting a memory barrier in a database operation data processing method according to an embodiment of the present application.

Fig. 5A to 5D are schematic diagrams illustrating a relationship between movement of a memory barrier and sequential and parallel processing of data in a database operation data processing method according to an embodiment of the present application.

Fig. 6 is a schematic diagram illustrating a system implementation environment of a processing device for database operation data according to an embodiment of the present application.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described herein, and thus the present invention is not limited to the specific embodiments disclosed below.

As used herein, the terms "a," "an," "the," and/or "the" are not intended to be inclusive and include the plural unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.

It should be noted that the terms "first", "second", and the like are used to define the components, and are only used for convenience of distinguishing the corresponding components, and the terms have no special meanings unless otherwise stated, and therefore, the scope of protection of the present application is not to be construed as being limited. Further, although the terms used in the present application are selected from publicly known and used terms, some of the terms mentioned in the specification of the present application may be selected by the applicant at his or her discretion, the detailed meanings of which are described in relevant parts of the description herein. Further, it is required that the present application is understood not only by the actual terms used but also by the meaning of each term lying within.

Flow charts are used herein to illustrate operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, various steps may be processed in reverse order or simultaneously. Meanwhile, other operations are added to or removed from these processes.

The embodiment of the application describes a method and a device for processing database operation data.

As shown in fig. 1, the method for processing database operation data includes, step 101, setting a plurality of processing operations and a sequence of the processing operations for each piece of operation data in a plurality of pieces of operation data, where the plurality of pieces of operation data have a predetermined processing sequence; 102, dividing and/or merging the plurality of processing operations according to the physical resources consumed by each processing operation in the plurality of processing operations to form N processing stages, wherein the sequence of the N processing stages corresponds to the sequence of the plurality of processing operations; n is a positive integer greater than or equal to 2; 103, adjusting thread resources of each processing stage in the N processing stages when the processing stage runs in the memory, so that a difference value of the running time of each processing stage is within a predetermined range; step 104, setting a memory barrier at the foremost end of the memory address where the data processed in each processing stage are located, wherein the memory barrier in the later processing stage does not exceed the memory barrier in the earlier processing stage; in step 105, each memory barrier is moved with the corresponding processing of each run data in each processing stage.

Specifically, in step 101, a plurality of processing operations and a sequence of the processing operations of each piece of running data in the plurality of pieces of running data are set. The plurality of pieces of operation data have a predetermined processing order.

In some embodiments, the database operating data includes, for example, Binlog data of a MySQL database, or other types of logging data. The database operation data may also be operation data of other types of databases. MySQL Bilog is a binary log file, and Bilog data can be used for recording data update of MySQL.

The processing operations on the operational data include, for example, data reception, data parsing, data conversion, data filtering, and/or data transmission. Data parsing is, for example, parsing received data according to the format of the data, for example, parsing encoded data in binary form into data that can represent specific meaning.

The plurality of pieces of operation data have a predetermined processing order. For example, for database operation data, the result of performing an addition operation before performing a deletion operation and performing a deletion operation before performing an addition operation may be completely different. For example, if an addition operation is performed first and then a deletion operation is performed, the result may be no new data; the deletion operation is performed first and then the addition operation is performed, and as a result, new data can be generated. Therefore, it is very important to maintain the predetermined processing sequence of the database operation data when the database operation data is processed.

In step 102, according to the physical resources consumed by each processing operation in the plurality of processing operations, dividing and/or merging the plurality of processing operations to form N processing stages, wherein the sequence of the N processing stages corresponds to the sequence of the plurality of processing operations; n is a positive integer greater than or equal to 2.

The physical resources consumed include, for example, memory resources and processor resources. The plurality of processing operations may be divided and/or merged, for example, the received data and the parsed data may be merged into one stage, or the conversion and filtering operations on the data may be specified as one stage, or the filtering operation on the data may be divided into two stages.

In some embodiments, the processing stages that process the data include a data receiving stage, a data parsing stage, a data processing stage, and a data sending stage.

In step 103, the thread resources of each processing stage in the N processing stages when running in the memory are adjusted so that the difference of the running time of each processing stage is within a predetermined range. The predetermined range includes, for example, 10 milliseconds (ms) or less, and more specifically, 1 ms or less, 3 ms, 5 ms, and the like.

In some embodiments, adjusting the thread resources of each of the N processing stages when the processing stage runs in the memory such that the difference in the running time of each processing stage is within a predetermined range comprises:

and determining the thread resource of each processing stage when the processing stage runs in the memory according to the task amount required to be processed by each processing stage in the N processing stages, so that the difference value of the running time of each processing stage is in a preset range. The amount of tasks is determined, for example, by the amount of processing of the data and the complexity of the calculations.

In more particular embodiments, the data receive phase and the data transmit phase may be adjusted to single threaded execution, for example. The adjustment in the data processing stage can be adjusted to multi-thread execution due to more or more complicated operations, and the specific thread number can be further set. Data parsing operations may also be similarly configured.

In step 104, a memory barrier is set at the forefront of the memory address of the data processed in each processing stage, and the memory barrier in the later processing stage does not exceed the memory barrier in the earlier processing stage.

In some embodiments, the memory barriers point to specific memory logical addresses, and may also point to specific physical addresses. The memory logical address can be obtained by encapsulating and mapping the physical address.

In step 105, each memory barrier is moved with the corresponding processing of each run data in each processing stage.

The

memory spaces

201, 202, 203, 204, and 205 corresponding to the memory addresses in fig. 2A and 2B are shown, and the other memory spaces in fig. 2A and 2B are not further labeled. The ellipses in fig. 2A and 2B indicate that there may be more storage space.

Referring to fig. 2A and 2B, a memory barrier is set at the foremost end of the memory address where data processed in each processing stage is located, and the memory barrier in the subsequent processing stage does not exceed the memory barrier in the previous processing stage. Each memory barrier is moved with a corresponding processing of each run data in each processing stage. As previously mentioned, the processing stages for processing data may include, for example, a data receiving stage, a data parsing stage, a data processing stage, and a data sending stage. Accordingly, a memory barrier is set at the forefront of the memory address where the data processed in each processing stage is located. The memory barriers are exemplified by arrows (r), (g), (d), (e) and (g) in fig. 2A and 2B. The meaning of the memory barrier (the 'total bar') indicated by the arrow (c) in fig. 2A and 2B will be explained below.

In some embodiments of the present application, the step of moving each memory barrier with the corresponding processing of each piece of operating data in each processing stage comprises: circularly moving the memory barriers in a defined number of memory address spaces, wherein memory barriers sequentially in a last processing stage do not exceed memory barriers sequentially in a first order. The arrows R in fig. 2A and 2B may indicate the memory barrier in a defined amount of memory space.

Fig. 3A to fig. 3D are schematic process diagrams of a processing method of database operation data according to an embodiment of the present application. In other words, fig. 3A, fig. 3B, fig. 3C, and fig. 3D show schematic diagrams of processes of a processing method of database operation data according to an embodiment of the present application. In fig. 3A-3D, Id may be used as a representation of the data sequence, such as the sequence in which the data is received, and also as a representation of the sequence in which the data is to be processed. If the data is deleted, the sequence is sequentially complemented.

In fig. 3A, 301 represents a storage space corresponding to a memory logical address, for example. As described above, in the technical solution of the present application, the memory barrier is set at the foremost end of the memory address where the data processed in each processing stage is located, and the memory barrier in the subsequent processing stage does not exceed the memory barrier in the previous processing stage. Each memory barrier is moved with a corresponding processing of each run data in each processing stage. The memory barriers are exemplified by arrows (r), (g), (D), and c) in fig. 3A-3D.

In order to prevent the memory barrier in the first processing stage from exceeding the memory barrier in the last processing stage, a memory barrier, such as the 'global bar' in fig. 3A-3D, may be provided at the front end of the memory address where the data whose processing operation has been completed is located. The 'general rail' in fig. 2A-2B can also be referred to in a similar sense. In some cases, the memory barrier of the data transmission phase (r) may also be used as a 'total rail'.

When the memory barriers are moved in a cycle in a limited number of memory address spaces, the arrows R in fig. 3A to 3D can indicate that the memory barriers are moved in a cycle in a limited number of memory address spaces, in this case, the memory barrier (i) in the data receiving stage does not exceed the memory barrier (v) when the memory barrier (i) is in a cycle, i.e., the memory barrier (i.e., the memory barrier at the forefront of the memory address where the data whose processing operation has been completed is located (i.e., the 'total rail').

Referring to fig. 3A to 3D, in fig. 3A, '0 x 1010100 x 1001001' indicates received data, which is stored in a memory address space (which may also be referred to as a memory space corresponding to a memory address) 302, and the received data is analyzed. In fig. 3B, the memory address space 302 stores the analyzed data, and the memory barrier corresponding to the data analysis stage has moved to the forefront position of the memory address 302, so that the analyzed data will be processed in the processing stage. In FIG. 3C, the data at memory address 302 is processed, and the memory barrier C corresponding to the data processing stage has also been moved to the front-most location of memory address 302. The processing stage here is, for example, data filtering, more specifically, a data deleting operation, for example, an illustration of drawing lines on a part of data in the memory address space 302 in fig. 3C. As mentioned above, in some embodiments, the memory barrier points to a particular memory logical address. In particular implementations, the memory barrier is implemented, for example, by a pointer in the programmed code to an address.

The data after the processing stage enters the sending stage, and in fig. 3D, the data in the memory address 302 is sent, and the memory barrier (r) corresponding to the data sending stage has also moved to the foremost position of the memory address 302.

The data sent has been processed, so the memory barrier, ('total rail'), indicating that the data has been processed will also be moved to the front most position of the memory address 302 in the next time period.

The foregoing process described with reference to fig. 3A-3D may indicate that a memory barrier is disposed at the foremost end of a memory address where data processed in each processing stage in the technical solution of the present application is located, and the memory barrier in the subsequent processing stage does not exceed the memory barrier in the previous processing stage; the specific representation of each memory barrier is moved with the corresponding processing of each run data in each processing stage. In the above embodiments, the memory barrier movement may be understood as continuously setting the memory barrier or performing an update setting of the memory barrier.

The step of moving each memory barrier with a corresponding processing of each run data in each processing stage comprises: circularly moving the memory barriers in a defined number of memory address spaces, wherein memory barriers sequentially in a first processing phase do not exceed memory barriers sequentially in a last processing phase. In fig. 3A-3D, since the memory address pointed by the memory barrier (i.e., 'total balustrade') indicates the memory address of the data whose processing operation is completed, the address space after the memory barrier (i.e., 'total balustrade') can be reclaimed or covered as the data has been processed. Therefore, when the memory barriers are moved cyclically in a limited number of memory address spaces, the memory barrier (i) corresponding to the data receiving stage does not exceed the memory barrier (i.e., 'total rail'), because the storage space after the memory barrier (i.e., 'total rail') is a storage space that can be recycled or covered, and the data of the storage space before the memory barrier (i.e., 'total rail') is still in the process of processing, in the embodiment of the technical solution of the present application, the memory barriers are moved cyclically in the limited number of memory address spaces, wherein the memory barrier in the earliest processing stage does not exceed the memory barrier in the rearmost order.

In the technical scheme of the application, when each data processing stage adjusts the thread resource of the data processing stage, the difference value of the running time of each processing stage is within a predetermined range (which can be understood as approximately equal in an expanded manner), by setting the memory barrier, sequential parallel processing of different pieces of running data can be realized, that is, while a previous processing stage is performed on the previous running data, a next processing stage is performed on the next running data, where the previous data and the next data are, for example, two adjacent data, and the previous processing stage and the next processing stage are, for example, two adjacent processing stages.

As illustrated in fig. 4:

receiving first data (data 1) in a first period, wherein the foremost memory address of the first data (or called the foremost memory address of the first data, or called the foremost memory logical address of the first data) is a first address; the memory address or the memory logical address is, for example, a 16-bit or 32-bit address according to a hardware configuration condition of the memory. When implemented by means of pointers, for example pointers, the memory space of the 16-bit or 32-bit address is pointed to.

Receiving second data (data 2) in a second time interval, wherein the memory address at the forefront of the memory address where the second data is located is a second address, and simultaneously moving the memory barrier in an analysis stage to the first address to analyze the first data;

receiving third data (data 3) in a third time interval, wherein the memory address at the forefront of the memory address where the third data is located is the third address, and meanwhile, moving the memory barrier in the analysis stage to the second address, analyzing the second data, moving the memory barrier in the processing stage to the first address, and processing the first data;

receiving fourth data (data 4) in a fourth time period, wherein the memory address at the forefront of the memory address where the fourth data is located is the fourth address, simultaneously moving the memory barrier in the analysis stage to the third address, analyzing the third data, moving the memory barrier in the processing stage to the second address, processing the second data, moving the memory barrier in the sending stage to the first address, and sending the first data;

receiving fifth data (data 5) in a fifth time period, wherein the memory address at the forefront of the memory address where the fifth data is located is the fifth address, simultaneously moving the memory barrier in the analysis stage to the fourth address, analyzing the fourth data, moving the memory barrier in the processing stage to the third address, processing the third data, moving the memory barrier in the sending stage to the second address, and sending the second data.

……

When the available memory address resources have been used for one round in the limited amount of memory address space, the cycle is back to the first address, which may be

When the 4M +1 th data (or referred to as 4M +1 th data) is received in the 4M +1 th period (data 4M +1), the memory address at the forefront of the memory address where the 4M +1 th data is located is the first address;

receiving 4M +2 data (data 2) in a 4M +2 th period, where a memory address at the head of a memory address where the 4M +2 data is located is a second address, and moving a memory barrier in an analysis stage to the first address to analyze the first data;

receiving 4M +3 data (data 3) in a 4M +3 th period, where a memory address at the forefront of a memory address where the 4M +3 data is located is a third address, and simultaneously moving a memory barrier in an analysis stage to the second address, analyzing the second data, and moving a memory barrier in a processing stage to the first address, and processing the first data;

receiving 4M +4 data (data 4) in a 4M +4 th period (or referred to as a 4(M +1) th period), where a memory address at the forefront of a memory address of the 4M +4 data is a fourth address, moving a memory barrier in a parsing stage to the third address, parsing the third data, moving a memory barrier in a processing stage to the second address, processing the second data, moving a memory barrier in a sending stage to the first address, and sending the first data;

receiving 4(M +1) +1 data (data 4(M +1) +1) in a 4(M +1) +1 time period, wherein a memory address at the forefront of a memory address where the 4(M +1) +1 data is located is the fifth address, moving a memory barrier in an analysis stage to the fourth address, analyzing the fourth data, moving a memory barrier in a processing stage to the third address, processing the third data, moving a memory barrier in a sending stage to the second address, and sending the second data.

……

M is a natural number, M may represent the number of rounds (or the number of rounds) a defined number of memory address spaces have been cycled through, M is 0 may represent that a round has not been cycled through, M is 1 may represent that a round has been cycled through, and so on.

The in-order parallel processing of data may also be understood with reference to fig. 3A-3D, for example, in fig. 3A-3D, when data is to be received into memory address space 301, the data already received in memory address space 302 is to be parsed, the data already parsed in memory address space 303 is to be processed, and the data already processed in memory address space 304 is to be transmitted. Correspondingly, the memory barrier corresponding to each processing stage also moves, and the memory barrier (i.e., 'total rail') indicating that the data has been processed also moves correspondingly. When the memory barriers are circularly moved in a limited number of memory address spaces, the memory barrier in the most front processing stage does not exceed the memory barrier in the most rear processing stage, i.e. the memory barrier in the data receiving stage does not exceed the 'total rail' of the memory barriers, so as to prevent the data which is not processed from being covered and the correct processing result cannot be obtained.

In order to more intuitively show the technical scheme of the present application, so as to more clearly understand the scheme, reference may be further made to fig. 5A to 5D, which further show the principle of the memory barrier moving and the process of implementing the sequential parallel processing of data in the database operation data processing method of the present application. Sequential parallel processing may be understood as processing data in a predetermined order and performing processing operations on a plurality of pieces of data in the same period.

In fig. 5A-5D, when the processing time of each processing stage is approximately equal through adjusting the thread resources used by each processing stage, the memory barriers move approximately uniformly (i.e., move sequentially), and at this time, the parallel processing of the operation data in sequence can also be realized, which greatly improves the processing efficiency of the data compared with the serial processing of the data, for example, in the embodiment shown in fig. 5A-5D, when the data size is large, for example, thousands of data or more, the processing time can be replied to 1/4 when the processing time is shortened to the serial processing.

According to the processing method of the database running data, the data are processed in sequence and in parallel through division and adjustment of the data processing stages and setting of the memory barriers of the data processing stages, the requirement of the data processing sequence is met, and meanwhile the data processing efficiency can be greatly improved.

The present application further provides a processing apparatus for database operation data, including: a memory for storing instructions executable by the processor; and a processor for executing the instructions to implement the method as previously described.

Fig. 6 is a schematic diagram illustrating a system implementation environment of a processing device for database operation data according to an embodiment of the present application. The processing device 600 for database execution data may include an internal communication bus 601, a Processor (Processor)602, a Read Only Memory (ROM)603, a Random Access Memory (RAM)604, and a communication port 605. The processing device 600 for database operation data is connected to the network through the communication port to realize connection with other systems or devices. The internal communication bus 601 may enable data communication among the components of the processing device 600 of database operating data. Processor 602 may make the determination and issue a prompt. In some embodiments, the processor 602 may be comprised of one or more processors. The communication port 605 may enable sending and receiving information and data from the network. The database-run-data processing apparatus 600 may also include various forms of program storage units and data storage units, such as a Read Only Memory (ROM)603 and a Random Access Memory (RAM)604, capable of storing various data files for computer processing and/or communication, and possibly program instructions for execution by the processor 602. The processor executes these instructions to implement the main parts of the method. The results processed by the processor are communicated to the user device through the communication port and displayed on the user interface.

The processing device 600 for database operation data described above can be implemented as a computer program, stored in a memory, and recorded in the processor 602 for execution, so as to implement the database operation data processing method of the present application.

The present application also provides a computer readable medium storing computer program code which, when executed by a processor, implements the method of processing database run data of the present application as described above.

Aspects of the present application may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software. The above hardware or software may be referred to as "data block," module, "" engine, "" unit, "" component, "or" system. The processor may be one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), digital signal processing devices (DAPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or a combination thereof. Furthermore, aspects of the present application may be represented as a computer product, including computer readable program code, embodied in one or more computer readable media. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips … …), optical disks (e.g., Compact Disk (CD), Digital Versatile Disk (DVD) … …), smart cards, and flash memory devices (e.g., card, stick, key drive … …).

The computer readable medium may comprise a propagated data signal with the computer program code embodied therein, for example, on a baseband or as part of a carrier wave. The propagated signal may take any of a variety of forms, including electromagnetic, optical, and the like, or any suitable combination. The computer readable medium can be any computer readable medium that can communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device. Program code on a computer readable medium may be propagated over any suitable medium, including radio, electrical cable, fiber optic cable, radio frequency signals, or the like, or any combination of the preceding.

Similarly, it should be noted that in the preceding description of embodiments of the application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the embodiments. This method of disclosure, however, is not intended to require more features than are expressly recited in the claims. Indeed, the embodiments may be characterized as having less than all of the features of a single embodiment disclosed above.

Numerals describing the number of components, attributes, etc. are used in some embodiments, it being understood that such numerals used in the description of the embodiments are modified in some instances by the use of the modifier "about", "approximately" or "substantially". Unless otherwise indicated, "about", "approximately" or "substantially" indicates that the number allows a variation of ± 20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that may vary depending upon the desired properties of the individual embodiments. In some embodiments, the numerical parameter should take into account the specified significant digits and employ a general digit preserving approach. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the range are approximations, in the specific examples, such numerical values are set forth as precisely as possible within the scope of the application.

Although the present application has been described with reference to the present specific embodiments, it will be recognized by those skilled in the art that the foregoing embodiments are merely illustrative of the present application and that various changes and substitutions of equivalents may be made without departing from the spirit of the application, and therefore, it is intended that all changes and modifications to the above-described embodiments that come within the spirit of the application fall within the scope of the claims of the application.

Claims

1. A processing method of database operation data comprises the following steps:

setting a plurality of processing operations of each piece of running data in a plurality of pieces of running data and the sequence of the processing operations, wherein the running data has a preset processing sequence;

according to the physical resources consumed by each processing operation in the plurality of processing operations, dividing and/or merging the plurality of processing operations to form N processing stages, wherein the sequence of the N processing stages corresponds to the sequence of the plurality of processing operations; n is a positive integer greater than or equal to 2;

adjusting thread resources of each processing stage in the N processing stages when the processing stage runs in a memory, so that the difference value of the running time of each processing stage is in a preset range;

setting a memory barrier at the foremost end of a memory address where data processed in each processing stage is located, wherein the memory barrier of the processing stage in the later order does not exceed the memory barrier of the processing stage in the earlier order;

each memory barrier is moved with a corresponding processing of each run data in each processing stage.

2. The method of claim 1, wherein the step of moving each memory barrier with the corresponding processing of each piece of operation data in each processing stage comprises: circularly moving the memory barriers in a defined number of memory address spaces, wherein memory barriers sequentially in a first processing phase do not exceed memory barriers sequentially in a last processing phase.

3. The method of claim 1, wherein the memory barrier points to a specific memory logical address.

4. The method of claim 1, wherein the predetermined range comprises 10 milliseconds or less.

5. The method of claim 1, wherein adjusting the thread resources of each of the N processing stages when the processing stage runs in the memory to make the difference of the running time of each processing stage within a predetermined range comprises:

6. The method for processing the database operation data according to claim 1, wherein the database operation data comprises Binlog data of a MySQL database.

7. The method for processing the database operation data according to claim 1, wherein the processing operation comprises data receiving, data parsing, data conversion, data filtering and/or data sending.

8. The method for processing database operation data according to claim 7, wherein the processing stages comprise a data receiving stage, a data parsing stage, a data processing stage and a data sending stage.

9. A processing apparatus for database operation data, comprising:

a memory for storing instructions executable by the processor; and

a processor for executing the instructions to implement the method of any one of claims 1-8.

10. A computer-readable medium having stored thereon computer program code which, when executed by a processor, implements the method of any of claims 1-8.