CN117131071B - Data processing method, device, electronic equipment and computer readable medium - Google Patents
Data processing method, device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN117131071B CN117131071B CN202311404917.3A CN202311404917A CN117131071B CN 117131071 B CN117131071 B CN 117131071B CN 202311404917 A CN202311404917 A CN 202311404917A CN 117131071 B CN117131071 B CN 117131071B
- Authority
- CN
- China
- Prior art keywords
- primary key
- application
- read
- upstream
- numerical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 22
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 182
- 238000012545 processing Methods 0.000 claims abstract description 147
- 238000000034 method Methods 0.000 claims abstract description 42
- 230000008569 process Effects 0.000 claims abstract description 20
- 238000004590 computer program Methods 0.000 claims description 9
- 238000011084 recovery Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 14
- 230000004044 response Effects 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a device, electronic equipment and a computer readable medium, and relates to the technical field of data processing, wherein the method comprises the following steps: receiving a data processing request, and acquiring a corresponding upstream application identifier and a downstream application identifier; acquiring an upstream application output table, calling a corresponding first database transaction, and writing a first numerical primary key into a primary key unread log table; calling a downstream application, reading a first numerical primary key in a primary key unread log table, and inquiring a corresponding first numerical primary key value; calling a downstream application, and executing a business processing process based on the first numerical primary key value; calling a second database transaction, writing service processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table; and comparing the primary key unread log table with the primary key read log table, and updating the downstream application output table based on the determined target primary key. Therefore, the data processing flow is simpler, and the occurrence of unread reading of the database is avoided.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a computer readable medium.
Background
At present, in a data computation intensive scene, a plurality of clients read and write a certain data set at the same time, so that high performance and transverse expansibility cannot be maintained, the situation that the data is not heavy, lost and disordered cannot be guaranteed, and the database unread reading problem caused by concurrent reading and writing is easy to occur.
Disclosure of Invention
In view of this, the embodiments of the present application provide a data processing method, apparatus, electronic device, and computer readable medium, which can solve the problems that the existing multiple clients read and write a certain data set at the same time, cannot maintain high performance and lateral expansibility, cannot ensure that the data is not duplicated, lost or disordered, and is easy to generate database unread reading due to concurrent reading and writing.
To achieve the above object, according to one aspect of the embodiments of the present application, there is provided a data processing method, including:
receiving a data processing request, and acquiring a corresponding upstream application identifier and a downstream application identifier;
acquiring an upstream application output table corresponding to the upstream application identifier, and calling a first database transaction corresponding to the upstream application output table to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table;
Calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in a primary key unread log table, and inquiring a corresponding first numerical primary key value in an upstream application output table according to the read first numerical primary key;
invoking a downstream application to execute a business processing process based on the first numerical primary key value and output business processing result data;
calling a second database transaction, writing service processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table;
and comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key.
Optionally, before invoking the downstream application corresponding to the downstream application identifier, the method further includes:
the upstream application output table is locked based on the upstream application identification.
Optionally, locking the upstream application output table based on the upstream application identification includes:
acquiring a preset application identification set, and determining a read-only application identification according to the preset application identification set and an upstream application identification;
and giving the read-only permission of the upstream application output table to the application corresponding to the read-only application identifier.
Optionally, determining the target primary key includes:
and determining the primary key which is not in the read log table and is not in the read log table as the target primary key.
Optionally, writing the service processing result data to a downstream application output table includes:
and in response to the down of the downstream application when the business processing result data is written into the downstream application output table, the first numerical primary key is put back into the primary key unread log table, and in response to the recovery of the application program of the downstream application, the first numerical primary key is obtained again from the primary key unread log table, so that the business processing result data obtained through the first numerical primary key value processing corresponding to the first numerical primary key is written into the downstream application output table.
Optionally, the numerical primary keys in the primary key unread log table and the primary key read log table are generated in an incremental form.
Optionally, before invoking the downstream application corresponding to the downstream application identifier, the method further includes:
responding to an editing request of an application program to an upstream application output table, and acquiring a corresponding application identifier;
and in response to the application identifier being a read-only application identifier, prompting read-only information, and preventing the editing operation of the application program on the upstream application output table.
In addition, the application also provides a data processing device, which comprises:
the receiving unit is configured to receive the data processing request and acquire a corresponding upstream application identifier and a corresponding downstream application identifier;
the acquisition unit is configured to acquire an upstream application output table corresponding to the upstream application identifier, call a first database transaction corresponding to the upstream application output table, and write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table;
the query unit is configured to call a downstream application corresponding to the downstream application identifier so as to read a first numerical primary key in the primary key unread log table and query a corresponding first numerical primary key value in the upstream application output table according to the read first numerical primary key;
the execution unit is configured to call a downstream application to execute a business processing process based on the first numerical type main key value and output business processing result data;
the writing unit is configured to call a second database transaction, write the business processing result data into a downstream application output table, and write the read first numerical primary key into a primary key read log table;
and an updating unit configured to compare the primary key unread log table with the primary key read log table to determine a target primary key and further update the downstream application output table based on the target primary key.
Optionally, the data processing apparatus further comprises a locking unit configured to:
the upstream application output table is locked based on the upstream application identification.
Optionally, the locking unit is further configured to:
acquiring a preset application identification set, and determining a read-only application identification according to the preset application identification set and an upstream application identification;
and giving the read-only permission of the upstream application output table to the application corresponding to the read-only application identifier.
Optionally, the updating unit is further configured to:
and determining the primary key which is not in the read log table and is not in the read log table as the target primary key.
Optionally, the execution unit is further configured to:
and in response to the down of the downstream application when the business processing result data is written into the downstream application output table, the first numerical primary key is put back into the primary key unread log table, and in response to the recovery of the application program of the downstream application, the first numerical primary key is obtained again from the primary key unread log table, so that the business processing result data obtained through the first numerical primary key value processing corresponding to the first numerical primary key is written into the downstream application output table.
Optionally, the numerical primary keys in the primary key unread log table and the primary key read log table are generated in an incremental form.
Optionally, the querying element is further configured to:
responding to an editing request of an application program to an upstream application output table, and acquiring a corresponding application identifier;
and in response to the application identifier being a read-only application identifier, prompting read-only information, and preventing the editing operation of the application program on the upstream application output table.
In addition, the application also provides data processing electronic equipment, which comprises: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data processing method as described above.
In addition, the application also provides a computer readable medium, on which a computer program is stored, which when executed by a processor implements a data processing method as described above.
One embodiment of the above invention has the following advantages or benefits: the method comprises the steps of obtaining corresponding upstream application identifiers and downstream application identifiers by receiving a data processing request; acquiring an upstream application output table corresponding to the upstream application identifier, and calling a first database transaction corresponding to the upstream application output table to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table; calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in a primary key unread log table, and inquiring a corresponding first numerical primary key value in an upstream application output table according to the read first numerical primary key; invoking a downstream application to execute a business processing process based on the first numerical primary key value and output business processing result data; calling a second database transaction, writing service processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table; and comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key. The high-performance streaming computing method based on the database is suitable for real-time and off-line batch processing scenes, the data processing flow is simpler, the situation that the data set is not lost or disordered when read-write processing is carried out on the data set can be guaranteed, and the occurrence of unread reading of the database is avoided.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the present application and are not to be construed as unduly limiting the present application. Wherein:
FIG. 1 is a schematic diagram of the main flow of a data processing method provided according to one embodiment of the present application;
FIG. 2 is a schematic diagram of the main flow of a data processing method provided according to one embodiment of the present application;
FIG. 3 is a schematic view of an application scenario of a data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of the main units of a data processing apparatus according to an embodiment of the present application;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present application may be applied;
fig. 6 is a schematic diagram of a computer system suitable for use in implementing the terminal device or server of the embodiments of the present application.
Detailed Description
Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. The data acquisition, storage, use, processing and the like in the technical scheme meet the relevant regulations of national laws and regulations.
Fig. 1 is a schematic diagram of main flow of a data processing method according to an embodiment of the present application, and as shown in fig. 1, the data processing method includes:
step S101, receiving a data processing request, and acquiring a corresponding upstream application identifier and downstream application identifier.
In this embodiment, the execution body (for example, may be a server) of the data processing method may receive the data processing request by means of a wired connection or a wireless connection. The data processing request may be, for example, a request for reading, writing, storing or updating data, and the content of the data processing request is not particularly limited in the embodiments of the present application. After receiving the data processing request, the executing body can acquire the upstream application identifier and the downstream application identifier carried in the request. The upstream application identifier may be a name or a number corresponding to an upstream application to be used in data processing, and the downstream application identifier may be a name or a number corresponding to a downstream application to be used in data processing.
Step S102, an upstream application output table corresponding to the upstream application identifier is obtained, and a first database transaction corresponding to the upstream application output table is called to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table.
In this embodiment of the present application, the uninterrupted output data table (i.e., the upstream application output table in this application) of the upstream application corresponding to the upstream application identifier and the primary key unread log table use the same database transaction together for writing, and in this embodiment of the present application, the first database transaction is used for representing the data table. To ensure transaction consistency of the data table (i.e., the upstream application output table in this application) with the primary key unread log table.
In this embodiment of the present application, the upstream application output table is used to store the service processing result of the upstream application. The main key is not used for reading the log list and is used for storing the numerical main key of the service processing result of the upstream application program. In a relational database, the first database transaction may be an SQL statement, a set of SQL statements, or an entire program, the transaction being the basic unit of recovery and concurrency control. By using the same database transaction, i.e., the first database transaction, when writing data, for the upstream application output table and the primary key unread log table, the transaction consistency of the two tables can be ensured.
Step S103, calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in the primary key unread log table, and inquiring a corresponding first numerical primary key value in the upstream application output table according to the read first numerical primary key.
And storing the numerical primary key of the service processing result of the upstream application program in the primary key unread log table. The downstream application reads the numerical primary key in the primary key unread log table to obtain corresponding data, namely a first numerical primary key value, from the upstream application output table according to the read numerical primary key.
Specifically, before calling the downstream application corresponding to the downstream application identifier, the method further comprises: the upstream application output table is locked based on the upstream application identification.
The upstream application output table is locked based on the upstream application identifier, so that the upstream application output table can be edited only by the upstream application corresponding to the upstream application identifier (the upstream application output table can refer to writing operation here), and the phenomenon of query unread reading caused by other applications editing the upstream application output table at the same time is avoided.
Step S104, calling the downstream application to execute the business processing process based on the first numerical type primary key value and output business processing result data.
And the downstream application executes the business processing according to the first numerical primary key value obtained from the upstream application output table to obtain business processing result data.
Step S105, calling a second database transaction, writing the business processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table.
In this embodiment of the present application, the uninterrupted output data table (i.e., the downstream application output table in this application) of the downstream application corresponding to the downstream application identifier and the primary key read log table use the same database transaction together for writing, and in this embodiment of the present application, the data table is represented by a second database transaction. To ensure transaction consistency of the data table (i.e., the downstream application output table in this application) with the primary key read log table.
The downstream application performs the submitting of the downstream application business process result data and the writing of the primary key read log table through the second database transaction. Specifically, when the downstream application output table exists, the execution body may determine and call a second database transaction corresponding to the downstream application output table, and write the read first numerical primary key to the primary key read log table based on the second database transaction. To ensure transaction consistency of the downstream application output table and the primary key read log table.
Specifically, the downstream application output table stores the downstream application business processing results. Writing the business processing result data into a downstream application output table, comprising: and in response to the down of the downstream application when the business processing result data is written into the downstream application output table, the first numerical primary key is put back into the primary key unread log table, and in response to the recovery of the application program of the downstream application, the first numerical primary key is obtained again from the primary key unread log table, so that the business processing result data obtained through the first numerical primary key value processing corresponding to the first numerical primary key is written into the downstream application output table.
Under abnormal conditions, if an application downtime occurs when a downstream application submits a service output result, the downstream application submits service processing result data failure, because the downstream application service processing result data output and the numerical primary key writing of the primary key read log table use the same database transaction, the read upstream application numerical primary key also fails to submit (i.e. fails to submit to the primary key read log table), after the application program of the downstream application is restored, the application program of the downstream application queries the upstream application output result according to the comparison difference of the primary key unread log table and the primary key read log table, because the first numerical primary key value (i.e. the upstream application service processing result data) corresponding to the first numerical primary key which is not successfully written into the primary key read log table by downtime is read again by the downstream application program, thereby ensuring that the first numerical primary key value (i.e. the upstream application service processing result data) is read without loss under abnormal conditions. And the downstream application processes the service according to the read first numerical primary key value to obtain service processing result data, and writes the service processing result data into the downstream application output table.
And S106, comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key.
Specifically, determining the target primary key includes: and determining the primary key which is not in the read log table and is not in the read log table as the target primary key.
Specifically, a primary key which is obtained after comparison and is only present in a primary key unread log table and is not read is determined as a target primary key, so that a corresponding primary key value (for example, a) is searched in an upstream application output table through the target primary key, a downstream application performs service processing according to the searched primary key value (for example, a service processing result of the downstream application, and the downstream application output table is updated according to the obtained service processing result of the downstream application.
The embodiment obtains the corresponding upstream application identifier and downstream application identifier by receiving the data processing request; acquiring an upstream application output table corresponding to the upstream application identifier, and calling a first database transaction corresponding to the upstream application output table to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table; calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in a primary key unread log table, and inquiring a corresponding first numerical primary key value in an upstream application output table according to the read first numerical primary key; calling a downstream application to execute a business processing process based on the first numerical primary key value, outputting business processing result data, and writing the business processing result data into a downstream application output table; calling a second database transaction corresponding to the downstream application output table to write the read first numerical primary key into the primary key read log table; and comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key. The high-performance streaming computing method based on the database is suitable for real-time and off-line batch processing scenes, the data processing flow is simpler, the situation that the data set is not lost or disordered when read-write processing is carried out on the data set can be guaranteed, and the occurrence of unread reading of the database is avoided.
Fig. 2 is a main flow diagram of a data processing method according to an embodiment of the present application, and as shown in fig. 2, the data processing method includes:
step S201, receiving a data processing request, and obtaining a corresponding upstream application identifier and downstream application identifier.
The data processing request may be triggered by the end of the processing of the upstream application service, for example, the detection of the processing result of the upstream application service, triggering the sending of the data processing request, and after receiving the data processing request, the executing body may obtain the upstream application identifier corresponding to the upstream application and the downstream application identifier corresponding to the downstream application.
Step S202, an upstream application output table corresponding to the upstream application identifier is obtained, and a first database transaction corresponding to the upstream application output table is called to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table.
The first database transaction is used for executing the writing of the upstream application output table and the writing of the primary key unread log table so as to ensure the transaction consistency of the upstream application output table and the primary key unread log table.
Step S203, a preset application identification set is obtained, and a read-only application identification is determined according to the preset application identification set and the upstream application identification.
And determining the identifiers except the upstream application identifier in the preset application identifier set as read-only application identifiers. It can be understood that the application corresponding to the read-only application identifier only has read-only permission to the upstream application output table, and no permission to edit writing is available. The content and form of the read-only application identifier are not particularly limited in the embodiments of the present application.
Step S204, the read-only application identifier is given to the upstream application output table read-only permission.
And giving the read-only application permission to the application corresponding to the read-only application identifier, so that the application corresponding to the read-only application identifier cannot edit the upstream application output table, thereby avoiding the occurrence of database magic reading and ensuring the accuracy of data writing and reading.
Step S205, calling the downstream application corresponding to the downstream application identifier to read the first numerical primary key in the primary key unread log table, and inquiring the corresponding first numerical primary key value in the upstream application output table according to the read first numerical primary key.
Specifically, before calling the downstream application corresponding to the downstream application identifier, the method further comprises: responding to an editing request of an application program to an upstream application output table, and acquiring a corresponding application identifier; and in response to the application identifier being a read-only application identifier, prompting read-only information, and preventing the editing operation of the application program on the upstream application output table.
When receiving an editing request for the upstream application output table, the execution main body can judge whether an application identifier corresponding to the application sending the request is a read-only application identifier, if so, the execution main body generates prompt information and outputs the prompt information, and calls a blocking program to organize the editing operation of the application sending the request on the upstream application output table so as to ensure the data safety and accuracy of the upstream application output table.
Step S206, calling the downstream application to execute the business processing process based on the first numerical primary key value and output the business processing result data.
And calling the downstream application to take the first numerical primary key value as the input of the business processing process so as to output corresponding business processing result data.
Step S207, call the second database transaction, write the business processing result data into the downstream application output table, and write the read first numerical primary key into the primary key read log table.
Specifically, the numerical primary keys in the primary key unread log table and the primary key read log table are generated in an incremental form.
When the upstream application program writes the log list which is not read by the main key, the written numerical main key is generated in an incremental mode, and when the downstream application program writes the log list which is read by the main key, the written numerical main key is generated in an incremental mode so as to prevent unread reading.
Step S208, comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key.
Specifically, a primary key that exists in a primary key unread log table and that does not exist in a primary key read log table may be determined as a target primary key, that is, a primary key that is not read to be used for processing a service, and the downstream application may acquire a corresponding target primary key value according to the target primary key, use the target primary key value to perform service processing to obtain service processing result data, and update the service processing result data into the downstream application output table.
Fig. 3 is an application scenario diagram of a data processing method according to an embodiment of the present application. The data processing method is applied to scenes of upstream and downstream application cooperative processing business. In the embodiment of the application, the streaming calculation: streaming computation is a data processing model for real-time computation and analysis of infinite or continuous streaming data, in which data enters a system in a continuous stream form, is then processed and converted through a series of operations, and is finally output in a corresponding form. Upstream application output table: and storing the service processing result of the upstream application program. The primary key does not read the log table: and storing the numerical type primary key of the service processing result of the upstream application program. Downstream application output table: and storing the downstream application business processing result. The primary key has read the log table: the numerical type primary key of the output result of the upstream application is stored and read by the downstream application.
The upstream application may configure an output source to which all of the running results (e.g., record n+m+k, record n+m+k+1, record n+m+k+2, …) are written. The downstream application may configure an input source (which may store record n, record n+1, record n+2, …) that points to the same table in the database at the same time (e.g., record n+m+k, record n+m+k+1, record n+m+k+2, … may be stored in the table). Based on this, it can be ensured that the output of the upstream application is just the input of the downstream application. The upstream application can continuously read the output result of the upstream application while continuously outputting the output.
When the upstream application writes the business processing result data into the upstream application output table of the database, the numerical primary key of the upstream application output table is written into the primary key unread log table in the same database transaction, and the downstream application reads the primary key in the primary key unread log table and queries the corresponding primary key value of the upstream application output table according to the primary key to perform business processing.
After the downstream application processes the primary key value read in the upstream application output table, the data of the service processing result is written into the downstream application output table, and the primary key of the read upstream application output table is written into the primary key read log table in the same database transaction.
The downstream application can continuously scan the main key data of the main key unread log table and compare the main key data with the main key data in the main key read log table, so that the downstream application can find the main key which is not read in the upstream application output table, and inquire the main key value recorded in the upstream application output table according to the unread main key.
The data processing method of the embodiment of the application supports the database table as a stream processing carrier, and the upstream and downstream systems can simultaneously perform concurrent insertion and concurrent reading operations on the same table. The intermediate processing result in the business processing process can be recorded, and when the business program is abnormal, the cost for positioning and recovering the program abnormality can be reduced; the service module consumes the data without being lost or disordered, and can ensure that the data read and written by the service module can be accurately processed only once and is read strictly according to the data writing sequence even under the condition of discontinuous primary key values. The method has the advantages of realizing high performance, supporting concurrent processing and transverse expansion, being capable of transversely expanding a single-node service module, supporting writing data to an output source by using a concurrent mode and supporting reading data from an input source by using the concurrent mode. The method can avoid the unread reading problem of the database caused by concurrent reading and writing on the basis of the isolation level of the repeatable reading of the database. Core processing logic in a high performance streaming computing method based on a database: normal processing: when the upstream application writes the service processing result data into the upstream application output table, the numerical primary key of the upstream application output table is written into the primary key unread log table in the same database transaction. The precondition is: the upstream application output table can only be modified by the upstream application program, and other programs can only be read and cannot be modified. Because the upstream application output table and the main key unread log table use the same database transaction when writing data, the transaction consistency of the two tables can be ensured, namely, the upstream application result corresponding to the numerical main key existing in the main key unread log table is necessarily stored in the upstream application output table. The data are not repeated: under normal conditions, when the downstream application submits the service output result, the data of the log table read by the primary key is submitted in the same database transaction. Because the same database transaction is used, it can be guaranteed that each read of upstream data is recorded to the primary key read log table. The downstream application program uses the main key read log table and the main key unread log table to compare and read the upstream business result data, so as to ensure that the upstream data is not read again. Data are not lost: under abnormal conditions, if the downstream application fails to submit the service output result due to application downtime when the downstream application submits the service output result, because the downstream application result output and the main key writing of the main key read log table use the same database transaction, the read upstream application main key value also fails to submit, and after the application program is recovered, the downstream application program queries the upstream application output result according to the comparison difference of the main key unread log table and the main key read log table, because the upstream service data corresponding to the numerical main key which is not successfully written into the main key read log table by downtime (namely, the numerical main key value corresponding to the numerical main key) can be re-read by the downstream application program, thereby ensuring that the upstream application result data is not lost in abnormal conditions. Preventing magic reading: when the upstream and downstream application programs write the output result, the numerical primary keys of the output result are generated in an incremental mode. And the downstream application program reads the output result of the upstream application according to the comparison result of the main key unread log table and the main key read log table. Based on the uniqueness of the compared unread upstream application output main key, the upstream application is incrementally written when writing the output result, so that the business processing result data written into the database can be prevented from being modified by the upstream application program, and meanwhile, as other application programs only have read-only permission on the upstream application output table, the downstream application program can be ensured to query the upstream output result for multiple times according to the same numerical value type main key, only the same upstream output result data can be read, and the unread reading phenomenon during database query can be avoided. The high-performance streaming computing method based on the database can persist the intermediate process data into the database, and is suitable for real-time and offline batch processing scenes.
Fig. 4 is a schematic diagram of main units of a data processing apparatus according to an embodiment of the present application. As shown in fig. 4, the data processing apparatus 400 includes a receiving unit 401, an acquiring unit 402, a querying unit 403, an executing unit 404, a writing unit 405, and an updating unit 406.
The receiving unit 401 is configured to receive the data processing request, and obtain the corresponding upstream application identifier and downstream application identifier.
The obtaining unit 402 is configured to obtain an upstream application output table corresponding to the upstream application identifier, and call a first database transaction corresponding to the upstream application output table, so as to write a first numerical primary key corresponding to the upstream application output table into the primary key unread log table.
The query unit 403 is configured to call the downstream application corresponding to the downstream application identifier, so as to read the first numerical primary key in the primary key unread log table, and query the corresponding first numerical primary key value in the upstream application output table according to the read first numerical primary key.
And an execution unit 404 configured to invoke the downstream application to execute the service processing procedure based on the first numerical primary key value and output service processing result data.
And a writing unit 405 configured to invoke a second database transaction, write the business processing result data into the downstream application output table, and write the read first numerical primary key into the primary key read log table.
An updating unit 406 configured to compare the primary key unread log table and the primary key read log table to determine a target primary key, and further update the downstream application output table based on the target primary key.
In some embodiments, the data processing apparatus further comprises a locking unit, not shown in fig. 4, configured to: the upstream application output table is locked based on the upstream application identification.
In some embodiments, the locking unit is further configured to: acquiring a preset application identification set, and determining a read-only application identification according to the preset application identification set and an upstream application identification; and giving the read-only permission of the upstream application output table to the application corresponding to the read-only application identifier.
In some embodiments, the updating unit 406 is further configured to: and determining the primary key which is not in the read log table and is not in the read log table as the target primary key.
In some embodiments, the execution unit 404 is further configured to: and in response to the down of the downstream application when the business processing result data is written into the downstream application output table, the first numerical primary key is put back into the primary key unread log table, and in response to the recovery of the application program of the downstream application, the first numerical primary key is obtained again from the primary key unread log table, so that the business processing result data obtained through the first numerical primary key value processing corresponding to the first numerical primary key is written into the downstream application output table.
In some embodiments, the numerical primary keys in the primary key unread log table and the primary key read log table are generated in an incremental form.
In some embodiments, the query unit 403 is further configured to: responding to an editing request of an application program to an upstream application output table, and acquiring a corresponding application identifier; and in response to the application identifier being a read-only application identifier, prompting read-only information, and preventing the editing operation of the application program on the upstream application output table.
In this application, the data processing method and the data processing apparatus have a corresponding relationship in terms of implementation content, and therefore, the description is not repeated.
Fig. 5 illustrates an exemplary system architecture 500 in which the data processing methods or data processing apparatus of embodiments of the present application may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 505 via the network 504 using the terminal devices 501, 502, 503 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 501, 502, 503, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be a variety of electronic devices having a data processing screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (by way of example only) providing support for data processing requests submitted by users using the terminal devices 501, 502, 503. The background management server can receive the data processing request and acquire the corresponding upstream application identifier and downstream application identifier; acquiring an upstream application output table corresponding to the upstream application identifier, and calling a first database transaction corresponding to the upstream application output table to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table; calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in a primary key unread log table, and inquiring a corresponding first numerical primary key value in an upstream application output table according to the read first numerical primary key; invoking a downstream application to execute a business processing process based on the first numerical primary key value and output business processing result data; calling a second database transaction, writing service processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table; and comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key. The high-performance streaming computing method based on the database is suitable for real-time and off-line batch processing scenes, the data processing flow is simpler, the situation that the data set is not lost or disordered when read-write processing is carried out on the data set can be guaranteed, and the occurrence of unread reading of the database is avoided.
It should be noted that, the data processing method provided in the embodiment of the present application is generally executed by the server 505, and accordingly, the data processing apparatus is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, a schematic diagram of a computer system 600 suitable for use in implementing the terminal device of an embodiment of the present application is shown. The terminal device shown in fig. 6 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the computer system 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a liquid crystal credit authorization query processor (LCD), and the like, and a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments disclosed herein include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented by software, or may be implemented by hardware. The described units may also be provided in a processor, for example, described as: a processor includes a receiving unit, an acquiring unit, a querying unit, an executing unit, a writing unit, and an updating unit. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
As another aspect, the present application also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, which when executed by one of the devices, cause the device to receive a data processing request, and obtain a corresponding upstream application identifier and downstream application identifier; acquiring an upstream application output table corresponding to the upstream application identifier, and calling a first database transaction corresponding to the upstream application output table to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table; calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in a primary key unread log table, and inquiring a corresponding first numerical primary key value in an upstream application output table according to the read first numerical primary key; invoking a downstream application to execute a business processing process based on the first numerical primary key value and output business processing result data; calling a second database transaction, writing service processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table; and comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key.
According to the technical scheme of the embodiment of the application, the high-performance streaming computing method based on the database is suitable for real-time and off-line batch processing scenes, the data processing flow is simpler, and the situation that the database is unread and read can be avoided when a certain data set is read and written at the same time.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (10)
1. A method of data processing, comprising:
receiving a data processing request, and acquiring a corresponding upstream application identifier and a downstream application identifier;
acquiring an upstream application output table corresponding to the upstream application identifier, and calling a first database transaction corresponding to the upstream application output table to write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table;
calling a downstream application corresponding to the downstream application identifier to read a first numerical primary key in the primary key unread log table, and inquiring a corresponding first numerical primary key value in the upstream application output table according to the read first numerical primary key;
Invoking the downstream application to execute a business processing process based on the first numerical primary key value and output business processing result data;
calling a second database transaction, writing the service processing result data into a downstream application output table, and writing the read first numerical primary key into a primary key read log table;
and comparing the primary key unread log table with the primary key read log table to determine a target primary key, and updating the downstream application output table based on the target primary key.
2. The method of claim 1, wherein prior to the invoking the downstream application identifying the corresponding downstream application, the method further comprises:
locking the upstream application output table based on the upstream application identification.
3. The method of claim 2, wherein the locking the upstream application output table based on the upstream application identification comprises:
acquiring a preset application identification set, and determining a read-only application identification according to the preset application identification set and the upstream application identification;
and giving the read-only application identifier corresponding application read-only permission to the upstream application output table.
4. The method of claim 1, wherein the determining the target primary key comprises:
And determining the primary key which exists in the primary key unread log table and does not exist in the primary key read log table as a target primary key.
5. The method of claim 1, wherein writing the business process result data to a downstream application output table comprises:
and responding to the down of the downstream application when the business processing result data is written into a downstream application output table, replacing the first numerical primary key with the primary key unread log table, and responding to the recovery of the application program of the downstream application, acquiring the first numerical primary key again from the primary key unread log table so as to write the business processing result data obtained through the first numerical primary key value processing corresponding to the first numerical primary key into the downstream application output table.
6. The method of claim 1, wherein the primary key unread log table and the numeric primary keys in the primary key read log table are generated in incremental form.
7. A method according to claim 3, wherein prior to said invoking the downstream application to identify the corresponding downstream application, the method further comprises:
responding to an editing request of an application program to the upstream application output table, and acquiring a corresponding application identifier;
And responding to the application identifier as the read-only application identifier, prompting read-only information, and preventing the application program from editing the upstream application output table.
8. A data processing apparatus, comprising:
the receiving unit is configured to receive the data processing request and acquire a corresponding upstream application identifier and a corresponding downstream application identifier;
the acquisition unit is configured to acquire an upstream application output table corresponding to the upstream application identifier, call a first database transaction corresponding to the upstream application output table, and write a first numerical primary key corresponding to the upstream application output table into a primary key unread log table;
the query unit is configured to call a downstream application corresponding to the downstream application identifier so as to read a first numerical primary key in the primary key unread log table and query a corresponding first numerical primary key value in the upstream application output table according to the read first numerical primary key;
an execution unit configured to invoke the downstream application to execute a business processing process based on the first numerical primary key value, and output business processing result data;
the writing unit is configured to call a second database transaction, write the business processing result data into a downstream application output table, and write the read first numerical primary key into a primary key read log table;
And an updating unit configured to compare the primary key unread log table and the primary key read log table to determine a target primary key, and further update the downstream application output table based on the target primary key.
9. A data processing electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311404917.3A CN117131071B (en) | 2023-10-26 | 2023-10-26 | Data processing method, device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311404917.3A CN117131071B (en) | 2023-10-26 | 2023-10-26 | Data processing method, device, electronic equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117131071A CN117131071A (en) | 2023-11-28 |
| CN117131071B true CN117131071B (en) | 2024-01-26 |
Family
ID=88858639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311404917.3A Active CN117131071B (en) | 2023-10-26 | 2023-10-26 | Data processing method, device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117131071B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020035076A1 (en) * | 2018-08-17 | 2020-02-20 | 第四范式（北京）技术有限公司 | Method and system for visualizing data processing step of machine learning process |
| WO2022134876A1 (en) * | 2020-12-24 | 2022-06-30 | 中兴通讯股份有限公司 | Data synchronization method and apparatus, and electronic device and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7251650B2 (en) * | 2003-08-25 | 2007-07-31 | International Business Machines Corporation | Method, system, and article of manufacture for processing updates to insert operations |
| US10223372B2 (en) * | 2016-01-26 | 2019-03-05 | International Business Machines Corporation | Log synchronization among discrete devices in a computer system |
| US11573930B2 (en) * | 2019-06-03 | 2023-02-07 | Zuora, Inc. | Self-healing data synchronization |
-
2023
- 2023-10-26 CN CN202311404917.3A patent/CN117131071B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020035076A1 (en) * | 2018-08-17 | 2020-02-20 | 第四范式（北京）技术有限公司 | Method and system for visualizing data processing step of machine learning process |
| WO2022134876A1 (en) * | 2020-12-24 | 2022-06-30 | 中兴通讯股份有限公司 | Data synchronization method and apparatus, and electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117131071A (en) | 2023-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108958959B (en) | Method and device for detecting hive data table | |
| CN111144926B (en) | Service request processing method, device, system, electronic equipment and readable medium | |
| US11809216B2 (en) | Providing external access to a processing platform | |
| CN111126948A (en) | Processing method and device for approval process | |
| CN112835885A (en) | Processing method, device and system for distributed table storage | |
| CN116701053B (en) | Method, device, equipment and medium for restoring data backup of production environment database | |
| CN117131071B (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN116775167A (en) | Service processing method, device, electronic equipment and computer readable medium | |
| CN112559233B (en) | Method, device, equipment and computer readable medium for identifying fault type | |
| CN110858199A (en) | Document data distributed computing method and device | |
| CN113946816A (en) | Cloud service-based authentication method and device, electronic equipment and storage medium | |
| CN109740027B (en) | Data exchange method, device, server and storage medium | |
| CN112559547A (en) | Method and device for determining consistency among multiple storage object copies | |
| CN113326038A (en) | Method, apparatus, device, storage medium and program product for providing service | |
| CN111008202A (en) | Distributed transaction processing method and framework | |
| CN113111120B (en) | Service data verification method and device | |
| CN109656519B (en) | Method and device for automatically accessing service data | |
| CN111914065B (en) | Short message content verification method, device, computer system and computer readable medium | |
| CN113268488B (en) | Method and device for data persistence | |
| CN112181937B (en) | Method and device for transferring data | |
| CN117675846A (en) | Cache cluster maintenance method, device, electronic equipment and computer readable medium | |
| CN118069539A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN112527802A (en) | Soft link method and device based on key value database | |
| CN116578572A (en) | Service date changing method, device, equipment, storage medium and program product | |
| CN116701220A (en) | Data synchronization test method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |