CN110765148A - Service data processing method and device - Google Patents
Service data processing method and device Download PDFInfo
- Publication number
- CN110765148A CN110765148A CN201911031520.8A CN201911031520A CN110765148A CN 110765148 A CN110765148 A CN 110765148A CN 201911031520 A CN201911031520 A CN 201911031520A CN 110765148 A CN110765148 A CN 110765148A
- Authority
- CN
- China
- Prior art keywords
- middleware
- results
- target table
- database
- calculation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 10
- 238000004364 calculation method Methods 0.000 claims abstract description 115
- 238000004458 analytical method Methods 0.000 claims abstract description 97
- 230000008859 change Effects 0.000 claims abstract description 77
- 238000000034 method Methods 0.000 claims abstract description 31
- BQCADISMDOOEFD-UHFFFAOYSA-N Silver Chemical compound [Ag] BQCADISMDOOEFD-UHFFFAOYSA-N 0.000 claims description 7
- 229910052709 silver Inorganic materials 0.000 claims description 7
- 239000004332 silver Substances 0.000 claims description 7
- 241001233242 Lontra Species 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 239000000470 constituent Substances 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 238000009472 formulation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the specification provides a service data processing method and device. The method comprises the following steps: the method comprises the steps that an offline computing platform firstly obtains a plurality of analysis results from a first middleware, wherein the analysis results are generated by analyzing a service data change log of a database by the first middleware; secondly, performing offline batch processing on the plurality of analysis results to obtain a first set of calculation results; acquiring a target table generated by respectively performing online calculation on at least one analysis result acquired from the second middleware by the real-time calculation platform; the at least one analysis result belongs to an analysis result generated by analyzing the service data change log of the database by the second middleware; then, comparing the first set with the target table; updating the target table based on the first set when the computation results in the first set and the computation results in the target table are inconsistent.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of computer information processing, and in particular, to a method and an apparatus for processing service data.
Background
Global mobility management of business resources is a major link in the management of large, cross-country companies. For example, the business resource may be a fund. In order to reduce the global liquidity risk of multinational companies, improve the utilization efficiency of capital and reduce the capital cost, the capital needs to be subjected to global liquidity management. The processing of the bank data influences the perception of the fund amount in the actual accounts of all parts of the world, thereby influencing the formulation of schemes such as fund transfer schemes and the like. The formulation of schemes such as fund transfer schemes and the like requires high-timeliness and high-quality processing results of the bank data.
Therefore, a method for processing service data is needed to obtain a processing result with high time efficiency and high quality of service data.
Disclosure of Invention
One or more embodiments of the present specification describe a service data processing method, which can ensure the accuracy of a processing result under the condition of ensuring the timeliness of service data processing.
According to a first aspect, a method for processing service data is provided, the method comprising:
the method comprises the steps that an offline computing platform obtains a plurality of analysis results from a first middleware, wherein the analysis results are generated by analyzing a service data change log of a database by the first middleware;
performing offline batch processing on the plurality of analysis results to obtain a first set of calculation results;
acquiring a target table generated by respectively performing online calculation on at least one analysis result acquired from the second middleware by the real-time calculation platform; the at least one analysis result belongs to an analysis result generated by analyzing the service data change log of the database by the second middleware;
comparing the first set with the target table;
updating the target table based on the first set when the computation results in the first set and the computation results in the target table are inconsistent.
In some embodiments, the business data is silver deposit data.
In some embodiments, wherein the first middleware is any one of canal, databus, keylet, otter;
the second intermediate piece is drc;
the link between the offline computing platform and the first middleware is different from the link between the real-time computing platform and the second middleware.
In some embodiments, the plurality of analysis results are results of analyzing, by the first middleware, change logs of the database in a preset period;
the at least one analysis result belongs to an analysis result generated by analyzing the change log of the database in the preset period by the second middleware.
In some embodiments, the plurality of analysis results are the results of the first middleware analyzing all change logs of the database;
the at least one analysis result belongs to an analysis result generated by analyzing a change log generated after the database is on line on the real-time computing platform by the second middleware.
In some embodiments, wherein said updating the target table based on the first set when the computation results in the first set and the computation results in the target table are inconsistent comprises:
when the first set has at least one more calculation result than the target table, writing the at least one calculation result into the target table to update the target table.
In some embodiments, wherein the first set includes first computation results corresponding to a first change log, the target table includes second computation results corresponding to the first change log; when the calculation results in the first set and the calculation results in the target table are inconsistent, updating the target table based on the first set includes:
and when the first calculation result is inconsistent with the second calculation result, correcting the second calculation result based on the first calculation result so as to update the target table.
In some embodiments, wherein the method further comprises: the initiating an alert when the first set and the target table are inconsistent.
According to a second aspect, there is provided a service data processing apparatus, the apparatus comprising:
the first acquisition unit is configured to acquire a plurality of analysis results from a first middleware by an offline computing platform, wherein the analysis results are generated by analyzing a service data change log of a database by the first middleware;
the batch processing unit is configured to perform offline batch processing on the plurality of analysis results to obtain a first set of calculation results;
the second acquisition unit is configured to acquire a target table generated by the real-time computing platform performing online computing on at least one analysis result acquired from the second middleware respectively; the at least one analysis result belongs to an analysis result generated by analyzing the service data change log of the database by the second middleware;
a comparison unit configured to compare the first set and the target table;
an updating unit configured to update the target table based on the first set when the calculation result in the first set and the calculation result in the target table are inconsistent.
According to a third aspect, there is provided a computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of the first aspect.
According to a fourth aspect, there is provided a computing terminal comprising a memory having stored therein executable code and a processor that, when executing the executable code, implements the method of the first aspect.
The method and the device provided by the embodiment of the specification can perform offline calculation and can compare the real-time calculation result of the offline calculation platform, so that the real-time calculation result can be corrected, and the accuracy of the service data processing result is further ensured under the condition of ensuring the timeliness of service data processing; and through the comparison of the offline calculation result and the real-time calculation result, abnormal data can be found conveniently, so that the fault causing data abnormality can be found conveniently, and the data quality can be further improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 illustrates an application scenario diagram according to one embodiment;
FIG. 2 shows a flow diagram of a business data processing method according to one embodiment;
fig. 3 shows a schematic block diagram of a traffic data processing apparatus according to an embodiment.
Detailed Description
The scheme provided by the specification is described below with reference to the accompanying drawings.
In many technical scenarios, the service resources are distributed in different databases in different parts of the world through distributed storage, which brings a difficult problem to the unified management of the service resources by the service party. Take the business resources as funds as an example. In order to make an efficient and accurate fund transfer scheme, the amount of funds in the bank accounts of different business branches in different regions needs to be sensed, and the bank deposit data of each business branch needs to be processed.
The bank deposit data is short for bank deposit data. Bank deposits are money stored at a bank. In one example, the banking data of each business branch of the business party may be funds stored in a respective corresponding banking account by each branch of the multinational company.
Each branch company of the multinational company can acquire the banking data of the branch company in the bank through the open specification interface of the branch company and the bank in the intercommunicating agreement range negotiated with the bank. Specifically, for any one of the international companies, the corresponding banking system may send the account change condition to the business system (e.g., irecon, ireserve, reservation, etc.) so that the business system obtains the banking data and stores the banking data in the database of the business system. It should be noted that, hereinafter, for convenience of description, the database of the business system herein is simply referred to as a database, and therefore, unless otherwise specified, the database hereinafter refers to the database of the business system herein.
According to one scheme, the middleware corresponding to the offline computing platform may analyze the change log in the log file of the database, and input the analysis result to the offline computing platform for offline computing according to a specific period (for example, the period duration may be 24 hours), so as to obtain an offline computing result.
According to the scheme, the off-line calculation results are generated in batches according to a specific period, and the requirements of real-time bills on high frequency and high real-time performance are difficult to meet. In addition, banks corresponding to different branch companies of the multinational company are distributed in different time zones, so that the sending time of the day end files of the branch companies is greatly different, and the off-line calculation of the bank storage data of the branch companies in the same batch cannot ensure that the day end bills of different time zones are generated in time.
According to another scheme, when the change log is generated in the log file, the middleware corresponding to the real-time computing platform analyzes the change log, inputs the analysis result to the real-time computing platform, and performs real-time computing to obtain a real-time computing result.
The link between the middleware corresponding to the real-time computing platform and the real-time computing platform is unstable, data loss is easy to occur, data omission occurs in real-time computing, and further the real-time computing result is incomplete.
The embodiment of the present specification provides a service data processing method, and an application architecture of the service data processing method may be as shown in fig. 1. After the change logs generated by the database are analyzed by the first middleware, offline batch processing can be performed by the offline computing platform; after being analyzed by the second middleware, the real-time computing platform can perform online processing.
The service data processing method provided by the embodiment of the specification utilizes the real-time computing platform to perform online computing on the change log, so that related personnel can sense service change in time, and the timeliness of the processing result of the change log is ensured.
The method can compare the off-line calculation with the real-time calculation result of the off-line calculation platform, thereby correcting the real-time calculation result and ensuring the accuracy of the service data processing result; and through the comparison of the offline calculation result and the real-time calculation result, abnormal data can be found conveniently, so that the fault causing data abnormality can be found conveniently, and the data quality can be further improved.
Next, referring to fig. 2, a service data processing method provided in an embodiment of the present specification is specifically described. The method may be performed by any apparatus, device, platform, cluster of devices having computing and processing capabilities. As shown in fig. 2, the method comprises the steps of: 200, the off-line computing platform acquires a plurality of analysis results from a first middleware, wherein the analysis results are generated by analyzing a service data change log of a database by the first middleware; 202, performing offline batch processing on the plurality of analysis results to obtain a first set of calculation results; step 204, acquiring a target table generated by the real-time computing platform respectively performing online computation on at least one analysis result acquired from the second middleware; the at least one analysis result is generated by analyzing the service data change log of the database by the second middleware; step 206, comparing the first set with the target table; and step 208, when the calculation result in the first set is inconsistent with the calculation result in the target table, updating the target table based on the first set.
Next, each step described above will be specifically described with reference to specific examples.
First, in step 200, the offline computing platform obtains a plurality of analysis results from a first middleware, where the plurality of analysis results are generated by the first middleware analyzing a service data change log of a database.
The database may be mysql, oracle, oceanbase, etc. When the service data recorded in the database changes, for example, service data is added, deleted, and modified, a corresponding change log is generated. The change log is recorded in a log file (e.g., binlog). For any database, the corresponding first middleware can analyze the change log to obtain an analysis result.
In some embodiments, the business data is silver deposit data.
For convenience of description, unless otherwise specified, the service data change log is simply referred to as a change log, and when the service data is the silver storage data, the change log refers to the silver storage data change log.
In some embodiments, the first middleware may be any one of canal, databus, keylet, otter, etc.
At the starting time (e.g., 0 point 0 minute 0 second every day) of a preset period (e.g., one period every 24 hours), the analysis results of the plurality of first middleware may be imported to the offline computing platform (e.g., OPDS) at the same time.
In some embodiments, the plurality of analysis results are the results of analyzing all the change logs of the database by the first middleware, that is, all the analysis results of the first middleware may be imported to the offline computing platform, in other words, the analysis results of all the change logs of the database analyzed by the first middleware are imported to the offline computing platform. Therefore, the accuracy of the calculation result obtained by the subsequent off-line batch processing is higher.
In some embodiments, the plurality of analysis results are results obtained by analyzing, by the first middleware, change logs of the database in a preset period, that is, the analysis results of the change logs of the first middleware analysis database in the preset period may be imported to the offline computing platform. Thereby saving computing resources in subsequent offline batch processing.
Next, in step 202, the multiple analysis results are processed in an offline batch manner, so as to obtain a first set of calculation results.
The offline computing platform can perform offline batch processing on the analysis results, and the computing results corresponding to the analysis results form a first set of computing results.
In some embodiments, each parsing result may correspond to one calculation result each.
In some embodiments, the offline computing platform may cluster a plurality of computing results corresponding to the plurality of parsing results, and use a cluster obtained by clustering as a constituent element of the first set.
In some embodiments, the offline computing platform may cluster the plurality of parsing results, then perform offline batch processing on the clustered clusters, and use the obtained computing results as the constituent elements of the first set.
In step 204, a target table generated by the real-time computing platform performing online computing on at least one analysis result obtained from the second middleware is obtained; the at least one analysis result is generated by analyzing the service data change log of the database by the second middleware.
The real-time computing platform may be a kepler (kepler) real-time computing platform.
In some embodiments, the second middleware is drc (data Replication center), which may provide the real-time computing platform with the parsed results of the real-time incremental change log. drc may support real-time synchronization of homogeneous or heterogeneous databases, real-time data record change subscription services, and the like.
In step 204, each time the database is changed, a change log is generated, and the second middleware corresponding to the database can analyze the change log; and immediately importing the analysis result into a real-time computing platform for online computing to obtain a real-time computing result, and writing the real-time computing result into a target table.
The offline computing platform obtains the target table from the real-time computing platform at a starting time (e.g., 0 o' clock 0 min 0 sec every day) of a preset period (e.g., one period every 24 hours).
It should be noted that due to the nature of online computing of real-time computing platforms, it requires that middleware be able to deliver data to it in a timely manner. While offline computing platforms perform offline batch processing, which has lower requirements for middleware. The second middleware is specially arranged and is used for transmitting data to the real-time computing platform in time and meeting the online computing characteristics of the real-time computing platform, and the first middleware can be universally arranged, mature and stable.
In general, the second middleware is unstable and is prone to lose data compared to the first middleware. And the link between the real-time computing platform and the second middleware is more complex than the link between the offline computing platform and the first middleware, and data is easily lost. In addition, the real-time computing platform only acquires the analysis result of the change log generated by the database on-line on the real-time computing platform from the second middleware to perform on-line computation, and does not process the analysis result of the change log before on-line.
For the above reasons, not all the analysis results after the second middleware analysis can be imported into the real-time computing platform. Therefore, the at least one analysis result is an analysis result of the second middleware successfully imported to the real-time computing platform, and belongs to an analysis result of the change log by the second middleware, and in most cases, the analysis result is not equal to the analysis result of the change log by the second middleware.
In some embodiments, when the plurality of analysis results are results of analyzing all change logs of the database by the first middleware, the at least one analysis result belongs to an analysis result generated by analyzing, by the second middleware, the change logs generated after the database is online on the real-time computing platform.
In some embodiments, when the plurality of analysis results are results of analyzing, by the first middleware, change logs of the database in a preset period, the at least one analysis result belongs to analysis results generated by analyzing, by the second middleware, change logs of the database in the preset period.
Next, in step 206, the first set is compared to the target table.
As described above, the elements in the first set are calculation results obtained by performing offline batch processing on the offline calculation platform, which may be referred to as offline calculation results for short. The calculation result in the target table is a calculation result obtained by performing online processing on the real-time calculation platform, and may be referred to as a real-time calculation result for short.
In some embodiments, in step 206, a one-to-one comparison between the offline calculation results in the first set and the real-time calculation results in the target table may be performed to compare the real-time calculation results in the target table to the offline calculation results in the first set. It is easily understood that for any change log, it has a primary key. Accordingly, both the offline calculation result and the real-time calculation result of the change log have the primary key. Therefore, the change log, the off-line calculation result and the real-time calculation result can be associated with each other by the primary key.
In some embodiments, in step 206, the offline calculation results and the real-time calculation results corresponding to the same change log may be compared for consistency.
Then, in step 208, when the computation results in the first set and the computation results in the target table are inconsistent, the target table is updated based on the first set.
In some embodiments, after performing the one-to-one comparison between the offline calculation results in the first set and the real-time calculation results in the target table, if some of the offline results do not correspond to the real-time calculation results, which indicates that the change log corresponding to the offline results is not analyzed by the second middleware or the analysis results are lost, the offline results may be written into the target table to update the target table.
In the embodiments, by comparing the offline calculation result in the first set with the real-time calculation result in the target table, it can be found whether data omission occurs, and further, a measure for the data omission phenomenon can be taken, so as to improve the subsequent data quality of the real-time calculation platform.
In some embodiments, if the offline calculation result and the real-time calculation result corresponding to the same change log are not consistent, the real-time calculation result is modified based on the offline calculation result to update the target table.
As described above, the second middleware is unstable compared to the first middleware, which may result in inaccurate analysis results and inaccurate real-time calculation results. Therefore, when the off-line calculation result is inconsistent with the real-time calculation result, the real-time calculation result is corrected by using the off-line calculation result.
In the embodiments, whether the off-line calculation result is consistent with the real-time calculation result or not is compared, so that whether the real-time calculation result is accurate or not can be determined, and further, a measure aiming at the phenomenon that the calculation result of the real-time calculation platform is inaccurate can be taken, so that the accuracy of the calculation result of the real-time calculation platform is improved.
In some embodiments, wherein the method further comprises: the initiating an alert when the first set and the target table are inconsistent.
In the embodiments, when the calculation result of the offline calculation platform is inconsistent with the calculation result of the real-time calculation platform, an alarm is initiated to remind relevant personnel to take measures such as checking data, modifying a business report generated according to the calculation result of the real-time calculation platform, and the like.
The service data processing method provided by the embodiment of the specification can perform offline calculation and can compare the real-time calculation result of the offline calculation platform, so that the real-time calculation result can be corrected, and the accuracy of the service data processing result is further ensured under the condition of ensuring the timeliness of service data processing; and through the comparison of the offline calculation result and the real-time calculation result, abnormal data can be found conveniently, so that the fault causing data abnormality can be found conveniently, and the data quality can be further improved.
In a second aspect, the present specification provides a service data processing apparatus 300. Referring to fig. 3, the apparatus 300 includes:
a first obtaining unit 310, configured to obtain, by an offline computing platform, multiple analysis results from a first middleware, where the multiple analysis results are generated by analyzing, by the first middleware, a service data change log of a database;
the batch processing unit 320 is configured to perform offline batch processing on the plurality of analysis results to obtain a first set of calculation results;
a second obtaining unit 330, configured to obtain a target table generated by the real-time computing platform performing online computation on at least one analysis result obtained from the second middleware; the at least one analysis result belongs to an analysis result generated by analyzing the service data change log of the database by the second middleware;
a comparing unit 340 configured to compare the first set and the target table;
an updating unit 350 configured to update the target table based on the first set when the calculation result in the first set and the calculation result in the target table are inconsistent.
In some embodiments, the business data is silver deposit data.
In some embodiments, the first middleware is any one of canal, databus, keylet, otter;
the second intermediate piece is drc;
the link between the offline computing platform and the first middleware is different from the link between the real-time computing platform and the second middleware.
In some embodiments, the plurality of analysis results are results of analyzing, by the first middleware, change logs of the database in a preset period;
the at least one analysis result belongs to an analysis result generated by analyzing the change log of the database in the preset period by the second middleware.
In some embodiments, the plurality of analysis results are the results of the first middleware analyzing all change logs of the database;
the at least one analysis result belongs to an analysis result generated by analyzing a change log generated after the database is on line on the real-time computing platform by the second middleware.
In some embodiments, the updating unit 350 is configured to write at least one calculation result to the target table to update the target table when the first set has more calculation results than the target table.
In some embodiments, the first set includes first computation results corresponding to a first change log, and the target table includes second computation results corresponding to the first change log; the updating unit 350 is configured to modify the second calculation result based on the first calculation result to update the target table when the first calculation result and the second calculation result are inconsistent.
In some embodiments, the apparatus further comprises an alert unit (not shown) configured to initiate an alert when the first set and the target table are inconsistent.
In another aspect, embodiments of the present specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method shown in fig. 2.
In another aspect, embodiments of the present description provide a computing terminal including a memory and a processor, where the memory stores executable code, and the processor executes the executable code to implement the method shown in fig. 2.
Those skilled in the art will recognize that, in one or more of the examples described above, the functions described in this invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.
The above-mentioned embodiments, objects, technical solutions and advantages of the present invention are further described in detail, it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made on the basis of the technical solutions of the present invention should be included in the scope of the present invention.
Claims (18)
1. A method for processing service data, the method comprising:
the method comprises the steps that an offline computing platform obtains a plurality of analysis results from a first middleware, wherein the analysis results are generated by analyzing a service data change log of a database by the first middleware;
performing offline batch processing on the plurality of analysis results to obtain a first set of calculation results;
acquiring a target table generated by respectively performing online calculation on at least one analysis result acquired from the second middleware by the real-time calculation platform; the at least one analysis result belongs to an analysis result generated by analyzing the service data change log of the database by the second middleware;
comparing the first set with the target table;
updating the target table based on the first set when the computation results in the first set and the computation results in the target table are inconsistent.
2. The method of claim 1, wherein the traffic data is silver deposit data.
3. The method of claim 1, wherein the first middleware is any one of canal, databus, keylet, otter;
the second intermediate piece is drc;
the link between the offline computing platform and the first middleware is different from the link between the real-time computing platform and the second middleware.
4. The method according to claim 1, wherein the plurality of analysis results are results of the first middleware analyzing change logs of the database in a preset period;
the at least one analysis result belongs to an analysis result generated by analyzing the change log of the database in the preset period by the second middleware.
5. The method of claim 1, wherein the plurality of parsing results are results of the first middleware parsing all change logs of the database;
the at least one analysis result belongs to an analysis result generated by analyzing a change log generated after the database is on line on the real-time computing platform by the second middleware.
6. The method of claim 1, wherein said updating the target table based on the first set when the computation results in the first set and the computation results in the target table are inconsistent comprises:
when the first set has at least one more calculation result than the target table, writing the at least one calculation result into the target table to update the target table.
7. The method of claim 1, wherein the first set includes first computation results corresponding to a first change log, the target table includes second computation results corresponding to the first change log; when the calculation results in the first set and the calculation results in the target table are inconsistent, updating the target table based on the first set includes:
and when the first calculation result is inconsistent with the second calculation result, correcting the second calculation result based on the first calculation result so as to update the target table.
8. The method of claim 1, wherein the method further comprises: the initiating an alert when the first set and the target table are inconsistent.
9. A traffic data processing apparatus, the apparatus comprising:
the first acquisition unit is configured to acquire a plurality of analysis results from a first middleware by an offline computing platform, wherein the analysis results are generated by analyzing a service data change log of a database by the first middleware;
the batch processing unit is configured to perform offline batch processing on the plurality of analysis results to obtain a first set of calculation results;
the second acquisition unit is configured to acquire a target table generated by the real-time computing platform performing online computing on at least one analysis result acquired from the second middleware respectively; the at least one analysis result belongs to an analysis result generated by analyzing the service data change log of the database by the second middleware;
a comparison unit configured to compare the first set and the target table;
an updating unit configured to update the target table based on the first set when the calculation result in the first set and the calculation result in the target table are inconsistent.
10. The apparatus of claim 9, wherein the traffic data is silver deposit data.
11. The apparatus of claim 9, wherein the first middleware is any one of canal, databus, keylet, otter;
the second intermediate piece is drc;
the link between the offline computing platform and the first middleware is different from the link between the real-time computing platform and the second middleware.
12. The apparatus according to claim 9, wherein the plurality of parsing results are results of parsing, by the first middleware, change logs of the database in a preset period;
the at least one analysis result belongs to an analysis result generated by analyzing the change log of the database in the preset period by the second middleware.
13. The apparatus of claim 9, wherein the plurality of parsing results are results of the first middleware parsing all change logs of the database;
the at least one analysis result belongs to an analysis result generated by analyzing a change log generated after the database is on line on the real-time computing platform by the second middleware.
14. The apparatus of claim 9, wherein the updating unit is configured to write at least one more computation result than the target table to the target table when the first set has the at least one more computation result than the target table to update the target table.
15. The apparatus of claim 9, wherein the first set comprises first computation results corresponding to a first change log, and the target table comprises second computation results corresponding to the first change log; the updating unit is configured to correct the second calculation result based on the first calculation result to update the target table when the first calculation result and the second calculation result are inconsistent.
16. The apparatus of claim 9, wherein the apparatus further comprises an alert unit configured to initiate an alert when the first set and the target table are inconsistent.
17. A computer-readable storage medium, on which a computer program is stored which, when executed in a computer, causes the computer to carry out the method of any one of claims 1-8.
18. A computing terminal comprising a memory having stored therein executable code and a processor that, when executing the executable code, implements the method of any of claims 1-8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911031520.8A CN110765148B (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
| CN202210652485.7A CN114969077A (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911031520.8A CN110765148B (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210652485.7A Division CN114969077A (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110765148A true CN110765148A (en) | 2020-02-07 |
| CN110765148B CN110765148B (en) | 2022-06-28 |
Family
ID=69333077
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911031520.8A Active CN110765148B (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
| CN202210652485.7A Pending CN114969077A (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210652485.7A Pending CN114969077A (en) | 2019-10-28 | 2019-10-28 | Service data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN110765148B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113742192A (en) * | 2021-09-13 | 2021-12-03 | 杭州安恒信息技术股份有限公司 | Log rule quality analysis method, system, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107784098A (en) * | 2017-10-24 | 2018-03-09 | 百味云科技股份有限公司 | Real-time data warehouse platform |
| CN107979477A (en) * | 2016-10-21 | 2018-05-01 | 苏宁云商集团股份有限公司 | A kind of method and system of business monitoring |
| US20190005086A1 (en) * | 2017-07-02 | 2019-01-03 | Equalum Ltd. | System and method for system for determining database relations based on data monitoring |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AR102628A1 (en) * | 2014-11-12 | 2017-03-15 | Fraunhofer Ges Forschung | DECODER TO DECODE A MULTIMEDIA SIGNAL AND CODIFIER TO CODIFY SECONDARY MULTIMEDIA DATA THAT INCLUDE METADATA OR CONTROL DATA FOR PRIMARY MULTIMEDIA DATA |
| CN106909598B (en) * | 2016-07-01 | 2020-10-13 | 阿里巴巴集团控股有限公司 | Processing method, device and system for guaranteeing consistency of calculation data |
| US10261958B1 (en) * | 2016-07-29 | 2019-04-16 | Microsoft Technology Licensing, Llc | Generating an association between confidential data and member attributes |
| CN107122431A (en) * | 2017-04-14 | 2017-09-01 | 浙江数链科技有限公司 | A kind of real-time calculating platform and the method for computing data based on real-time calculating platform |
| US10860612B2 (en) * | 2018-04-19 | 2020-12-08 | Sap Se | Parallel replication across formats |
| CN109871388A (en) * | 2019-02-19 | 2019-06-11 | 北京字节跳动网络技术有限公司 | Data cache method, device, whole electronic equipment and storage medium |
-
2019
- 2019-10-28 CN CN201911031520.8A patent/CN110765148B/en active Active
- 2019-10-28 CN CN202210652485.7A patent/CN114969077A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107979477A (en) * | 2016-10-21 | 2018-05-01 | 苏宁云商集团股份有限公司 | A kind of method and system of business monitoring |
| US20190005086A1 (en) * | 2017-07-02 | 2019-01-03 | Equalum Ltd. | System and method for system for determining database relations based on data monitoring |
| CN107784098A (en) * | 2017-10-24 | 2018-03-09 | 百味云科技股份有限公司 | Real-time data warehouse platform |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113742192A (en) * | 2021-09-13 | 2021-12-03 | 杭州安恒信息技术股份有限公司 | Log rule quality analysis method, system, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114969077A (en) | 2022-08-30 |
| CN110765148B (en) | 2022-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020186901A1 (en) | Block chain-based data check system and method, computing device, and storage medium | |
| CN111277639B (en) | Method and device for maintaining data consistency | |
| US8799230B2 (en) | Method and system for centralized issue tracking | |
| WO2020000720A1 (en) | Server, packet processing method, program, and computer-readable storage medium | |
| CN111125106B (en) | Batch running task execution method, device, server and storage medium | |
| US11816163B2 (en) | Systems and methods for improved transactional mainframes | |
| CN113505520A (en) | Method, device and system for supporting heterogeneous federated learning | |
| US12021996B2 (en) | Universal blockchain data model | |
| CN110765148B (en) | Service data processing method and device | |
| CN112835885A (en) | Processing method, device and system for distributed table storage | |
| CN111741080B (en) | Network file distribution method and device | |
| CN112950191A (en) | Service data processing method and device based on fee refunding service and computer equipment | |
| CN112527839A (en) | Multi-source data processing method, system, equipment and storage medium | |
| CN112148762A (en) | Statistical method and device for real-time data stream | |
| CN113077241B (en) | Approval processing method, device, equipment and storage medium | |
| CN110795259A (en) | Method and device for analyzing application collapse | |
| CN114691768A (en) | Data processing method, accounting system and related equipment | |
| CN111767318A (en) | Data statistical method, device, electronic equipment and medium | |
| WO2019056736A1 (en) | Breakpoint data acquisition method and apparatus, terminal device and medium | |
| CN107566478B (en) | Data pushing method, system and computer readable storage medium | |
| CN113076297A (en) | Data processing method, device and storage medium | |
| CN115098281A (en) | Block chain cross-chain task processing method, device, server and medium | |
| CN114490678A (en) | Abnormal account data processing method, device, equipment and medium | |
| CN115760328A (en) | Method and device for returning updated data record based on error code | |
| CN114372111A (en) | Service data uploading method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240206 Address after: Room 1408, No. 447 Nanquan North Road, China (Shanghai) Pilot Free Trade Zone, Pudong New Area, Shanghai, 200120 Patentee after: Shanghai Ant Chuangjiang Information Technology Co.,Ltd. Country or region after: China Address before: 310000 801-11 section B, 8th floor, 556 Xixi Road, Xihu District, Hangzhou City, Zhejiang Province Patentee before: Alipay (Hangzhou) Information Technology Co.,Ltd. Country or region before: China |