CN116226144A

CN116226144A - Data processing method and device, computer readable storage medium and electronic equipment

Info

Publication number: CN116226144A
Application number: CN202211733629.8A
Authority: CN
Inventors: 张立; 王墨飞; 胡瑞娟; 张明; 李京
Original assignee: Postal Savings Bank of China Ltd
Current assignee: Postal Savings Bank of China Ltd
Priority date: 2022-12-30
Filing date: 2022-12-30
Publication date: 2023-06-06

Abstract

The application provides a data processing method, a data processing device, a computer readable storage medium and electronic equipment. The method comprises the following steps: acquiring offline data in an offline database; acquiring real-time data in a real-time database, wherein the real-time data is data of real-time transaction in a business transaction process; checking the real-time data to determine whether the real-time data is abnormal; and under the condition that the real-time data is abnormal, updating the real-time data in the real-time database according to the offline data in the offline database to obtain updated real-time data. In the scheme, whether the real-time data is abnormal or not can be determined by checking the real-time data, and the real-time data can be updated by adopting the offline data under the condition that the real-time data is abnormal, so that the consistency between the updated real-time data and the offline data can be ensured, and the data quality of the real-time data can be improved by adopting the offline data to update the real-time data because the offline data is accurate.

Description

Data processing method and device, computer readable storage medium and electronic equipment

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method, a data processing device, a computer readable storage medium, and an electronic device.

Background

In the actual transaction data transmission process, due to business processing or technical problems and the like, the data acquired in real time can be lost, and the consistency of the real-time data can not be ensured.

Disclosure of Invention

The main objective of the present application is to provide a data processing method, apparatus, computer readable storage medium and electronic device, so as to solve the problem that the consistency of real-time data cannot be guaranteed in the prior art.

According to an aspect of an embodiment of the present invention, there is provided a data processing method, including: acquiring offline data in an offline database, wherein the offline data are all data of business transactions before the current moment; acquiring real-time data in a real-time database, wherein the real-time data is data of real-time transaction in a business transaction process; checking the real-time data to determine whether the real-time data is abnormal; and under the condition that the real-time data is abnormal, updating the real-time data in the real-time database according to the offline data in the offline database to obtain updated real-time data.

Optionally, the real-time data is checked to determine whether the real-time data is abnormal, including at least one of: extracting a key field in the real-time data, and determining that the real-time data is abnormal under the condition that the key field is empty, wherein the key field at least comprises a global service tracking number, the global service tracking number is a unique identifier for requesting to execute a target task, the global service tracking number is generated by an event, and the target task requested to be executed comprises a plurality of events; extracting a plurality of sub-data in the real-time data of the same transaction, and determining that the real-time data is abnormal under the condition that any two sub-data are different; and respectively extracting the real-time data of a target time period and the offline data of the target time period, and determining that the real-time data is abnormal when the real-time data of the target time period is different from the offline data of the target time period.

Optionally, under the condition that the real-time data is determined to be abnormal, updating the real-time data in the real-time database according to the offline data in the offline database to obtain updated real-time data, including: under the condition that the real-time data is abnormal, acquiring a logic relationship among a plurality of sub-data in the abnormal real-time data, wherein the logic relationship refers to a containing relationship and/or a reference relationship among the plurality of sub-data; determining data to be updated in the real-time data according to the logic relation, wherein the data to be updated is at least part of the data in the real-time data, and the data to be updated is abnormal data in the real-time data; and updating the data to be updated according to the offline data to obtain updated real-time data.

Optionally, determining the data to be updated in the real-time data according to the logical relationship includes: under the condition that the abnormal child data is determined to be of a parent type, determining that the abnormal child data of the parent type and the corresponding child data of the child type are the data to be updated; under the condition that the abnormal sub-data is of an intermediate type, determining the abnormal sub-data of the intermediate type and the sub-data which has a logic relation with the abnormal sub-data of the intermediate type are both the data to be updated; and under the condition that the abnormal sub-data is determined to be of the subtype, determining the abnormal sub-data of the subtype as the data to be updated.

Optionally, updating the data to be updated according to the offline data to obtain updated real-time data, including: determining a target time period of the data to be updated, and extracting the offline data of the corresponding target time period from the offline database to obtain target offline data; and updating the data to be updated by adopting the target offline data to obtain updated real-time data.

Optionally, the plurality of data to be updated are provided, the data to be updated is updated according to the offline data, so as to obtain updated real-time data, including: determining the priority order of a plurality of data to be updated according to the logic relation; and sequentially updating the plurality of data to be updated according to the priority order to obtain updated real-time data.

Optionally, in a case where the number of updates to the real-time data is greater than a number threshold, and/or in a case where the time to update the real-time data is greater than a target duration, the method further includes: determining the abnormality of the offline data, generating prompt information and sending the prompt information to a target terminal, wherein the prompt information characterizes the abnormality of the offline data and cannot update the real-time data.

According to another aspect of the embodiment of the present invention, there is also provided a data processing apparatus, including: the first acquisition unit is used for acquiring offline data in an offline database, wherein the offline data are all data of business transactions before the current moment; the second acquisition unit is used for acquiring real-time data in the real-time database, wherein the real-time data is data of real-time transaction in the business transaction process; the determining unit is used for checking the real-time data and determining whether the real-time data is abnormal or not; and the updating unit is used for updating the real-time data in the real-time database according to the offline data in the offline database under the condition that the real-time data is determined to be abnormal, so as to obtain updated real-time data.

According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium including a stored program, wherein the program performs any one of the methods.

According to still another aspect of the embodiment of the present invention, there is also provided an electronic device including: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods.

In the embodiment of the invention, firstly, the offline data in the offline database is acquired, then the real-time data in the real-time database is acquired, then the real-time data is checked to determine whether the real-time data is abnormal, and finally, under the condition that the real-time data is determined to be abnormal, the real-time data in the real-time database is updated according to the offline data in the offline database, so that the updated real-time data is obtained. In the scheme, whether the real-time data is abnormal or not can be determined by checking the real-time data, and the real-time data can be updated by adopting the offline data under the condition that the real-time data is abnormal, so that the consistency between the updated real-time data and the offline data can be ensured, and the data quality of the real-time data can be improved by adopting the offline data to update the real-time data because the offline data is accurate.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

FIG. 1 shows a flow diagram of a method of processing data according to an embodiment of the present application;

FIG. 2 shows a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 3 shows a flow diagram of yet another data processing method.

Detailed Description

It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Furthermore, in the description and in the claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element or "connected" to the other element through a third element.

For convenience of description, the following will describe some terms or terms related to the embodiments of the present application:

offline data: the date of the data is defined as T-day, and the offline data refers to all yesterday data processed t+1-day after the end of T-day, and generally refers to the date of t+1. For example, today's date t=2022-07-03, then the offline business data that can be embodied includes only the previous day data (data of t+1).

Real-time data: real-time data is data information obtained in the process of generating the transaction and the process of generating the data synchronously. Real-time data mainly refers to data delays that are small, e.g., delays on the order of milliseconds, seconds, minutes. The delay at the hour level is referred to as "near real-time data". For example, a unified monitoring platform of a data center, for a part of transaction data to be monitored, refreshes the transaction data in a statistics large screen in real time according to the minute level

Database: a data set organized according to a predetermined structure.

With the development of enterprise business and the development of real-time data technology, the application of communication interaction scenes among systems and services is gradually increased. Unlike offline batch data transmission, real-time data transmission often requires additional means to ensure the final consistency of the data, such as using transactional messages, asynchronous task updates, and the like. Besides guaranteeing data consistency to the greatest extent in mechanism, various problems in real-time data processing in a service scene are found in time. During the operation of the system, the real-time data is sometimes not changed according to the plan, for example, a user requests to stop paying a real-time transaction at the bank counter, and the inquiry system calls the paid account to stop paying after receiving the stop payment request, but for some reasons (such as system fault, service processing error reasons and the like), the stop payment request is not sent and validated, or a response of successful stop payment is not successfully returned to the system, and finally the inquiry result shows that the real-time transaction state is successful but not stopped, or the transaction success is not confirmed but stopped. The cause of such problems is generally: source library and target library anomalies, task job anomalies, base component faults, inter-system communication faults, and the like. Therefore, the data is lost in a real-time data scene, the consistency of the real-time data and the offline data cannot be ensured, and the data quality of the real-time data is low.

At present, the real-time data replication (CDC) is a relatively mature synchronous database changing technical scheme and is used for guaranteeing consistency in the process of adding, deleting and checking database tables in real time. In the process of real-time data processing, the CDC can be used as a sending end and a receiving end of a data source, and in the processes of real-time access and real-time data pushing, a lightweight snapshot mechanism (Checkpoint) is adopted to ensure the consistency of data. I.e., a snapshot of all task states is saved at a certain time. The snapshot saves a timeline where all tasks have processed the same data. And automatically executing a distributed consistency Checkpoint step in a time interval, asynchronously inserting a Barrier Checkpoint boundary, and automatically storing a memory state as a Checkpoint process file. Finally, the purpose of accurate single processing of the data is achieved. For common consistency problems of real-time access and push, there are typically several ways:

1. for the abnormal situations of the source database and the target database, if the abnormal exit of the source database is found and the abnormal writing of the target database is not possible, a restarting or reconnecting operation is usually executed, and at the moment, a snapshot is redone and the total data is sent to the target database so as to ensure the final consistency of the data.

2. For the abnormal condition of the real-time task, reconnection or reconnection operation is usually performed, and a snapshot is reworked at the moment to ensure the final consistency of the data.

3. For the condition that the snapshot is abnormal when the binary log (Binlog) is read, incremental data is acquired after the full data are synchronized, a real-time task can set an inspection task to perform timing inspection, displacement information and Schema information of the Binlog are read and stored in a stateBackend, and if the snapshot is abnormal, the following mode is adopted for recovery under result analysis:

a. for the abnormal condition of the source database, if the abnormal exit of the source database is found, the connection cannot be realized, and the writing of the target database is abnormal, a restarting or reconnecting operation is usually executed, and the data recovery is carried out from the latest check point.

b. For the abnormal condition of the real-time task, reconnection or reconnection operation is usually executed, the scene is consistent with the situation a, the data is recovered in a Binlog log synchronization mode, and the final consistency of the data is maintained.

c. For the abnormal condition of the target database, if the abnormal writing condition of the target database is found, the Binlog displacement information read by the CDC Connector is not updated any more, the two operations are atomic operations, and after the real-time task is recovered, data recovery is carried out from the latest check point, and finally consistency is maintained.

However, there are some drawbacks in the three recovery methods:

1. after the real-time task is started, the full-table scanning of the database is firstly performed, and when the full-table data is scanned, the snapshot has no site available for recovery, so that the Checkpoint cannot be executed in the full-table scanning stage. In order not to execute checkpoints, the CDC source table of the database may have the executing checkpoints wait all the time (global lock) and even the checkpoints timeout, then the scanning takes a long time if the database table data size is large.

2. Before the real-time task reads the Binlog log, all library tables in the corresponding database instance are locked, so that other connections cannot do DML operation. The lock can be released after the Binlog log is read, and other connections can normally do DML operation. All other database tables cannot do DML during the global lock table.

3. In the snapshot making stage, an abnormality of the data source database or an abnormality of the task itself may occur. Thus, the use of the update mechanism of the Flink itself may affect the writing actions of other programs and may also affect the efficiency of business processes.

Therefore, as can be seen from the above description, during the actual transaction data transmission process, due to reasons such as service processing or technical problems (such as service operation errors, code logic bug, mishandling of concurrent scenario, failure of basic components (network, database, middleware), when the real-time data is written into the real-time database, a part of the real-time data may be lost, so that consistency between the real-time data and the data in the real-time database cannot be ensured, and the data quality of the real-time data is low.

As described in the background art, in order to solve the foregoing problem, in an exemplary embodiment of the present application, a data processing method, apparatus, computer readable storage medium, and electronic device are provided.

According to an embodiment of the application, a data processing method is provided.

Fig. 1 is a flowchart of a method of processing data according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:

Step S101, obtaining off-line data in an off-line database, wherein the off-line data are all data of business transactions before the current moment;

specifically, the offline data is relatively accurate, the offline data is generally stored in an offline database in batches, the transaction of the day is stored in the offline database (only once a day) after the transaction of the day is processed, and the accuracy of the offline data is also detected, for example, the accuracy of the offline data is verified through service data (data in the service of a specific transaction, such as serial number of the transaction, type of the transaction and calculation result of the transaction), or the accuracy of the offline data can be detected manually, and if the offline data is abnormal, the offline data of the day is processed again in batches to ensure the accuracy of the offline data.

Step S102, acquiring real-time data in a real-time database, wherein the real-time data are data of real-time transaction in a business transaction process;

step S103, checking the real-time data to determine whether the real-time data is abnormal;

specifically, a verification service may be set, through which the offline database table corresponding to the real-time database table is sequentially called, and if it is determined that the real-time database table is abnormal, the work service that has been completed before the abnormal database table may be revoked.

In the step S103, by checking the real-time data, it can be determined whether the real-time data is abnormal, and if it is determined that the real-time data is abnormal, it can be determined that there is no consistency between the real-time data and the offline data, and if it is determined that the real-time data is not abnormal, it can be determined that there is consistency between the real-time data and the offline data.

Step S104, when the real-time data is determined to be abnormal, updating the real-time data in the real-time database according to the offline data in the offline database, and obtaining updated real-time data.

Specifically, an update service may be provided, by which service arrangement (timing custom update tasks) and update function is provided and automatically completed.

In the step S104, under the condition that the real-time data is determined to be abnormal, the abnormal real-time data can be updated by using the offline data, so that the consistency of the real-time data and the offline data can be ensured.

In the method, firstly, the offline data in the offline database is acquired, then the real-time data in the real-time database is acquired, then the real-time data is checked, whether the real-time data is abnormal or not is determined, and finally, under the condition that the real-time data is determined to be abnormal, the real-time data in the real-time database is updated according to the offline data in the offline database, and the updated real-time data is obtained. In the scheme, whether the real-time data is abnormal or not can be determined by checking the real-time data, and the real-time data can be updated by adopting the offline data under the condition that the real-time data is abnormal, so that the consistency between the updated real-time data and the offline data can be ensured, and the data quality of the real-time data can be improved by adopting the offline data to update the real-time data because the offline data is accurate.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

In the above scheme of the application, the specific verification time can be verification after running and batch every day, multiple times can be verified in one day, and the real-time data is updated in the evening, the business is processed in the daytime, so that the real-time data can be verified on the premise of not invading the business, the abnormality of the real-time data can be timely found, the real-time data is updated through the offline data in the evening on the premise of not influencing the efficiency of the real-time computing cluster, the pressure of the cluster on the daytime is reduced, the access times of the real-time data on the updating time line can be reduced, the performance of the updated real-time data is better, and the data quality is improved

In practical application, whether the real-time data is abnormal or not can be determined through different application scenes, and in one embodiment of the application, the real-time data is checked to determine whether the real-time data is abnormal or not, which includes at least one of the following steps:

step S201, extracting a key field in the real-time data, and determining that the real-time data is abnormal when the key field is empty, wherein the key field at least comprises a global service tracking number, the global service tracking number is a unique identifier for requesting to execute a target task, the global service tracking number is generated by an event, and the target task requested to be executed comprises a plurality of events;

For each transaction, the global service tracking number is a unique identification of the transaction, a transaction can be quickly located through the global service tracking number, if the global service tracking number is empty, then a specific transaction can not be located later, and then real-time data abnormality can be determined.

Step S202, extracting a plurality of sub-data in the real-time data of the same transaction, and determining that the real-time data is abnormal under the condition that any two sub-data are different;

for the case that some transactions are fixed transactions, 100 yuan of money transfer is required every day, if the data of one day of money transfer is 10 ten thousand yuan in the stored real-time data, the real-time data is abnormal for the fixed transactions, and the real-time data can be determined to be abnormal.

Step S203 of extracting the real-time data of the target time period and the offline data of the target time period, respectively, and determining that the real-time data is abnormal when the real-time data of the target time period is different from the offline data of the target time period.

Specifically, whether the real-time data is abnormal or not can be determined by comparing the consistency of the real-time data and the offline data in the target time period, and the real-time data is abnormal if the real-time data in the target time period is different from the offline data because the offline data is already batched in advance and is accurate.

In the above steps S201 to S203, whether the real-time data is abnormal may be further accurately determined by checking the key fields in the real-time data, whether the real-time data is abnormal may be further accurately determined by checking a plurality of sub-data in the same transaction, and whether the real-time data is abnormal may be further accurately determined by comparing the consistency of the real-time data and the offline data.

Of course, whether the real-time data is abnormal can be further accurately determined by checking key fields in the real-time data and checking a plurality of sub-data in the same transaction; the key fields in the real-time data can be checked, and consistency of the real-time data and the offline data is compared to further accurately determine whether the real-time data is abnormal or not; the real-time data can be further accurately determined whether to be abnormal or not by checking a plurality of sub-data in the same transaction and comparing the consistency of the real-time data and the offline data; whether the real-time data is abnormal or not can be further accurately determined by verifying key fields in the real-time data, verifying a plurality of sub-data in the same transaction and comparing consistency of the real-time data and the offline data.

For verification operation of real-time data, the difference between the real-time data and the offline data is not simply compared, or the state before the traffic occurs is compared through rollback operation, and other concurrent requests can also change the state, so that the update of the real-time data can be completed in a reverse operation mode.

For example, in the calculation of the real-time balance type, the table a can provide the transaction serial number (i.e. the global service tracking number), the table B can provide the transaction abstract information, the balance transaction type can be calculated after the summary, if the error in the real-time calculation process or the abstract information is absent in the table B, the calculation of the balance type can fail, the complete table data information of the table a and the table B can be obtained again from the offline database, and after the balance transaction type is calculated again, the calculation result corresponding to the real-time data is updated.

In the case that it is determined that the real-time data is to be updated, not all the data in the real-time data are abnormal, so that only the abnormal data in the real-time data may be updated.

Step S301, when the real-time data is determined to be abnormal, acquiring a logic relationship among a plurality of sub-data in the abnormal real-time data, wherein the logic relationship refers to a containing relationship and/or a reference relationship among the plurality of sub-data;

step S302, determining data to be updated in the real-time data according to the logic relation, wherein the data to be updated is at least part of the data in the real-time data, and the data to be updated is abnormal data in the real-time data;

to implement updating of the real-time data, the scope of the real-time data to be updated may be determined when the real-time data is determined to be abnormal, and in a specific embodiment of the present application, determining the data to be updated in the real-time data according to the logical relationship includes: under the condition that the abnormal child data is determined to be the parent type, the abnormal parent type child data and the corresponding child type child data are determined to be the data to be updated; under the condition that the abnormal sub-data is of an intermediate type, the abnormal sub-data of the intermediate type and the sub-data which has a logic relation with the abnormal sub-data of the intermediate type are both the data to be updated; and under the condition that the abnormal sub-data is determined to be of the sub-type, determining the abnormal sub-data of the sub-type as the data to be updated. In the scheme, if the abnormal child data is of a parent type, the child data of the parent type can comprise the child data of the child type, and the child data of the child type can be abnormal when the child data of the parent type is abnormal, so that when the child data of the parent type is abnormal, the child data of the parent type and the child data of the corresponding child type are abnormal real-time data, if the abnormal child data is of an intermediate type, the child data of the intermediate type can have a logical relationship with the child data of a non-intermediate type, the child data of the intermediate type can also be abnormal when the child data of the intermediate type is abnormal, therefore, when the child data of the intermediate type is abnormal, the child data of the intermediate type and the child data of the corresponding child data of the intermediate type are all abnormal real-time data, if the child data of the child type is not abnormal, the child data of the corresponding type can only be determined to be abnormal real-time data, thus the update of the update required range can be updated accurately, and the update of the following data can be guaranteed to be updated more effectively in real time than the following data.

If the plurality of sub-data with the logic relationship is not updated successfully, the updating operation can be terminated in time, and only if the plurality of sub-data with the logic relationship is updated, the updating operation is calculated to be completed.

After the data to be updated is determined, an update serial number can be generated, the update serial number can correspond to the range of the data to be updated, which is updated once, and then the real-time data can be updated according to the update serial number.

Step S303, updating the data to be updated according to the offline data to obtain updated real-time data.

In particular, the service check (whether the offline data is correct or not) can be synchronously performed under the condition that the real-time data is required to be updated, and the check can be performed through the verification service, so that the robustness of the verification service is improved. For example, after the transaction amount calculation time-out in the collection business is over, the collection amount is recalculated and updated, and theoretically, the collection amount data in the offline database is only required to be updated to complete the updating operation. However, for example, it is also possible to update the serial number and exchange rate, and it is possible to determine whether the calculation of the amount of the sales exchange is correct.

When the updated real-time data is abnormal (namely, the updated service is abnormal), the completion state of each updated service can be synchronously recorded through the verification service, meanwhile, the business serial number related to the transaction of the abnormal real-time data is recorded, and the updated service is executed again later. The update interface of the update service of the real-time data may not only take the difference of the verification service as the update standard on the premise of verifying the service verification rule of the service request.

In the case of updating the data to be updated, if the offline data in the time period is arbitrarily selected to cause confusion of the data, the offline data corresponding to the data to be updated in the target time period can be selected, so that the data updating efficiency is higher, and meanwhile, no error occurs in the updating process.

Step S3031, determining a target time period of the data to be updated, and extracting the offline data of the corresponding target time period from the offline database to obtain target offline data;

Step S3032, updating the data to be updated by using the target offline data to obtain updated real-time data.

In the steps S3031 to S3032, the offline data corresponding to the data to be updated in the target time period can be adopted to update the data to be updated, so that the data quality of the real-time data in the target time period can be further ensured, the consistency of the real-time data and the offline data is further ensured, the development complexity is reduced, the processing efficiency of the real-time data is greatly enhanced, and the data quality of the real-time data can be further improved by adopting the offline data to update the real-time data because the offline data is accurate data.

In the process of updating the real-time data by using the offline data, the method may not be executed according to the reverse order of occurrence of the service, the priority order may be determined according to the importance degree of the service requirement, the updating order may be executed according to the priority order, or the method may be executed concurrently.

Step S3033, determining the priority order of a plurality of data to be updated according to the logic relation;

step S3034, updating the plurality of data to be updated sequentially according to the priority order to obtain updated real-time data.

Specifically, the plurality of data to be updated may be updated sequentially in order of priority from high to low, or may be updated sequentially in order of priority from low to high.

In the steps S3033 to S3034, the priority order of the plurality of data to be updated can be determined first, and then the plurality of data to be updated can be updated sequentially according to the priority order, so that the data which is more important or has higher influence on other data can be updated first, and further, the higher efficiency of data update can be ensured.

In the steps S301 to S303, the data to be updated is abnormal data in the real-time data, and only the abnormal data in the real-time data can be updated, that is, only the data to be updated is updated, so that the non-abnormal data in the real-time data can not be updated, and the updating speed of the data is ensured to be higher.

In another embodiment of the present application, in a case where the number of updates for updating the real-time data is greater than a number threshold, and/or in a case where the time for updating the real-time data is greater than a target duration, the method further includes: determining the abnormality of the offline data, generating prompt information and sending the prompt information to a target terminal, wherein the prompt information characterizes the abnormality of the offline data and cannot update the real-time data. In the embodiment, whether the offline data is abnormal or not can be determined, so that if the offline data is abnormal, the abnormal offline data is adopted to update the real-time data, and errors can be generated, so that the offline data can be checked, and the effectiveness and the accuracy of the offline data are further ensured.

The embodiment of the application also provides a data processing device, and it should be noted that the data processing device of the embodiment of the application can be used for executing the data processing method provided by the embodiment of the application. The following describes a data processing apparatus provided in an embodiment of the present application.

Fig. 2 is a schematic diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 2, the apparatus includes:

a first obtaining unit 10, configured to obtain offline data in an offline database, where the offline data is all data of a business transaction before a current time;

A second acquiring unit 20, configured to acquire real-time data in a real-time database, where the real-time data is data of a real-time transaction in a business transaction process;

a determining unit 30 for checking the real-time data and determining whether the real-time data is abnormal;

The above-mentioned determination unit, through carrying out the verification to real-time data, can confirm whether real-time data is unusual, under the circumstances that confirm real-time data is unusual, can confirm that there is not the uniformity between real-time data and the off-line data, under the circumstances that confirm real-time data is non-unusual, can confirm that there is the uniformity between real-time data and the off-line data, this scheme can carry out the verification to real-time data fast, can reduce the complexity of developing and improve the efficiency of verification.

And an updating unit 40, configured to update the real-time data in the real-time database according to the offline data in the offline database, and obtain updated real-time data when the real-time data is determined to be abnormal.

According to the updating unit, under the condition that the real-time data is abnormal, the abnormal real-time data can be updated by adopting the offline data, so that the consistency of the real-time data and the offline data can be ensured.

In the above device, the first acquiring unit acquires the offline data in the offline database, the second acquiring unit acquires the real-time data in the real-time database, the determining unit performs verification on the real-time data to determine whether the real-time data is abnormal, and the updating unit updates the real-time data in the real-time database according to the offline data in the offline database under the condition that the real-time data is determined to be abnormal, so as to obtain updated real-time data. In the scheme, whether the real-time data is abnormal or not can be determined by checking the real-time data, and the real-time data can be updated by adopting the offline data under the condition that the real-time data is abnormal, so that the consistency between the updated real-time data and the offline data can be ensured, and the data quality of the real-time data can be improved by adopting the offline data to update the real-time data because the offline data is accurate.

In the above scheme of this application, specific verification time can run out each day and run out and carry out the verification, can check many times in the day to update real-time data in the time of evening, business is handled in daytime, consequently can carry out the verification to real-time data under the prerequisite of not invading business, in time discover real-time data's unusual, and under the prerequisite that does not influence real-time calculation cluster efficiency, update real-time data through offline data in evening, reduce the pressure of on-line cluster in daytime, also can reduce the access number of times of on-line real-time data at the update time, and then the real-time data's after the update performance is better, improves data quality.

In practical application, whether real-time data is abnormal or not can be determined through different application scenes, and in one embodiment of the application, the determining unit at least comprises at least one of a first determining module, a second determining module and a third determining module, and functions of the modules are as follows:

the first determining module is configured to extract a key field in the real-time data, determine that the real-time data is abnormal when the key field is empty, where the key field includes at least a global service tracking number, the global service tracking number is a unique identifier of a target task requested to be executed, the global service tracking number is generated by an event, and the target task requested to be executed includes a plurality of events;

The second determining module is used for extracting a plurality of sub-data in the real-time data of the same transaction, and determining that the real-time data is abnormal under the condition that any two sub-data are different;

And a third determining module configured to extract the real-time data of the target time period and the offline data of the target time period, respectively, and determine that the real-time data is abnormal when the real-time data of the target time period is different from the offline data of the target time period.

The first determining module, the second determining module and the third determining module can further accurately determine whether the real-time data is abnormal or not through checking key fields in the real-time data, can further accurately determine whether the real-time data is abnormal or not through checking a plurality of sub-data in the same transaction, and can further accurately determine whether the real-time data is abnormal or not through comparing consistency of the real-time data and the offline data.

In the case that it is determined that the real-time data is to be updated, not all data in the real-time data is abnormal, so that only the abnormal data in the real-time data may be updated.

The acquisition module is used for acquiring a logic relationship among a plurality of sub-data in the abnormal real-time data under the condition that the real-time data is determined to be abnormal, wherein the logic relationship refers to a containing relationship and/or a reference relationship among the plurality of sub-data;

a fourth determining module, configured to determine, according to the logical relationship, data to be updated in the real-time data, where the data to be updated is at least part of the real-time data, and the data to be updated is abnormal data in the real-time data;

to implement updating of real-time data, a range of real-time data to be updated may be determined when it has been determined that the real-time data is abnormal, and in a specific embodiment of the present application, the fourth determining module includes a first determining submodule, a second determining submodule, and a third determining submodule, where the first determining submodule is configured to determine, when it is determined that the abnormal child data is of a parent type, that both the abnormal parent type child data and the corresponding child type child data are the data to be updated; the second determining submodule is used for determining that the abnormal intermediate type of the sub data and the sub data which has a logical relation with the abnormal intermediate type of the sub data are all the data to be updated under the condition that the abnormal intermediate type of the sub data is determined; and the third determining submodule is used for determining that the abnormal sub-data of the abnormal sub-type is the data to be updated under the condition that the abnormal sub-data is determined to be of the sub-type. In the scheme, if the abnormal child data is of a parent type, the child data of the parent type can comprise the child data of the child type, and the child data of the child type can be abnormal when the child data of the parent type is abnormal, so that when the child data of the parent type is abnormal, the child data of the parent type and the child data of the corresponding child type are abnormal real-time data, if the abnormal child data is of an intermediate type, the child data of the intermediate type can have a logical relationship with the child data of a non-intermediate type, the child data of the intermediate type can also be abnormal when the child data of the intermediate type is abnormal, therefore, when the child data of the intermediate type is abnormal, the child data of the intermediate type and the child data of the corresponding child data of the intermediate type are all abnormal real-time data, if the child data of the child type is not abnormal, the child data of the corresponding type can only be determined to be abnormal real-time data, thus the update of the update required range can be updated accurately, and the update of the following data can be guaranteed to be updated more effectively in real time than the following data.

And the updating module is used for updating the data to be updated according to the offline data to obtain updated real-time data.

Under the condition of updating the data to be updated, if the offline data in the time period is arbitrarily selected to cause confusion of the data, the offline data corresponding to the data to be updated in the target time period can be selected, so that higher data updating efficiency can be ensured, meanwhile, no error occurs in the updating process.

A fourth determining submodule, configured to determine a target time period of the data to be updated, and extract the offline data of the corresponding target time period from the offline database to obtain target offline data;

And the first updating sub-module is used for updating the data to be updated by adopting the target offline data to obtain updated real-time data.

The fourth determining sub-module and the first updating sub-module can update the data to be updated by adopting the offline data corresponding to the data to be updated in the target time period, so that the data quality of the real-time data in the target time period can be further ensured, the consistency of the real-time data and the offline data is further ensured, the development complexity is reduced, the processing efficiency of the real-time data is greatly enhanced, and the data quality of the real-time data can be further improved by adopting the offline data to update the real-time data because the offline data is accurate data.

In the process of updating the real-time data by using the offline data, the updating process may not be performed according to the reverse order of the occurrence of the service, the priority order may be determined according to the importance degree of the service requirement, the updating process may be performed according to the priority order, or the updating process may be performed concurrently.

A fifth determining submodule, configured to determine a priority order of the plurality of data to be updated according to the logical relationship;

and the second updating sub-module is used for sequentially updating the plurality of data to be updated according to the priority order to obtain updated real-time data.

The fifth determining sub-module and the second updating sub-module can determine the priority order of the plurality of data to be updated first, and then update the plurality of data to be updated in turn according to the priority order, so that the data which is more important or has higher influence on other data can be updated first, and further the data updating efficiency can be ensured to be higher.

The acquisition module, the fourth determination module and the updating module can update only abnormal data in the real-time data, namely update only the data to be updated, so that non-abnormal data in the real-time data can not be updated, and the updating speed of the data is ensured to be higher.

In another embodiment of the present application, the apparatus further includes a processing unit, where the processing unit is configured to determine, when the number of updates for updating the real-time data is greater than a number threshold, and/or when the time for updating the real-time data is greater than a target duration, determine that the offline data is abnormal, and generate prompt information to send the prompt information to a target terminal, where the prompt information characterizes that the offline data is abnormal, and cannot update the real-time data. In the embodiment, whether the offline data is abnormal or not can be determined, so that if the offline data is abnormal, the abnormal offline data is adopted to update the real-time data, and errors can be generated, so that the offline data can be checked, and the effectiveness and the accuracy of the offline data are further ensured.

The data processing device comprises a processor and a memory, wherein the first acquisition unit, the second acquisition unit, the determining unit, the updating unit and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one kernel, and the problem that the consistency of real-time data cannot be ensured in the prior art is solved by adjusting the kernel parameters.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program which, when executed by a processor, implements the above-described data processing method.

The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute the data processing method.

The present application also provides an electronic device comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods described above.

The embodiment of the invention provides equipment, which comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the processor realizes at least the following steps when executing the program:

The device herein may be a server, PC, PAD, cell phone, etc.

The present application also provides a computer program product adapted to perform a program initialized with at least the following method steps when executed on a data processing device:

In order that those skilled in the art can more clearly understand the technical solutions of the present application, the technical solutions and technical effects of the present application will be described below with reference to specific embodiments.

Examples

The present embodiment relates to a data processing method, as shown in fig. 3, including:

the service system can acquire service processing data, calculate real-time data through a real-time engine and select real-time data of T days;

the large data platform processes off-line data in batches, processes the data in batches at night, and conducts data importing to obtain corresponding off-line data;

and checking the real-time data through a checking service or checking service processing data according to service logic to obtain abnormal problem data (namely abnormal real-time data), uniformly inquiring the offline data according to the checking result to obtain offline data of T+1 days, scheduling the corresponding offline data to update, executing a data updating task through an updating service to update the real-time data (the real-time data of T+1 days can be updated), and inquiring the service according to a receipt and payment detail under the condition of uniformly inquiring the offline data.

In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units may be a logic function division, and there may be another division manner when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units described above, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the above-mentioned method of the various embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

1) According to the data processing method, firstly, offline data in an offline database is acquired, then, real-time data in a real-time database is acquired, then, verification is carried out on the real-time data, whether the real-time data are abnormal or not is determined, and finally, under the condition that the real-time data are abnormal, the real-time data in the real-time database are updated according to the offline data in the offline database, and updated real-time data are obtained. In the scheme, whether the real-time data is abnormal or not can be determined by checking the real-time data, and the real-time data can be updated by adopting the offline data under the condition that the real-time data is abnormal, so that the consistency between the updated real-time data and the offline data can be ensured, and the data quality of the real-time data can be improved by adopting the offline data to update the real-time data because the offline data is accurate.

2) According to the data processing device, the first acquisition unit acquires the offline data in the offline database, the second acquisition unit acquires the real-time data in the real-time database, the determination unit performs verification on the real-time data to determine whether the real-time data is abnormal, and the update unit updates the real-time data in the real-time database according to the offline data in the offline database under the condition that the real-time data is determined to be abnormal, so that updated real-time data is obtained. In the scheme, whether the real-time data is abnormal or not can be determined by checking the real-time data, and the real-time data can be updated by adopting the offline data under the condition that the real-time data is abnormal, so that the consistency between the updated real-time data and the offline data can be ensured, and the data quality of the real-time data can be improved by adopting the offline data to update the real-time data because the offline data is accurate.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method of processing data, comprising:

acquiring offline data in an offline database, wherein the offline data are all data of business transactions before the current moment;

acquiring real-time data in a real-time database, wherein the real-time data is data of real-time transaction in a business transaction process;

checking the real-time data to determine whether the real-time data is abnormal;

and under the condition that the real-time data is abnormal, updating the real-time data in the real-time database according to the offline data in the offline database to obtain updated real-time data.

2. The method of claim 1, wherein verifying the real-time data to determine if the real-time data is anomalous comprises at least one of:

extracting a key field in the real-time data, and determining that the real-time data is abnormal under the condition that the key field is empty, wherein the key field at least comprises a global service tracking number, the global service tracking number is a unique identifier for requesting to execute a target task, the global service tracking number is generated by an event, and the target task requested to be executed comprises a plurality of events;

Extracting a plurality of sub-data in the real-time data of the same transaction, and determining that the real-time data is abnormal under the condition that any two sub-data are different;

and respectively extracting the real-time data of a target time period and the offline data of the target time period, and determining that the real-time data is abnormal when the real-time data of the target time period is different from the offline data of the target time period.

3. The method according to claim 1, wherein in case it is determined that the real-time data is abnormal, updating the real-time data in the real-time database according to the offline data in the offline database, to obtain updated real-time data, comprises:

under the condition that the real-time data is abnormal, acquiring a logic relationship among a plurality of sub-data in the abnormal real-time data, wherein the logic relationship refers to a containing relationship and/or a reference relationship among the plurality of sub-data;

determining data to be updated in the real-time data according to the logic relation, wherein the data to be updated is at least part of the data in the real-time data, and the data to be updated is abnormal data in the real-time data;

And updating the data to be updated according to the offline data to obtain updated real-time data.

4. A method according to claim 3, wherein determining the data to be updated in the real-time data according to the logical relationship comprises:

under the condition that the abnormal child data is determined to be of a parent type, determining that the abnormal child data of the parent type and the corresponding child data of the child type are the data to be updated;

under the condition that the abnormal sub-data is of an intermediate type, determining the abnormal sub-data of the intermediate type and the sub-data which has a logic relation with the abnormal sub-data of the intermediate type are both the data to be updated;

and under the condition that the abnormal sub-data is determined to be of the subtype, determining the abnormal sub-data of the subtype as the data to be updated.

5. A method according to claim 3, wherein updating the data to be updated according to the offline data to obtain updated real-time data comprises:

determining a target time period of the data to be updated, and extracting the offline data of the corresponding target time period from the offline database to obtain target offline data;

And updating the data to be updated by adopting the target offline data to obtain updated real-time data.

6. The method of claim 3, wherein the plurality of data to be updated, the updating the data to be updated according to the offline data, and obtaining updated real-time data, comprises:

determining the priority order of a plurality of data to be updated according to the logic relation;

and sequentially updating the plurality of data to be updated according to the priority order to obtain updated real-time data.

7. The method according to claim 1, wherein in case the number of updates to the real-time data is greater than a number threshold and/or in case the time to update the real-time data is greater than a target duration, the method further comprises:

determining the abnormality of the offline data, generating prompt information and sending the prompt information to a target terminal, wherein the prompt information characterizes the abnormality of the offline data and cannot update the real-time data.

8. A data processing apparatus, comprising:

the first acquisition unit is used for acquiring offline data in an offline database, wherein the offline data are all data of business transactions before the current moment;

The second acquisition unit is used for acquiring real-time data in the real-time database, wherein the real-time data is data of real-time transaction in the business transaction process;

the determining unit is used for checking the real-time data and determining whether the real-time data is abnormal or not;

and the updating unit is used for updating the real-time data in the real-time database according to the offline data in the offline database under the condition that the real-time data is determined to be abnormal, so as to obtain updated real-time data.

9. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein the program performs the method of any one of claims 1 to 7.

10. An electronic device, comprising: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 1-7.