WO2022068316A1

WO2022068316A1 - Data reconciliation method and apparatus, device, and storage medium

Info

Publication number: WO2022068316A1
Application number: PCT/CN2021/106203
Authority: WO
Inventors: 黄恒; 谢永恒; 万月亮
Original assignee: 北京锐安科技有限公司
Priority date: 2020-09-29
Filing date: 2021-07-14
Publication date: 2022-04-07
Also published as: CN112162976A

Abstract

A data reconciliation method and apparatus, a device, and a storage medium. The data reconciliation method comprises: acquiring, from a data source, data to be linked (S110); performing a first accounting on the data to be linked to obtain first accounting content (S120); linking to a target database the data to be linked to obtain linked data (S130); performing a second accounting on the linked data to obtain second accounting content (S140); and if the first accounting content and the second accounting content are the same, determining that the reconciliation is successful (S150), the first accounting content comprising: the number of data pieces of the data to be linked and fingerprint information, and the second accounting content comprising: the number of the data pieces of the linked data and fingerprint information.

Description

Data reconciliation method, device, equipment and storage medium

This application claims the priority of the Chinese patent application with application number 202011052704.5 filed with the China Patent Office on September 29, 2020, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the field of computer information technology, for example, to a data reconciliation method, apparatus, device, and storage medium.

Background technique

Data access is an indispensable link in the use of big data systems. A series of logical processing is required in the process of data source from data access to data storage, such as: data discarding, field filling, conversion, de- Heavy and so on. In the process of these logical processing, there may be garbled data, inability to parse, data errors, etc., which will make the data source before data access and the data after data storage inconsistent.

SUMMARY OF THE INVENTION

The present application provides a data reconciliation method, device, equipment and storage medium, so as to achieve the technical effect of accurately reconciling data before and after data access.

This application provides a data reconciliation method, which includes:

Obtain the data to be accessed from the data source;

Perform first accounting for the data to be accessed to obtain first accounting content;

Accessing the data to be accessed to a target database to obtain access data;

Carrying out the second accounting for the access data to obtain the second accounting content;

In the case that the first accounting content and the second accounting content are the same, it is determined that the reconciliation is successful;

The first accounting content includes: the number of data bars and fingerprint information of the data to be accessed, and the second accounting content includes: the number of data bars and fingerprint information of the access data.

The application also provides a data reconciliation device, the device comprising:

a data acquisition module, set to acquire the data to be accessed in the data source;

The first bookkeeping module is set to carry out the first bookkeeping to the data to be accessed, and obtain the first bookkeeping content;

a data access module, configured to access the data to be accessed to a target database to obtain access data;

A second accounting module, configured to perform second accounting for the access data to obtain second accounting content;

A reconciliation module, configured to determine that the reconciliation is successful when the first and second accounting contents are the same;

The present application also provides a computer device, wherein the computer device includes:

at least one processor;

a storage device configured to store at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the above data reconciliation method.

The present application also provides a computer-readable storage medium storing a computer program, wherein when the program is executed by the processor, the above-mentioned data reconciliation method is implemented.

Description of drawings

1 is a flowchart of a data reconciliation method provided in Embodiment 1 of the present application;

1a is a schematic diagram of a data reconciliation method provided in Embodiment 1 of the present application;

FIG. 1b is a schematic diagram of a data reconciliation method provided in Embodiment 1 of the present application;

2 is a structural diagram of a data reconciliation device provided in Embodiment 2 of the present application;

FIG. 3 is a structural diagram of a computer device according to Embodiment 3 of the present application.

Detailed ways

The present application will be described below with reference to the accompanying drawings and embodiments.

The multiple steps described in the method embodiments of the present application may be performed in different orders and/or in parallel.

Concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units relation.

Example 1

FIG. 1 is a flowchart of a data reconciliation method provided in Embodiment 1 of the present application. This embodiment can be applied to a scenario where data access is required. The method can be executed by a data reconciliation device, which can use software and/or hardware, and can be integrated into an electronic device that has storage and computing capabilities and can perform text processing. As shown in Figure 1, the technical solution provided by this embodiment includes the following steps:

Step S110, acquiring the data to be accessed in the data source.

In the embodiment of the present application, the data source is the acquisition source of the data to be accessed, which is equivalent to the location where the required data to be accessed is stored. The data source may be a collection of to-be-accessed data corresponding to multiple access tasks, or may be an access source, a database, or a file system of the to-be-accessed data. Wherein, the data structure of each data bar in the data source includes the unique identifier of each data bar, and the unique identifier of each data bar will not change during the data access process. The data to be accessed is the data that needs to be accessed, and the data to be accessed has a corresponding relationship with the access task.

In this embodiment of the present application, acquiring the data to be accessed in the data source may be to determine the data to be accessed according to the access task, that is, the data to be accessed, and correspondingly acquire the data to be accessed from the data source. An access task is a specified event that requires data access. For example, start the access task at the set time, and the access task is to access the input data of yesterday's file system, then determine that the input data of yesterday's file system is the data to be accessed, and obtain the input data of yesterday's file system from the file system. data. For example: at the set time at 0:00 on September 16, 2020, start the access task of the entered data of the file system yesterday, then the access task is to access the data entered in the file system on September 15, 2020, and from the file Obtain the data entered in the file system on September 15, 2020 in the system.

Step S120: Perform first accounting for the data to be accessed to obtain first accounting content.

In the embodiment of the present application, after the data to be accessed is obtained from the data source, the first accounting is performed on the obtained data to be accessed. The first accounting is to record the data volume and data content of the data to be accessed. The first accounting content is the data volume and data content information of the data to be accessed obtained after the first accounting of the data to be accessed.

In the embodiment of the present application, after the data to be accessed is obtained from the data source, the first accounting is performed on the obtained data to be accessed. The first accounting of the data to be accessed is to record the data to be accessed as the original data, so that after the access action is completed, the access data and the data to be accessed are checked, and the data bars and data fields before and after access are judged. Whether the content has changed.

Performing the first accounting of the data to be accessed to obtain the first accounting content includes: acquiring the number of data bars of the data to be accessed; determining the data bar according to the data content corresponding to each data bar The core field corresponding to each data bar; the fingerprint information of the core field corresponding to each data bar is determined by the information digest algorithm.

In the embodiment of the present application, the data in the data source is stored in sequence according to the data strips, and the number of data strips of the data to be accessed obtained from the data source reflects the data volume of the data to be accessed; The fingerprint information corresponding to the data bar reflects the data content of the data to be accessed. The core field corresponding to each data bar is a field whose data content corresponding to each data bar is fixed.

In the embodiment of the present application, the fingerprint information corresponding to each data bar is the fingerprint information generated according to the core field corresponding to each data bar by using a message digest algorithm, for example, by using a message-digest algorithm 5 (Message-Digest algorithm 5, MD5) The MD5 value is generated according to the core field corresponding to each data strip, and the fingerprint information generated corresponding to each data strip includes the unique identifier of the data strip and the MD5 value corresponding to the data strip.

In the embodiment of the present application, the first accounting content of the data to be accessed includes a data bar accounting statement and a data fingerprint information record table, wherein the data bar accounting statement is used to record the data bar of the data to be accessed. Quantity; the data fingerprint information record table is used to record the fingerprint information corresponding to each data bar of the data to be accessed. The fingerprint information generated corresponding to each data bar of the data to be accessed contains the unique identifier of each data bar, because one MD5 value obtained by using the MD5 algorithm in the information digest algorithm may correspond to multiple data bars, so set each data bar. The unique identifier of the data strip is used in combination with the MD5 value to make the fingerprint information generated by the information digest algorithm unique.

Exemplarily, the first billing content obtained by performing the first billing of the data to be accessed includes a bill for the number of data pieces, and a sample bill for the number of data pieces is shown in Table 1 below:

Table 1

In Table 1, the bill for the number of data pieces of the first billing content of the data to be accessed includes: billing date, number of data pieces, and billing type. The accounting date is the date when the data to be accessed is entered into the data source, and the accounting type is the data information source. For example, the billing type "1" in Table 1 indicates that the data comes from a data source, and the billing type "2" indicates that the data comes from an access database.

Exemplarily, the fingerprint information corresponding to the data bar of the data to be accessed is accounted for, and the data fingerprint information record table is shown in Table 2 below:

Table 2

In Table 2, the data fingerprint information record table of the first accounting content of the data to be accessed includes: date, data bar ID, accounting type, and data fingerprint information. Among them, the date is the date when the fingerprint information is generated from the data to be accessed; the data bar ID is the ID of each data bar of the data to be accessed, and the data bar ID will not change during the data access process; the accounting type is data Information source; data fingerprint information is the fingerprint information generated by the information digest algorithm according to the core field corresponding to each data bar. Wherein, the data bar IDs "1" and "2" in Table 2 are only exemplary to show that the unique identifier of each data bar will not change before and after data access, and are not actual data bar IDs. Among them, the accounting type "1" in Table 2 indicates that the data comes from a data source, and the accounting type "2" indicates that the data comes from an access database.

The determining of the core field corresponding to each data bar according to the data content corresponding to each data bar includes: matching the corresponding data bar corresponding to each data bar in a preset core field library according to the data content corresponding to each data bar. core fields.

In the embodiment of the present application, the core field corresponding to each data bar is a field whose data content corresponding to each data bar is fixed. The fields whose data content is fixed and unchanged corresponding to each data bar are fields that will not change after logical processing during the data access process. The preset core field library is a database that stores fields that will not change after logical processing.

Exemplarily, it is assumed that the data to be accessed in the data source has the following structure:

Table name: t_express_info (express information table);

Field:

id,code,name,weight,origin_name,origin_tel,dest_name,dest_tel,delivery_addr,send_time,receive_time,remark,create_time,update_time;

The access data of the access target database has the following structure:

Table name: t_delivery (delivery management);

Field:

id,dvcode,dvname,dvweight,dvoriginname,dvoriginphone,dvmanname,dvmanphone,dvaddress,dvmantime,dvreceivetime,dvremark,dvcreatetime,dvupdatetime,dvtype,dvlongitude,dvlatitude,input_time;

Exemplarily, FIG. 1a is a schematic diagram of a data reconciliation method provided in Embodiment 1 of the present application. As shown in FIG. 1a, the process of determining the core field of the data to be accessed is as follows:

A field whose data content corresponding to each data bar in the data to be accessed is fixed and unchanged is a field that will not change after logical processing during the data access process. As shown in Figure 1a: id, code, name, weight, origin_name, origin_teldest_namedest_tel, delivery_addr, send_time, receive_time, remark, create_time, update_time are the core fields of the data content of the data bar. Among them, in the process of accessing the data to be accessed into the target database, the four fields of dvtype, dvlongitude, dvlatitude, and input_time are extended fields. After the data bar in the data to be accessed has undergone a series of logical processing in the data access process, it may happen that the data content of the data bar has not changed in substance, but the structure of the data bar has changed, but the Changes in the structure of the data strip do not affect the normal use of the data strip, and fingerprint information can be generated according to the core field of the data content of the data strip to check the data content of the access data.

In the embodiment of the present application, matching is performed in the preset core field database according to the data content corresponding to each data bar, and the field in the preset core field database that is successfully matched with the data content corresponding to each data bar is used as the corresponding field of the current data bar The core field of the data content.

Step S130: Access the data to be accessed to a target database to obtain access data.

In the embodiment of the present application, the target database is a database that stores the access data after accessing the data to be accessed. The access data is the data stored in the target database after being accessed.

In the embodiment of the present application, in the process of accessing the data to be accessed to the target database, the data to be accessed needs to undergo a series of logical processing, such as data discarding, field filling, conversion, deduplication, and so on. The data to be accessed is stored in the target database after a series of logical processing, and the data stored in the target database is used as the access data.

Step S140: Perform second accounting for the access data to obtain second accounting content.

In the embodiment of the present application, the data to be accessed is stored in the target database after being accessed, and the data stored in the target database is used as the access data, and the second time accounting is performed on the access data. The second accounting is to record the data volume and data content of the access data. The second accounting content is the data volume and data content information of the access data obtained after the second accounting of the access data is performed.

In the embodiment of the present application, after the access data is obtained from the target database, the second time accounting is performed on the obtained access data. The second accounting of the access data is to record the access data as the original data, so that after the access action is completed, the access data and the data to be accessed are checked, and the content of the data bar and data field before and after the access is judged. changes.

Step S150, in the case that the first accounting content and the second accounting content are the same, it is determined that the reconciliation is successful; wherein, the first accounting content includes: the data bar of the data to be accessed. The number and fingerprint information, and the second accounting content includes: the number and fingerprint information of the data bars of the access data.

In this embodiment of the present application, it is checked whether the first billing content and the second billing content are the same, that is, the number and fingerprint information of the data to be accessed in the first billing content and the access data in the second billing content are checked. Whether the number of data strips and the content of fingerprint information are consistent respectively. According to the number of data strips and the fingerprint information of the data content of the data strip, it is checked whether the data to be accessed is consistent with the data to be accessed. , then the data to be accessed is consistent with the access data, that is, the data before and after access are consistent, and the account reconciliation is successful.

The first accounting content further includes: the identifier of each data bar of the data to be accessed, and the second accounting content further includes: the identifier of each data bar of the access data; In the case that the first accounting content and the second accounting content are the same, determining that the reconciliation is successful includes: acquiring the identifier and the identifier of each data bar of the data to be accessed in the first accounting content The identifier of each data bar of the data to be accessed in the second accounting content; there is a data bar with the same identifier as each data bar of the data to be accessed in the access data, and The fingerprint information of the data to be accessed and the data strips with the same identification in the access data are the same, and the number of data strips in the first accounting content is the same as the number of data strips in the second accounting content. If the number is the same, it is determined that the reconciliation is successful.

Exemplarily, FIG. 1b is a schematic diagram of a data reconciliation method provided in Embodiment 1 of the present application. As shown in FIG. 1b, the data reconciliation process is as follows:

Determine the data to be accessed according to the access task and obtain the data to be accessed from the data source, perform first accounting on the acquired data to be accessed in the accounting module 1, obtain the first accounting content, and obtain the accounting The first accounting content in the accounting module 1 is stored in the reconciliation module. After the first accounting content is obtained, the data to be accessed is stored in the target database through the data access service module. Taking the inbound data as the access data, the second accounting is performed in the accounting module 2, and the second accounting content is obtained and stored in the reconciliation module. After the second accounting content is stored in the reconciliation module, the first accounting content and the second accounting content in the reconciliation module are checked, and the number of data bars in the first accounting content and the value of each data bar are checked. Whether the fingerprint information is consistent with the number of data bars in the second accounting content and the fingerprint information of each data bar respectively, if the number of data bars in the first accounting content and the fingerprint information of each data bar are consistent with the second accounting content If the number of data bars and the fingerprint information of each data bar are consistent, the reconciliation is successful, and the reconciliation is completed.

In this embodiment of the present application, it is checked whether the number of data bars of the first accounting content is consistent with the number of data bars of the second accounting content. If the number of bars is the same, then according to the property of the unique identifier of each data bar that does not change before and after data access, make a one-to-one correspondence with the unique identifiers of all data bars in the first accounting content and the second accounting content, and check. Whether the fingerprint information of the data strips with the same unique identifier in the first accounting content and the second accounting content are consistent, if the unique identifiers of all data strips in the first accounting content can be found in the second accounting content, and have If the fingerprint information of the data strips with the same unique identification is consistent, the reconciliation is successful.

The data reconciliation method further includes: in the case that the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, in the case of the second accounting content After the accounting time interval preset time, the third accounting is performed on the access data to obtain the third accounting content; in the case that the first accounting content and the third accounting content are different, Add 1 to the number of account failures; repeatedly perform the next accounting of the access data after a preset time interval with the accounting time interval of the previous accounting, and obtain the next accounting content, in the first If the accounting content is different from the next accounting content, the operation of adding 1 to the number of failed reconciliations is performed until the number of failed reconciliations is greater than or equal to the preset number of failed reconciliations, and it is determined that reconciliation fails.

In this embodiment of the present application, the preset time is the period of time for re-obtaining access data through refresh. The third accounting is to record the data volume and data content of the re-obtained access data after being refreshed by a preset time interval with the accounting time interval of the second accounting. The third accounting content is the data volume and data content information of the access data obtained by performing third accounting on the access data obtained after refresh. The preset number of reconciliation failures is the maximum number of reconciliation attempts allowed during the set reconciliation process. The number of reconciliation failures is the cumulative number of reconciliation failures during the reconciliation process.

In this embodiment of the present application, if the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, the data to be accessed has missing data bars or data access failure during the access process. complete situation. For example, the number of data bars in the first accounting content is 10,000, while the number of data bars in the second accounting content is 9,000, and the remaining 1,000 data bars may be data bars being accessed or in the process of accessing missing. If the data to be accessed has incomplete data access during the access process, the access data needs to be refreshed after a preset time interval from the accounting time interval for the second accounting, and the refreshed access data needs to be refreshed. Enter the data for the third accounting, and obtain the third accounting content. If the first accounting content is different from the third accounting content, the number of failed accounting is incremented by 1. Setting the preset time can avoid incomplete data access even after the system is refreshed. Accumulate the number of reconciliation failures, and compare the accumulated number of reconciliation failures this time with the preset number of reconciliation failures, which can prevent the reconciliation time from being extended due to excessive refresh times. After refreshing, the access data is obtained again, the access data is recorded for the third time to obtain the third accounting content, and the first accounting content and the third accounting content are checked, which can avoid waiting due to incomplete data access. The access data and the access data are inconsistent. Exemplarily, the number of data bars in the first accounting content is 10,000, and the number of data bars in the second accounting content is 9,000, and the remaining 1,000 data bars may still be accessed. Refresh the access data every 15 minutes to perform the third accounting, and the number of data bars in the third accounting content is still 9000, then the reconciliation fails, the number of reconciliation failures is increased by 1, and it is judged that the reconciliation fails. Whether the number of reconciliation failures of the account is less than the preset number of reconciliation failures, if the number of reconciliation failures in this reconciliation is less than the preset number of reconciliation failures, refresh at an interval of 15 minutes to obtain access data, and verify the access data. Carry out the fourth bookkeeping, obtain the number of data bars in the fourth bookkeeping content and check the number of data bars in the fourth bookkeeping content with the number of data bars in the first bookkeeping content, if the first bookkeeping The number of data bars in the accounting content is greater than the number of data bars in the fourth accounting content, and the number of reconciliation failures is added by 1 to determine whether the number of reconciliation failures in this reconciliation is equal to the preset number of reconciliation failures. If the number of reconciliation failures in reconciliation is equal to the preset number of reconciliation failures, the current reconciliation fails and the reconciliation is stopped.

The data reconciliation method further includes: in the case that the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, in the case of the second accounting content After the accounting time interval preset time, the third accounting is performed on the access data to obtain the third accounting content; in the case that the first accounting content and the third accounting content are different, The reconciliation time is recorded according to the preset time; after the preset time interval between the accounting time of the previous accounting and the previous accounting is performed repeatedly, the next accounting is performed on the access data, and the next accounting content is obtained. In the case where the first billing content and the next billing content are different, the operation of recording the account reconciliation time according to the preset time, until the account reconciliation time is greater than or equal to the preset account reconciliation time, determine Reconciliation failed.

In the embodiment of the present application, the reconciliation time is the accumulated reconciliation time in the reconciliation process. The preset reconciliation time is the reconciliation time length preset according to the access task. If the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, the data to be accessed may be missing data bars or incomplete data access during the access process. Increasing the preset time and setting the reconciliation after refreshing the access data may increase the overall reconciliation time of this reconciliation. Multiple reconciliation attempts will greatly increase the reconciliation time, increase the setting of the preset reconciliation time, and avoid the large amount of reconciliation attempts caused by multiple reconciliation attempts when the data to be accessed and the amount of access data are large. increase in magnitude.

The data reconciliation method further includes: the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, and the pending accounting content in the first accounting content If the result of the comparison between the identifier of each data bar of the incoming data and the identifier of each data bar of the access data in the second accounting content is that the intermediate data bar is missing, it is determined that the reconciliation fails.

In this embodiment of the present application, if the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, the data to be accessed has missing data bars or data access failure during the access process. complete situation. The unique identifier of each data bar in the first accounting content is compared with the unique identifier of each data bar in the second accounting content, if the unique identifiers of all data bars in the second accounting content are less than those of the first accounting content The unique identifier of all the data bars in the accounting content, and compared with the first accounting content, the unique identifier of the missing data bar in the second accounting content is in the middle of the unique identifiers of all the data bars in the first accounting content, Then, compared with the first accounting content, the second accounting content is missing an intermediate data bar.

In this embodiment of the present application, it is checked whether the number of data bars of the first accounting content is consistent with the number of data bars of the second accounting content. If the number of data bars of the first accounting content and the data bars of the second accounting content are consistent If the number of data bars is the same, then according to the property of the unique identifier of each data bar that does not change before and after access, make a one-to-one correspondence with the unique identifiers of all data bars in the first accounting content and the second accounting content, and check the first accounting content. Whether the fingerprint information of the data bar with the same unique identifier in the accounting content and the second accounting content is consistent, if the unique identifiers of all data bars in the first accounting content can be found in the second accounting content, and have the same unique identifier If the fingerprint information of the identified data strip is consistent, the account reconciliation is successful.

In this application, the first accounting is performed on the data to be accessed, and after obtaining the first accounting content, the data to be accessed is connected to the target database to obtain the access data; the second accounting is performed on the access data to obtain the second accounting Check whether the first billing content and the second billing content are consistent; to check the number of data bars and fingerprint information of the data to be accessed and the number and fingerprint information of the data bars to access data, so as to achieve the connection between the data The technical effect of accurate reconciliation of the data before and after the entry.

Embodiment 2

FIG. 2 is a structural diagram of a data reconciliation device provided in Embodiment 2 of the present application. As shown in FIG. 2 , an embodiment of the present application also provides a data reconciliation device, which includes: a data acquisition module 21, which sets In order to obtain the data to be accessed in the data source; the first accounting module 22 is configured to perform first accounting on the data to be accessed to obtain the first accounting content; the data access module 23 is configured to The data to be accessed is accessed to the target database to obtain access data; the second accounting module 24 is configured to perform second accounting for the access data to obtain second accounting content; the accounting module 25 , set to determine that the reconciliation is successful when the first accounting content and the second accounting content are the same; wherein, the first accounting content includes: the data of the data bar of the data to be accessed The number and fingerprint information, and the second accounting content includes: the number and fingerprint information of the data bars of the access data.

The first accounting module 22 is configured to: obtain the number of data bars of the data to be accessed; determine the core field corresponding to each data bar according to the data content corresponding to each data bar; use an information digest algorithm to determine each data bar. Fingerprint information of the core field corresponding to the data bar.

The first accounting module 22 is configured to determine the core field corresponding to each data bar according to the data content corresponding to each data bar in the following manner: match all the data in the preset core field library according to the data content corresponding to each data bar. Describe the core fields corresponding to each data bar.

The first accounting content further includes: an identifier of each data bar of the data to be accessed, and the second accounting content further includes: an identifier of each data bar of the access data; an account reconciliation module 25 is set to: obtain the identifier of each data bar of the data to be accessed in the first accounting content and the identifier of each data bar of the access data in the second accounting content; There is a data bar with the same identifier as each data bar of the to-be-accessed data in the access data, and the to-be-accessed data and the access data have the same fingerprint information of the data bar with the same identifier, and If the number of data bars in the first accounting content is the same as the number of data bars in the second accounting content, it is determined that the reconciliation is successful.

The reconciliation module 25 is further configured to: in the case that the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, in the case of After the accounting time interval preset time, the third accounting is performed on the access data to obtain the third accounting content; in the case that the first accounting content and the third accounting content are different, Add 1 to the number of account failures; repeatedly perform the next accounting of the access data after a preset time interval with the accounting time interval of the previous accounting, and obtain the next accounting content, in the first If the accounting content is different from the next accounting content, the operation of adding 1 to the number of failed reconciliations is performed until the number of failed reconciliations is greater than or equal to the preset number of failed reconciliations, and it is determined that reconciliation fails.

The reconciliation module 25 is further configured to: in the case that the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, in the case of After the accounting time interval preset time, the third accounting is performed on the access data to obtain the third accounting content; in the case that the first accounting content and the third accounting content are different, The reconciliation time is recorded according to the preset time; after the preset time interval between the accounting time of the previous accounting and the previous accounting is performed repeatedly, the next accounting is performed on the access data, and the next accounting content is obtained. In the case where the first billing content and the next billing content are different, the operation of recording the account reconciliation time according to the preset time, until the account reconciliation time is greater than or equal to the preset account reconciliation time, determine Reconciliation failed.

The account reconciliation module 25 is further configured to: the number of data bars in the first bookkeeping content is greater than the number of data bars in the second bookkeeping content, and the number of data bars in the first bookkeeping content If the result of the comparison between the identifier of each data bar of the incoming data and the identifier of each data bar of the access data in the second accounting content is that the intermediate data bar is missing, it is determined that the reconciliation fails.

A data reconciliation device provided by an embodiment of the present application can execute the data reconciliation method provided by any embodiment of the present application, and has corresponding functional modules and effects for executing the method.

Embodiment 3

FIG. 3 is a schematic structural diagram of a computer device according to Embodiment 3 of the present application. Figure 3 shows a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present application. The computer device 12 shown in FIG. 3 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 3, computer device 12 takes the form of a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , system memory 28 , and a bus 18 connecting various system components including system memory 28 and processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, enhanced ISA bus, Video Electronics Standards Association (Video Electronics Standards Association) Association, VESA) local bus and Peripheral Component Interconnect (PCI) bus.

Computer device 12 includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including both volatile and nonvolatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . Computer device 12 may include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 3, commonly referred to as a "hard disk drive"). Although not shown in Figure 3, a magnetic disk drive may be provided for reading and writing to removable non-volatile magnetic disks, such as "floppy disks", and to removable non-volatile optical disks, such as Compact Disc Read-Only Memory (Compact Disc) Read-Only Memory, CD-ROM), Digital Versatile Disc Read-Only Memory (DVD-ROM) or other optical media, CD-ROM drive for reading and writing. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. The memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of the embodiments of the present application.

A program/utility 40 having a set (at least one) of program modules 42, which may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data , each or a combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

Computer device 12 may also communicate with one or more external devices 14 (eg, keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with computer device 12, and/or communicate with Any device (eg, network card, modem, etc.) that enables the computer device 12 to communicate with one or more other computing devices. Such communication may take place through an input/output (I/O) interface 22 . Furthermore, the computer device 12 may also communicate with one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through the network adapter 20. As shown in FIG. 3 , network adapter 20 communicates with other modules of computer device 12 via bus 18 . It should be understood that, although not shown in FIG. 3, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, redundant independent disks Array (Redundant Arrays of Independent Disks, RAID) systems, tape drives and data backup storage systems.

The processing unit 16 executes a variety of functional applications and data processing by running the program stored in the system memory 28, for example, implementing the data reconciliation method provided by the embodiment of the present application, the method includes: acquiring the data to be accessed in the data source perform first accounting on the data to be accessed to obtain first accounting content; access the data to be accessed to the target database to obtain access data; perform a second accounting on the access data The second accounting content is obtained, and the second accounting content is obtained; if the first accounting content and the second accounting content are the same, it is determined that the reconciliation is successful; wherein, the first accounting content includes: the The number and fingerprint information of the data pieces of the data to be accessed, and the second accounting content includes: the number and fingerprint information of the data pieces of the access data.

Embodiment 4

Embodiment 4 of the present application provides a computer-readable storage medium storing a computer program, and when the program is executed by a processor, is used to execute a data reconciliation method. The method includes: acquiring a data source to be accessed in a data source data; perform first accounting on the data to be accessed to obtain first accounting content; access the data to be accessed to the target database to obtain access data; perform second accounting on the access data The second accounting content is obtained, and the second accounting content is obtained; if the first accounting content and the second accounting content are the same, it is determined that the reconciliation is successful; wherein, the first accounting content includes: the The number of data bars and fingerprint information of the data to be accessed, and the second accounting content includes: the number of data bars and fingerprint information of the data to be accessed.

The computer storage medium of the embodiments of the present application may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. Examples (non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, RAM, Read-Only Memory (ROM), erasable Erasable Programmable Read Only Memory (EPROM) or flash memory, optical fiber, CD-ROM, optical storage device, magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present application may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional A procedural programming language, such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a LAN or WAN, or may be connected to an external computer, such as through the Internet using an Internet service provider.

Claims

A data reconciliation method, including:

Obtain the data to be accessed from the data source;

Perform first accounting for the data to be accessed to obtain first accounting content;

Accessing the data to be accessed to a target database to obtain access data;

Carrying out the second accounting for the access data to obtain the second accounting content;

In the case that the first accounting content and the second accounting content are the same, it is determined that the reconciliation is successful;

The first accounting content includes: the number of data bars and fingerprint information of the data to be accessed, and the second accounting content includes: the number of data bars and fingerprint information of the access data.
The method according to claim 1, wherein the first accounting for the data to be accessed to obtain the first accounting content comprises:

obtaining the number of data bars of the data to be accessed;

Determine the core field corresponding to each data bar according to the data content corresponding to each data bar;

The information digest algorithm is used to determine the fingerprint information of the core field corresponding to each data bar.
The method according to claim 2, wherein the determining the core field corresponding to each data bar according to the data content corresponding to each data bar comprises:

The core field corresponding to each data bar is matched in the preset core field library according to the data content corresponding to each data bar.
The method according to claim 3, wherein the first accounting content further comprises: an identifier of each data bar of the data to be accessed, and the second accounting content further comprises: the access data the identification of each data strip;

In the case that the first accounting content and the second accounting content are the same, determining that the reconciliation is successful includes:

acquiring the identifier of each data bar of the data to be accessed in the first accounting content and the identifier of each data bar of the access data in the second accounting content;

In the access data, there is a data bar with the same identification as each data bar of the to-be-accessed data, and the to-be-accessed data and the data bar with the same identification in the access data have the same fingerprint information , and if the number of data bars in the first accounting content is the same as the number of data bars in the second accounting content, it is determined that the reconciliation is successful.
The method of claim 4, further comprising:

In the case that the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, after a preset time interval with the accounting time interval of the second accounting , carry out the third accounting for the access data, and obtain the third accounting content;

In the case that the first accounting content and the third accounting content are different, add 1 to the number of accounting failures;

Repeatedly perform the next accounting for the access data after the accounting time interval from the previous accounting for a preset time to obtain the next accounting content, where the first accounting content and the next accounting In the case of different accounting contents, the operation of adding 1 to the number of account reconciliation failures is performed until the number of account reconciliation failures is greater than or equal to the preset number of account reconciliation failures, and it is determined that the account reconciliation fails.
The method of claim 4, further comprising:

In the case that the number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, after a preset time interval with the accounting time interval of the second accounting , carry out the third accounting for the access data, and obtain the third accounting content;

In the case that the first accounting content and the third accounting content are different, record the reconciliation time according to the preset time;

Repeatedly perform the next accounting for the access data after the accounting time interval from the previous accounting for a preset time to obtain the next accounting content, where the first accounting content and the next accounting When the accounting contents are different, the operation of recording the account reconciliation time according to the preset time, until the account reconciliation time is greater than or equal to the preset account reconciliation time, it is determined that the account reconciliation fails.
The method of claim 4, further comprising:

The number of data bars in the first accounting content is greater than the number of data bars in the second accounting content, and the number of each data bar of the data to be accessed in the first accounting content If the result of the comparison between the identifier and the identifier of each data bar of the access data in the second accounting content is that the intermediate data bar is missing, it is determined that the reconciliation fails.
A data reconciliation device, comprising:

a data acquisition module, set to acquire the data to be accessed in the data source;

a first billing module, configured to perform first billing on the data to be accessed to obtain first billing content;

a data access module, configured to access the data to be accessed to a target database to obtain access data;

A second accounting module, configured to perform second accounting for the access data to obtain second accounting content;

A reconciliation module, configured to determine that the reconciliation is successful when the first accounting content and the second accounting content are the same;

Wherein, the first accounting content includes: the number of data bars of the data to be accessed and fingerprint information, and the second accounting content includes: the number of data bars and fingerprint information of the access data.
A computer device comprising:

at least one processor;

a storage device configured to store at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the data reconciliation method according to any one of claims 1-7.
A computer-readable storage medium storing a computer program, wherein when the program is executed by a processor, the data reconciliation method according to any one of claims 1-7 is implemented.