CN118114649A

CN118114649A - Data processing method and device, equipment, storage medium and program product

Info

Publication number: CN118114649A
Application number: CN202410327962.1A
Authority: CN
Inventors: 刘月
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2024-03-21
Filing date: 2024-03-21
Publication date: 2024-05-31

Abstract

The disclosure provides a data processing method, which can be applied to the technical field of big data. The data processing method comprises the following steps: obtaining a modified target field of a target service table in a preset historical time period, and obtaining an original value corresponding to the target field before modification and a modified value corresponding to the target field after modification, wherein the target service table is a service table to be checked by a user; determining a plurality of target linkage fields with association relation with the target field, and determining a field calculation rule between the target field and the plurality of target linkage fields; generating an ideal reference table based on the original value and the modified value of the target field, a plurality of target linkage fields and a field calculation rule; and generating a checking result aiming at the target service table according to the ideal reference table and the target service table. The present disclosure also provides a data processing apparatus, device, storage medium, and program product.

Description

Data processing method and device, equipment, storage medium and program product

Technical Field

The present disclosure relates to the field of big data technologies, and in particular, to a data processing method, apparatus, device, medium, and program product.

Background

The financial institution needs to report the supervision report data to the supervision institution regularly, and when reporting, the specific report data comprises a business report detail data table under each business scene, and the supervision institution performs data verification and feeds back the verification result to the financial institution aiming at the reported data. And then, the financial institution carries out data treatment based on the early warning items in the verification result, regenerates a treated business report detail data table, and re-reports the treated data to the supervision institution, wherein the data treatment relates to a small part of the total business data, but the total data still needs to be re-reported. In the process of data re-reporting, before the supervision is formally reported, the data in the data table needs to be checked to check whether the related change data item and field value are correct.

In the process of realizing the disclosed concept, the inventor finds that at least the following problems exist in the related art, because the data check table generated by recalculation and most of data of the data check table before data treatment are the same, the number of data bars is basically the same, and only the numerical value of a small number of data items is changed, but before checking, it cannot be determined which related data items are changed, and whether the numerical value is changed correctly, and because the number of data bars of the data table is very large, the existing checking method is mostly manual or computer execution check calculation piece by piece, the processing efficiency is low, and more time, manpower and calculation resources are consumed.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a data processing method, apparatus, device, medium, and program product.

In a first aspect of the present disclosure, there is provided a data processing method, including: obtaining a modified target field of a target service table in a preset historical time period, and obtaining an original value corresponding to the target field before modification and a modified value corresponding to the target field after modification, wherein the target service table is a service table to be checked by a user;

determining a plurality of target linkage fields with association relation with the target field, and determining a field calculation rule between the target field and the plurality of target linkage fields;

generating an ideal reference table based on the original value and the modified value of the target field, a plurality of target linkage fields and a field calculation rule;

And generating a checking result aiming at the target service table according to the ideal reference table and the target service table.

According to an embodiment of the present disclosure, determining a plurality of target linkage fields having an association relationship with a target field, and determining a field calculation rule between the target field and the plurality of target linkage fields includes:

Reading a service field association diagram pre-constructed for a target service table from a diagram database, wherein the service field association diagram comprises N nodes corresponding to N service fields, the types of the nodes are active nodes and/or linkage nodes, the nodes with association relations are connected through directed edges, the directed edges point to the linkage nodes from the active nodes, and the attributes of the directed edges comprise: a field calculation rule between service fields with association relation;

Based on the service field association map, a plurality of target linkage fields with association relation with the target fields are determined, and a field calculation rule between the target fields and the plurality of target linkage fields is determined.

According to an embodiment of the present disclosure, the data processing method further includes:

acquiring a history modification log related to a target business table;

a log analysis tool is called to analyze the history modification log, and N business fields included in a target business table, the association relation among the N business fields and the field calculation rule among the business fields with the association relation are determined;

Based on N service fields, the association relation among the N service fields and the field calculation rule among the service fields with the association relation, constructing a service field association map aiming at the target service table.

acquiring a history modification log related to a target business table;

The method comprises the steps of calling a log analysis tool to analyze a history modification log, determining a plurality of target linkage fields with association relation with target fields, and determining field calculation rules between the target fields and the plurality of target linkage fields.

According to an embodiment of the present disclosure, invoking a log parsing tool parses a history modification log, determining a plurality of target linkage fields having an association relationship with a target field, and determining a field calculation rule between the target field and the plurality of target linkage fields includes:

Calling a log analysis tool to analyze a history modification log and determining a linkage modification log related to a target field, wherein the linkage modification log is generated based on modification operation trigger of the target field;

Extracting a plurality of target linkage fields, first field value change information of the target fields and second field value change information of each target linkage field from the linkage modification log;

A field calculation rule is generated that determines between the target field and the plurality of target linkage fields based on the first field value change information and the second field value change information using a predetermined numerical fit algorithm.

According to an embodiment of the present disclosure, generating an ideal reference table based on an original value and a modified value of a target field, a plurality of target linkage fields, a field calculation rule includes:

Based on the original value and the modified value of the target field and the field calculation rules between the target field and the plurality of target linkage fields, calculating corresponding reference values after the field values of the target linkage fields are triggered to change under the condition that the target field is changed from the original value to the modified value;

an ideal reference table is generated based on the modified values of the target fields, the reference values of the respective target linkage fields.

According to an embodiment of the present disclosure, generating a collation result for a target service table from an ideal reference table and the target service table includes:

inquiring a target service table by taking a field name of a target linkage field as a main key to obtain a field value to be checked, corresponding to the target linkage field, in the target service table;

and checking the field value to be checked corresponding to the target linkage field with the reference value to generate a checking result aiming at the target business table.

A second aspect of the present disclosure provides a data processing apparatus comprising: the first acquisition module is used for acquiring the modified target field of the target service table in a preset historical time period, and acquiring an original value corresponding to the target field before modification and a modification value corresponding to the target field after modification, wherein the target service table is a service table to be checked by a user;

The first determining module is used for determining a plurality of target linkage fields with association relation with the target fields and determining field calculation rules between the target fields and the plurality of target linkage fields;

The first generation module is used for generating an ideal reference table based on the original value and the modified value of the target field, a plurality of target linkage fields and a field calculation rule; and

And the second generation module is used for generating a checking result aiming at the target service table according to the ideal reference table and the target service table.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

A fourth aspect of the present disclosure also provides a computer readable storage medium having stored thereon a computer program or instructions which, when executed by a processor, implement the steps of the above method.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program or instructions which, when executed by a processor, performs the steps of the method described above.

According to the embodiment of the disclosure, an ideal reference table is generated by acquiring an association relationship between a target field and a target linkage field and determining a field calculation rule between the target field and the target linkage field based on an original value and a modified value of the target field. Since the ideal reference table is derived from the target field and the target linkage field associated with the target field, only the field value of the modified traffic field associated with the target field can be reflected in the ideal reference table. Thus, the data change amount including only the changed service field and each field value is clearly obtained in the ideal reference table. Furthermore, the target service table can be compared based on the ideal reference table, and only the changed service data is actually checked without checking all the data, so that the data volume of the data to be checked is reduced to a great extent, and the data processing efficiency is improved.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a data processing method, apparatus, device, medium and program product according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the disclosure;

FIG. 3 schematically illustrates a flow chart of a method of determining field computation rules between a target field and a plurality of target linkage fields, in accordance with an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of generating an ideal reference table according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a method of determining field computation rules between a target field and a plurality of target linkage fields, according to another embodiment of the present disclosure;

FIG. 6 schematically illustrates a schematic diagram of a business field association map of a target business table of an embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of generating a collation result for a target business table in accordance with an embodiment of the disclosure;

FIG. 8 schematically illustrates a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and

Fig. 9 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a convention should be interpreted in accordance with the meaning of one of skill in the art having generally understood the convention (e.g., "a system having at least one of A, B and C" would include, but not be limited to, systems having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical solution of the present disclosure, the related user information (including, but not limited to, user personal information, user image information, user equipment information, such as location information, etc.) and data (including, but not limited to, data for analysis, stored data, displayed data, etc.) are information and data authorized by the user or sufficiently authorized by each party, and the related data is collected, stored, used, processed, transmitted, provided, disclosed, applied, etc. in compliance with relevant laws and regulations and standards, necessary security measures are taken, no prejudice to the public order colloquia is provided, and corresponding operation entries are provided for the user to select authorization or rejection.

In the scenario of using personal information to make an automated decision, the method, the device and the system provided by the embodiment of the disclosure provide corresponding operation inlets for users, so that the users can choose to agree or reject the automated decision result; if the user selects refusal, the expert decision flow is entered. The expression "automated decision" here refers to an activity of automatically analyzing, assessing the behavioral habits, hobbies or economic, health, credit status of an individual, etc. by means of a computer program, and making a decision. The expression "expert decision" here refers to an activity of making a decision by a person who is specializing in a certain field of work, has specialized experience, knowledge and skills and reaches a certain level of expertise.

After the reported early warning items are subjected to data treatment, the treated data are required to be re-reported to a supervision organization. Before reporting, the data generated by the recalculation still needs to be checked to generate a data check table. Then, the data for the early warning item is generally a small part of data, and the data generated after the treatment is the same as the data before the treatment (i.e. the data which is not treated or modified) in the majority of the data, and the number of the data is basically the same. For this portion of data, on the one hand, no verification of data that has not been remedied or modified is required; on the other hand, it cannot be confirmed which part of the data is the linkage modification behavior due to the modification of the data. The existing checking method is mostly implemented by manual or computer to check and calculate piece by piece, and has low processing efficiency and consumes more time, manpower and calculation resources.

In view of this, an embodiment of the present disclosure provides a data processing method, including:

obtaining a modified target field of a target service table in a preset historical time period, and obtaining an original value corresponding to the target field before modification and a modified value corresponding to the target field after modification, wherein the target service table is a service table to be checked by a user;

Fig. 1 schematically illustrates an application scenario diagram of data processing according to an embodiment of the present disclosure.

As shown in fig. 1, an application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium used to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 through the network 104 using at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages, etc. Various communication client applications, such as a shopping class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only) may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

The first terminal device 101, the second terminal device 102, the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the data processing method provided in the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally provided in the server 105. The data processing method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The data processing method of the disclosed embodiment will be described in detail below with reference to fig. 2 to 7 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the data processing of this embodiment includes operations S210 to S240.

In operation S210, the modified target field of the target service table in the predetermined history period is obtained, and the original value corresponding to the target field before being modified and the modified value corresponding to the target field after being modified are obtained, wherein the target service table is a service table to be checked by the user.

According to embodiments of the present disclosure, the target business table may be a business report detail data table in different business scenarios, including a plurality of business fields, for example, including but not limited to "transaction amount", "ticket amount", "interest rate", and the like. The target field is used to characterize the modified service field in the target service table, such as "transaction amount" or "interest rate" and the like. The predetermined historical time period may be a time period with a period unit of hours, days, weeks, months, years, etc., and may be a time period customized according to business requirements, for example, 30 days or 3 months, etc.

According to an embodiment of the present disclosure, one or more target fields in a target business table to be currently checked that have been modified within a predetermined history period are obtained. For example, all service fields of the service table to be checked are acquired within 3 months, and one or more modified service fields in all service fields are determined as target fields.

Further, after determining which fields are target fields, the original values corresponding to the target fields before being modified and the modified values corresponding to the target fields after being modified are obtained. Specifically, for all field values corresponding to at least one target field, an original value corresponding to each field value before being modified and a modified value corresponding to each field value after being modified are obtained. For example, the target field is "ticket amount", and one of the corresponding field values is "100 yuan" before being modified and "150 yuan" after being modified.

In operation S220, a plurality of target linkage fields having an association relationship with the target field are determined, and a field calculation rule between the target field and the plurality of target linkage fields is determined.

According to an embodiment of the present disclosure, the target linkage field is a service field having an association relationship with the modified target field. One embodiment of determining the target linkage field may be to determine a traffic field calculated in association with the presence of the target field as the target linkage field. For example, the target field is "interest rate", the target linkage field is "interest" and the like.

According to an embodiment of the present disclosure, after a plurality of target linkage fields are determined according to an association relationship with the target field, a field calculation rule between the target field and the plurality of target linkage fields is determined. Taking the target linkage field and the target field in the above embodiment as examples, when the target field is "interest rate" and the target linkage field is "interest," the field calculation rule between the target field and the target linkage field may be a multiplication or division calculation rule; when the target linkage field is "money collection amount" and the target field is "money transfer amount", the field calculation rule between the target field and the target linkage field may be an addition or subtraction calculation rule.

In operation S230, an ideal reference table is generated based on the original value and the modified value of the target field, the plurality of target linkage fields, and the field calculation rule.

In the embodiment of the present disclosure, since the modified service field is determined as the target field, the original value before modification and the modified value after modification thereof can be directly obtained. But for the field value in the target linkage field associated with it, only its original value (the field value before modification) can be obtained, and it cannot be determined which fields in the plurality of target linkage fields have been modified or updated. Further, since the field calculation rule between the target field and the plurality of target linkage fields is determined, a modified value that is modified or updated in the target linkage field in an ideal case can be calculated according to the calculation rule.

Specifically, the original value and the modified value of each field value in the target field and the original value of the field value in the target linkage field corresponding to the target field are substituted into the field calculation rule, so that the modified value of the field value in the target linkage field corresponding to the target field can be obtained, and an ideal reference table is generated according to the ideal modified value.

In operation S240, a collation result for the target service table is generated based on the ideal reference table and the target service table.

According to the embodiment of the disclosure, since the ideal reference table is generated according to the association relation between the target field and the target linkage field and the calculation rule thereof, the ideal reference table contains all the fields and field values which are possibly modified, but does not contain the fields and field values which are not modified. When checking the replay data, only the data which is possibly updated needs to be checked, and the part of the data which is not updated does not need to be checked, so that the comparison and the check can be directly performed according to an ideal reference table during the check.

According to an embodiment of the present disclosure, an ideal reference table is compared with a target traffic table. Specifically, all the fields in the ideal reference table are checked against all the fields in the target service table and a check result is generated. Specifically, when the checking result is that the data in the target service table is consistent with the data in the ideal reference table, the fact that the re-reported data in the target service table is error-free can be reported is indicated; the checking result is that the data in the target service table is inconsistent with the data in the ideal reference table, which indicates that the re-reported data in the target service table is wrong, and the wrong data needs to be re-checked.

Fig. 3 schematically illustrates a flow chart of a method of determining field computation rules between a target field and a plurality of target linkage fields, in accordance with an embodiment of the present disclosure.

As shown in fig. 3, determining a plurality of target linkage fields having an association relationship with a target field, and determining a field calculation rule between the target field and the plurality of target linkage fields includes operation S310 and operation S320.

In operation S310, a history modification log related to a target service table is acquired.

In operation S320, the log parsing tool is called to parse the history modification log, determine a plurality of target linkage fields having an association relationship with the target field, and determine a field calculation rule between the target field and the plurality of target linkage fields.

According to the embodiment of the disclosure, the history modification log can be directly parsed by using the log parsing tool, the association relation between the service fields and the target fields is obtained, and the partial service fields are determined as the target linkage fields. Any of the prior art may be used for the log parsing tool, and no limitation is made herein.

Further, after the target field, the plurality of target linkage fields and the association relation between the target field and the plurality of target linkage fields are obtained, a field calculation rule between the target field and the plurality of target linkage fields is determined. One specific implementation of determining the field calculation rule may refer to operation S220 in the above embodiment, which is not described herein.

Further, another embodiment of determining the field calculation rule may include step 11 and step 13.

In step 11, a log parsing tool is invoked to parse the historical modification log to determine a linked modification log associated with the target field, wherein the linked modification log is generated based on a modification operation trigger to the target field.

According to embodiments of the present disclosure, since there is a potential business association between the target field and the target linkage field, the method of embodiments of the present disclosure aims to mine such potential business association rules and explicitly embody the potential business association between the target field and the target linkage field in various calculation rules. When the field value of the target field is modified, the field value of other service fields may be changed, for example, the target field is "interest rate", and when the corresponding field value is modified, the field value of the service field associated with the target field is "interest rate" is changed according to the historical calculation rule, and a linkage modification log is generated. The log analysis tool can be used for directly obtaining the partial log from the historical modification log and determining the partial log as the linkage modification log. The linked modification log is generated based on a modification operation trigger to the target field, and may include related modification records for the target field, as well as including modification records related to the target linked field.

In step 12, a plurality of target linkage fields, first field value change information of the target fields, and second field value change information of each target linkage field are extracted from the linkage modification log.

According to the embodiment of the disclosure, the linkage modification log contains the original value and the modified value corresponding to the target field before and after modification, and the linkage modification log can be directly read. Further, according to the original value and the modified value corresponding to the target field before and after being modified, calculating the first field value change information of the target field, specifically, differencing the original value and the modified value corresponding to the target field before and after being modified, and obtaining the first field value change information of the target field. Illustratively, for a certain field value in the target field, which is "15%" before modification and "25%" after modification, the first field value change information of the target field is 25% -15% =10%.

According to the embodiment of the disclosure, the linkage modification log contains the original value and the modified value corresponding to the target linkage field before and after modification, and the target linkage field can be directly read by reading the linkage modification log. Further, according to the original value and the modified value of the target linkage field before and after being modified, second field value change information of the target linkage field is calculated. Specifically, the original value and the modified value corresponding to the target linkage field before and after being modified are differenced to obtain second field value change information of the target linkage field. The difference calculation process and the calculation process of the first field value change information are not illustrated here.

In step 13, a field calculation rule is generated that determines between the target field and the plurality of target linkage fields based on the first field value change information and the second field value change information using a predetermined numerical fitting algorithm.

According to the embodiments of the present disclosure, in an actual application scenario, there may be a plurality of target fields (target field 1, target field 2, target field 3, etc.) and a plurality of target linkage fields (target linkage field 1, target linkage field 2, target linkage field 3, target linkage field 4, etc.), and the field calculation rule may be the same or different between each target field and each target linkage field. Therefore, a plurality of field calculation rules (field calculation rule 1, field calculation rule 2, field calculation rule 3, etc.) may be included between the plurality of target fields and the plurality of target linkage fields. The following is a specific embodiment illustrating the generation of field calculation rules for only one target field and one target linkage field.

According to an embodiment of the present disclosure, in step 12, the first field value change information Δx of the target field and the second field value change information Δy of the target link field extracted from the link modification log are actually extracted to obtain a plurality of first field value change information (Δx1, Δx2, Δx3, Δx4, etc.) and a plurality of second field value change information (Δy1, Δy2, Δy3, Δy4, etc.). Further, fitting the plurality of first field value change information and the plurality of second field value change information according to a predetermined numerical fitting algorithm to obtain a field calculation rule between the target field and the plurality of target linkage fields.

Illustratively, the target field is "interest", the target linkage field is "interest", and the calculation rule thereof satisfies the linear change, that is, satisfies the predetermined numerical fitting algorithm y=a×b×x, where a is principal, b is a lifetime, x is interest, and y is interest. Substituting the plurality of first field value change information (Δx1, Δx2, Δx3, Δx4, etc.) and the plurality of second field value change information (Δy1, Δy2, Δy3, Δy4, etc.) into y=a×b×x, the values of a and b can be obtained by fitting, that is, determining a field calculation rule between the target field and the plurality of target linkage fields, where y=25000×15×x.

Fig. 4 schematically illustrates a flow chart of generating an ideal reference table according to an embodiment of the present disclosure.

As shown in fig. 4, generating the ideal reference table includes operations S410 to S420 based on the original value and the modified value of the target field, the plurality of target linkage fields, and the field calculation rule.

In operation S410, based on the original value and the modified value of the target field and the field calculation rules between the target field and the plurality of target linked fields, the corresponding reference value after the field value of each target linked field is triggered to change is calculated when the target field is changed from the original value to the modified value.

According to an embodiment of the present disclosure, in the above-described embodiment, it is exemplarily shown that a field calculation rule between a target field and a plurality of target linkage fields is y=25000×15×x. Further, according to the field calculation rule, a reference value corresponding to the changed field value of the target linkage field is calculated, for example, the modified value x of the target field is substituted into the field calculation rule of y=25000×15×x, so that the reference value y corresponding to the changed field value of the target linkage field can be directly calculated.

In operation S420, an ideal reference table is generated based on the modified values of the target fields, the reference values of the respective target linkage fields.

Fig. 5 schematically illustrates a flow chart of a method of determining a field computation rule between a target field and a plurality of target linkage fields according to another embodiment of the present disclosure. Fig. 6 schematically illustrates a schematic diagram of a business field association map of a target business table of an embodiment of the present disclosure. The following is a specific explanation in connection with fig. 5 and 6.

As shown in fig. 5, determining a plurality of target linkage fields having an association relationship with a target field, and determining a field calculation rule between the target field and the plurality of target linkage fields includes operation S510 and operation S520.

In operation S510, a service field association graph pre-constructed for the target service table is read from the graph database, where the service field association graph includes N nodes corresponding to N service fields, the types of the nodes are active nodes and/or linked nodes, the nodes with association relationships are connected by directional edges, the directional edges point to the linked nodes from the active nodes, and the attribute of the directional edges includes: the field calculation rule between the service fields having the association relationship.

According to an embodiment of the present disclosure, a pre-constructed service field association diagram is shown in fig. 6, in the service field association diagram, each service field is used as a node, and N service fields correspond to N nodes, as shown in service field 1 to service field N in fig. 6. The node type of one service field may be an active node or a linkage node, or may be both an active node and a linkage node, for example, service field 1, service field 2, service field 3, and service field 5 in fig. 6. The active node is used to represent the directly modified traffic field, and the linked node is used to represent the node whose field value is updated as the active node corresponding to the node is modified. Each node (traffic field) is connected by a directed edge, as indicated by the arrow in fig. 6, which is directed by the active node to the linked node, indicating which node is the active node and which node is the passive node. For example, traffic field 2 is both an active node and a linked node. In the case that the service field 2 is used as the active node, the service fields 1,3 and 4 are linked nodes associated with the active node (in the case that the service field 2 is modified, the field values of the service fields 1,3 and 4 are triggered to change); in the case where the service field 2 is used as the link node, the active node corresponding to the link node is the service field 1 (in the case where the service field 1 is modified, the field value of the trigger service field 2 is changed). Wherein, the attribute of the directed edge is a field calculation rule between service fields with association relation.

In operation S520, a plurality of target linkage fields having an association relationship with the target field are determined based on the traffic field association map, and a field calculation rule between the target field and the plurality of target linkage fields is determined.

According to the embodiment of the disclosure, a plurality of target linkage fields which have an association relationship with a target field are directly read from a service field association map, and further, a field calculation rule between the target field and the plurality of target linkage fields is determined. In this embodiment, specific implementation manners of determining the field calculation rule between the target field and the plurality of target linkage fields may refer to steps 11 to 13, which are not described herein again.

According to the embodiment of the disclosure, as the service field association graph is pre-constructed and stored in the graph database, required data can be directly read from the graph database when data processing is performed, and a field calculation rule does not need to be calculated in real time in the whole data processing process, so that steps of the data processing process are reduced, and the speed and efficiency of data processing can be further improved.

According to an embodiment of the present disclosure, a specific implementation manner of constructing a service field association map of a target service table includes steps 21 to 23.

In step 21, a history modification log associated with the target business table is obtained.

In step 22, a log parsing tool is invoked to parse the history modification log, and determine N service fields included in the target service table, association relationships between the N service fields, and field calculation rules between the service fields having the association relationships.

According to an embodiment of the present disclosure, in step S22, a log parsing tool is invoked to parse the history modification log, and according to the parsing result, read the service fields in the target service table, and the specific implementation of the association relationship between the service fields may refer to operation S320. Further, the obtaining manner of the field calculation rule in step 22 may be determined according to the specific implementation manner in operation S320 or operation S520, which is not described in detail in this embodiment.

In step 23, a service field association map for the target service table is constructed based on the N service fields, the association relationships between the N service fields, and the field calculation rules between the service fields having the association relationships.

According to the embodiment of the disclosure, a service field association map as shown in fig. 6 is constructed based on N service fields, association relations among the N service fields, and field calculation rules among the service fields having the association relations.

Fig. 7 schematically illustrates a flowchart of generating a collation result for a target business table according to an embodiment of the disclosure.

As shown in fig. 7, generating the collation result for the target service table from the ideal reference table and the target service table includes operations S710 to S720.

In operation S710, the field name of the target linkage field is used as a primary key to query the target service table, so as to obtain a field value to be checked corresponding to the target linkage field in the target service table.

Illustratively, the field name of the target linkage field is "interest", the target business table to be checked is searched for the query "interest", the "interest" field may be located, and the field value under the target linkage field is obtained, for example, 30, 300, 3001, 30000, etc.

In operation S720, the field value to be checked corresponding to the target linkage field is checked with the reference value, and a check result for the target service table is generated.

Illustratively, the field name of the target linkage field is "interest", the "interest" field may be located by searching the ideal reference table, and the reference value under the target linkage field may be obtained, for example, 30, 300, 3001, 30000, etc.

Further, the field value to be checked in the target service table is checked with the reference value in the ideal reference report item by item. Wherein, the field value is 30 yuan, 300 yuan, 30000 yuan, etc. and the reference value are all the same, then the field value of the part is free of errors; the field value in the target service table is 3001 yuan, and the reference value in the ideal reference table is 3000 yuan, which indicates that the field value in the target service table is wrong, and the error source needs to be modified again or searched.

Based on the data processing method, the disclosure also provides a data processing device. The device will be described in detail below in connection with fig. 8.

Fig. 8 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 8, the data processing apparatus 800 of this embodiment includes a first acquisition module 810, a first determination module 820, a first generation module 830, and a second generation module 840.

The first obtaining module 810 is configured to obtain a modified target field of the target service table in a predetermined history period, and obtain an original value corresponding to the target field before being modified and a modified value corresponding to the target field after being modified, where the target service table is a service table to be checked by a user. In an embodiment, the first obtaining module 810 may be configured to perform the operation S210 described above, which is not described herein.

The first determining module 820 is configured to determine a plurality of target linkage fields having an association relationship with the target field, and determine a field calculation rule between the target field and the plurality of target linkage fields. In an embodiment, the first determining module 820 may be used to perform the operation S220 described above, which is not described herein.

The first generation module 830 is configured to generate an ideal reference table based on the original value and the modified value of the target field, the plurality of target linkage fields, and the field calculation rule. In an embodiment, the first generating module 830 may be configured to perform the operation S230 described above, which is not described herein.

The second generating module 840 is configured to generate a verification result for the target service table according to the ideal reference table and the target service table. In an embodiment, the second generating module 840 may be configured to perform the operation S240 described above, which is not described herein.

According to the embodiment of the disclosure, based on the original value and the modified value of the target field, the association relation between the target field and the target linkage field determined by the first determining module is determined, the field calculation rule between the target field and the target linkage field is determined, and the ideal reference table is generated by the first generating module. Since the ideal reference table is derived from the target field and the target linkage field associated with the target field, only the field value of the modified traffic field associated with the target field can be reflected in the ideal reference table. Thus, the data change amount including only the changed service field and each field value is clearly obtained in the ideal reference table. Furthermore, the second generation module compares the target service table based on the ideal reference table, and only the changed service data is actually checked without checking all the data, so that the data quantity of the data to be checked is reduced to a great extent, and the data processing efficiency is improved.

According to an embodiment of the present disclosure, any of the first acquisition module 810, the first determination module 820, the first generation module 830, and the second generation module 840 may be combined in one module to be implemented, or any of the modules may be split into a plurality of modules. Or at least some of the functionality of one or more of the modules may be combined with, and implemented in, at least some of the functionality of other modules. According to embodiments of the present disclosure, at least one of the first acquisition module 810, the first determination module 820, the first generation module 830, and the second generation module 840 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or as hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or as any one of or a suitable combination of any of the three. Or at least one of the first acquisition module 810, the first determination module 820, the first generation module 830 and the second generation module 840 may be at least partially implemented as computer program modules which, when executed, may perform the respective functions.

According to an embodiment of the present disclosure, the first determining module 820 includes: a reading sub-module and a first determining sub-module.

The reading submodule is used for reading a service field association graph which is pre-constructed for a target service table from a graph database, wherein the service field association graph comprises N nodes corresponding to N service fields, the types of the nodes are active nodes and/or linkage nodes, the nodes with association relations are connected through directed edges, the directed edges point to the linkage nodes from the active nodes, and the attributes of the directed edges comprise: a field calculation rule between service fields with association relation;

the first determining submodule is used for determining a plurality of target linkage fields with association relation with the target fields based on the service field association map and determining field calculation rules between the target fields and the plurality of target linkage fields.

According to an embodiment of the present disclosure, the data processing apparatus 800 of this embodiment further includes: the device comprises a second acquisition module, a second determination module and a construction module.

The second acquisition module is used for acquiring a history modification log related to the target business table;

the second determining module is used for calling a log analyzing tool to analyze the history modification log and determining N service fields included in the target service table, the association relation among the N service fields and the field calculation rule among the service fields with the association relation;

the construction module is used for constructing a service field association map aiming at the target service table based on N service fields, the association relation among the N service fields and the field calculation rule among the service fields with the association relation.

According to an embodiment of the present disclosure, the first determining module 820 further includes: the sub-module and the second determination sub-module are obtained.

An acquisition sub-module for acquiring a history modification log related to the target service table;

the second determining submodule is used for calling a log analyzing tool to analyze the history modification log, determining a plurality of target linkage fields with association relation with the target fields, and determining field calculation rules between the target fields and the plurality of target linkage fields.

According to an embodiment of the present disclosure, the second determination submodule includes: a determining unit, an extracting unit and a generating unit.

The determining unit is used for calling the log analysis tool to analyze the historical modification log and determining the linkage modification log related to the target field, wherein the linkage modification log is generated based on the modification operation trigger of the target field;

The extraction unit is used for extracting a plurality of target linkage fields, first field value change information of the target fields and second field value change information of each target linkage field from the linkage modification log;

and the generation unit is used for generating a field calculation rule between the determined target field and the plurality of target linkage fields based on the first field value change information and the second field value change information by using a preset numerical fitting algorithm.

According to an embodiment of the present disclosure, the first generation module 830 includes: a calculation sub-module and a first generation sub-module.

The calculation sub-module is used for calculating corresponding reference values after the field values of all the target linkage fields are triggered to change under the condition that the target fields are changed from the original values to the modified values based on the original values and the modified values of the target fields and the field calculation rules between the target fields and the plurality of target linkage fields;

and the first generation sub-module is used for generating an ideal reference table based on the modified values of the target fields and the reference values of the target linkage fields.

According to an embodiment of the present disclosure, the second generating module 840 includes: a sub-module and a second generation sub-module are obtained.

The obtaining submodule is used for inquiring the target service table by taking the field name of the target linkage field as a main key to obtain a field value to be checked corresponding to the target linkage field in the target service table;

And the second generation sub-module is used for checking the field value to be checked corresponding to the target linkage field with the reference value to generate a checking result aiming at the target service table.

As shown in fig. 9, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. The processor 901 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 901 may also include on-board memory for caching purposes. Processor 901 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. The processor 901 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the program may be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the disclosure, the electronic device 900 may also include an input/output (I/O) interface 905, the input/output (I/O) interface 905 also being connected to the bus 904. The electronic device 900 may also include one or more of the following components connected to an input/output (I/O) interface 905: an input section 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 908 including a hard disk or the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to an input/output (I/O) interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 910 so that a computer program read out therefrom is installed into the storage section 908 as needed.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 902 and/or RAM 903 and/or one or more memories other than ROM 902 and RAM 903 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code means for causing a computer system to carry out the data processing methods provided by the embodiments of the present disclosure when the computer program product is run on the computer system.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed, and downloaded and installed in the form of a signal on a network medium, via communication portion 909, and/or installed from removable medium 911. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 909 and/or installed from the removable medium 911. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. These examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A method of data processing, the method comprising:

Obtaining a target field of a target service table which is modified in a preset historical time period, and obtaining an original value corresponding to the target field before modification and a modified value corresponding to the target field after modification, wherein the target service table is a service table to be checked by a user;

Generating an ideal reference table based on the original value and the modified value of the target field, the plurality of target linkage fields, the field calculation rule;

2. The method of claim 1, wherein determining a plurality of target linkage fields that have an association with the target field, and determining a field calculation rule between the target field and the plurality of target linkage fields comprises:

Reading a service field association map which is pre-constructed for the target service table from a graph database, wherein the service field association map comprises N nodes corresponding to N service fields, the types of the nodes are active nodes and/or linkage nodes, the nodes with association relations are connected through directed edges, the directed edges point to the linkage nodes from the active nodes, and the attributes of the directed edges comprise: a field calculation rule between service fields with association relation;

Based on the business field association map, a plurality of target linkage fields with association relation with the target field are determined, and a field calculation rule between the target field and the plurality of target linkage fields is determined.

3. The method as recited in claim 2, further comprising:

acquiring a history modification log related to the target service table;

A log analysis tool is called to analyze the history modification log, and N service fields included in the target service table, the association relation among the N service fields and the field calculation rule among the service fields with the association relation are determined;

And constructing a service field association map aiming at the target service table based on the N service fields, the association relation among the N service fields and the field calculation rule among the service fields with the association relation.

4. The method of claim 1, wherein determining a plurality of target linkage fields that have an association with the target field, and determining a field calculation rule between the target field and the plurality of target linkage fields comprises:

acquiring a history modification log related to the target service table;

And calling a log analysis tool to analyze the history modification log, determining a plurality of target linkage fields with association relation with the target fields, and determining field calculation rules between the target fields and the plurality of target linkage fields.

5. The method of claim 4, wherein invoking a log parsing tool parses the history modification log, determining a plurality of target linked fields that have an association with the target field, and determining a field calculation rule between the target field and the plurality of target linked fields comprises:

Calling a log analysis tool to analyze the historical modification log and determining a linkage modification log related to the target field, wherein the linkage modification log is generated based on modification operation trigger of the target field;

Extracting the plurality of target linkage fields, first field value change information of the target fields and second field value change information of each target linkage field from the linkage modification log;

generating a field computation rule determining between the target field and the plurality of target linkage fields based on the first field value change information and the second field value change information using a predetermined numerical fitting algorithm.

6. The method of claim 1, wherein generating an ideal reference table based on the original and modified values of the target field, the plurality of target linkage fields, the field calculation rule comprises:

Calculating a corresponding reference value after triggering the field value of each target linkage field to change under the condition that the target field is changed from the original value to the modified value based on the original value and the modified value of the target field and a field calculation rule between the target field and the plurality of target linkage fields;

the ideal reference table is generated based on the modified value of the target field, the reference value of each of the target linkage fields.

7. The method of claim 1, wherein generating a collation result for the target service table based on the ideal reference table and the target service table comprises:

Inquiring the target service table by taking the field name of the target linkage field as a primary key to obtain a field value to be checked corresponding to the target linkage field in the target service table;

8. A data processing apparatus, the apparatus comprising:

The first acquisition module is used for acquiring a modified target field of a target service table in a preset historical time period, and acquiring an original value corresponding to the target field before modification and a modified value corresponding to the target field after modification, wherein the target service table is a service table to be checked by a user;

the first determining module is used for determining a plurality of target linkage fields with association relation with the target field and determining a field calculation rule between the target field and the plurality of target linkage fields;

A first generation module, configured to generate an ideal reference table based on the original value and the modified value of the target field, the plurality of target linkage fields, and the field calculation rule; and

And the second generation module is used for generating a check result aiming at the target service table according to the ideal reference table and the target service table.

9. An electronic device, comprising:

One or more processors;

a memory for storing one or more computer programs,

Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program or instructions is stored, characterized in that the computer program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

11. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.