CN116226111A - Data management method, device, equipment, storage medium and program product - Google Patents

Data management method, device, equipment, storage medium and program product Download PDF

Info

Publication number
CN116226111A
CN116226111A CN202310255918.XA CN202310255918A CN116226111A CN 116226111 A CN116226111 A CN 116226111A CN 202310255918 A CN202310255918 A CN 202310255918A CN 116226111 A CN116226111 A CN 116226111A
Authority
CN
China
Prior art keywords
data
information
abnormal
scene
verification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310255918.XA
Other languages
Chinese (zh)
Inventor
程翎峰
徐俊皓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
CCB Finetech Co Ltd
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202310255918.XA priority Critical patent/CN116226111A/en
Publication of CN116226111A publication Critical patent/CN116226111A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/02—Banking, e.g. interest calculation or account maintenance

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • Technology Law (AREA)
  • General Business, Economics & Management (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • General Factory Administration (AREA)

Abstract

The invention discloses a data management method, a device, equipment, a storage medium and a program product, and relates to the technical field of data processing. The method comprises the following steps: obtaining a verification result by verifying the business data of the current batch; the verification result comprises normal and abnormal; generating abnormal data information and abnormal result statistical information according to the verification result; generating reasonable scene information according to the abnormal result statistical information; marking the business scene in the abnormal data information based on the reasonable scene information; and managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information. According to the embodiment of the disclosure, the abnormal data information, the abnormal result statistical information and the reasonable scene information are obtained by checking the current batch of business data, and the business data of the next batch is treated according to the abnormal data information, the abnormal result statistical information and the reasonable scene information, so that the efficiency of data treatment can be improved.

Description

Data management method, device, equipment, storage medium and program product
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data management method, a device, equipment, a storage medium and a program product.
Background
With the increasing strictness of data quality requirements, financial institutions are required to monitor data in all directions, and therefore, data management is increasingly important.
At present, the directions in the field of data management include: data standards, metadata, data quality, data models, data distribution, data exchange, storage, and the like. Along with the development of the change of financial (such as banking) business, the emphasis of data management is also continuously adjusting. The existing data management scheme has the problems of common problems of low management efficiency, poor inter-tissue communication coordination capability, inconsistent management content and production content, incapability of continuously performing effective management and the like.
Disclosure of Invention
The embodiment of the invention provides a data management method, a device, equipment, a storage medium and a program product, which can improve the efficiency of data management.
In a first aspect, an embodiment of the present invention provides a data management method, including: checking the business data of the current batch to obtain a checking result; wherein the verification result comprises normal and abnormal; generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information; generating reasonable scene information according to the abnormal result statistical information; labeling the business scene in the abnormal data information based on the reasonable scene information; and managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
In a second aspect, an embodiment of the present invention further provides a data management apparatus, including: the verification module is used for verifying the business data of the current batch to obtain a verification result; wherein the verification result comprises normal and abnormal; the abnormal information generation module is used for generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information; the reasonable scene information generation module is used for generating reasonable scene information according to the abnormal result statistical information; the business scene labeling module is used for labeling the business scenes in the abnormal data information based on the reasonable scene information; and the treatment module is used for treating the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a data management method according to any one of the embodiments of the present invention when the processor executes the program.
In a fourth aspect, embodiments of the present invention further provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data governance method according to any of the embodiments of the present invention.
In a fifth aspect, embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements a data governance method as described in any of the embodiments of the present invention.
In the embodiment of the invention, the verification result is obtained by verifying the business data of the current batch; wherein the verification result comprises normal and abnormal; generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information; generating reasonable scene information according to the abnormal result statistical information; labeling the business scene in the abnormal data information based on the reasonable scene information; and managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information. According to the embodiment of the disclosure, the abnormal data information, the abnormal result statistical information and the reasonable scene information are obtained by checking the current batch of business data, and the business data of the next batch is treated according to the abnormal data information, the abnormal result statistical information and the reasonable scene information, so that the efficiency of data treatment can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data management method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data management method according to an embodiment of the present invention;
FIG. 3 is a flow chart of another method for data management according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data management device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance. The data acquisition, storage, use, processing and the like in the technical scheme meet the relevant regulations of national laws and regulations.
The embodiment of the invention discloses a data management method, which belongs to the field of data standard and data quality management in the field of data management.
Fig. 1 is a flowchart of a data management method according to an embodiment of the present invention, where the method may be applied to a data management situation, and the method may be performed by a data management device, where the data management device may be implemented in hardware and/or software, and the data management device may be configured in an electronic device, where the electronic device may be a mobile terminal, a PC or a server, etc. As shown in fig. 1, the method includes:
s110, checking the business data of the current batch to obtain a checking result.
Wherein the verification result comprises normal and abnormal. In this embodiment, the service data of the current batch may include a plurality of pieces of service data, and for each piece of service data, the service data may be checked according to a check rule corresponding to the service data, so as to obtain a check result.
S120, generating abnormal data information and abnormal result statistical information according to the verification result.
The abnormal data information comprises a service scene corresponding to service data; the abnormal result statistical information comprises business scene feedback information. The business scenario feedback information may be a reasonable scenario. In this embodiment, abnormal data information may be generated according to the relevant information corresponding to the abnormal service data and the abnormal service data that are the verification results; the abnormal result statistical information may be generated according to the total amount of the abnormal service data and another related information corresponding to the abnormal service data.
S130, generating reasonable scene information according to the abnormal result statistical information.
The reasonable scene information may include a check rule, a reasonable scene judgment rule, a reasonable scene description, and the like. In this embodiment, it may be determined that the abnormal result statistics information is abnormal data of a reasonable scene, and reasonable scene information is generated according to further related information corresponding to the abnormal data of the reasonable scene.
Optionally, the abnormal data information, the abnormal result statistical information and the reasonable scene information are embodied in a table form.
In this embodiment, the abnormal data information, the abnormal result statistical information and the reasonable scene information may be embodied in a table form, so as to facilitate searching and displaying related problem information, and improve searching and displaying efficiency.
And S140, marking the business scene in the abnormal data information based on the reasonable scene information.
In this embodiment, the service scene in the abnormal data information may be marked as a reasonable scene according to the reasonable scene information, so that the abnormal data marked as the reasonable scene need not to be adjusted during the subsequent treatment.
S150, treating the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
Specifically, according to the marked abnormal data information and the abnormal result statistical information, the abnormal data in the next batch of business data can be subjected to data adjustment, and the adjustment state of the abnormal data can be modified, for example, the detailed process of abnormal data adjustment (such as an adjuster, adjustment time, adjustment scheme, adjustment result and the like) and the solution result of the abnormal data are recorded.
In the embodiment of the invention, the verification result is obtained by verifying the business data of the current batch; wherein the verification result comprises normal and abnormal; generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information; generating reasonable scene information according to the abnormal result statistical information; labeling the business scene in the abnormal data information based on the reasonable scene information; and managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information. According to the embodiment of the disclosure, the abnormal data information, the abnormal result statistical information and the reasonable scene information are obtained by checking the current batch of business data, and the business data of the next batch is treated according to the abnormal data information, the abnormal result statistical information and the reasonable scene information, so that the efficiency of data treatment can be improved.
FIG. 2 is a flow chart of another data management method according to an embodiment of the present invention. The embodiment of the invention is embodied on the basis of the embodiment of the invention, and referring to fig. 2, the method provided by the embodiment of the invention specifically comprises the following steps:
S210, acquiring a check rule corresponding to each service data.
It should be noted that, each service data is stored in the form of a data table, and each data table includes service data to be checked and a field corresponding to the data table, and a corresponding check rule may be invoked through the field. Wherein the verification rule may include at least one of: integrity verification, format verification, data entity content verification and associated information verification. The integrity check may be understood as performing non-empty check on the data item in the service data, for example, the service data includes multiple data items such as borrower identity information and loan amount, and performing non-empty check on the borrower identity information of the data item, that is, the content corresponding to the data item to be checked should be non-empty. For a specific data item to be checked, this embodiment is not limited to this, and may be set according to experience of service personnel. Format verification may be understood as a verification of the data type and data length of the traffic data. The verification of the content of the data entity may be understood as verification of the actual content in the service data, such as checking the deposit, debit/credit balance, principal balance/overdue month index in the service data for a total score (or checking a specific amount). The verification of the association information may be verification of association information of service data such as association code, license number, accounting code, customer unit code, detail code, account number, contract number, and borrowing number, for example, verification of association information such as whether the association code, license number, accounting code, customer unit code, account number, contract number, and borrowing number are correct, whether the association information corresponds to user information corresponding to the service data, and the like.
Optionally, the mode of acquiring the check rule corresponding to each service data may be: determining a data table in which service data are located; extracting key fields of a data table; and determining a check rule corresponding to the service data according to the key field.
In this embodiment, the data table where the service data is located may be determined according to the corresponding relationship between the service data and the data table, the key field is extracted from the data table, and the verification rule corresponding to the service data is determined according to the key field. The key fields and the check rules have a corresponding relationship, and the key fields and the check rules can be one-to-one or one-to-many. The key field may be any field capable of identifying the data table, such as an identity unique identification ID, a term of art in the business data.
S220, checking the service data based on the checking rule to obtain a checking result.
In this embodiment, after the verification rule is obtained, the service data may be verified according to the verification rule, so that a verification result may be obtained. The checking rule is an integrity check, non-empty check is performed on the content of the user information in the service data, if the content of the user information in the service data is non-empty, the check result is normal, and if the content of the user information in the service data is empty, the check result is abnormal.
S230, generating abnormal data information and abnormal result statistical information according to the verification result.
Optionally, the method for generating the abnormal data information according to the verification result may be: acquiring service data with abnormal verification results as abnormal data; acquiring first related information corresponding to the abnormal data; and adding the first related information as a new field into the abnormal data to obtain abnormal data information.
Wherein the anomalous data information may be characterized as an error list. Wherein, optionally, the first related information may include at least one of the following: data generation time, data source information, check rules and business scenes. The data generation time may be understood as time of generating service data, the data source information may be understood as source of the service data (e.g. from an upstream system), the check rule in the first related information may be a check rule for checking the service data to generate an exception, and the service scenario may be understood as a scenario where the service data is located, for example, a loan service scenario, a deposit service scenario, and the like.
Specifically, a blank error list can be newly created, in the original data list, the service data with the abnormal verification result is taken as abnormal data, the abnormal data is copied into the error list, first relevant information is added to the abnormal data, namely the first relevant information is added into the error list, and the abnormal data corresponds to the first relevant information, so that verification rules or data source information and the like generating verification abnormality can be rapidly located from the error list.
Optionally, the method for generating the abnormal result statistical information according to the verification result may be: for each check rule, determining abnormal data corresponding to the check rule; counting the total amount of the abnormal data, and determining second related information corresponding to each abnormal data; and generating abnormal result statistical information based on the total amount of the abnormal data and the second related information.
The abnormal result statistical information can be characterized as a quality problem tracking table, and can be understood as statistics of the number of abnormal results for each check rule. Optionally, the second related information may include at least one of: data source, abnormal reason, business scene feedback information and data adjustment scheme. The abnormal cause may be obtained according to a verification rule that generates a verification abnormality, for example, for an integrity verification rule, if the verification result is abnormal, the abnormal cause is that the data is null. The traffic scenario feedback information may include a reasonable scenario. The data adjustment scheme can be understood as a scheme for adjusting abnormal data, and the abnormal data can be checked by checking rules after the abnormal data is adjusted by the data adjustment scheme.
Specifically, for the service data of the current batch, for each verification rule, the abnormal data corresponding to the verification rule can be determined, the total amount of the abnormal data, namely the number of error data, is counted, the second relevant information corresponding to each different data is determined, and a quality problem tracking table is generated based on the total amount of the abnormal data and the second relevant information. The quality problem tracking table may further include a verification rule (verification rule serial number), a quality problem number, an original error number, a data source, a problem description, a proposer, a proposing time, a data batch, a home business group, a home physical subsystem, an allocated person, an allocated time, a problem classification, service scene feedback information, problem analysis, a problem processing state, a data adjustment scheme, a correction data range, a planned production time, a correction scheme project group confirmation, a circulated party, an actual solution time, communication with a service department, a service department feedback, a need for upgrading, an upgrading result, a rising problem number, a non-uniform opinion of a service and a number pipe part, a different opinion of a service and a number pipe part, a need for providing rules to a supervision project group, a need for providing rules to a component, a correction scheme verification description, a correction scheme verification result, a verifier, a verification time, a verification data batch, a need for providing a legacy error number, a project group suggestion and a remark, and the like.
The quality problem tracking table can directly know the number of errors of each check rule, the source of error data, the cause (abnormal cause) of the problem, the service scene corresponding to the error data, the data adjustment scheme, the problem verification and other conditions, and can trace back the state of each problem in each flow and the current responsible person.
S240, generating reasonable scene information according to the abnormal result statistical information.
S250, marking the business scene in the abnormal data information based on the reasonable scene information.
S260, treating the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
FIG. 3 is a flow chart of another data management method according to an embodiment of the present invention. The embodiment of the invention is embodied on the basis of the embodiment of the invention, and referring to fig. 3, the method provided by the embodiment of the invention specifically comprises the following steps:
s310, checking the business data of the current batch to obtain a checking result.
S320, generating abnormal data information and abnormal result statistical information according to the verification result.
The service scene feedback information comprises reasonable scenes. For reasonable scenes, the business scenes corresponding to the business data meeting the reasonable scene conditions can be preset as the reasonable scenes according to historical experience and actual conditions.
S330, extracting abnormal data of which the business scene feedback information is a reasonable scene from the abnormal result statistical information.
In this embodiment, after obtaining the statistical information of the abnormal result, if the feedback information of the service scenario is a reasonable scenario, the abnormal data corresponding to the reasonable scenario may be extracted.
S340, acquiring third related information corresponding to the abnormal data.
In this embodiment, the abnormal data in the reasonable scenario may further have corresponding third related information.
Optionally, the third related information includes at least one of: data source, check rule, reasonable scene judgment rule and reasonable scene description.
The reasonable scene judgment rule can be understood as a rule and a sentence for judging a reasonable scene, and the reasonable scene description can be understood as an explanation of a reasonable scene, and the explanation is performed by describing the situation of the related reasonable scene.
And S350, generating reasonable scene information based on the third related information.
Wherein the reasonable scene information can be characterized by a reasonable scene tracking table. In this embodiment, a reasonable scene tracking table may be generated according to the third related information. Because some business scenes corresponding to abnormal data belong to reasonable scenes, a reasonable scene tracking table can be designed. The reasonable scene tracking table can also contain a check rule (check rule serial number), a check type, a check table name, a data item name, a data source, a first responsibility component, a first responsibility system, a responsible person, a reasonable scene type (reasonable scene, partial reasonable scene, etc.), whether to identify a reasonable scene, a reasonable scene description, a verification result, a reasonable scene judgment rule, etc.
S360, marking the business scene in the abnormal data information based on the reasonable scene information.
In this embodiment, the service scenario in the error detail table may be marked as a reasonable scenario based on the reasonable scenario tracking table. For example, the abnormal data in the error detail table can be determined in the reasonable scene tracking table, and the business scene corresponding to the abnormal data in the error detail table is marked as the reasonable scene, so that the follow-up adjustment of the abnormal data is not needed.
S370, managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
In this embodiment, the abnormal data, the adjustment scheme corresponding to the abnormal data, the specific verification rule for generating the verification abnormality, and the like may be determined according to the error detail table, the quality problem tracking table, and the reasonable scene tracking table, so that the business data may be managed.
Optionally, after the business data of the next batch is treated based on the marked abnormal data information and the abnormal result statistical information, the method further comprises: verifying the business data of the next batch after treatment to obtain abnormal data information and abnormal result statistical information of the next batch; comparing the abnormal data information of the next batch with the abnormal data information of the current batch, and comparing the abnormal result statistical information of the next batch with the abnormal result statistical information of the current batch to obtain a comparison result; and determining a data management result according to the comparison result.
In this embodiment, the business data may be managed according to a set period, for example, the adjustment status of all business data is tracked weekly. Specifically, abnormal data information and abnormal result statistical information of the business data of the current batch are obtained, business data of the next batch is treated according to the abnormal data information and the abnormal result statistical information, and the business data of the next batch after treatment is checked to obtain the abnormal data information and the abnormal result statistical information of the next batch; comparing the abnormal data information of the next batch with the abnormal data information of the current batch to obtain a first comparison result, and comparing the abnormal result statistical information of the next batch with the abnormal result statistical information of the current batch to obtain a second comparison result; and determining a data governance result according to the first comparison result and the second comparison result. The first comparison result and/or the second comparison result can indicate whether the abnormal data of the current batch is effectively solved or whether the business data of the next batch is treated or whether new abnormal data or whether the number of the abnormal data is reduced or not, and the like, and the business data of the next batch can be treated again according to the abnormal data information, the abnormal result statistical information and the reasonable scene information.
According to the technical scheme disclosed by the embodiment, the business data of the next batch is managed based on the marked abnormal data information and the abnormal result statistical information, and the business data is pushed from top to bottom based on the business data problem. Meanwhile, abnormal data information, abnormal result statistical information and reasonable scene information record the flow state and feedback information of each link, so that problem tracking, problem tracing and problem responsibility fixing are facilitated. Because the method belongs to the push from top to bottom, the problem of low treatment efficiency of the existing data treatment scheme can be solved. Meanwhile, the flow state and feedback information of each link are recorded, different components can check the feedback information, and the problems that the inter-organization communication coordination capacity is poor, the management content is inconsistent with the production content, the effective treatment cannot be continuously carried out and the like can be solved.
Fig. 4 is a schematic structural diagram of a data management device according to an embodiment of the present invention. As shown in fig. 4, the apparatus includes a verification module 410, an anomaly information generation module 420, a reasonable scene information generation module 430, a business scene labeling module 440, and a treatment module 450;
the verification module 410 is configured to verify the service data of the current batch to obtain a verification result; wherein the verification result comprises normal and abnormal;
The abnormal information generating module 420 is configured to generate abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information;
a reasonable scene information generating module 430, configured to generate reasonable scene information according to the abnormal result statistics information;
the service scene labeling module 440 is configured to label a service scene in the abnormal data information based on the reasonable scene information;
and the governance module 450 is used for governance of the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
In the embodiment of the invention, the service data of the current batch is checked by the check module to obtain a check result; wherein the verification result comprises normal and abnormal; generating abnormal data information and abnormal result statistical information according to the verification result through an abnormal information generation module; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information; generating reasonable scene information through a reasonable scene information generation module and a business scene labeling module according to the abnormal result statistical information; marking the business scene in the abnormal data information based on the reasonable scene information through a business scene marking module; and treating the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information through a treatment module. According to the embodiment of the disclosure, the abnormal data information, the abnormal result statistical information and the reasonable scene information are obtained by checking the current batch of business data, and the business data of the next batch is treated according to the abnormal data information, the abnormal result statistical information and the reasonable scene information, so that the efficiency of data treatment can be improved.
Optionally, the verification module is specifically configured to: acquiring a verification rule corresponding to each service data; wherein the verification rule includes at least one of: integrity verification, format verification, data entity content verification and associated information verification; and verifying the service data based on the verification rule to obtain a verification result.
Optionally, the verification module is further configured to: determining a data table in which the service data are located; extracting key fields of the data table; determining a verification rule corresponding to the service data according to the key field; and the key fields have corresponding relations with the verification rules.
Optionally, the anomaly information generation module is specifically configured to: acquiring service data with abnormal verification results as abnormal data; acquiring first related information corresponding to the abnormal data; and adding the first related information as a new field into the abnormal data to obtain abnormal data information.
Optionally, the first related information includes at least one of: data generation time, data source information, check rules and business scenes.
Optionally, the anomaly information generation module is further configured to: for each check rule, determining abnormal data corresponding to the check rule; counting the total amount of the abnormal data, and determining second related information corresponding to each abnormal data; and generating abnormal result statistical information based on the total amount of the abnormal data and the second related information.
Optionally, the second related information includes at least one of: data source, abnormal reason, business scene feedback information and data adjustment scheme.
Optionally, the reasonable scene information generating module is specifically configured to: extracting abnormal data of which the business scene feedback information is a reasonable scene from the abnormal result statistical information; acquiring third related information corresponding to the abnormal data; and generating reasonable scene information based on the third related information.
Optionally, the third related information includes at least one of: data source, check rule, reasonable scene judgment rule and reasonable scene description.
Optionally, the abnormal data information, the abnormal result statistical information and the reasonable scene information are embodied in a table form.
Optionally, the abatement module is further configured to: verifying the business data of the next batch after treatment to obtain abnormal data information and abnormal result statistical information of the next batch; comparing the abnormal data information of the next batch with the abnormal data information of the current batch, and comparing the abnormal result statistical information of the next batch with the abnormal result statistical information of the current batch to obtain a comparison result; and determining a data treatment result according to the comparison result.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. Fig. 5 shows a block diagram of an electronic device 12 suitable for use in implementing embodiments of the present invention. The electronic device 12 shown in fig. 5 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention. Device 12 is a typical electronic device that implements data governance.
As shown in fig. 5, the electronic device 12 is in the form of a general purpose computing device. Components of the electronic device 12 may include, but are not limited to: one or more processors 16, a memory 28, a bus 18 that connects the various system components, including the memory 28 and the processor 16.
Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include industry standard architecture (IndustryStandardArchitecture, ISA) bus, micro channel architecture (MicroChannel Architecture, MCA) bus, enhanced ISA bus, video electronics standards association (video electronics StandardsAssociation, VESA) local bus, and peripheral component interconnect (PeripheralComponent Interconnect, PCI) bus.
Electronic device 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by electronic device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
Memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RandomAccessMemory, RAM) 30 and/or cache memory 32. The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard disk drive"). Although not shown in fig. 5, a disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk such as a compact disc-ReadOnlyMemory, CD-ROM, digital video disc (digital video disc-ReadOnlyMemory, DVD-ROM), or other optical media, may be provided. In such cases, each drive may be coupled to bus 18 through one or more data medium interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the invention.
Programs 36 having a set (at least one) of program modules 26 may be stored, for example, in the storage 28, such program modules 26 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 26 generally perform the functions and/or methods of the embodiments described herein.
The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, camera, display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and/or any devices (e.g., network card, modem, etc.) that enable the electronic device 12 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 22. Also, the electronic device 12 may communicate with one or more networks, such as a local area Network (LocalAreaNetwork, LAN), a Wide Area Network (WAN), and/or a public Network, such as the internet, via the Network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 over the bus 18. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 12, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk array (redundant arrays ofIndependentDisks, RAID) systems, tape drives, data backup storage systems, and the like.
The processor 16 executes various functional applications and data processing, such as implementing the data governance methods provided by the above-described embodiments of the present invention, by running programs stored in the memory 28.
The embodiment of the invention provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and when the program is executed by a processing device, the data management method in the embodiment of the invention is realized. The computer readable medium of the present invention described above may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperTextTransfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: checking the business data of the current batch to obtain a checking result; wherein the verification result comprises normal and abnormal; generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information; generating reasonable scene information according to the abnormal result statistical information; labeling the business scene in the abnormal data information based on the reasonable scene information; and managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements a data governance method as provided in any of the embodiments of the present application.
Computer program product in the implementation, the computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims (15)

1. A method of data management comprising:
checking the business data of the current batch to obtain a checking result; wherein the verification result comprises normal and abnormal;
generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information;
generating reasonable scene information according to the abnormal result statistical information;
labeling the business scene in the abnormal data information based on the reasonable scene information;
And managing the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
2. The method of claim 1, wherein verifying the business data of the current lot to obtain the verification result comprises:
acquiring a verification rule corresponding to each service data; wherein the verification rule includes at least one of: integrity verification, format verification, data entity content verification and associated information verification;
and verifying the service data based on the verification rule to obtain a verification result.
3. The method of claim 2, wherein obtaining the check rule corresponding to each service data comprises:
determining a data table in which the service data are located;
extracting key fields of the data table;
determining a verification rule corresponding to the service data according to the key field; and the key fields have corresponding relations with the verification rules.
4. The method of claim 2, wherein generating exception data information based on the verification result comprises:
acquiring service data with abnormal verification results as abnormal data;
acquiring first related information corresponding to the abnormal data;
And adding the first related information as a new field into the abnormal data to obtain abnormal data information.
5. The method of claim 4, wherein the first related information comprises at least one of: data generation time, data source information, check rules and business scenes.
6. The method of claim 2, wherein generating anomalous results statistics from the verification results comprises:
for each check rule, determining abnormal data corresponding to the check rule;
counting the total amount of the abnormal data, and determining second related information corresponding to each abnormal data;
and generating abnormal result statistical information based on the total amount of the abnormal data and the second related information.
7. The method of claim 6, wherein the second related information comprises at least one of: data source, abnormal reason, business scene feedback information and data adjustment scheme.
8. The method of claim 1, wherein the business scenario feedback information comprises a reasonable scenario, and wherein generating reasonable scenario information based on the anomaly result statistics comprises:
Extracting abnormal data of which the business scene feedback information is a reasonable scene from the abnormal result statistical information;
acquiring third related information corresponding to the abnormal data;
and generating reasonable scene information based on the third related information.
9. The method of claim 8, wherein the third related information comprises at least one of: data source, check rule, reasonable scene judgment rule and reasonable scene description.
10. The method of claim 1, wherein the anomaly data information, anomaly result statistics, and the reasonable scene information are embodied in a table.
11. The method of claim 1, further comprising, after administering the next batch of business data based on the annotated abnormal data information and the abnormal result statistics:
verifying the business data of the next batch after treatment to obtain abnormal data information and abnormal result statistical information of the next batch;
comparing the abnormal data information of the next batch with the abnormal data information of the current batch, and comparing the abnormal result statistical information of the next batch with the abnormal result statistical information of the current batch to obtain a comparison result;
And determining a data treatment result according to the comparison result.
12. A data governance device, comprising:
the verification module is used for verifying the business data of the current batch to obtain a verification result; wherein the verification result comprises normal and abnormal;
the abnormal information generation module is used for generating abnormal data information and abnormal result statistical information according to the verification result; wherein, the abnormal data information comprises a service scene corresponding to the service data; the abnormal result statistical information comprises business scene feedback information;
the reasonable scene information generation module is used for generating reasonable scene information according to the abnormal result statistical information;
the business scene labeling module is used for labeling the business scenes in the abnormal data information based on the reasonable scene information;
and the treatment module is used for treating the business data of the next batch based on the marked abnormal data information and the abnormal result statistical information.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor implements the data governance method of any of claims 1 to 11 when executing the computer program.
14. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a data governance method according to any of claims 1 to 11.
15. A computer program product comprising a computer program which, when executed by a processor, implements the data governance method of any of claims 1 to 11.
CN202310255918.XA 2023-03-16 2023-03-16 Data management method, device, equipment, storage medium and program product Pending CN116226111A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310255918.XA CN116226111A (en) 2023-03-16 2023-03-16 Data management method, device, equipment, storage medium and program product

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310255918.XA CN116226111A (en) 2023-03-16 2023-03-16 Data management method, device, equipment, storage medium and program product

Publications (1)

Publication Number Publication Date
CN116226111A true CN116226111A (en) 2023-06-06

Family

ID=86576790

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310255918.XA Pending CN116226111A (en) 2023-03-16 2023-03-16 Data management method, device, equipment, storage medium and program product

Country Status (1)

Country Link
CN (1) CN116226111A (en)

Similar Documents

Publication Publication Date Title
CN110263024B (en) Data processing method, terminal device and computer storage medium
CN110121727B (en) Method and system for providing authenticated, auditable and immutable input for intelligent contracts
CN109359277B (en) Data monitoring method, device and computer storage medium
CN113034274A (en) Supply chain financial service system and method based on block chain and terminal equipment
CN111260465A (en) Business processing method, device, server and storage medium
CN112184005B (en) Operation task classification method, device, equipment and storage medium
CN111144697A (en) Data processing method, data processing device, storage medium and electronic equipment
CN111046184A (en) Text risk identification method, device, server and storage medium
CN110781971A (en) Merchant type identification method, device, equipment and readable medium
CN112445866A (en) Data processing method and device, computer readable medium and electronic equipment
CN113205402A (en) Account checking method and device, electronic equipment and computer readable medium
CN112101939A (en) Node management method and system based on block chain
CN114840527A (en) Data processing method, device and computer readable storage medium
CN111899111A (en) Capital matching method, device, server and storage medium
CN112965986B (en) Service consistency processing method, device, equipment and storage medium
CN113034275B (en) Management system and method based on block chain network and terminal equipment
CN108062423A (en) Information-pushing method and device
CN112579468A (en) Method and device for testing service system
CN110070383B (en) Abnormal user identification method and device based on big data analysis
CN111367776A (en) Recording method, device, equipment and storage medium of resource transfer service
CN116071152A (en) Data processing method and device, electronic equipment and storage medium
CN116226111A (en) Data management method, device, equipment, storage medium and program product
CN112561689B (en) Transaction data recording processing method, device, equipment and storage medium
CN110377269B (en) Service approval system collocation method, device and storage medium
CN116993525B (en) Medical industry supply chain settlement processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination