CN117235133A - Bank business data quality detection method, device, storage medium and equipment - Google Patents
Bank business data quality detection method, device, storage medium and equipment Download PDFInfo
- Publication number
- CN117235133A CN117235133A CN202311197542.8A CN202311197542A CN117235133A CN 117235133 A CN117235133 A CN 117235133A CN 202311197542 A CN202311197542 A CN 202311197542A CN 117235133 A CN117235133 A CN 117235133A
- Authority
- CN
- China
- Prior art keywords
- data
- detected
- task
- detection
- check
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 113
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000005192 partition Methods 0.000 claims abstract description 31
- 238000004364 calculation method Methods 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims description 8
- 238000011161 development Methods 0.000 description 8
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000011022 operating instruction Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- RWSOTUBLDIXVET-UHFFFAOYSA-N Dihydrogen sulfide Chemical compound S RWSOTUBLDIXVET-UHFFFAOYSA-N 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The application discloses a banking data quality detection method, a banking data quality detection device, a storage medium and banking data quality detection equipment, which can be applied to the field of blockchain or the field of finance, wherein the method comprises the following steps: firstly, acquiring system service data of each service system in a bank; storing the system service data of each service system into a preset database, then storing the system service data of each service system onto different partitions of a block chain in a distributed manner, and configuring detection rules according to a data table, fields to be detected and preset built-in general rules by the system service data of each different partition; and configuring the service data to be detected, the detection rule and a calculation engine Flink window to form a task to be detected, storing the task to be detected in Kafka, and further transmitting the task to be detected in Kafka to the Flink for real-time data quality detection to obtain a detection result. Therefore, the real-time performance and accuracy of the quality detection of the banking business data can be effectively improved, and the stable operation of each banking business system is ensured.
Description
Technical Field
The present application relates to the field of blockchain technologies, and in particular, to a method, an apparatus, a storage medium, and a device for detecting quality of banking data.
Background
Along with the high-speed development of social economy, each big bank serves as a financial service center, more and more customers transact various financial services through the banks, so that data to be processed in a big data platform of the banks are more and more, the sources of the service data are various, the reliability and timeliness of the data are directly influenced whether correct conclusions can be obtained in the use process of the banking services, data analysis and data mining are not separated from high-quality data, and if the data do not have integrity, planning and consistency, the data inevitably cause inaccurate service results in the subsequent service development process, and the difficult-to-estimate loss possibly can be caused.
However, the existing method for detecting the quality of the banking data is usually near real-time operation, so that real-time detection of the data cannot be realized, and the banking data is continuously generated in real time, so that the problems of accumulation of the banking data, incapability of accurately operating each banking system and the like are caused, a great deal of manpower and material resources are required to be consumed to process the problems, and the quality detection of the banking data is high in cost and poor in detection effect.
Disclosure of Invention
The embodiment of the application mainly aims to provide a method, a device, a storage medium and equipment for detecting the quality of banking data, which can effectively improve the detection effect of the quality of banking data and reduce the detection cost so as to ensure the stable operation of each banking system.
In a first aspect, an embodiment of the present application provides a method for detecting quality of banking data, where the method includes:
acquiring system service data of each service system in a bank; storing the system service data into a preset database;
the system service data of each service system is stored in a distributed mode on different partitions of a block chain;
configuring detection rules according to the system service data on each different partition of the block chain, the fields to be detected and preset built-in general rules;
configuring service data to be detected, the detection rule and a calculation engine Flink window to form a task to be detected, and storing the task to be detected into a distributed publish-subscribe message system card Kafka;
and transmitting the task to be detected in the Kafka to the Flink in real time to perform real-time data quality detection, so as to obtain a detection result.
Optionally, the preset database is a relational database Mysql.
Optionally, the preset built-in general rule includes at least one of data format check, data range check, null value check, data uniqueness check, integrity check and regular expression check.
Optionally, after the configuring the service data to be detected, the detection rule and the calculation engine link window to form the task to be detected, the method further includes:
and adding a unique identifier for the task to be detected.
In a second aspect, an embodiment of the present application further provides a banking data quality detection apparatus, where the apparatus includes:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring system service data of each service system in a bank; storing the system service data into a preset database;
the storage unit is used for storing the system service data of each service system to different partitions of the block chain in a distributed manner;
the first configuration unit is used for configuring detection rules according to the system service data on each different partition of the block chain, the fields to be detected and preset built-in general rules;
the second configuration unit is used for configuring the service data to be detected, the detection rule and the calculation engine Flink window to form a task to be detected, and storing the task to be detected into a distributed publish-subscribe message system Kafka;
and the detection unit is used for transmitting the task to be detected in the Kafka to the Flink in real time to perform real-time data quality detection, so as to obtain a detection result.
Optionally, the preset database is a relational database Mysql.
Optionally, the preset built-in general rule includes at least one of data format check, data range check, null value check, data uniqueness check, integrity check and regular expression check.
Optionally, the apparatus further includes:
and the adding unit is used for adding the unique identification for the task to be detected.
The embodiment of the application also provides a banking data quality detection device, which comprises: a processor, memory, system bus;
the processor and the memory are connected through the system bus;
the memory is configured to store one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform any of the implementations of the banking data quality detection method described above.
The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions run on the terminal equipment, the terminal equipment is caused to execute any implementation mode of the banking data quality detection method.
The embodiment of the application provides a method for detecting the quality of banking data, which comprises the steps of firstly, acquiring system business data of each business system in a bank; storing the system service data into a preset database, then storing the system service data of each service system onto different partitions of the block chain in a distributed manner, and configuring detection rules according to a data table, fields to be detected and preset built-in general rules by the system service data of each different partition of the block chain; and then, configuring the service data to be detected, configured detection rules and a calculation engine Flink window to form a task to be detected, storing the task to be detected into Kafka, and further transmitting the task to be detected in Kafka into the Flink in real time to perform real-time data quality detection to obtain a detection result. Therefore, the real-time performance and the accuracy of the quality detection of the banking business data can be effectively improved, the detection cost is reduced, and the stable operation of each banking business system is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a method for detecting quality of banking data according to an embodiment of the present application;
FIG. 2 is an overall process diagram of quality detection of banking data provided by an embodiment of the present application;
fig. 3 is a schematic diagram of a banking data quality detection device according to an embodiment of the present application.
Detailed Description
The current banking data quality detection method generally adopts a stream processing framework, most technologies adopt Spark Streaming, and although the near real-time response can be achieved, the fault tolerance cost is low, batch operation and stream operation can be combined, delay is unavoidable based on the realization principle of the method, and the accurate real-time performance cannot be achieved. The banking data is always generated continuously in real time, so that the existing data quality detection method can cause the problems of accumulation of the banking data, incapability of accurately running each banking system and the like, and therefore, how to realize real-time data quality detection of a large number of bounded or unbounded banking data streams of the banks and ensure the real-time performance of data processing are the problems to be solved urgently at present.
In order to solve the above-mentioned defect and improve the detection effect to the quality of the business data of the bank, the embodiment of the application provides a business data quality detection method of the bank, obtain the system business data of every business system in the bank at first; storing system service data into a preset database, then storing the system service data of each service system onto different partitions of a block chain in a distributed manner, and configuring detection rules according to a data table, fields to be detected and preset built-in general rules by the system service data on each different partition of the block chain; and configuring the service data to be detected, the detection rule and the calculation engine Flink window to form a task to be detected, storing the task to be detected in Kafka, and further transmitting the task to be detected in Kafka to the Flink for real-time data quality detection to obtain a detection result. Therefore, the real-time performance and the accuracy of the quality detection of the banking business data can be effectively improved, the detection cost is reduced, and the stable operation of each banking business system is ensured.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
First embodiment
Referring to fig. 1, a flow chart of a banking data quality detection method provided in this embodiment includes the following steps:
s101: acquiring system service data of each service system in a bank; and storing the system service data into a preset database.
In this embodiment, in order to effectively improve the detection effect of the quality of the bank service data and reduce the detection cost so as to ensure the stable operation of each bank service system, the application provides a method for detecting the quality of the service data of each system in a big bank data development platform based on a blockchain, which specifically comprises the following steps: firstly, uplink-storing data partitions in a blockchain through each service system in a big data development platform of a bank, configuring corresponding detection rules for system service data in each partition in the blockchain according to a data table and fields to be detected, forming a task after configuring data sources to be detected and detection rules of different tasks and a Flink window, configuring a unique identification task ID of the task, transmitting the task to Kafka for storage, and transmitting the task to the Flink in real time for actual data quality detection.
Specifically, in the present embodiment, first, system service data of each service system (such as real-time loan system, real-time transfer system, etc.) in the bank may be acquired, and these system service data may be stored in a preset database to perform the subsequent step S102. The acquired business systems in the acquired bank refer to the banking business systems which use a bank big data development platform to carry out development tasks, including but not limited to real-time data warehouse projects, data lake projects and the like, wherein the system business data can include but not limited to data in formats such as Excel table formats, database formats and the like.
It should be noted that, the specific composition of the preset database is not limited, and the database can be set according to actual conditions and experience values. For example, a relational database Mysql may be used as the preset database.
S102: system service data of each service system is stored in a distributed manner on different partitions of the blockchain.
In this embodiment, after the system service data of each service system in the bank is obtained in step S101, the system service data of each service system may be further stored in a distributed manner on different partitions of the blockchain. To ensure that one service system corresponds to one blockchain partition, and the system service data of different service systems are stored in different partitions, thereby ensuring the security of the data and being used for executing the subsequent step S103.
S103: and configuring detection rules according to the system service data on each different partition of the block chain, the fields to be detected and preset built-in general rules.
In this embodiment, after the system service data of each service system of the bank is stored in a distributed manner on different partitions of the blockchain in step S102, the system service data on each different partition of the blockchain may be further configured with a detection rule according to the data table, the field to be detected and a preset built-in general rule, and the process may be performed until the subsequent step S104.
The application does not limit the specific format and content of the data table, the field to be detected and the preset built-in general rule, and can be set according to actual conditions and experience values. For example, the data table may be a system_config table in a real-time number bin. The fields to be detected may include ID, name, value, description, type wait to detect fields, and the built-in general rules include format check of data, data range check, null check, uniqueness check of data, integrity check, and regular expression check.
An alternative implementation may include, but is not limited to, at least one of a format check, a data range check, a null check, a data uniqueness check, an integrity check, and a regular expression check for the data. The detection rule is similar to the meaning of the built-in general rule, and also comprises, but is not limited to, format check, data range check, null value check, data uniqueness check, integrity check and regular expression check of the data. The data format check is to detect whether the format of the data accords with the specification; the data range check is to check whether the range of the data meets the specification; the null value test of the data is to detect whether null values exist in a field to be detected of the data; the uniqueness check of the data is to ensure that the value in the data of the primary key value of the data is unique in the whole data set; the integrity check of the data is to check that the data accords with a predefined condition to ensure the integrity of the data; regular expression checking of data is to detect that data in a format such as mailbox address, phone, date, etc. meets its specifications for verifying that text meets a particular format or pattern.
S104: and configuring the business data to be detected, the detection rules and the calculation engine Flink window to form a task to be detected, and storing the task to be detected into Kafka.
In this embodiment, after the detection rule is configured in step S103, further, after the banking data to be detected is obtained, the banking data to be detected, the configured detection rule and a calculation engine (Flink) window may be configured to form a task to be detected, and after a unique identifier is added to the task to be detected, the task to be detected is stored in Kafka. The specific content of the identifier is not limited, and may be set according to actual conditions and empirical values, for example, id (Identity document) having uniqueness may be used as a unique identifier.
It should be noted that the link window configuration may include, but is not limited to, a window type and a delay or statistics time type, where the window type may include, but is not limited to, a sliding window or a rolling window, and the delay or statistics time type may include, but is not limited to, a processing time or a traffic time.
S105: and transmitting the task to be detected in Kafka to the Flink in real time to perform real-time data quality detection, so as to obtain a detection result.
In this embodiment, the to-be-detected service data, the detection rule and the calculation engine link window are configured in step S104 to form a to-be-detected task, and after the to-be-detected task is stored in Kafka, further, the to-be-detected task in Kafka may be transmitted to the link in real time, and the data quality of the to-be-detected service data is detected according to the configured detection rule, and the detection result is output.
The detection result comprises that the service data to be detected is effective data or the service data to be detected is ineffective data.
For example, one task in the real-time bin: and (3) the data are dropped into the Hbase database, the data of the whole task are data in an Excel table format, the data of the whole task are immediately transmitted to the Flink for real-time data quality detection through Kafka after being uploaded to a bank big data development platform, namely, the quality of the data is detected by using a detection rule (similar to a built-in general rule), and a detection result is obtained.
In order to facilitate understanding of the banking data quality detection method provided by the present application, the present application also provides an overall process diagram of banking data quality detection, as shown in fig. 2, specifically: in order to obtain the system service data of each service system (such as real-time loan system, real-time transfer system, etc.) in the big data development platform of the bank, and store the system service data in different partitions of the blockchain in a distributed manner so as to ensure the safety of the data, and then configure the system service data on each different partition of the blockchain according to a data table, fields to be detected and preset built-in general rules to obtain the detection rules. And then configuring the business data to be detected, the detection rule and the Flink window to form a task to be detected and storing the task to be detected into Kafka. And then the task to be detected in Kafka is transmitted to the Flink in real time for real-time data quality detection, so that the detection result that the business data to be detected are effective data or invalid data is accurately detected in real time, the real-time performance and accuracy of the quality detection of the banking business data are improved, and the stable operation of each banking business system is ensured.
In summary, in the method for detecting quality of banking data provided in this embodiment, first, system service data of each service system in a bank is obtained; storing the system service data into a preset database, then storing the system service data of each service system onto different partitions of the block chain in a distributed manner, and configuring detection rules according to a data table, fields to be detected and preset built-in general rules by the system service data of each different partition of the block chain; and then, configuring the service data to be detected, configured detection rules and a calculation engine Flink window to form a task to be detected, storing the task to be detected into Kafka, and further transmitting the task to be detected in Kafka into the Flink in real time to perform real-time data quality detection to obtain a detection result. Therefore, the real-time performance and the accuracy of the quality detection of the banking business data can be effectively improved, the detection cost is reduced, and the stable operation of each banking business system is ensured.
Second embodiment
The present embodiment will be described with reference to a banking data quality detection apparatus, and related content is referred to the above method embodiment.
Referring to fig. 3, a schematic composition diagram of a banking data quality detection apparatus provided in this embodiment specifically includes:
an acquiring unit 301, configured to acquire system service data of each service system in a bank; storing the system service data into a preset database;
a storage unit 302, configured to store system service data of each service system in a distributed manner onto different partitions of a blockchain;
a first configuration unit 302, configured to configure the detection rules for the system service data on each different partition of the blockchain according to the data table, the field to be detected, and a preset built-in general rule;
the second configuration unit 304 is configured to configure the service data to be detected, the detection rule and the calculation engine link window to form a task to be detected, and store the task to be detected in Kafka;
and the detection unit 305 is configured to transmit the task to be detected in the Kafka to the link in real time to perform real-time data quality detection, so as to obtain a detection result.
In one implementation of this embodiment, the preset database is a relational database Mysql.
In one implementation manner of this embodiment, the preset built-in general rule includes at least one of format check, data range check, null check, uniqueness check, integrity check and regular expression check of data.
In one implementation of this embodiment, the apparatus further includes:
and the adding unit is used for adding the unique identification for the task to be detected.
In summary, the present embodiment provides a device for detecting quality of banking data, first, acquiring system service data of each service system in a bank; storing the system service data into a preset database, then storing the system service data of each service system onto different partitions of the block chain in a distributed manner, and configuring detection rules according to a data table, fields to be detected and preset built-in general rules by the system service data of each different partition of the block chain; and then, configuring the service data to be detected, configured detection rules and a calculation engine Flink window to form a task to be detected, storing the task to be detected into Kafka, and further transmitting the task to be detected in Kafka into the Flink in real time to perform real-time data quality detection to obtain a detection result. Therefore, the real-time performance and the accuracy of the quality detection of the banking business data can be effectively improved, the detection cost is reduced, and the stable operation of each banking business system is ensured.
Further, the embodiment of the application also provides a banking data quality detection device, which comprises: a processor, memory, system bus;
the processor and the memory are connected through the system bus;
the memory is for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform any of the implementations of the banking data quality detection method described above.
Wherein the number of processors and the memory may be one or more.
The memory may include read only memory and random access memory and provide instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, where the operating instructions may include various operating instructions for performing various operations. The operating system may include various system programs for implementing various underlying services and handling hardware-based tasks.
The processor controls the operation of the terminal device, which may also be referred to as a CPU.
The method disclosed by the embodiment of the application can be applied to a processor or realized by the processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor described above may be a general purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
Further, the embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions run on the terminal equipment, the terminal equipment is caused to execute any implementation method of the banking data quality detection method.
In the context of the present application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable medium of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
When introducing elements of various embodiments of the present application, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements.
It should be noted that, it will be understood by those skilled in the art that all or part of the above-mentioned method embodiments may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and the program may include the above-mentioned method embodiments when executed. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random-access Memory (Random Access Memory, RAM), or the like.
Computer program code for carrying out operations of the present application may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points. The apparatus embodiments described above are merely illustrative, wherein the units and modules illustrated as separate components may or may not be physically separate. In addition, some or all of the units and modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. Those of ordinary skill in the art will understand and implement the present application without undue burden.
In addition, it should be noted that the method, the device, the storage medium and the equipment for detecting the quality of banking data provided by the application can be used in the blockchain field or the financial field. The foregoing is merely an example, and the application fields of the banking data quality detection method, the banking data quality detection device, the storage medium and the banking data equipment provided by the present application are not limited.
The foregoing is merely a preferred embodiment of the present application, and the present application has been disclosed in the above description of the preferred embodiment, but is not limited thereto. Any person skilled in the art can make many possible variations and modifications to the technical solution of the present application or modifications to equivalent embodiments using the methods and technical contents disclosed above, without departing from the scope of the technical solution of the present application. Therefore, any simple modification, equivalent variation and modification of the above embodiments according to the technical substance of the present application still fall within the scope of the technical solution of the present application.
Claims (10)
1. A method for detecting quality of banking data, the method comprising:
acquiring system service data of each service system in a bank; storing the system service data into a preset database;
the system service data of each service system is stored in a distributed mode on different partitions of a block chain;
configuring detection rules according to the system service data on each different partition of the block chain, the fields to be detected and preset built-in general rules;
configuring service data to be detected, the detection rule and a calculation engine Flink window to form a task to be detected, and storing the task to be detected into a distributed publish-subscribe message system card Kafka;
and transmitting the task to be detected in the Kafka to the Flink in real time to perform real-time data quality detection, so as to obtain a detection result.
2. The method of claim 1, wherein the predetermined database is a relational database Mysql.
3. The method of claim 1, wherein the pre-set built-in general rules include at least one of data format check, data range check, null check, data uniqueness check, integrity check, and regular expression check.
4. A method according to any one of claims 1-3, wherein said configuring the traffic data to be detected with the detection rules and a calculation engine link window, after forming the task to be detected, further comprises:
and adding a unique identifier for the task to be detected.
5. A banking data quality detection apparatus, the apparatus comprising:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring system service data of each service system in a bank; storing the system service data into a preset database;
the storage unit is used for storing the system service data of each service system to different partitions of the block chain in a distributed manner;
the first configuration unit is used for configuring detection rules according to the system service data on each different partition of the block chain, the fields to be detected and preset built-in general rules;
the second configuration unit is used for configuring the service data to be detected, the detection rule and the calculation engine Flink window to form a task to be detected, and storing the task to be detected into a distributed publish-subscribe message system Kafka;
and the detection unit is used for transmitting the task to be detected in the Kafka to the Flink in real time to perform real-time data quality detection, so as to obtain a detection result.
6. The apparatus of claim 5, wherein the predetermined database is a relational database Mysql.
7. The apparatus of claim 5, wherein the pre-set built-in general rules include at least one of a format check, a data range check, a null check, a uniqueness check, an integrity check, and a regular expression check of the data.
8. The apparatus according to any one of claims 5-7, further comprising:
and the adding unit is used for adding the unique identification for the task to be detected.
9. A banking data quality detection apparatus, comprising: a processor, memory, system bus;
the processor and the memory are connected through the system bus;
the memory is for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform the banking data quality detection method of any of claims 1-4.
10. A computer readable storage medium having instructions stored therein which, when executed on a terminal device, cause the terminal device to perform the banking data quality detection method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311197542.8A CN117235133A (en) | 2023-09-15 | 2023-09-15 | Bank business data quality detection method, device, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311197542.8A CN117235133A (en) | 2023-09-15 | 2023-09-15 | Bank business data quality detection method, device, storage medium and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117235133A true CN117235133A (en) | 2023-12-15 |
Family
ID=89082114
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311197542.8A Pending CN117235133A (en) | 2023-09-15 | 2023-09-15 | Bank business data quality detection method, device, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117235133A (en) |
-
2023
- 2023-09-15 CN CN202311197542.8A patent/CN117235133A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11830001B2 (en) | Blockchain consensus method, accounting node and node | |
| CN109933514B (en) | Data testing method and device | |
| CN112800095B (en) | Data processing method, device, equipment and storage medium | |
| US20180212986A1 (en) | Network attack detection method and device | |
| US9507761B2 (en) | Comparing webpage elements having asynchronous functionality | |
| CN109325744B (en) | Payment processing method, payment processing device, payment processing medium and electronic equipment | |
| CN112084179A (en) | Data processing method, device, equipment and storage medium | |
| CN111858576A (en) | Real-time message processing method, system, device, storage medium and electronic equipment | |
| CN117235133A (en) | Bank business data quality detection method, device, storage medium and equipment | |
| CN116483888A (en) | Program evaluation method and device, electronic equipment and computer readable storage medium | |
| CN117216736A (en) | Abnormal account identification method, data scheduling platform and graph computing platform | |
| CN112364285B (en) | Method and device for establishing abnormality detection model based on UEBA (unified extensible firmware interface) and related products | |
| CN113095891A (en) | Data processing method and device, storage medium and electronic device | |
| CN111681005A (en) | Data interaction method and device and electronic equipment | |
| CN110493205B (en) | Subway data processing method and device | |
| CN114979100B (en) | Cloud resource inspection method and related device | |
| CN110532561B (en) | Data detection method and device, storage medium and electronic device | |
| CN109857632B (en) | Test method, test device, terminal equipment and readable storage medium | |
| CN114172725B (en) | Illegal website processing method and device, electronic equipment and storage medium | |
| CN116627827A (en) | Parallel method, device, equipment and storage medium for testing environment software | |
| CN106372178A (en) | Transaction file processing method and device as well as server | |
| CN109474447B (en) | Alarm method and device for real-time monitoring system | |
| CN114925406A (en) | Data verification method, device and computer program product | |
| CN113961632A (en) | Message data processing method and device and electronic equipment | |
| CN117689132A (en) | Processing method and device of flow form, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |