CN115080805A - Data checking method, device and system and electronic equipment - Google Patents

Abstract

The application discloses a data checking method, a device, a system and an electronic device, wherein the method comprises the following steps: receiving and analyzing a check task to acquire configuration information of the check task, wherein the check task is issued to the cloud service platform by the check management platform in response to a check task establishment instruction; establishing a check flow application of the check task according to the configuration information, wherein the check flow application at least comprises a pulling component, at least one processing component and a convergence component; calling a pulling component, and pulling first data from at least one target data source according to the configuration information; calling a processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to configuration information; and calling the convergence component, and summarizing the result obtained by checking to obtain a data checking result. The method and the device realize cross-system checking of the data; and online configuration and one-key deployment can be supported, and flexible configuration of data checking and quick iteration of change are realized.

Description

Data checking method, device and system and electronic equipment

Technical Field

The present application relates to the field of data processing, and in particular, to a data checking method, apparatus, system, and electronic device.

Background

In the development process of software products, a lot of data are generated, data circulation among systems is easy to cause data inconsistency due to various reasons, and many abnormal problems are caused if valuable data are inconsistent, so that in some service scenes, data from different service systems need to be checked to find abnormal data.

Cross-system data verification presents a number of problems: for example, data islanding and chimney trouble problems, service links are longer and longer, data are independently stored among systems, distributed system problems, network errors, system abnormalities and the like are easily caused in the complex links, and it is very difficult to find out the data inconsistency in time, so that many data inconsistency problems among the links are known only after operation feedback is received and cannot be found out in time when changes or faults occur; the change is frequent and the logic is complex, the business is changed continuously, the reconciliation rule and the reconciliation data are also changed, and each data check needs to carry out quick change on the checking rule by developing codes for workers familiar with the business, which consumes very large labor and time cost.

In the prior art, a traditional programming mode is adopted for data checking or acquisition, or the traditional programming mode is mainly focused on single system internal checking or has no universality, so that the requirement of a large number of cross-system data checking is difficult to meet.

Disclosure of Invention

In view of the above situation, embodiments of the present application provide a data checking method, apparatus, system and electronic device to overcome or partially overcome the disadvantages of the prior art.

In a first aspect, an embodiment of the present application provides a data checking method, where the method is implemented by a data checking system, where the checking system includes a checking management platform and a cloud service platform that are in communication connection;

the method is applied to the cloud service platform and comprises the following steps:

receiving and analyzing a check task to acquire configuration information of the check task, wherein the check task is issued to the cloud service platform by the check management platform in response to a check task establishment instruction;

establishing a check flow application of a check task according to the configuration information, wherein the check flow application at least comprises a pulling component, at least one processing component and a convergence component;

calling the pulling component, and pulling first data from a plurality of target business systems according to the configuration information;

calling the processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to the configuration information;

and calling the convergence component, and summarizing the result obtained by checking to obtain a data checking result.

In a second aspect, an embodiment of the present application further provides a data collation method, where the method is implemented by a data collation system, and the collation system includes a collation management platform and a cloud service platform that are in communication connection;

the method is applied to the checking management platform and comprises the following steps:

providing a configuration interface, wherein the configuration interface comprises a checking task configuration control, and the checking task configuration control comprises a data source configuration sub-control and a checking rule configuration sub-control;

acquiring configuration information of the checking task through a configuration interface;

and generating a checking task according to the configuration information, and issuing the checking task to the cloud service platform.

In a third aspect, an embodiment of the present application further provides a data checking apparatus, where the data checking apparatus may implement any one of the methods described above.

In a fourth aspect, an embodiment of the present application further provides a data verification system, where the data verification system includes a verification management platform and a cloud service platform that are in communication connection, and the verification management platform is deployed with a data verification device, so that the data verification method applied to the verification management platform as described above can be implemented; the cloud service platform is provided with the data checking device, so that the data checking method applied to the cloud service platform can be realized.

In a fifth aspect, an embodiment of the present application further provides an electronic device, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform any of the methods described above.

In a sixth aspect, this application embodiment also provides a computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform any of the methods described above.

The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects:

the checking system comprises a checking management platform and a cloud service platform which are in communication connection, the checking management platform is regarded as a client side of the cloud service platform, checking personnel can configure checking tasks on the checking management platform and issue the checking tasks to the cloud service platform, the cloud service platform can establish checking application according to configuration information of the checking management platform after receiving the management tasks, the checking application comprises a plurality of flow applications which can be configured in advance, the flow applications pull first data and second data from the systems according to the configuration information, and data checking and result summarizing work is completed. The data of different data sources can be gathered together for comparison processing, and cross-system checking of the data is realized; the online configuration and one-key deployment can be supported, and the flexible configuration of data checking and the quick iteration of change are realized; and the service is separated from the check, and the data check does not influence the access of the service at all.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 shows a schematic block diagram of a data collation system according to an embodiment of the present application;

FIG. 2 shows a schematic flow diagram of a data collation method according to an embodiment of the present application;

FIG. 3 shows a schematic flow diagram of a data collation method according to another embodiment of the present application;

FIG. 4 shows a schematic flow diagram of a data collation method according to a further embodiment of the present application;

FIG. 5 shows a schematic structural diagram of a first data collation apparatus according to an embodiment of the present application;

FIG. 6 shows a schematic structural diagram of a second data collation apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

The data checking system comprises at least a communication connection checking management platform and a cloud service platform, the checking management platform serves as the cloud service platform, checking personnel can submit checking tasks on the management platform at any time, and the cloud service platform can pull corresponding data from a plurality of service systems according to the checking tasks to check the data.

Fig. 1 shows a schematic structural diagram of a data collation system according to an embodiment of the present application, and as can be seen from fig. 1, a data collation system 100 includes a collation management platform 110 and a cloud service platform 120, and the collation management platform 110 and the cloud service platform 120 are in communication connection. The cloud service platform 120 may be any cloud service in the prior art, such as a kubernets platform.

The Kubernetes platform is a portable, extensible, open source platform for managing containerized workloads and services that facilitate declarative configuration and automation. Kubernets possess a large and rapidly growing ecosystem with a wide range of services, support and tools.

Fig. 2 shows a schematic flowchart of a data checking method according to an embodiment of the present application, where the method is applied to the cloud service platform, and the method at least includes steps S210 to S250:

step S210: and receiving and analyzing a check task to acquire configuration information of the check task, wherein the check task is issued to the cloud service platform by the check management platform in response to a check task establishment instruction.

The data verification method of the present application can be implemented by the data verification system 100 shown in fig. 1, and is applied to the cloud service platform 120 of the data verification system 100, in a use process, the verification management platform 110 serves as a client of the cloud service platform 120, the verification management platform 110 is a platform facing a verifier, the verifier can verify configuration of a verification task performed by the management platform 110, after the configuration is completed, the verifier can initiate a task establishment instruction on the verification management platform 110, and the verification management platform 110 issues the verification task to the cloud service platform 120 in response to the task establishment instruction.

The configuration of the checking task performed by the checking personnel on the checking management platform 110 mainly includes two core configurations, one is a data source to be processed, and the other is a checking rule how to check the data.

After receiving a check task, the cloud service platform 120 first analyzes the check task to obtain configuration information of the check task, where the configuration information at least includes a data source of the first data and a check rule.

Step S220: and establishing a check flow application of a check task according to the configuration information, wherein the check flow application at least comprises a pulling component, at least one processing component and a convergence component.

After obtaining the configuration information of the verification task, a verification flow application of the verification task needs to be established on the cloud service platform 120, where the verification flow application at least includes the following 3 components: the system comprises a pulling component, a processing component and a convergence component.

So-called stream applications are handled or loaded by streams (streams), i.e. used for side-loading. The data is checked, and a common flow is to pull the data, process the data, and summarize a result, so the stream application of the present application at least includes 3 components, namely, a pull component, a process component, and a convergence component, but is not limited thereto. The stream application composed of the 3 components of the pull component, the processing component and the convergence component can be understood as a general stream application template for data verification, and is deployed on the cloud service platform 120 in advance, and can be directly called when data verification is performed, and only relevant contents in the stream application need to be replaced according to configuration information, such as a data source and a verification rule.

It should be noted that the stream applications are deployed on the cloud service platform 120 in advance by one key, but the so-called "deployment" is not equal to installation, and in some embodiments, in order to save hardware expense and occupation of the cloud service platform 120, the stream applications are not installed when not in use, and are installed and used by means of stream loading when in use.

In addition, the developed data checking script can also be formed into one component according to the personalized requirement of data checking, the component is stored on the cloud service platform 120, and when the stream application is established according to one checking task, the required component can be inquired and loaded in a plurality of functional components stored in the cloud service platform 120 according to the setting of configuration information, so that the rapid establishment of the stream application by checking is rapidly realized.

In some cases, the data processing methods of each service system may be different, and thus the universality of the processing components may be poor, and for this reason, in order to ensure smooth data checking, a corresponding processing component may be configured for each service, and when a stream application is established, the corresponding processing component is invoked according to the current service requirements. If the processing component 1 corresponds to the service a, the processing component 2 corresponds to the service B. The streaming application deployed on the cloud service platform 120 may add a container as needed to accommodate more components and support checking of massive business data.

After the core stream application is established, each component in the core stream application is called, and corresponding operations are sequentially executed according to the function of each component, in this embodiment, steps S230 to S250 are sequentially executed:

step S230: and calling the pulling component, and pulling the first data from at least one target data source according to the configuration information.

Step S240: and calling the processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to the configuration information.

Step S250: and calling the convergence component, and summarizing the result obtained by checking to obtain a data checking result.

The method comprises the steps that a pull component is called firstly, and the pull component is mainly responsible for pulling first data to be checked from an upstream first data source configured in configuration information.

And then calling a processing component, wherein the processing component can pull second data of the corresponding business system and match the first data with the second data. The service system corresponding to the processing component may be a service system where the verification management platform 110 is located, that is, a current service system, or may be a service system of a third party. For example, if the upstream service system is system a, and the service system where the checking management platform 110 is located is system B, the checking person may configure the target data source as service system a in the checking management platform 110, and default that the current service system is system B, that is, the service system corresponding to the processing element is system B, when performing data checking, the pulling element pulls the first data from the service system a, and the processing element may pull the second data from the service system B. For another example, if the upstream service system is system a and the third-party service system is system C, the checking personnel may configure the target data source as service system a and the third-party service system as system C in the checking management platform 110, and when performing data checking, the pulling component pulls the first data from the service system a, and the processing component pulls the second data from the service system C, and checks the first data and the second data.

In the checking process, the checking rule can be obtained from the configuration information, and the comparison and matching can be carried out according to the checking rule, so that the abnormal condition can be found.

And the aggregation component is mainly used for summarizing the result obtained by checking to obtain the data checking result.

As can be seen from the method shown in fig. 2, the verification system is designed, and the verification system includes a verification management platform and a cloud service platform which are in communication connection, the verification management platform is regarded as a client of the cloud service platform, a verifier can configure a verification task on the verification management platform and issue the verification task to the cloud service platform, after receiving the management task, the cloud service platform can establish a verification application according to configuration information of the management task, the verification application includes a plurality of flow applications which can be configured in advance, and the flow application pulls first data and second data from the plurality of systems according to the configuration information, and completes data verification and result summarizing. The data of different data sources can be gathered together for comparison processing, and cross-system checking of the data is realized; the online configuration and one-key deployment can be supported, and the flexible configuration of data checking and the quick iteration of change are realized; and the service is separated from the check, and the data check does not influence the access of the service at all.

In some embodiments of the present application, in the above method, the collation system further includes: the visual observation platform is connected with the cloud server platform; the method further comprises the following steps: sending the data checking result to the visual observation platform so that the visual observation platform displays the data checking result; and/or sending the abnormal result to a specified terminal through a specified communication mode.

In order to facilitate the visual check of the checking result by the checking personnel, in some embodiments of the application, an observable platform may be further provided in the checking system, the observable platform may be a Grafana (open observability platform, which has no unified Chinese name in the industry) platform or a prometheus (a monitoring system and a time series database, which has no unified Chinese name in the industry) platform, and the observable platform is in communication connection with the cloud server platform, and the cloud server platform may send the checking result to the observable platform for display after the checking result is confirmed. If the checking result contains abnormal data, the abnormal data can be sent to the terminal equipment of the person in charge of the checking task, such as a mobile phone, through a specified communication means, such as a short message, so as to remind the checking personnel to inquire the abnormal data on the observable platform.

In an embodiment of the application, in the above method, the creating a collation flow application of a collation task according to the configuration information includes: and providing a resource scheduling interface, responding to an interface calling instruction corresponding to the check task, and establishing check flow application of the check task on the cloud service platform.

Referring to fig. 1 again, when the verification management platform 110 issues the verification task on the cloud service platform 120, the verification task is implemented through a resource scheduling interface provided by the cloud service platform 120. The verification management platform 110 may be regarded as a client of the cloud service platform 120, the verification management platform 110 sends an interface call instruction corresponding to the verification task to the cloud service platform 120 by calling the interface, and the resource scheduling interface establishes a verification stream application of the verification task in the environment of the cloud service platform 120 after receiving the interface call instruction.

In some embodiments of the present application, in the above method, the configuration information includes at least one target data source and a collation rule; the calling the pull component, and pulling the first data from at least one target data source according to the configuration information, includes: establishing a communication connection with the target data source; first data is pulled from the target data source based on the communication connection. The checking the first data and the second data includes: and calculating the first data and the second data according to the checking rule to confirm whether the first data and/or the second data are different.

When pulling data from a target data source, a pulling component first establishes a communication connection with the target data source, specifically, may send a connection request to the target data source, and then receives feedback of the target data source, if the feedback indicates that a connection can be established, a communication connection with the target data source is established, where the communication connection may be a plaintext-based connection, or a secret key-based (public key needs to be matched) connection, such as a Talent connection.

The target data source can be a MySQL database, an oracle database and the like, and can also be a source file, a webpage and the like. When data is pulled from the database, retrieval can be carried out according to fields, keywords and the like, and the retrieved data is pulled; when data is pulled from a source file and a webpage, the data can be extracted in a regular expression mode, and then relevant data is pulled.

When the processing component checks the data, the processing component compares and matches the first data with the second data according to a check rule analyzed from the configuration information, and considers the data meeting the check rule as normal data; and regarding the data which does not meet the checking rule as abnormal data. During checking, the first data or the second data or the first data and the second data can be determined to have abnormality through comparison, matching and the like.

If the first data is an order number acquired from a webpage, the second data is a corresponding order number in the current service system, the check rule is limited to use the first data as a reference, whether the second data is consistent with the first data or not is determined, if so, the second data is determined to be normal, and if not, the second data is determined to be abnormal. In the present application, the collation rule is not limited, and may be configured according to a specific service.

In some embodiments of the present application, in the above method, the number of the processing components is multiple, each processing component corresponds to one service system, and the multiple processing components are connected according to a dependency relationship of corresponding services; the calling the processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to the configuration information, further comprising: and the pre-processing component transmits at least one of the first data, the second data of the corresponding service and the self checking result to the post-processing component so as to enable the post-processing component to check the data.

In the actual process, there are often dependencies or relationships between services, so that one data may be checked with data in multiple service systems, at this time, multiple processing components may be set, each processing component corresponds to one service system, the multiple components are connected according to the dependency relationship of the services, for example, if a service C depends on a service B, and a service B depends on a service a, 3 processing components may be set, where a processing component 1 corresponds to the service a, a processing component 2 corresponds to the service B, a component 3 corresponds to the service C, and a processing component 1 is in communication connection with the service system a of the service a; the processing component 2 is in communication connection with a service system B of the service B; the processing component 3 is communicatively connected to a service system C of the service C. When data is checked, the pulling component firstly pulls first data from a target data source, the pulling component firstly transmits the first data to the processing component 1, the processing component 1 pulls second data of the service A from the corresponding service system A, and the first data and the second data are checked according to a checking rule to obtain a first checking result; then, the processing component 1 transmits the first data and/or the second data and the first check result (if necessary) to the processing component 2, the processing component 2 can pull the third data of the service B from the corresponding service system B, and the third data is compared with at least one of the first data and the second data according to the check rule to obtain a second check result; by analogy, the processing component 3 gets a third check result. If more components are provided, data can be passed to more components, versus more business data. However, it should be noted that, based on the capacity of the cloud service platform, the capacity of the container of the cloud service platform needs to be expanded while the number of components is increased.

Fig. 3 is a schematic flow chart illustrating a data collation method according to another embodiment of the present application, and as can be seen from fig. 3, the method of the present embodiment is applied to the collation management platform 110 (fig. 1), and the method includes:

step 310: and providing a configuration interface, wherein the configuration interface comprises a checking task configuration control, and the checking task configuration control comprises a data source configuration sub-control and a checking rule configuration sub-control.

As described above, the verification management platform 110, which is a client of the cloud service platform 120, is oriented to the verifier, and provides a configuration interface for operations of the verifier, where a plurality of task configuration controls are deployed in the configuration interface, and the verification task configuration controls include a data source configuration sub-control and a verification rule configuration sub-control. The collation personnel may add the target data source, and the collation rules directly to the configuration interface provided by the collation management platform 110. It should be noted that the configuration information is not limited to a scenario in which the configuration information is a relatively simple option or a simple expression, but may also be a script of the collation data, and the script may be directly imported into the configuration items of the collation management platform 110.

Step 320: and acquiring configuration information of the checking task through the configuration interface.

The reconciliation management platform 110 obtains configuration information of the reconciliation task via the configuration interface, the configuration information including, but not limited to, at least one target data source, reconciliation rules, and the like.

Step 330: and generating a checking task according to the configuration information, and issuing the checking task to the cloud service platform.

After the configuration information is acquired, a check task is generated in the check management platform 110, and the check task is issued to the cloud service platform 120, specifically, a resource scheduling interface provided by the cloud service platform 120 is called, and an interface calling instruction of the check task is sent to the cloud service platform 120, so as to implement the issue of the check task. As described above, the cloud service platform 120 receives the interface call instruction through the resource scheduling interface, and establishes a check flow application of the check task, which indicates that the check task is successfully issued.

Fig. 4 is a schematic flow chart illustrating a data verification method according to still another embodiment of the present application, and as can be seen from fig. 4, the present embodiment includes:

the checking management platform receives the configuration information of the checking task and generates the checking task according to the configuration information; and calling a resource scheduling interface of the cloud service platform, and issuing the check task.

The cloud service platform receives and analyzes the check task to obtain a target data source and a check rule; establishing core convection applications according to configuration rules

And the cloud service platform calls the pulling component according to the configuration information, establishes connection with the target data source and pulls the first data.

And the cloud service platform calls the processing component according to the configuration information, establishes connection with the corresponding business system, pulls the second data, calls the processing component, and checks the first data and the second data according to the check rule.

And the cloud service platform calls the convergence component and summarizes the check result.

And the cloud service platform responds to the query request and sends the checking result to the observable platform.

Fig. 5 shows a schematic structural diagram of a first data collating apparatus 500 according to an embodiment of the present application, the first data collating apparatus 500 being disposed in the cloud service platform 120 of the data collating system 100, the first data collating apparatus 500 including:

a receiving and analyzing unit 510, configured to receive and analyze a check task to obtain configuration information of the check task, where the check task is issued to the cloud service platform by the check management platform in response to a check task establishment instruction;

an application establishing unit 520, configured to establish a check stream application of a check task according to the configuration information, where the check stream application at least includes a pull component, at least one processing component, and a convergence component;

a calling unit 530, configured to call the pull component, and pull first data from at least one target data source according to the configuration information; calling the processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to the configuration information; and calling the convergence component, and summarizing the result obtained by checking to obtain a data checking result.

In some embodiments of the present application, the collation system further comprises: the visual observation platform is connected with the cloud server platform; the first data verification device 500 further includes a result sending unit, configured to send the data verification result to the visual observation platform, so that the visual observation platform displays the data verification result; and/or sending the abnormal result in the data checking result to a specified terminal through a specified communication mode.

In some embodiments of the present application, in the first data collating device 500, the application establishing unit 520 is configured to provide a resource scheduling interface, and in response to an interface call instruction corresponding to the collation task, establish, at the cloud service platform, a collation flow application of the collation task.

In some embodiments of the present application, in the first data collating device 500, the configuration information includes at least one target data source and a collating rule; a calling unit 530, configured to establish a communication connection with the target data source; pulling first data from the target data source based on the communication connection; the invoking unit 530 is further configured to perform an operation on the first data and the second data according to the check rule, and determine whether the first data and/or the second data are different.

In some embodiments of the present application, in the first data checking apparatus 500, the number of the processing components is multiple, each processing component corresponds to one business system, and the multiple processing components are connected according to a dependency relationship of the corresponding business;

the invoking unit 530 is further configured to invoke the pre-processing component, and transfer at least one of the first data, the second data of the corresponding service, and the check result of the pre-processing component to the post-processing component, so that the post-processing component performs data check.

FIG. 6 shows a schematic structural diagram of a second data collation apparatus 600 according to an embodiment of the present application, the second data collation apparatus 600 being disposed in the collation management platform 110 of the data collation system 100, the second data collation apparatus 600 including:

a configuration unit 610, configured to provide a configuration interface, where the configuration interface includes a check task configuration control, and the check task configuration control includes a data source configuration sub-control and a check rule configuration sub-control;

an obtaining unit 620, configured to obtain configuration information of the checking task through the configuration interface;

the issuing unit 630 is configured to generate a check task according to the configuration information, and issue the check task to the cloud service platform.

In some embodiments of the present application, in the second data collating device 600, the issuing unit 630 is configured to invoke a resource scheduling interface provided by the cloud service platform, and send an interface calling instruction of a collation task to the cloud service platform, so as to implement issuing of the collation task.

It should be noted that, the first data verification apparatus 500 and the second data verification apparatus 600 can respectively implement the data verification methods described above, and are not described herein again.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Referring to fig. 7, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus.

And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the data checking device on the logic level. And the processor is used for executing the program stored in the memory and is specifically used for executing the method.

The method performed by the data collating device according to the embodiment shown in fig. 5 or 6 of the present application may be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.

The electronic device may further execute the method executed by the data checking apparatus in fig. 5 or 6, and implement the functions of the data checking apparatus in the embodiment shown in fig. 5 or 6, which are not described herein again in this embodiment of the present application.

Embodiments of the present application also provide a computer-readable storage medium storing one or more programs, where the one or more programs include instructions, which when executed by an electronic device including a plurality of application programs, enable the electronic device to perform the method performed by the data collating apparatus in the embodiment shown in fig. 5 or 6, and are specifically configured to perform the foregoing method.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of another element in the same process, method, article, or apparatus that comprises the element.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims (10)

1. A data checking method is characterized in that the method is realized through a data checking system, and the checking system comprises a checking management platform and a cloud service platform which are in communication connection;

the method is applied to the cloud service platform and comprises the following steps:

receiving and analyzing a check task to acquire configuration information of the check task, wherein the check task is issued to the cloud service platform by the check management platform in response to a check task establishment instruction;

establishing a check flow application of a check task according to the configuration information, wherein the check flow application at least comprises a pulling component, at least one processing component and a convergence component;

calling the pulling component, and pulling first data from at least one target data source according to the configuration information;

calling the processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to the configuration information;

and calling the convergence component, and summarizing the result obtained by checking to obtain a data checking result.

2. The method of claim 1, wherein the verification system further comprises: the visual observation platform is connected with the cloud server platform;

the method further comprises the following steps:

sending the data checking result to the visual observation platform so that the visual observation platform displays the data checking result;

and/or the presence of a gas in the gas,

and sending the abnormal result in the data checking result to a specified terminal through a specified communication mode.

3. The method of claim 1, wherein establishing a reconciliation flow application of reconciliation tasks based on the configuration information comprises:

and providing a resource scheduling interface, responding to an interface calling instruction corresponding to the check task, and establishing check flow application of the check task on the cloud service platform.

4. The method of claim 1, wherein the configuration information includes at least one target data source and collation rules;

the calling the pull component, and pulling the first data from at least one target data source according to the configuration information, includes:

establishing a communication connection with the target data source;

pulling first data from the target data source based on the communication connection;

the checking the first data and the second data includes:

and calculating the first data and the second data according to the checking rule to confirm whether the first data and/or the second data are different.

5. The method according to claim 4, wherein the number of the processing components is plural, each processing component corresponds to a service system, and the plural processing components are connected according to the dependency relationship of the corresponding service;

the calling the processing component, pulling second data from a corresponding service system, and checking the first data and the second data according to the configuration information, further comprising:

and the pre-processing component transmits at least one of the first data, the second data of the corresponding service and the self checking result to the post-processing component so as to enable the post-processing component to check the data.

6. A data checking method is characterized in that the method is realized through a data checking system, and the checking system comprises a checking management platform and a cloud service platform which are in communication connection;

the method is applied to the checking management platform and comprises the following steps:

providing a configuration interface, wherein the configuration interface comprises a checking task configuration control, and the checking task configuration control comprises a data source configuration sub-control and a checking rule configuration sub-control;

acquiring configuration information of the checking task through the configuration interface;

and generating a checking task according to the configuration information, and issuing the checking task to the cloud service platform.

7. The method of claim 6, wherein said publishing said reconciliation task to said cloud service platform comprises:

and calling a resource scheduling interface provided by the cloud service platform, and sending an interface calling instruction of a checking task to the cloud service platform so as to realize the release of the checking task.

8. A data collation apparatus characterised in that it implements a method as claimed in any one of claims 1 to 5 or 6 to 7.

9. A data checking system, characterized in that the data checking system comprises a checking management platform and a cloud service platform which are connected in communication, the checking management platform is provided with the data checking device of claim 8, and the method of claim 6 or 7 can be realized; the cloud service platform is provided with the data checking device as claimed in claim 8, and can implement the method as claimed in any one of claims 1 to 5.

10. An electronic device, comprising:

a processor; and

a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method of claims 1 to 5 or 6 to 7.

Images