CN115203158A

CN115203158A - Data comparison method, device, equipment and storage medium

Info

Publication number: CN115203158A
Application number: CN202210790520.1A
Authority: CN
Inventors: 李文轲; 王杰; 程强
Original assignee: Beijing Ruian Technology Co Ltd
Current assignee: Beijing Ruian Technology Co Ltd
Priority date: 2022-07-05
Filing date: 2022-07-05
Publication date: 2022-10-18

Abstract

The invention discloses a data comparison method, a data comparison device, data comparison equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: acquiring a target processing result of to-be-tested data in a target data packet in a to-be-tested case at a target processing stage; wherein the target processing phase comprises at least one of a server output phase, a distributed file backup phase and a distributed database storage phase; and comparing the target processing results based on a preset comparison rule to obtain a comparison result. Through the technical scheme, the data comparison efficiency can be improved.

Description

Data comparison method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data comparison method, apparatus, device, and storage medium.

Background

With the increasing of the data types and data sources, the number of strategies for data processing is also increasingly huge. In contrast to thousands of processing strategies, if manual coverage testing is needed, time and labor are consumed, requirements for personnel are high, and final results are not necessarily accurate. Thus, improvements are needed.

Disclosure of Invention

The invention provides a data comparison method, a data comparison device, data comparison equipment and a storage medium, which are used for improving data comparison efficiency and saving human resources.

According to an aspect of the present invention, there is provided a data alignment method, the method comprising:

acquiring a target processing result of to-be-tested data in a target data packet in a to-be-tested case at a target processing stage; wherein the target processing phase comprises at least one of a server output phase, a distributed file backup phase and a distributed database storage phase;

and comparing the target processing results based on a preset comparison rule to obtain a comparison result.

According to another aspect of the present invention, there is provided a data alignment apparatus, comprising:

the target processing result acquisition module is used for acquiring a target processing result of to-be-tested data in a target data packet in a to-be-tested case at a target processing stage; wherein the target processing phase comprises at least one of a server output phase, a distributed file backup phase and a distributed database storage phase;

and the comparison result determining module is used for comparing the target processing results based on a preset comparison rule to obtain a comparison result.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data comparison method of any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the data alignment method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme of the embodiment of the invention, the target processing result of the to-be-tested data in the target data packet in the to-be-tested case in the target processing stage is obtained, wherein the target processing stage comprises at least one of a server output stage, a distributed file backup stage and a distributed database storage stage, and then the target processing result is compared based on the preset comparison rule to obtain the comparison result. Compared with the situation that manual coverage test is carried out aiming at a large number of processing strategies in the prior art, the technical scheme provided by the invention can automatically test data, so that time and human resources are saved, and the data testing efficiency can be improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a flowchart of a data comparison method according to an embodiment of the present invention;

FIG. 2 is a flowchart of a data comparison method according to a second embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a data comparison apparatus according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device implementing the data comparison method according to the embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

It is to be understood that the terms "target" and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a data comparison method according to an embodiment of the present invention. The embodiment is applicable to how to compare data, and the method may be executed by a data comparison apparatus, which may be implemented in the form of hardware and/or software and may be integrated in an electronic device carrying a data comparison function. As shown in fig. 1, the data comparison method of this embodiment may include:

s110, obtaining a target processing result of to-be-tested data in a target data packet in the to-be-tested case at a target processing stage.

In this embodiment, the test case to be tested refers to a test case to be tested, which is compiled by a tester; optionally, a test case may be written according to the actual requirement document and the policy file, for example, one test case given in table 1 below. Further, the use case to be tested may include at least one of a use case ID, a use case name, a package name, a transport directory (i.e., a directory where the package is located), an expected result (i.e., a result after data processing in the package), an hdfs comparison path (i.e., a distributed file comparison path), and an execution result.

TABLE 1 example of use cases to be tested

The target data packet refers to a data packet corresponding to a case name in a case to be tested, which needs to be tested; optionally, each use case name corresponds to one data packet, and each use case name can be used as a target data packet. Furthermore, the corresponding target data packet can be searched from the carrying catalog according to the name of the data packet in the use case to be tested; and then the data to be tested in the target data packet can be obtained.

The target processing stage refers to different stages in the processing process of the data to be tested; optionally, at least one of a server output phase, a distributed file backup phase and a distributed database storage phase may be included; the server output stage is a stage of outputting a test result through an output server after the data to be tested is processed by the distributed server; the distributed file backup stage refers to a stage of backing up a test result output by an output server; the distributed database storage stage is a stage of storing the test result into the database after distributed file backup.

Optionally, based on the use case name in the use case to be tested, the corresponding target data packet is searched from the transport directory, the data to be tested in the target data packet is obtained, and the data to be tested is processed. Specifically, the data to be tested is tested based on the distributed server kafka to obtain a target processing result of the data to be tested, and the target processing result is output through an output server, namely, a server output stage; then, backing up the target processing result through the distributed file hdfs to obtain a backup result, namely a distributed file backup stage; and further storing the target processing result into a distributed database hbase, namely a distributed database storage stage.

And S120, comparing the target processing results based on a preset comparison rule to obtain a comparison result.

Optionally, determining whether the data volume of the output server increases in the server output stage; and comparing the target processing result based on a preset comparison rule under the condition that the data volume of the output server is increased to obtain a comparison result.

Specifically, in the server output stage, whether the data volume of the output server increases or not can be monitored, and if yes, the preset comparison rule is used for comparing the target processing result to obtain a comparison result. If not, the target processing result is not correctly accessed to the output server, namely the test fails, the log is output to a specific document, and the test is stopped, namely the test on the data to be tested in the target data packet is carried out.

Optionally, determining whether data is queried in a distributed file comparison path corresponding to the target data packet in a distributed file backup stage; if yes, comparing the target processing result based on a preset comparison rule to obtain a comparison result.

Specifically, in the distributed file backup stage, whether newly added data exists in a distributed file comparison path corresponding to a target data packet is inquired, if the newly added data exists, a plaintext script is called to clarify data on hdfs, namely, an encrypted target processing result is decrypted to obtain a decrypted target processing result, and a preset comparison rule is used for comparing the target processing result to obtain a comparison result. It should be noted that, in the distributed file backup stage, only data needs to be encrypted. If not, the test of the data to be tested in the target data packet fails, and the test of the next case is continued.

Optionally, determining whether there is storage data corresponding to the target data packet in the stage of storing the distributed database; if yes, comparing the target processing result based on a preset comparison rule to obtain a comparison result.

Specifically, in the distributed database storage stage, whether a target processing result exists in a database corresponding to the target processing result is inquired; if yes, comparing the target processing result based on a preset comparison rule to obtain a comparison result. If not, continuing the test of the next case.

It can be understood that comparing the target processing results at different stages can ensure the correctness of the target processing results in the data transfer process and avoid the problem of modification in the data transfer process.

Example two

Fig. 2 is a flowchart of a data comparison method according to a second embodiment of the present invention. This example is further optimized on the basis of the above examples, and provides an alternative implementation scheme. As shown in fig. 2, the data comparison method of this embodiment may include:

s210, determining whether the target data packet in the use case to be tested is transmitted to the input server according to the data volume in the input server.

In this embodiment, the input server refers to a distributed server that receives a target data packet in a use case to be tested.

Specifically, if the data volume in the input server increases, it is determined whether the target data packet in the to-be-tested use case is transmitted to the input server.

S220, if yes, obtaining a target processing result of the to-be-tested data in the target data packet in the to-be-tested case in the target processing stage.

Wherein the target processing phase comprises at least one of a server output phase, a distributed file backup phase, and a distributed database storage phase.

And S230, comparing the target processing results based on a preset comparison rule to obtain a comparison result.

Specifically, for each field in the data to be tested, whether the target processing result of the field is consistent with the expected processing result is compared to obtain a comparison result.

S240, backfilling the comparison result to a test result field corresponding to the target data packet in the to-be-tested case.

In this embodiment, the comparison result may be success or failure; further, a failure reason and the like may be included.

Specifically, the comparison result may be refilled into a test result field corresponding to the target data packet in the to-be-tested case.

According to the technical scheme of the embodiment of the invention, whether the target data packet in the case to be tested is transmitted to the input server is determined according to the data volume in the input server, then the target processing result of the data to be tested in the target data packet in the case to be tested in the target processing stage is obtained, then the target processing result is compared based on the preset comparison rule to obtain the comparison result, and finally the comparison result is backfilled to the test result field corresponding to the target data packet in the case to be tested. Compared with the situation that manual coverage test is carried out aiming at a large number of processing strategies in the prior art, the technical scheme provided by the invention can automatically test data, so that time and human resources are saved, and the data testing efficiency can be improved.

On the basis of the above-described embodiments, a specific example is given.

(1) Reading the name of the data packet in the use case to be tested, and transporting the data zip packet with the corresponding name to a specified DPP (DPP) directory (namely, a temporary directory).

(2) And (3) monitoring the content in the DPP directory, judging whether the data packet in the step (1) exists, if so, judging that the data packet fails, outputting the log to a document, and stopping running the use case script to be tested. If the monitoring DPP directory is empty, the target data packet is normally carried, and the next step is continued.

(3) Monitoring the data amount of the kafka (namely an input server) corresponding to the topic, if the amount in the topic is not increased, judging that the data is not accessed correctly, outputting the log into a document, and stopping the script. If the amount of data grows, the next step is continued.

(4) And monitoring the volume of topic data of the kafka (namely, an output server), if the volume of the topic data is not increased, judging that a target processing result corresponding to the target data packet is not output correctly, outputting the log into a document, and stopping the script. If the amount of data grows, the next step is continued.

(5) And monitoring data in topic of the kafka (namely the output server), checking whether a corresponding field value (a target processing result corresponding to a field) is consistent with an expected result in the excel or not, if not, backfilling the result into the excel, filling the execution result into failure, and reading the next use case. If the field value and the expected result are constant, the next step is continued.

(6) And entering new data under the hdfs comparison path corresponding to the comparison path query target data packet in the hdfs, if the new data cannot be queried, backfilling the result into the excel, filling the execution result as failure, and reading the next use case. If the corresponding data can be found, the next step is continued.

(7) And calling a plaintext script to clarify the data on hdfs, comparing a target processing result of a field in the data to be tested in the target data packet with an expected result corresponding to the field, and if the comparison is inconsistent, backfilling a failure result to excel and reading the next use case. If the alignment is consistent, the next step is continued.

(8) And entering a distributed database hbase corresponding to the target data packet to inquire the accessed data, if the data cannot be inquired, the data storage fails, and backfilling a failure result to the excel. If the data can be queried, the next step is continued.

(9) And comparing the values of the corresponding fields of the data in the hbase and the expected result, if the comparison is inconsistent, backfilling the failure result to excel, and if the comparison is continued, backfilling the success result to excel. Whether successful or not, after execution is complete, the next use case is read. And circulating until the use case is executed. Exemplary test results for the use cases to be tested are shown in Table 2 below after backfilling.

TABLE 2 example of execution results of use cases to be tested

EXAMPLE III

Fig. 3 is a schematic structural diagram of a data comparison apparatus according to a third embodiment of the present invention. The embodiment is applicable to how to compare data, and the device can be implemented in the form of hardware and/or software and can be integrated into an electronic device carrying a data comparison function. As shown in fig. 3, the data comparing apparatus of the present embodiment may include:

a target processing result obtaining module 310, configured to obtain a target processing result of to-be-tested data in a target data packet in a to-be-tested case at a target processing stage; the target processing stage comprises at least one of a server output stage, a distributed file backup stage and a distributed database storage stage;

the comparison result determining module 320 is configured to compare the target processing results based on a preset comparison rule to obtain a comparison result.

Optionally, the comparison result determining module 320 is specifically configured to:

determining whether the data volume of the output server increases in the server output phase;

and comparing the target processing result based on a preset comparison rule under the condition that the data volume of the output server is increased to obtain a comparison result.

determining whether data is inquired in a distributed file comparison path corresponding to the target data packet or not in a distributed file backup stage;

if yes, comparing the target processing result based on a preset comparison rule to obtain a comparison result.

determining whether storage data corresponding to the target data packet exists in a distributed database storage stage;

and for each field in the data to be tested, comparing whether the target processing result of the field is consistent with the expected processing result to obtain a comparison result.

Optionally, the apparatus further comprises:

and the data input determining module is used for determining whether the target data packet in the use case to be tested is transmitted to the input server according to the data volume in the input server.

Optionally, the apparatus further comprises:

and the test result backfilling module is used for backfilling the comparison result to a test result field corresponding to the target data packet in the to-be-tested case.

The data comparison device provided by the embodiment of the invention can execute the data comparison method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

In the technical scheme of the invention, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the related data to be tested and the like all accord with the regulations of related laws and regulations without violating the good customs of the public order.

Example four

Fig. 4 is a schematic structural diagram of an electronic device implementing the data comparison method according to the embodiment of the present invention. FIG. 4 shows a schematic block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. Processor 11 performs the various methods and processes described above, such as a data alignment method.

In some embodiments, the data alignment method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data alignment method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data alignment method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired result of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of data alignment, comprising:

2. The method of claim 1, wherein the comparing the target processing result based on a preset comparison rule to obtain a comparison result comprises:

determining whether an amount of data of an output server increases in the server output phase;

3. The method according to claim 1, wherein the comparing the target processing result based on a preset comparison rule to obtain a comparison result comprises:

determining whether data is inquired in a distributed file comparison path corresponding to the target data packet or not in the distributed file backup stage;

and if so, comparing the target processing result based on a preset comparison rule to obtain a comparison result.

4. The method of claim 1, wherein the comparing the target processing result based on a preset comparison rule to obtain a comparison result comprises:

determining whether stored data corresponding to a target data packet exists in the distributed database storage stage;

5. The method according to any one of claims 1 to 4, wherein the comparing the target processing results based on a preset comparison rule to obtain a comparison result comprises:

6. The method of claim 1, wherein before obtaining the target processing result of the target data to be tested in the target data packet in the use case to be tested in the target processing stage, the method further comprises:

and determining whether the target data packet in the use case to be tested is transmitted to the input server or not according to the data volume in the input server.

7. The method of claim 1, wherein after comparing the target processing results based on a preset comparison rule to obtain a comparison result, the method further comprises:

and backfilling the comparison result to a test result field corresponding to the target data packet in the to-be-tested case.

8. A data alignment apparatus, comprising:

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data alignment method of any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions for causing a processor to implement the data alignment method of any one of claims 1-7 when executed.