CN112732322A

CN112732322A - Data verification method and device, electronic equipment and computer storage medium

Info

Publication number: CN112732322A
Application number: CN202110353831.7A
Authority: CN
Inventors: 曹莉英
Original assignee: Shanghai Smk Network Technology Co ltd
Current assignee: Shanghai Smk Network Technology Co ltd
Priority date: 2021-04-01
Filing date: 2021-04-01
Publication date: 2021-04-30

Abstract

The application provides a data verification method, a data verification device, electronic equipment and a computer storage medium. The data verification method comprises the following steps: acquiring an interface data file; and in the case of using the continuous integration tool for timing triggering, verifying the interface data file by using a data analysis package tool. According to the embodiment of the application, the data verification efficiency can be improved.

Description

Data verification method and device, electronic equipment and computer storage medium

Technical Field

The present application belongs to the field of data verification technologies, and in particular, to a data verification method, an apparatus, an electronic device, and a computer storage medium.

Background

With the development of big data technology, the requirements for big data and application scenes are gradually enhanced, and the requirements for verification of generated data are increasingly increased. At present, manual verification is mainly used, but more and more scenes need to regularly generate related data every day, for the tasks, the manual verification is difficult to ensure the timely verification every day, and the verification has high repeatability and low efficiency.

Therefore, how to improve the data verification efficiency is a technical problem that needs to be solved urgently by those skilled in the art.

Disclosure of Invention

The embodiment of the application provides a data verification method, a data verification device, electronic equipment and a computer storage medium, and the data verification efficiency can be improved.

In a first aspect, an embodiment of the present application provides a data verification method, including: acquiring an interface data file; and in the case of using the continuous integration tool for timing triggering, verifying the interface data file by using a data analysis package tool.

Optionally, the continuous integration tool is Jenkins, and the data analysis package tool is pandas; in the case of using a persistent integration tool for timed triggering, verifying an interface data file using a data analysis package tool, comprising: under the condition of using Jenkins for timing triggering, compiling a plurality of test points by using pandas; the interface data file is verified using a plurality of test points.

Optionally, in the case of using the persistent integration tool for timing triggering, after the interface data file is verified by using the data analysis package tool, the method further includes: test reports are generated using pytest and all.

Optionally, the obtaining the interface data file includes: json data files are downloaded through the interface.

In a second aspect, an embodiment of the present application provides a data verification apparatus, including: the acquisition module is used for acquiring the interface data file; and the verification module is used for verifying the interface data file by using the data analysis package tool under the condition of using the continuous integration tool for timing trigger.

Optionally, the continuous integration tool is Jenkins, and the data analysis package tool is pandas; a verification module, comprising: the compiling unit is used for compiling a plurality of test points by utilizing the pandas under the condition of using Jenkins for timing triggering; and the verification unit is used for verifying the interface data file by utilizing the plurality of test points.

Optionally, the apparatus further comprises: and the generating module is used for generating a test report by using the pytest and the all.

Optionally, the obtaining module includes: and the acquisition unit is used for downloading the json data file through the interface.

In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a data validation method as illustrated in the first aspect.

In a fourth aspect, the present application provides a computer storage medium, on which computer program instructions are stored, and when executed by a processor, the computer program instructions implement the data verification method shown in the first aspect.

The data verification method, the data verification device, the electronic equipment and the computer storage medium can improve data verification efficiency. The data verification method comprises the steps of obtaining an interface data file; under the condition of using the continuous integration tool for timing triggering, the data analysis package tool is used for verifying the interface data file, so that the data verification efficiency can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a data verification method according to an embodiment of the present application.

Fig. 2 is a schematic flow chart of a data verification method according to another embodiment of the present application.

FIG. 3 is a schematic diagram of the process flow of python and pandas code provided by one embodiment of the present application.

Fig. 4 is a schematic structural diagram of a data verification apparatus according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to be limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

In order to solve the prior art problems, embodiments of the present application provide a data verification method, apparatus, device, and computer storage medium. The following first introduces a data verification method provided in the embodiments of the present application.

Fig. 1 shows a schematic flow chart of a data verification method according to an embodiment of the present application. As shown in fig. 1, the data verification method includes: s101, acquiring an interface data file.

In one embodiment, obtaining an interface data file comprises: and downloading a JavaScript Object Notation (json) data file through an interface.

And S102, under the condition of using the continuous integration tool for timing trigger, verifying the interface data file by using a data analysis package tool.

In one embodiment, the persistent integration tool is Jenkins and the data analysis package tool is pandas; in the case of using a persistent integration tool for timed triggering, verifying an interface data file using a data analysis package tool, comprising: under the condition of using Jenkins for timing triggering, compiling a plurality of test points by using pandas; the interface data file is verified using a plurality of test points.

In one embodiment, after validating the interface data file using the data analysis package tool with the persistent integration tool timing trigger, the method further comprises: test reports are generated using pytest and all. Among them, pytest is a test tool, and all is a test report tool.

The data verification method comprises the steps of firstly obtaining an interface data file, and then verifying the interface data file by using a data analysis package tool under the condition of using a continuous integration tool for timing triggering, so that the data verification efficiency can be improved.

The above technical solution is specifically described below with an example.

As shown in fig. 2, the data verification method includes: writing codes by using python and pandas, and verifying interface data; uploading the code to the gitlab; jenkins calls python scripts regularly; generating a test report and displaying the test report on Jenkins; and sending the mail after the execution is finished.

Among these, python and pandas code processing is detailed as follows: the realization idea is as follows: downloading a json data file through an interface; reading the file into pandas; and (5) verifying the file (designing a test case according to actual requirements, and performing data verification by using pandas).

As shown in fig. 3, the interface is called to obtain the return content, and whether the interface status code is 200 is determined, if the interface status code is not 200, the interface is failed to be called, and the verification is determined to be failed; if the interface status code is 200, judging whether a status =500 record exists in the returned content, and if the status =500 record exists in the returned content, failing to verify; if no status =500 record is returned, the data is read into the Pandas dataframe, and a result report is generated according to the verification result of each test case.

In the embodiment, on the basis of generating the single file data interface at a fixed time every day, a traditional method of only testing the function of the interface is not adopted in the process of automatically verifying the file data at a fixed time every day. Because the primary content is the interface that returns the data results, testing needs to be done for the data content in addition to the interface functional testing. The traditional method of only testing the interface function can not test the data content, but also needs a large amount of codes to realize, and consumes a large amount of time and energy.

In the embodiment, the interface data files generated regularly every day are triggered regularly by Jenkins, so that the verification period is ensured, data can be verified timely every day, data verification is performed by using pandas, the test points can be compiled quickly, and data verification of a plurality of verification points can be performed only by loading data once. And finally, a test report and related details can be generated by using the pytest and the all, so that the test results can be conveniently checked, shared and stored.

As shown in fig. 4, an embodiment of the present application further provides a data verification apparatus, including: an obtaining module 401, configured to obtain an interface data file; a verification module 402 for verifying the interface data file using the data analysis package tool with a persistent integration tool timed trigger.

In one embodiment, the persistent integration tool is Jenkins and the data analysis package tool is pandas; a verification module 402 comprising: the compiling unit is used for compiling a plurality of test points by utilizing the pandas under the condition of using Jenkins for timing triggering; and the verification unit is used for verifying the interface data file by utilizing the plurality of test points.

In one embodiment, the apparatus further comprises: and the generating module is used for generating a test report by using the pytest and the all.

In one embodiment, the obtaining module 401 includes: and the acquisition unit is used for downloading the json data file through the interface.

Each module/unit in the apparatus shown in fig. 4 has a function of implementing each step in fig. 1, and can achieve the corresponding technical effect, and for brevity, the description is not repeated here.

Fig. 5 shows a schematic structural diagram of an electronic device provided in an embodiment of the present application.

The electronic device may comprise a processor 501 and a memory 502 in which computer program instructions are stored.

Specifically, the processor 501 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 502 may include mass storage for data or instructions. By way of example, and not limitation, memory 502 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 502 may include removable or non-removable (or fixed) media, where appropriate. The memory 502 may be internal or external to the electronic device, where appropriate. In particular embodiments, memory 502 may be non-volatile solid-state memory.

In one example, the Memory 502 may be a Read Only Memory (ROM). In one example, the ROM may be mask programmed ROM, programmable ROM (prom), erasable prom (eprom), electrically erasable prom (eeprom), electrically rewritable ROM (earom), or flash memory, or a combination of two or more of these.

The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement any of the data verification methods in the above embodiments.

In one example, the electronic device can also include a communication interface 503 and a bus 510. As shown in fig. 5, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 to complete communication therebetween.

The communication interface 503 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiments of the present application.

Bus 510 comprises hardware, software, or both to couple the components of the online data traffic billing device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 510 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

In addition, the embodiment of the application can be realized by providing a computer storage medium. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any of the data validation methods in the above embodiments.

It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions or change the order between the steps after comprehending the spirit of the present application.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.

Aspects of the present application are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware for performing the specified functions or acts, or combinations of special purpose hardware and computer instructions.

As described above, only the specific embodiments of the present application are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims

1. A method of data verification, comprising:

acquiring an interface data file;

and in the case of using the continuous integration tool for timing triggering, verifying the interface data file by using a data analysis package tool.

2. The data validation method of claim 1, wherein the persistent integration tool is Jenkins and the data analysis package tool is pandas; the verifying the interface data file by using a data analysis package tool under the condition of using a continuous integration tool for timing triggering comprises the following steps:

under the condition of using Jenkins for timing triggering, compiling a plurality of test points by using pandas;

and verifying the interface data file by using a plurality of test points.

3. The data verification method of claim 1, wherein after verifying the interface data file using a data analysis package tool with persistent integration tool timing triggers, the method further comprises:

test reports are generated using pytest and all.

4. The data verification method of claim 1, wherein the obtaining the interface data file comprises:

json data files are downloaded through the interface.

5. A data verification apparatus, comprising:

the acquisition module is used for acquiring the interface data file;

and the verification module is used for verifying the interface data file by using a data analysis package tool under the condition of using the continuous integration tool for timing trigger.

6. The data validation device of claim 5, wherein the persistent integration tool is Jenkins and the data analysis package tool is pandas; the verification module comprises:

the compiling unit is used for compiling a plurality of test points by utilizing the pandas under the condition of using Jenkins for timing triggering;

and the verification unit is used for verifying the interface data file by utilizing a plurality of test points.

7. The data validation device of claim 5, wherein the device further comprises:

and the generating module is used for generating a test report by using the pytest and the all.

8. The data verification device of claim 5, wherein the obtaining module comprises:

and the acquisition unit is used for downloading the json data file through the interface.

9. An electronic device, characterized in that the electronic device comprises: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements a data validation method as claimed in any of claims 1-4.

10. A computer storage medium having computer program instructions stored thereon which, when executed by a processor, implement the data validation method of any of claims 1-4.