CN110764998A - Data comparison method, device and equipment based on Django framework and storage medium - Google Patents

Data comparison method, device and equipment based on Django framework and storage medium Download PDF

Info

Publication number
CN110764998A
CN110764998A CN201910842151.4A CN201910842151A CN110764998A CN 110764998 A CN110764998 A CN 110764998A CN 201910842151 A CN201910842151 A CN 201910842151A CN 110764998 A CN110764998 A CN 110764998A
Authority
CN
China
Prior art keywords
data
test
scheduling task
name
testing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910842151.4A
Other languages
Chinese (zh)
Other versions
CN110764998B (en
Inventor
王彦葛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Health Insurance Company of China Ltd
Original Assignee
Ping An Health Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Health Insurance Company of China Ltd filed Critical Ping An Health Insurance Company of China Ltd
Priority to CN201910842151.4A priority Critical patent/CN110764998B/en
Publication of CN110764998A publication Critical patent/CN110764998A/en
Application granted granted Critical
Publication of CN110764998B publication Critical patent/CN110764998B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3688Test management for test execution, e.g. scheduling of test suites
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3692Test management for test results analysis
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention discloses a data comparison method based on a Django framework, which comprises the following steps: acquiring a data test request input by a data test platform, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script; verifying the data test request; if the verification is successful, calling an azkaban scheduling platform to execute the scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse; executing the oracle test script through an ORM technology, and inquiring expected data from a data table specified by the database name and table name information; when the scheduling task and the oracle test script are successfully executed, executing the hive test script to obtain data to be tested; and displaying the data to be tested, the expected data and the comparison result of the data to be tested and the expected data through the data testing platform. The invention solves the problems of low efficiency, high error rate and long research and development period of a test tool in the prior art when big data test is carried out.

Description

Data comparison method, device and equipment based on Django framework and storage medium
Technical Field
The invention relates to the technical field of information, in particular to a data comparison method, device, equipment and storage medium based on a Django framework.
Background
With the development trend of enterprises and more valuable information brought by big data, the traditional functional test cannot cope with the test scene brought by massive data. Currently, big data testing is still in an exploration stage, the predecessor is from BI testing, and with the development of technology, big data testers need more comprehensive testing technology to meet the faced requirements. The existing big data open source testing tools are few, complex data aggregation logic is mainly achieved by taking an ash box as a main part, a final purpose is achieved by manually comparing scripts or using testing tools carried by components to be matched with traditional testing tools in an auxiliary mode, efficiency is low, and the error rate of testing personnel is high. The existing data testing platform based on the java framework has high requirements on the code capability of testers, and the research and development time of testing tools is long.
Therefore, it is a technical problem to be urgently needed by those skilled in the art to find a method for solving the current situations of low efficiency, high error rate and long development cycle of testing tools when performing big data testing in the prior art.
Disclosure of Invention
The embodiment of the invention provides a data comparison method, a data comparison device, data comparison equipment and a data comparison storage medium based on a Django framework, and aims to solve the problems of low efficiency, high error rate and long research and development period of a test tool in the prior art when a big data test is carried out.
A data comparison method based on a Django framework comprises the following steps:
when data testing is executed, acquiring a data testing request input by a user through a data testing platform constructed based on a Django framework, wherein the data testing request comprises a scheduling task name, an oracle testing script, a database name, table name information and a hive testing script;
verifying the data test request;
if the data test request is successfully verified, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relational mapping technology, and inquiring expected data from a data table specified by the database name and table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain to-be-tested data;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
Optionally, the verifying the data test request includes:
when the data test request is successfully submitted, checking whether a database corresponding to the database name in the data test request exists or not and whether a data table corresponding to the table name information exists or not;
if the database corresponding to the database name exists and the data tables corresponding to the table name information exist, checking whether the scheduling task name in the scheduling task request exists on the azkaban scheduling platform;
if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform, establishing the record information of the scheduling task request in a local database, and initializing an execution state.
Optionally, the method further comprises:
and if the database corresponding to the database name and/or the data table corresponding to the table name information in the data test request do not exist, generating first verification prompt information, and outputting the first verification prompt information through the data test platform.
Optionally, the method further comprises:
and if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, generating second checking prompt information, and outputting the second checking prompt information through the data testing platform.
Optionally, during the process of executing the scheduling task corresponding to the scheduling task name and executing the oracle test script, the method further includes:
monitoring the execution state of the scheduling task and the execution result of the oracle test script;
when the execution state of the scheduling task is in progress, if the oracle test script executes error reporting, terminating the execution process of the scheduling task, and outputting error reporting information through the data test platform;
and when the execution state of the scheduling task is failure, inquiring the log information of the linux server, outputting the log information through the data test platform, and terminating the execution process of the oracle test script.
Optionally, in the process of executing the hive test script, the method further includes:
and if the hive test script executes error reporting, outputting error reporting information through the data test platform.
A data comparison device based on a Django framework comprises:
the data testing system comprises a request acquisition module, a data testing module and a data testing module, wherein the request acquisition module is used for acquiring a data testing request input by a user through a data testing platform constructed based on a Django framework when data testing is performed, and the data testing request comprises a scheduling task name, an oracle testing script, a database name, table name information and a hive testing script;
the verification module is used for verifying the data test request;
the task execution module is used for calling an azkaban scheduling platform to execute the scheduling task corresponding to the scheduling task name if the data test request is verified successfully, and storing a task execution result in a hive data warehouse;
the first testing module is used for executing the oracle testing script through an object relational mapping technology and inquiring expected data from a data table specified by the database name and table name information;
the second testing module is used for executing the hive testing script and inquiring the execution result of the scheduling task to obtain to-be-tested data when the scheduling task is successfully executed and the oracle testing script is successfully executed;
and the comparison module is used for comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data test platform.
Optionally, the apparatus further comprises:
the monitoring module is used for monitoring the execution state of the scheduling task and the execution result of the oracle test script;
the oracle error reporting module is used for terminating the execution process of the scheduling task and outputting error reporting information through the data testing platform if the oracle testing script executes error reporting when the execution state of the scheduling task is in progress;
and the failure prompt module is used for inquiring the log information of the linux server when the execution state of the scheduling task is failure, outputting the log information through the data test platform and terminating the execution process of the oracle test script.
A computer device comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the computer program to realize the data comparison method based on the Django framework.
A computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the data comparison method based on the Django framework.
The embodiment of the invention constructs a data test platform based on a Django framework, and when executing data test, acquires a data test request input by a user through the data test platform, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script; verifying the data test request; if the data test request is successfully verified, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse; executing the oracle test script through an object relational mapping technology, and inquiring expected data from a data table specified by the database name and table name information; when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain to-be-tested data; comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform, so that a tester does not need to manually call an azkaban scheduling platform and query a service database, the data testing efficiency is effectively improved, the testing time is reduced, useless repeated operation is avoided, and the data testing process is greatly simplified; and the data test platform is established based on the django framework, the development period is short, the framework language is popular and easy to understand, and the maintenance and secondary development are convenient.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a flowchart of a data comparison method based on Django framework in an embodiment of the present invention;
FIG. 2 is a flowchart of step S102 of the data comparison method based on Django framework in an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a snooping mechanism in a data comparison method based on the Django framework according to an embodiment of the present invention;
FIG. 4 is a schematic block diagram of a data comparison apparatus based on Django framework according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a computer device according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data alignment method based on the Django framework provided in this embodiment is described in detail below. In this embodiment, in order to solve the problems of low efficiency, large error rate and long development cycle of test tools existing in the prior art when carrying out big data test, the development is carried out on the Django classic frame of Python based on the original purpose of simplicity, high efficiency and easy use, and a data test platform convenient to manage is built by adopting the MTV mode of the Django classic frame.
The MTV mode of the Django framework is a Model (Model) -template (Templates) -view (Views) mode. Wherein the Model layer (Model), also called data access layer, is responsible for mapping the business objects to the database. The model is a unique, authoritative source of information, containing the necessary fields and behaviors of the stored data. Django creates tables in the database according to models, and usually one model corresponds to a unique table in the database. The data testing platform in the embodiment of the present invention processes all services related to data, such as accessing data and verifying validity, through the Model layer (Model). The Templates layer (Templates), also called presentation layer, is responsible for presenting pages to users (HTML), and HTML can be dynamically generated using template methods, the Templates containing static parts of the required HTML output, special syntax for inserting dynamic content. The data testing platform in the embodiment of the present invention processes the performance-related services through the template layer (Templates), for example, displays the services in a page or other types of documents. The view layers (Views), also referred to as business logic layers, contain the associated logic to access the Model and invoke the appropriate Template as a bridge between the Model layer and the Template layer. The data test platform in the embodiment of the invention encapsulates the logic responsible for processing the user request and returning the response through the view layer (Views), reads the data required by the front end from the database to the front end, and writes the data required to be stored by the user to the database.
In the embodiment of the invention, the data test platform is constructed based on the Django framework by adopting the Python language, and the Python language is easy to operate and has simpler grammar compared with Java and the like, so that the construction period of the data test platform is shorter, and a tester can use the data test platform only by knowing the Python basic grammar, thereby reducing the requirement on the code capability of the tester.
The data testing platform mainly adopts a BS framework, namely a browser and server framework mode. The browser is generally a browser on user equipment and is used for accessing the data test platform, initiating a data test request and displaying a data test result by a user; the background server is used for executing data test logic according to the data test request. In this embodiment, the data testing method based on the Django framework is applied to a server. In this embodiment, a server is used as an execution subject, and the data testing method based on the Django framework includes:
in step S101, when a data test is performed, a data test request input by a user through a data test platform constructed based on a Django framework is obtained, where the data test request includes a scheduling task name, an oracle test script, a database name, table name information, and a hive test script.
When the data test is executed, a user can access the data test platform through a browser on user equipment, operate the data test platform, submit the scheduling task name, the oracle test script, the database name, the table name information and the hive test script to a form and generate a data test request. And then the user equipment sends the data test request to a server. And the server receives the data test request and acquires a scheduling task name, an oracle test script, a database name, table name information and a hive test script which are input by a user through the data test platform.
Here, the scheduling task name refers to a name of a scheduling task that has been configured in the azkaban scheduling platform. The scheduling task on the azkaban scheduling platform completes a data processing task by running an extract, a transpose, a load (ETL for short) script, and a script for table structure operation. In the embodiment of the invention, the scheduling task name is determined by an ETL development engineer and is informed to developers and testers of the data testing platform. And the developer maintains the scheduling tasks which can be called by the data test platform into the task configuration table in advance by a mode of creating the task configuration table.
The oracle test script is script information written by a user and used for reading expected data from an oracle database. Here, the embodiment of the present invention reads the expected result of the data test from the business database directly by means of the oracle test script. The service database comprises various types, and the data tables in the database also comprise various types, so that the database and the data tables to be read by the oracle test script need to be specified. The database name refers to a database name to be read by the oracle test script, such as a specific database information MIS library in the oracle. The table name information refers to a data table name to be read by an oracle test foot. According to the databases and data tables involved in the data test, a plurality of databases and data tables can be configured in the data test platform in advance in a configuration file mode.
The hive test script is script information which is written by a user and used for reading data to be tested from a hive data warehouse and is used for reading an execution result of a scheduling task. Here, after receiving the scheduling task name, the azkaban scheduling platform runs an ETL script corresponding to the scheduling task name in the big data cluster, and then returns a data result processed by the ETL script, that is, a task execution result, and stores the data result in a data warehouse. And reading the task execution result from the data warehouse through the hive test script.
In step S102, the data test request is verified.
Here, because the scheduling task name, the oracle test script, the database name, the table name information, and the hive test script in the data test request are all input by the tester, errors are easily caused, and in order to reduce the error rate of the data test, the efficiency and the effect of the data test are improved. After the data test request is acquired, the embodiment of the invention further performs verification on the data test request to judge whether the information contained in the data test request is correct. Optionally, as shown in fig. 2, the verifying the data test request in step S102 includes:
in step S201, when the data test request is successfully submitted, it is checked whether a database corresponding to the database name in the data test request exists or not and whether a data table corresponding to the table name information exists or not.
As described above, in the embodiment of the present invention, a plurality of databases and data tables are configured in the data test platform in advance in a configuration file manner, so as to obtain a database configuration file and a data table configuration file. After a data test request is obtained, matching a database name in the data test request with the database configuration file to verify the database name, matching table name information in the data test request with the data table configuration file to verify the table name information, and judging whether the database name and the table name information input by a user are valid.
In step S202, if the database corresponding to the database name and the data tables corresponding to the table name information all exist, it is checked whether the scheduling task name in the scheduling task request exists on the azkaban scheduling platform.
If the database name exists in the database configuration file, indicating that the database corresponding to the database name exists; if the table name information exists in the data table configuration file, the data table corresponding to the table name information is also indicated to exist, so that the verification of the database name and the table name information in the data test request is completed, at this time, the scheduling task to be tested is retrieved in the azkaban scheduling platform based on the scheduling task name, specifically, the scheduling task name is matched with the task configuration table maintained in the data test platform.
In step S203, if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform, record information of the scheduling task request is established in a local database, and an execution state is initialized.
If the scheduling task name exists in the task configuration table, the scheduling task corresponding to the scheduling task name exists, the checking of the scheduling task name in the data test request is completed, and if the data test request is successfully checked, the recording information of the scheduling task request is established in a local database, and the execution state is initialized. Then step S103 is performed.
Optionally, as a preferred example of the present invention, the step S102 may further include:
in step S204, if a database corresponding to the database name in the data test request and/or a data table corresponding to the table name information do not exist, first verification prompt information is generated, and the first verification prompt information is output through the data test platform.
If the database name does not exist in the database configuration file, indicating that the database corresponding to the database name does not exist, and possibly inputting an incorrect database name by a user; if the table name information does not exist in the data table configuration file, it also indicates that the data table corresponding to the table name information does not exist, and the user may input wrong table name information. If the data table corresponding to the database name and/or the table name information in the data test request does not exist, the embodiment of the invention generates first verification prompt information according to the database name and/or the table name information, obtains error reporting information related to the database and the table name, returns the first verification prompt information to the user equipment, and displays the first verification prompt information through the data test platform to prompt the user to correct.
Optionally, as another preferred example of the present invention, the step S102 may further include:
in step S205, if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, a second check prompt message is generated, and the second check prompt message is output through the data testing platform.
Further, if the scheduling task name does not exist in the task configuration table, it indicates that the scheduling task corresponding to the scheduling task name does not exist, and the user may input an incorrect scheduling task name. According to the embodiment of the invention, second check prompt information is generated according to the scheduling task name, error reporting information related to the scheduling task name is obtained, the second check prompt information is returned to the user equipment, and the second check prompt information is displayed through the data test platform to prompt a user to correct.
In the embodiment of the invention, the scheduling task name, the database name and the table name information in the data test request are verified and error prompt is reported through the steps S1021 to S1025, so that the error rate of data test is reduced, and the efficiency and the effect of data test are improved.
In step S103, if the data test request is successfully verified, the azkaban scheduling platform is called to execute the scheduling task corresponding to the scheduling task name, and the task execution result is stored in the hive data warehouse.
In the embodiment of the present invention, when both the database corresponding to the database name in the data test request and the data table corresponding to the table name information exist, and the scheduling task exists on the azkaban scheduling platform, the data test request is verified successfully. At this time, calling an azkaban scheduling platform, sending the scheduling task name to the azkaban scheduling platform, and executing the scheduling task corresponding to the scheduling task name through the azkaban scheduling platform. As described above, after receiving a scheduling task, the azkaban scheduling platform runs an ETL script corresponding to the name of the scheduling task in a big data cluster, and then returns a data result processed by the ETL script, that is, a task execution result, and stores the data result in a data warehouse.
In step S104, the oracle test script is executed through an object relational mapping technique, and expected data is queried from the data table specified by the database name and table name information.
Here, the expected data refers to target data in the data test as reference data in the data test. The embodiment of the invention executes the oracle test script through an object relational mapping technology, and inquires expected data from a specified service data table according to the name of the database and the name information of the table. The expected data is data at the source of the service. The Object Relational Mapping (ORM, or O/RM, or O/R Mapping) is a program technology for implementing conversion between data of different types of systems in an Object-oriented programming language. The embodiment of the invention uses the self-contained ORM technology of Django to execute the oracle test script, so that the data test platform can be connected with the database, the connection specification is unified, and the convenience is brought to testers without learning other database access methods.
Optionally, as a preferred example of the present invention, in order to reduce the time for the tester to wait for task execution and script execution, the embodiment of the present invention adds a listening mechanism in the process of scheduling task and oracle test script execution. As shown in fig. 3, in the process of executing the scheduling task corresponding to the scheduling task name in step S103 and executing the oracle test script in step S104, the method further includes:
in step S301, the execution state of the scheduling task and the execution result of the oracle test script are monitored.
Here, in order to improve the efficiency of data testing, the step 103 may be performed synchronously by executing the scheduling task through the azkaban scheduling platform and the step S104 may be performed by executing the oracle test script through the ORM. The embodiment of the invention can synchronously monitor the scheduling task and the oracle test script. For the scheduling task, an obtaining interface of the execution state of the scheduling task may be preset. And when the azkaban scheduling platform executes a scheduling task, acquiring the execution state of the scheduling task through the acquisition interface at regular time. For the oracle test script, the SQLException thrown out in the script execution process can be captured, and the execution result of the oracle test script is obtained according to the SQLException cue.
Here, the execution status of the scheduling task includes in-progress, execution success, and execution failure. Wherein the in-process representation shows that the azkaban scheduling platform is executing the scheduling task; the successful execution indicates that the azkaban scheduling platform has successfully executed the scheduling task; an execution failure indicates that the azkaban scheduling platform has not completed executing the scheduled task but has stopped executing.
In step S302, if the oracle test script executes an error report when the execution state of the scheduling task is in progress, the execution process of the scheduling task is terminated, and error report information is output through the data test platform.
In the process of executing the scheduling task by the azkaban scheduling platform, if an SQLException thrown by an oracle test script is captured and the oracle test script executes error reporting, the azkaban scheduling platform is directly terminated to execute the scheduling task, error reporting prompt information is generated according to an SQLException prompt, the error reporting prompt information is returned to user equipment, and the data test platform displays the error reporting prompt information to prompt a user.
In step S303, when the execution state of the scheduling task is a failure, the log information of the linux server is queried, the log information is output through the data testing platform, and the execution process of the oracle testing script is terminated.
When the execution state of the scheduling task is failure, obtaining the reason of the execution failure by inquiring the log information of the linux server, generating error-reporting prompt information, returning the error-reporting prompt information to the user equipment, and displaying the error-reporting prompt information through the data test platform to prompt the user. And simultaneously, terminating the execution of the oracle test script.
In this case, the embodiment of the present invention effectively reduces the execution time of the tester waiting for the scheduling task or the test script through the monitoring mechanism. According to the remote fort machine in the prior art, the remote fort machine can automatically fall off when not operated for a long time, and by means of the monitoring mechanism provided by the embodiment of the invention, a tester does not need to pay attention to the state of the fort machine all the time, so that the problem of disconnection of the fort machine is effectively avoided, and the testing efficiency is improved. When any one of the steps of verifying the data test request, monitoring the execution state of the scheduling task and monitoring the execution result of the oracle test script fails, other processes are terminated, so that the resource waste is greatly reduced. When the regression test is executed after the old code is modified, the tester does not need to manually operate complicated and repeated test work, and the regression test time is greatly reduced.
Optionally, as described above, in the embodiment of the present invention, a piece of record information is established in the local database for the data test request, and the execution state is initialized. If after step S103, when the scheduling task is successfully executed, the method may further include:
and updating the execution state of the data test request to be executed.
Here, in the embodiment of the present invention, by recording the execution state of the data test request, the operation record of the data test request is saved, and the operation record is inserted into the database as an operation log through the Model layer, so that history archiving of each data test request on the data test platform is realized, and it is beneficial to perform subsequent tracking and checking on the data test request.
In step S105, when the scheduling task is successfully executed and the oracle test script is successfully executed, the hive test script is executed, and the execution result of the scheduling task is queried to obtain the data to be tested.
When the execution state of the scheduling task is successful, indicating that the azkaban scheduling platform has successfully executed the scheduling task and storing the task execution result into the data warehouse; when the oracle test script is successfully executed, the data testing platform is indicated to inquire expected data from the business database. After the azkaban executes the scheduling task, the task execution result falls to the data warehouse, so that the task execution result needs to be taken out of the data warehouse. In the embodiment of the invention, if and only if the scheduling task is successfully executed and the oracle test script is successfully executed, the data test platform submits the hive test script, and the execution result of the scheduling task is inquired from the data warehouse to obtain the data to be tested. Therefore, the embodiment of the invention integrates the process of inquiring the data to be tested and the expected data through the data test platform, so that the tester only needs to fill the relevant information of the data test request on the data test platform interface of the user equipment and submit the relevant information to background processing, thereby effectively reducing the work of manual execution, system switching comparison and other series of low efficiency of the tester, greatly simplifying the process of data test, reducing the test error rate and improving the test efficiency.
Optionally, as a preferred example of the present invention, after step S105, the method further includes:
and if the hive test script executes error reporting, outputting error reporting information through the data test platform.
If hive test script carries out and reports errors, then generates and reports error prompt information, will report error prompt information and return to user equipment, through data test platform demonstrates report error prompt information is with the suggestion user to through monitoring mechanism, the tester need not to pay close attention to fortress machine state constantly, has avoided the problem of fortress machine broken string effectively, has improved the efficiency of test. And simultaneously, the data test is terminated to reduce the waste of unnecessary resources. It should be appreciated that in embodiments of the invention, the tester accesses the database through the login bastion machine.
In step S106, the data to be tested and the expected data are compared to obtain a comparison result, and the data to be tested, the expected data and the comparison result are output through the data testing platform.
Here, comparing the data to be tested with the expected data means comparing whether the two are consistent to judge whether the result of the azkaban scheduling platform after executing the scheduling task meets the expected requirement. Optionally, when the data testing platform is applied to insurance services, according to the content of the scheduling task, the obtained data to be tested includes, but is not limited to, field information such as a policy number, applicant information, insured person information, address information, and table information. Step S106 is to directly compare the data to be tested with the expected data to judge whether the field information or the table information reaches the expectation. The comparison result includes the conclusion of whether the comparison is identical or not and the difference point.
Illustratively, assume the Oracle test script is:
select polno from pol_main a where a.agent=’test’;
inquiring and obtaining the field information 1 of the agent of the business database table pol _ main, namely polno in test;
the hive test script is:
select polno from pol_main_hive b where b.agent=’test’;
the query obtains field information 2 of the agent of pol _ main _ live in the data warehouse, i.e. the polno in the test.
And comparing the field information 1 with the field information 2 to obtain a comparison result, sending the field information 1, the field information 2 and the comparison result to user equipment, and displaying the result to a user through the data test platform.
In summary, in the embodiment of the present invention, a data test platform is constructed, a data test request of a user is received through the data test platform, an azkaban scheduling platform is called to execute a scheduling task according to the data test request, an ORM test script is executed by using an ORM technology to obtain expected data, then a hive test script is executed after the azkaban scheduling platform executes the scheduling task to obtain to-be-tested data, the to-be-tested data and the expected data are compared, and the to-be-tested data, the expected data and the comparison result are output, so that a tester does not need to manually operate the azkaban scheduling platform and query a service database, data test efficiency is effectively improved, test time is reduced, useless repeated operation is avoided, and a data test process is greatly simplified; and the data test platform is established based on the django framework, the development period is short, the framework language is popular and easy to understand, and the maintenance and secondary development are convenient.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, a data comparison device based on a Django frame is provided, and the data comparison device based on the Django frame corresponds to the data comparison method based on the Django frame in the embodiment one to one. As shown in fig. 4, the data comparison apparatus based on the Django framework includes a request obtaining module 41, a verifying module 42, a task executing module 43, a first testing module 44, a second testing module 45, and a comparison module 46. The functional modules are explained in detail as follows:
the request obtaining module 41 is configured to obtain a data test request input by a user through a data test platform constructed based on a Django framework when a data test is performed, where the data test request includes a scheduling task name, an oracle test script, a database name, table name information, and a hive test script;
the checking module 42 is used for checking the data test request;
the task execution module 43 is configured to, if the data test request is successfully verified, invoke an azkaban scheduling platform to execute the scheduling task corresponding to the scheduling task name, and store a task execution result in the hive data warehouse;
the first testing module 44 is configured to execute the oracle testing script through an object relational mapping technique, and query expected data from a data table specified by the database name and table name information;
the second testing module 45 is configured to execute the hive testing script and query an execution result of the scheduling task to obtain to-be-tested data when the scheduling task is successfully executed and the oracle testing script is successfully executed;
and a comparison module 46, configured to compare the to-be-tested data with the expected data to obtain a comparison result, and output the to-be-tested data, the expected data, and the comparison result through the data testing platform.
Optionally, the verification module 42 includes:
the first checking unit is used for checking whether a database corresponding to the database name in the data test request exists or not and whether a data table corresponding to the table name information exists or not when the data test request is successfully submitted;
a second checking unit, configured to check whether a scheduling task name in the scheduling task request exists on an azkaban scheduling platform if both a database corresponding to the database name and a data table corresponding to the table name information exist;
and the recording unit is used for establishing the recording information of the scheduling task request in a local database and initializing the execution state if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform.
Optionally, the verification module 42 further includes:
and the first verification prompt unit is used for generating first verification prompt information if a database corresponding to the database name in the data test request and/or a data table corresponding to the table name information do not exist, and outputting the first verification prompt information through the data test platform.
Optionally, the verification module 42 further includes:
and the second check prompting unit is used for generating second check prompting information if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, and outputting the second check prompting information through the data testing platform.
Optionally, the apparatus further comprises:
the monitoring module is used for monitoring the execution state of the scheduling task and the execution result of the oracle test script;
the oracle error reporting module is used for terminating the execution process of the scheduling task and outputting error reporting information through the data testing platform if the oracle testing script executes error reporting when the execution state of the scheduling task is in progress;
and the failure prompt module is used for inquiring the log information of the linux server when the execution state of the scheduling task is failure, outputting the log information through the data test platform and terminating the execution process of the oracle test script.
Optionally, the apparatus further comprises:
and the hive error reporting module is used for outputting error reporting information through the data test platform if the hive test script executes error reporting.
For specific limitations of the data alignment apparatus based on the Django framework, reference may be made to the above limitations of the data alignment method based on the Django framework, and details are not repeated here. All or part of each module in the data comparison device based on the Django framework can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to realize a data comparison method based on the Django framework.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
when data testing is executed, acquiring a data testing request input by a user through a data testing platform constructed based on a Django framework, wherein the data testing request comprises a scheduling task name, an oracle testing script, a database name, table name information and a hive testing script;
verifying the data test request;
if the data test request is successfully verified, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relational mapping technology, and inquiring expected data from a data table specified by the database name and table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain to-be-tested data;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
when data testing is executed, acquiring a data testing request input by a user through a data testing platform constructed based on a Django framework, wherein the data testing request comprises a scheduling task name, an oracle testing script, a database name, table name information and a hive testing script;
verifying the data test request;
if the data test request is successfully verified, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relational mapping technology, and inquiring expected data from a data table specified by the database name and table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain to-be-tested data;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, databases, or other media used in embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.

Claims (10)

1. A data comparison method based on Django framework is characterized by comprising the following steps:
when data testing is executed, acquiring a data testing request input by a user through a data testing platform constructed based on a Django framework, wherein the data testing request comprises a scheduling task name, an oracle testing script, a database name, table name information and a hive testing script;
verifying the data test request;
if the data test request is successfully verified, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relational mapping technology, and inquiring expected data from a data table specified by the database name and table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain to-be-tested data;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
2. The Django framework-based data comparison method as claimed in claim 1, wherein the verifying the data test request comprises:
when the data test request is successfully submitted, checking whether a database corresponding to the database name in the data test request exists or not and whether a data table corresponding to the table name information exists or not;
if the database corresponding to the database name exists and the data tables corresponding to the table name information exist, checking whether the scheduling task name in the scheduling task request exists on the azkaban scheduling platform;
if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform, establishing the record information of the scheduling task request in a local database, and initializing an execution state.
3. The Django-framework-based data comparison method as claimed in claim 2, further comprising:
and if the database corresponding to the database name and/or the data table corresponding to the table name information in the data test request do not exist, generating first verification prompt information, and outputting the first verification prompt information through the data test platform.
4. The Django framework-based data comparison method of claim 2, further comprising:
and if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, generating second checking prompt information, and outputting the second checking prompt information through the data testing platform.
5. The Django framework-based data comparison method as claimed in claim 1, wherein in the process of executing the scheduling task corresponding to the scheduling task name and executing the oracle test script, the method further comprises:
monitoring the execution state of the scheduling task and the execution result of the oracle test script;
when the execution state of the scheduling task is in progress, if the oracle test script executes error reporting, terminating the execution process of the scheduling task, and outputting error reporting information through the data test platform;
and when the execution state of the scheduling task is failure, inquiring the log information of the linux server, outputting the log information through the data test platform, and terminating the execution process of the oracle test script.
6. The Django framework-based data comparison method as claimed in claim 1, wherein in the process of executing the hive test script, the method further comprises:
and if the hive test script executes error reporting, outputting error reporting information through the data test platform.
7. A data comparison device based on Django framework is characterized by comprising:
the data testing system comprises a request acquisition module, a data testing module and a data testing module, wherein the request acquisition module is used for acquiring a data testing request input by a user through a data testing platform constructed based on a Django framework when data testing is performed, and the data testing request comprises a scheduling task name, an oracle testing script, a database name, table name information and a hive testing script;
the verification module is used for verifying the data test request;
the task execution module is used for calling an azkaban scheduling platform to execute the scheduling task corresponding to the scheduling task name if the data test request is verified successfully, and storing a task execution result in a hive data warehouse;
the first testing module is used for executing the oracle testing script through an object relational mapping technology and inquiring expected data from a data table specified by the database name and table name information;
the second testing module is used for executing the hive testing script and inquiring the execution result of the scheduling task to obtain to-be-tested data when the scheduling task is successfully executed and the oracle testing script is successfully executed;
and the comparison module is used for comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data test platform.
8. The Django framework-based data alignment apparatus of claim 7, further comprising:
the monitoring module is used for monitoring the execution state of the scheduling task and the execution result of the oracle test script;
the oracle error reporting module is used for terminating the execution process of the scheduling task and outputting error reporting information through the data testing platform if the oracle testing script executes error reporting when the execution state of the scheduling task is in progress;
and the failure prompt module is used for inquiring the log information of the linux server when the execution state of the scheduling task is failure, outputting the log information through the data test platform and terminating the execution process of the oracle test script.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the Django framework-based data alignment method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the Django framework-based data comparison method according to any one of claims 1 to 6.
CN201910842151.4A 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework Active CN110764998B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910842151.4A CN110764998B (en) 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910842151.4A CN110764998B (en) 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework

Publications (2)

Publication Number Publication Date
CN110764998A true CN110764998A (en) 2020-02-07
CN110764998B CN110764998B (en) 2024-04-02

Family

ID=69330368

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910842151.4A Active CN110764998B (en) 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework

Country Status (1)

Country Link
CN (1) CN110764998B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782489A (en) * 2020-07-16 2020-10-16 上海乾臻信息科技有限公司 Data table monitoring method and device
CN112231181A (en) * 2020-12-08 2021-01-15 平安科技(深圳)有限公司 Data abnormal update detection method and device, computer equipment and storage medium
CN112463836A (en) * 2020-12-15 2021-03-09 广州品唯软件有限公司 Data query method and device based on Django framework, computer equipment and medium
CN112527661A (en) * 2020-12-17 2021-03-19 平安消费金融有限公司 Data script verification method and device and computer equipment
CN112559331A (en) * 2020-12-10 2021-03-26 京东数字科技控股股份有限公司 Test method and device
CN112527661B (en) * 2020-12-17 2024-04-26 平安消费金融有限公司 Verification method and device of data script and computer equipment

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104239148A (en) * 2013-06-06 2014-12-24 腾讯科技(深圳)有限公司 Distributed task scheduling method and device
WO2017063470A1 (en) * 2015-10-16 2017-04-20 北京源创云网络科技有限公司 Method, device and system for verifying consistency of electronic data and certificate verification platform
CN107577599A (en) * 2017-08-21 2018-01-12 同程网络科技股份有限公司 A kind of automatic interface testing method and platform based on custom script
CN108549606A (en) * 2018-04-16 2018-09-18 成都医云科技有限公司 interface test method and device
CN109101415A (en) * 2018-06-25 2018-12-28 平安科技(深圳)有限公司 Interface test method, system, equipment and the storage medium compared based on database
CN109308251A (en) * 2017-07-27 2019-02-05 阿里巴巴集团控股有限公司 The method of calibration and device of test data
CN109542765A (en) * 2018-10-18 2019-03-29 深圳壹账通智能科技有限公司 Database script verification method, device, computer equipment and storage medium
CN110069572A (en) * 2019-03-19 2019-07-30 深圳壹账通智能科技有限公司 HIVE method for scheduling task, device, equipment and storage medium based on big data platform

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104239148A (en) * 2013-06-06 2014-12-24 腾讯科技(深圳)有限公司 Distributed task scheduling method and device
WO2017063470A1 (en) * 2015-10-16 2017-04-20 北京源创云网络科技有限公司 Method, device and system for verifying consistency of electronic data and certificate verification platform
CN109308251A (en) * 2017-07-27 2019-02-05 阿里巴巴集团控股有限公司 The method of calibration and device of test data
CN107577599A (en) * 2017-08-21 2018-01-12 同程网络科技股份有限公司 A kind of automatic interface testing method and platform based on custom script
CN108549606A (en) * 2018-04-16 2018-09-18 成都医云科技有限公司 interface test method and device
CN109101415A (en) * 2018-06-25 2018-12-28 平安科技(深圳)有限公司 Interface test method, system, equipment and the storage medium compared based on database
CN109542765A (en) * 2018-10-18 2019-03-29 深圳壹账通智能科技有限公司 Database script verification method, device, computer equipment and storage medium
CN110069572A (en) * 2019-03-19 2019-07-30 深圳壹账通智能科技有限公司 HIVE method for scheduling task, device, equipment and storage medium based on big data platform

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李一风;: "基于Celery和Django的分布式自动化测试系统设计", 信息技术, no. 05 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782489A (en) * 2020-07-16 2020-10-16 上海乾臻信息科技有限公司 Data table monitoring method and device
CN112231181A (en) * 2020-12-08 2021-01-15 平安科技(深圳)有限公司 Data abnormal update detection method and device, computer equipment and storage medium
CN112559331A (en) * 2020-12-10 2021-03-26 京东数字科技控股股份有限公司 Test method and device
CN112463836A (en) * 2020-12-15 2021-03-09 广州品唯软件有限公司 Data query method and device based on Django framework, computer equipment and medium
CN112527661A (en) * 2020-12-17 2021-03-19 平安消费金融有限公司 Data script verification method and device and computer equipment
CN112527661B (en) * 2020-12-17 2024-04-26 平安消费金融有限公司 Verification method and device of data script and computer equipment

Also Published As

Publication number Publication date
CN110764998B (en) 2024-04-02

Similar Documents

Publication Publication Date Title
CN110764998A (en) Data comparison method, device and equipment based on Django framework and storage medium
CN110069572B (en) HIVE task scheduling method, device, equipment and storage medium based on big data platform
CN110941546A (en) Automatic test method, device, equipment and storage medium for WEB page case
CN110716870B (en) Automatic service testing method and device
CN108446223B (en) Business scene automatic testing method based on dynamic compilation
JP2010231782A (en) Method and system for function automation
CN105138461A (en) Interface testing method and device for application program
CN109902017A (en) A kind of method and terminal based on RobotFramework test Dubbo interface
CN106991046B (en) Application testing method and device
CN113127347B (en) Interface testing method, device, equipment and readable storage medium
US9892019B2 (en) Use case driven stepping component automation framework
CN110716878B (en) Automatic interface testing method, device and system
CN111522728A (en) Method for generating automatic test case, electronic device and readable storage medium
CN113010413B (en) Automatic interface testing method and device
US20190243750A1 (en) Test reuse exchange and automation system and method
CN109933521A (en) Automated testing method, device, computer equipment and storage medium based on BDD
CN112286806A (en) Automatic testing method and device, storage medium and electronic equipment
CN110503544A (en) Collage-credit data reports and submits method, apparatus, system, computer equipment and storage medium
CN112035363A (en) Automatic interface testing method and device
CN112597014A (en) Automatic testing method, device, medium and electronic equipment based on data driving
US20210048999A1 (en) Automated generation of status chains for software updates
CN110597714A (en) Kafka message testing method and device, computer equipment and storage medium
CN113190443A (en) Test method, test device, computer equipment and storage medium
CN111694734A (en) Software interface checking method and device and computer equipment
US9612944B2 (en) Method and system for verifying scenario based test selection, execution and reporting

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant