CN110764998B - Data comparison method, device, equipment and storage medium based on Django framework - Google Patents

Data comparison method, device, equipment and storage medium based on Django framework Download PDF

Info

Publication number
CN110764998B
CN110764998B CN201910842151.4A CN201910842151A CN110764998B CN 110764998 B CN110764998 B CN 110764998B CN 201910842151 A CN201910842151 A CN 201910842151A CN 110764998 B CN110764998 B CN 110764998B
Authority
CN
China
Prior art keywords
data
test
name
platform
scheduling task
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910842151.4A
Other languages
Chinese (zh)
Other versions
CN110764998A (en
Inventor
王彦葛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Health Insurance Company of China Ltd
Original Assignee
Ping An Health Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Health Insurance Company of China Ltd filed Critical Ping An Health Insurance Company of China Ltd
Priority to CN201910842151.4A priority Critical patent/CN110764998B/en
Publication of CN110764998A publication Critical patent/CN110764998A/en
Application granted granted Critical
Publication of CN110764998B publication Critical patent/CN110764998B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3692—Test management for test results analysis
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention discloses a data comparison method based on a Django framework, which comprises the following steps: acquiring a data test request input by a data test platform, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script; checking the data test request; if the verification is successful, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse; executing the oracle test script through an ORM technology, and inquiring expected data from a data table appointed by the database name and the table name information; when the scheduling task and the oracle test script are successfully executed, executing the hive test script to obtain data to be tested; and displaying the data to be tested, the expected data and the comparison result of the data to be tested and the expected data through the data test platform. The invention solves the problems of low efficiency, high error rate and long research and development period of the testing tool in the prior art when testing big data.

Description

Data comparison method, device, equipment and storage medium based on Django framework
Technical Field
The present invention relates to the field of information technologies, and in particular, to a data comparison method, apparatus, device, and storage medium based on a Django framework.
Background
With the development trend of enterprises and more valuable information brought by big data, the traditional functional test cannot cope with the test scene brought by massive data. Big data testing is still under exploration at present, and precursors are derived from BI testing, so that big data testers need more comprehensive testing technology to cope with the faced demands as technology is developed. The existing big data open source testing tools are fewer, for complex data aggregation logic, mainly gray boxes are used, the final purpose is achieved by manually comparing scripts or using the testing tools of the components to match with the auxiliary matching of the traditional testing tools, the efficiency is low, and the error rate of the testing personnel is high. The existing java frame-based data testing platform has high requirements on the code capacity of testers, and the research and development time of testing tools is long.
Therefore, finding a method for solving the current situations of low efficiency, high error rate and long research and development period of test tools in the prior art when testing big data is a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The embodiment of the invention provides a data comparison method, device, equipment and storage medium based on a Django framework, which are used for solving the problems of low efficiency, high error rate and long research and development period of a test tool in the prior art when big data test is carried out.
A data comparison method based on a Django framework, comprising:
when a data test is executed, a data test request input by a user through a data test platform constructed based on a Django framework is obtained, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script;
checking the data test request;
if the data test request is successfully checked, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relation mapping technology, and inquiring expected data from a data table appointed by the database name and the table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain data to be tested;
And comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
Optionally, the verifying the data test request includes:
when the data test request is submitted successfully, checking whether a database corresponding to a database name in the data test request exists or not and whether a data table corresponding to table name information exists or not;
if the database corresponding to the database name and the data table corresponding to the table name information exist, checking whether the scheduling task name in the scheduling task request exists on an azkaban scheduling platform or not;
if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform, establishing record information of the scheduling task request in a local database, and initializing an execution state.
Optionally, the method further comprises:
and if the database corresponding to the database name and/or the data table corresponding to the table name information in the data test request does not exist, generating first verification prompt information, and outputting the first verification prompt information through the data test platform.
Optionally, the method further comprises:
and if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, generating second checking prompt information, and outputting the second checking prompt information through the data testing platform.
Optionally, in the process of executing the scheduled task corresponding to the scheduled task name and executing the oracle test script, the method further includes:
monitoring the execution state of the scheduling task and the execution result of the oracle test script;
when the execution state of the scheduling task is in progress, if the oracle test script executes error reporting, terminating the execution process of the scheduling task, and outputting error reporting information through the data test platform;
and when the execution state of the scheduling task is failure, inquiring log information of a linux server, outputting the log information through the data testing platform, and terminating the execution process of the oracle test script.
Optionally, in the process of executing the hive test script, the method further includes:
and if the hive test script executes error reporting, outputting error reporting information through the data test platform.
A data alignment device based on a Django framework, comprising:
The request acquisition module is used for acquiring a data test request input by a user through a data test platform constructed based on a Django framework when the data test is executed, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script;
the verification module is used for verifying the data test request;
the task execution module is used for calling the azkaban scheduling platform to execute the scheduling task corresponding to the scheduling task name and storing the task execution result in the hive data warehouse if the data test request is successfully checked;
the first test module is used for executing the oracle test script through an object relation mapping technology and inquiring expected data from a data table appointed by the database name and table name information;
the second test module is used for executing the hive test script when the scheduled task is successfully executed and the oracle test script is successfully executed, inquiring the execution result of the scheduled task and obtaining data to be tested;
and the comparison module is used for comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data test platform.
Optionally, the apparatus further comprises:
the monitoring module is used for monitoring the execution state of the scheduling task and the execution result of the oracle test script;
the oracle error reporting module is used for terminating the execution process of the scheduling task and outputting error reporting information through the data testing platform if the oracle test script executes error reporting when the execution state of the scheduling task is in progress;
and the failure prompting module is used for inquiring the log information of the linux server when the execution state of the scheduling task is failure, outputting the log information through the data testing platform and terminating the execution process of the oracle testing script.
A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the data alignment method based on the Django framework described above when executing the computer program.
A computer readable storage medium storing a computer program which when executed by a processor implements the data alignment method based on the Django framework described above.
According to the embodiment of the invention, the data test platform is constructed based on the Django framework, and when the data test is executed, a data test request input by a user through the data test platform is obtained, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script; checking the data test request; if the data test request is successfully checked, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse; executing the oracle test script through an object relation mapping technology, and inquiring expected data from a data table appointed by the database name and the table name information; when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain data to be tested; comparing the data to be tested with expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data test platform, so that a tester does not need to call an azkaban scheduling platform and query a service database manually, the data test efficiency is effectively improved, the test time is reduced, useless repeated operation is avoided, and the process of data test is greatly simplified; and the data test platform is built based on the django framework, the development period is short, the framework language is popular and easy to understand, and the maintenance and secondary development are convenient.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data alignment method based on a Django framework according to an embodiment of the present invention;
FIG. 2 is a flowchart of step S102 in a data alignment method based on Django framework according to an embodiment of the present invention;
FIG. 3 is a flow chart of a listening mechanism in a Django frame-based data alignment method according to an embodiment of the present invention;
FIG. 4 is a schematic block diagram of a data alignment device based on the Django framework according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a computer device in accordance with an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The data comparison method based on the Django framework provided in this embodiment is described in detail below. In this embodiment, in order to solve the problems of low efficiency, high error rate and long research and development period of a test tool existing in the prior art when testing big data, based on the original purposes of simplicity, high efficiency and easy use, the data test platform convenient to manage is built by adopting the MTV mode of the data test platform and using the Django classical framework of Python.
The MTV mode of the Django framework is the Model-template-view mode. Wherein the Model layer (Model), also called data access layer, is responsible for mapping business objects to databases. The model is a unique, authoritative source of information that contains the necessary fields and behavior of the stored data. Django creates tables in the database based on the model, typically one model corresponds to a unique table in the database. The data test platform in the embodiment of the invention processes all services related to data, such as accessing data and verifying validity, through the Model layer (Model). The Templates layer (Templates), also called presentation layer, is responsible for presenting pages to the user (HTML), and HTML can be dynamically generated using template methods, the Templates containing the static part of the desired HTML output, the special syntax for inserting dynamic content. The data testing platform in the embodiment of the invention processes the business related to the performance through the template layer (Templates), such as displaying in pages or other types of documents. The view layers (Views), also referred to as business logic layers, include related logic that accesses the Model and invokes the appropriate Template, as a bridge between the Model and Template layers. The data testing platform in the embodiment of the invention processes logic of user request and response return through the view layer (Views) package, reads data wanted by the front end from the database to the front end, and writes the data wanted to be saved by the user to the database.
In the embodiment of the invention, the data testing platform is built based on the Django framework by adopting the Python language, and the Python language is easy to start and the grammar is simpler than Java and the like, so that the building period of the data testing platform is shorter, and a tester only needs to know the Python basic grammar, so that the data testing platform can be used, and the code capability requirement on the tester is reduced.
The data testing platform mainly adopts a BS architecture, namely a browser and server architecture mode. The browser is usually a browser on user equipment and is used for a user to access the data test platform and initiate a data test request to display a data test result; the background server is used for executing data test logic according to the data test request. In this embodiment, the data testing method based on the Django framework is applied to a server. In this embodiment, a server is taken as an execution body, and the data testing method based on the Django framework includes:
in step S101, when performing a data test, a data test request input by a user through a data test platform constructed based on the Django framework is acquired, where the data test request includes a scheduling task name, an oracle test script, a database name, table name information, and a hive test script.
When the data test is executed, a user can access the data test platform through a browser on user equipment and operate the data test platform, and the scheduling task name, the oracle test script, the database name, the table name information and the hive test script submit a form to generate a data test request. The user device then sends the data test request to the server. And the server receives the data test request and acquires the scheduling task name, the oracle test script, the database name, the table name information and the hive test script which are input by the user through the data test platform.
Here, the scheduling task name refers to a name of a scheduling task that has been configured in the azkaban scheduling platform. The scheduling task on the azkaban scheduling platform completes the data processing task by running an extract, transpose transform, load (ETL) script and a script operating on a table structure. In the embodiment of the invention, the scheduling task name is determined by an ETL development engineer and informs developers and testers of the data test platform. And maintaining the callable scheduling task of the data test platform into the task configuration table by a developer in advance through a mode of creating the task configuration table.
The oracle test script is script information written by a user and used for reading expected data from an oracle database. Here, the embodiment of the invention reads the expected result of the data test from the service database directly by way of the oracle test script. The service database includes multiple types, and the data table in the database also includes multiple types, so that the database and the data table to be read by the oracle test script need to be specified. The database name refers to a database name to be read by an oracle test script, such as a specific database information MIS library in the oracle. The table name information refers to the name of the data table to be read by the oracle test pin. According to the databases and data tables related to the data test, a plurality of databases and data tables can be configured in the data test platform in advance in a configuration file mode.
The hive test script is script information written by a user and used for reading data to be tested from a hive data warehouse and reading an execution result of a scheduling task. After receiving the scheduling task name, the azkaban scheduling platform runs an ETL script corresponding to the scheduling task name in the big data cluster, and then returns a data result processed by the ETL script, namely a task execution result, and stores the data result in a data warehouse. To obtain the task execution results, the hive test script needs to be read from the data warehouse.
In step S102, the data test request is verified.
Here, because the scheduling task name, the oracle test script, the database name, the table name information and the hive test script in the data test request are all input by the tester, errors are easy to occur, and in order to reduce the error rate of the data test, the efficiency and the effect of the data test are improved. After the data test request is obtained, the embodiment of the invention further executes verification on the data test request to judge whether the information contained in the data test request is correct or not. Optionally, as shown in fig. 2, the verifying the data test request in step S102 includes:
in step S201, when the data test request is submitted successfully, it is checked whether a database corresponding to a database name in the data test request exists or not and whether a data table corresponding to table name information exists or not.
As described above, in the embodiment of the present invention, a plurality of databases and data tables are configured in the data test platform in advance by means of configuration files, so as to obtain database configuration files and data table configuration files. After the data test request is obtained, matching a database name in the data test request with the database configuration file to verify the database name, and matching table name information in the data test request with the data table configuration file to verify the table name information to judge whether the database name and the table name information input by a user are valid or not.
In step S202, if the database corresponding to the database name and the data table corresponding to the table name information exist, it is checked whether the scheduling task name in the scheduling task request exists on the azkaban scheduling platform.
If the database name exists in the database configuration file, indicating that a database corresponding to the database name exists; if the table name information exists in the data table configuration file, the data table corresponding to the table name information is also indicated to exist, so that verification of database names and table name information in a data test request is completed, and at the moment, the proposed scheduling task is searched in an azkaban scheduling platform based on the scheduling task name, and particularly, the scheduling task name is matched with the task configuration table maintained in the data test platform.
In step S203, if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform, record information of the scheduling task request is established in a local database, and an execution state is initialized.
And if the scheduling task name exists in the task configuration table, indicating that the scheduling task corresponding to the scheduling task name exists, finishing verification of the scheduling task name in the data test request, and if the data test request is successfully verified, establishing record information of the scheduling task request in a local database and initializing an execution state. Then step S103 is performed.
Alternatively, as a preferred example of the present invention, the step S102 may further include:
in step S204, if the database corresponding to the database name and/or the data table corresponding to the table name information in the data test request does not exist, a first verification prompt message is generated, and the first verification prompt message is output through the data test platform.
If the database name does not exist in the database configuration file, indicating that the database corresponding to the database name does not exist, and possibly inputting an incorrect database name by a user; if the table name information does not exist in the data table configuration file, the fact that the data table corresponding to the table name information does not exist is also indicated, and a user may input wrong table name information. If a database corresponding to a database name and/or a data table corresponding to table name information in the data test request does not exist, the embodiment of the invention generates first verification prompt information according to the database name and/or the table name information to obtain error reporting information related to the database and the table name, returns the first verification prompt information to user equipment, and displays the first verification prompt information through the data test platform to prompt a user to correct.
Alternatively, as another preferred example of the present invention, the step S102 may further include:
in step S205, if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, a second verification prompt message is generated, and the second verification prompt message is output through the data testing platform.
Further, if the scheduled task name does not exist in the task configuration table, it indicates that the scheduled task corresponding to the scheduled task name does not exist, and the user may input an incorrect scheduled task name. According to the embodiment of the invention, the second verification prompt information is generated according to the dispatching task name, the error reporting information related to the dispatching task name is obtained, the second verification prompt information is returned to the user equipment, and the second verification prompt information is displayed through the data test platform so as to prompt the user to correct.
According to the embodiment of the invention, the scheduling task name, the database name and the table name information in the data test request are checked and the error reporting prompt is carried out through the steps S1021 to S1025, so that the error rate of the data test is reduced, and the efficiency and the effect of the data test are improved.
In step S103, if the data test request is successfully checked, the azkaban scheduling platform is called to execute the scheduling task corresponding to the scheduling task name, and the task execution result is stored in the hive data warehouse.
In the embodiment of the invention, when the database corresponding to the database name and the data table corresponding to the table name information in the data test request exist and the scheduling task exists on the azkaban scheduling platform, the data test request is successfully checked. At this time, an azkaban scheduling platform is called, the scheduling task name is sent to the azkaban scheduling platform, and a scheduling task corresponding to the scheduling task name is executed through the azkaban scheduling platform. And after receiving the scheduling task, the azkaban scheduling platform runs an ETL script corresponding to the scheduling task name in the big data cluster, returns a data result processed by the ETL script, namely a task execution result, and stores the data result in a data warehouse.
In step S104, the oracle test script is executed by an object relationship mapping technique, and desired data is queried from the data table specified by the database name and table name information.
Here, the expected data is target data in the data test as reference data in the data test. The embodiment of the invention executes the oracle test script through the object relation mapping technology, and queries expected data from the appointed service data table according to the database name and the table name information. The expected data is data on a service source head. The object relational mapping (Object Relational Mapping, ORM for short, or O/RM, or O/R mapping) is a procedural technique for implementing transformations between data of different types of systems in an object-oriented programming language. According to the embodiment of the invention, the ORACLE test script is executed by using the ORM technology of the Django, so that the data test platform can be connected with the database, the connection specification is unified, and the convenience is brought to a tester without learning other database access methods.
Optionally, as a preferred example of the present invention, in order to reduce the time for a tester to wait for task execution and script execution, embodiments of the present invention add a listening mechanism in scheduling task and oracle test script execution. As shown in fig. 3, in the process of executing the scheduled task corresponding to the scheduled task name in step S103 and executing the oracle test script in step S104, the method further includes:
in step S301, the execution state of the scheduling task and the execution result of the oracle test script are monitored.
Here, in order to improve the efficiency of the data test, the step 103 may be performed synchronously by executing the scheduling task through the azkaban scheduling platform and the step 104 by executing the oracle test script through the ORM. The embodiment of the invention can synchronously monitor the scheduling task and the oracle test script. For the scheduled task, an acquisition interface of the execution state of the scheduled task may be preset. And when the azkaban scheduling platform executes the scheduling task, the execution state of the scheduling task is acquired through the acquisition interface at regular time. For the oracle test script, SQLExreception thrown in the executing process of the script can be captured, and the executing result of the oracle test script is obtained according to the SQLExreception prompt.
Here, the execution state of the scheduled task includes in-progress, execution success, and execution failure. Wherein in progress indicates that the azkaban scheduling platform is executing the scheduling task; the successful execution indicates that the azkaban scheduling platform has successfully executed the scheduling task; the execution failure indicates that the azkaban scheduling platform has not completed the scheduled task but has stopped executing.
In step S302, when the execution state of the scheduled task is in progress, if the oracle test script executes error reporting, the execution process of the scheduled task is terminated, and error reporting information is output through the data test platform.
In the process of executing the scheduling task by the azkaban scheduling platform, if the SQLException thrown by the oracle test script is captured, the oracle test script executes the error reporting, the azkaban scheduling platform is directly stopped to execute the scheduling task, error reporting prompt information is generated according to the SQLException prompt, the error reporting prompt information is returned to the user equipment, and the error reporting prompt information is displayed by the data testing platform to prompt a user.
In step S303, when the execution state of the scheduling task is failure, the log information of the linux server is queried, and the log information is output through the data test platform, and the execution process of the oracle test script is terminated.
When the execution state of the scheduling task is failure, obtaining the reason of the failure execution by inquiring the log information of the linux server, generating error reporting prompt information, returning the error reporting prompt information to the user equipment, and displaying the error reporting prompt information through the data test platform to prompt a user. And simultaneously, terminating executing the oracle test script.
The embodiment of the invention effectively reduces the execution time of the test personnel waiting for the dispatching task or the test script through the monitoring mechanism. In the prior art, the remote fort machine automatically drops when not operated for a long time, and by the monitoring mechanism provided by the embodiment of the invention, a tester does not need to pay attention to the fort machine state at any time, so that the problem of fort machine disconnection is effectively avoided, and the testing efficiency is improved. When any one of the verification data test request, the monitoring of the execution state of the scheduling task and the monitoring of the execution result of the oracle test script fails, other processes are terminated, so that the resource waste is greatly reduced. When the regression test is executed after the old code is modified, the testers do not need to manually operate complicated and repeated test work, so that the regression test time is greatly reduced.
Optionally, as described above, the embodiment of the present invention establishes a record in the local database for the data test request, and initializes the execution status. If the step S103 is performed, when the scheduled task is executed successfully, the method may further include:
updating the execution state of the data test request to be executed.
In this embodiment of the present invention, by recording the execution state of the data test request, the operation record of the data test request is saved, and is inserted into the database as an operation log through the Model layer, thereby implementing the history archiving of each data test request on the data test platform, and facilitating the follow-up tracking and checking of the data test request.
In step S105, when the execution of the scheduling task is successful and the execution of the oracle test script is successful, the hive test script is executed, and the execution result of the scheduling task is queried to obtain data to be tested.
When the execution state of the scheduled task is that the execution is successful, the azkaban scheduling platform is indicated to have successfully executed the scheduled task and the task execution result is stored in a data warehouse; and when the oracle test script is successfully executed, indicating that the data test platform has queried expected data from the service database. Since the task execution results of azkaban after executing the scheduled task fall to the data warehouse, the task execution results need to be fetched from the data warehouse. In the embodiment of the invention, if and only if the execution of the scheduling task is successful and the execution of the oracle test script is successful, the data test platform submits a hive test script again, and the execution result of the scheduling task is inquired from the data warehouse to obtain the data to be tested. Therefore, the embodiment of the invention integrates the process of inquiring the data to be tested and the expected data through the data test platform, so that a tester only needs to fill the related information of the data test request on the data test platform interface of the user equipment and submit the related information to the background processing, the work that the tester manually executes and switches the system to be lower than the work of equivalent series efficiency is effectively reduced, the process of data test is greatly simplified, the test error rate is reduced, and the test efficiency is improved.
Alternatively, as a preferred example of the present invention, after step S105, the method further includes:
and if the hive test script executes error reporting, outputting error reporting information through the data test platform.
If the hive test script executes error reporting, error reporting prompt information is generated, the error reporting prompt information is returned to the user equipment, and the data test platform displays the error reporting prompt information to prompt a user, so that a tester does not need to pay attention to the state of the fort machine all the time through a monitoring mechanism, the problem of broken wires of the fort machine is effectively avoided, and the test efficiency is improved. And meanwhile, the data test is terminated, so that unnecessary resource waste is reduced. It should be appreciated that in embodiments of the present invention the tester accesses the database by logging into the fort machine.
In step S106, the data to be tested and the expected data are compared to obtain a comparison result, and the data to be tested, the expected data and the comparison result are output through the data test platform.
Herein, comparing the data to be tested with the expected data refers to comparing whether the data to be tested and the expected data are consistent or not so as to judge whether the result of the azkaban scheduling platform after executing the scheduling task reaches the expected requirement or not. Optionally, when the data testing platform is applied to the insurance service, according to the content of the scheduling task, the obtained data to be tested includes, but is not limited to, field information or table information such as a policy number, applicant information, insured person information, address information and the like. Step S106 is to directly compare the data to be tested with the expected data to determine whether the field information or the table information reaches the expected value. The comparison result includes the conclusion of whether the same is not limited to the difference point.
Illustratively, assume the Oracle test script is:
select polno from pol_main a where a.agent＝’test’；
inquiring field information 1, namely polno in the agent=test of the service database table pol_main;
the hive test script is:
select polno from pol_main_hive b where b.agent＝’test’；
the query gets field information 2, polno in agent=test in the data warehouse, of table pol_main_hive.
And comparing the field information 1 with the field information 2 to obtain a comparison result, sending the field information 1, the field information 2 and the comparison result to user equipment, and displaying the results to a user through the data testing platform.
In summary, the embodiment of the invention receives the data test request of the user through the data test platform, calls the azkaban scheduling platform to execute the scheduling task and uses the ORM technology to execute the oracle test script to obtain the expected data according to the data test request, then executes the hive test script to obtain the data to be tested after the azkaban scheduling platform executes the scheduling task, compares the data to be tested with the expected data, and outputs the data to be tested with the expected data and the comparison result, thereby eliminating the need of a tester to manually inquire the service database by the azkaban scheduling platform, effectively improving the data test efficiency, reducing the test time, avoiding useless repeated operation and greatly simplifying the data test process; and the data test platform is built based on the django framework, the development period is short, the framework language is popular and easy to understand, and the maintenance and secondary development are convenient.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
In an embodiment, a data comparison device based on a Django frame is provided, where the data comparison device based on the Django frame corresponds to the data comparison method based on the Django frame in the above embodiment one by one. As shown in fig. 4, the data comparison device based on the Django framework includes a request acquisition module 41, a verification module 42, a task execution module 43, a first test module 44, a second test module 45, and a comparison module 46. The functional modules are described in detail as follows:
the request acquisition module 41 is configured to acquire a data test request input by a user through a data test platform constructed based on a Django framework when performing a data test, where the data test request includes a scheduling task name, an oracle test script, a database name, table name information, and a hive test script;
a verification module 42, configured to verify the data test request;
the task execution module 43 is configured to call an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name and store a task execution result in a hive data warehouse if the data test request is successfully checked;
A first test module 44, configured to execute the oracle test script through an object relationship mapping technology, and query desired data from a data table specified by the database name and table name information;
the second test module 45 is configured to execute the hive test script when the scheduled task is executed successfully and the oracle test script is executed successfully, and query an execution result of the scheduled task to obtain data to be tested;
the comparison module 46 is configured to compare the data to be tested with expected data to obtain a comparison result, and output the data to be tested, the expected data and the comparison result through the data testing platform.
Optionally, the verification module 42 includes:
the first verification unit is used for verifying whether a database corresponding to a database name in the data test request exists or not and whether a data table corresponding to table name information exists or not when the data test request is submitted successfully;
the second checking unit is used for checking whether the scheduling task name in the scheduling task request exists on the azkaban scheduling platform or not if the database corresponding to the database name and the data table corresponding to the table name information exist;
and the recording unit is used for establishing the recording information of the scheduling task request in a local database and initializing an execution state if the scheduling task name in the scheduling task request exists on the azkaban scheduling platform.
Optionally, the verification module 42 further includes:
and the first verification prompting unit is used for generating first verification prompting information if the database corresponding to the database name and/or the data table corresponding to the table name information in the data test request does not exist, and outputting the first verification prompting information through the data test platform.
Optionally, the verification module 42 further includes:
the second checking prompt unit is used for generating second checking prompt information if the scheduling task name in the scheduling task request does not exist on the azkaban scheduling platform, and outputting the second checking prompt information through the data testing platform.
Optionally, the apparatus further comprises:
the monitoring module is used for monitoring the execution state of the scheduling task and the execution result of the oracle test script;
the oracle error reporting module is used for terminating the execution process of the scheduling task and outputting error reporting information through the data testing platform if the oracle test script executes error reporting when the execution state of the scheduling task is in progress;
and the failure prompting module is used for inquiring the log information of the linux server when the execution state of the scheduling task is failure, outputting the log information through the data testing platform and terminating the execution process of the oracle testing script.
Optionally, the apparatus further comprises:
and the hive error reporting module is used for outputting error reporting information through the data testing platform if the hive test script executes error reporting.
For specific limitation of the data comparison device based on the Django framework, reference may be made to the limitation of the data comparison method based on the Django framework hereinabove, and the description thereof will not be repeated here. The modules in the data comparison device based on the Django framework can be all or partially realized by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a data alignment method based on the Django framework.
In one embodiment, a computer device is provided comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of when executing the computer program:
when a data test is executed, a data test request input by a user through a data test platform constructed based on a Django framework is obtained, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script;
checking the data test request;
if the data test request is successfully checked, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relation mapping technology, and inquiring expected data from a data table appointed by the database name and the table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain data to be tested;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
when a data test is executed, a data test request input by a user through a data test platform constructed based on a Django framework is obtained, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script;
checking the data test request;
if the data test request is successfully checked, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relation mapping technology, and inquiring expected data from a data table appointed by the database name and the table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain data to be tested;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.

Claims (8)

1. A data comparison method based on a Django framework, comprising:
when a data test is executed, a data test request input by a user through a data test platform constructed based on a Django framework is obtained, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script;
Checking the data test request; the verifying the data test request includes:
when the data test request is submitted successfully, checking whether a database corresponding to a database name in the data test request exists or not and whether a data table corresponding to table name information exists or not;
if the database corresponding to the database name and the data table corresponding to the table name information exist, checking whether the scheduling task name in the data test request exists on an azkaban scheduling platform or not;
if the scheduling task name in the data test request exists on the azkaban scheduling platform, establishing record information of the data test request in a local database, and initializing an execution state;
if the data test request is successfully checked, calling an azkaban scheduling platform to execute a scheduling task corresponding to the scheduling task name, and storing a task execution result in a hive data warehouse;
executing the oracle test script through an object relation mapping technology, and inquiring expected data from a data table appointed by the database name and the table name information;
when the scheduling task is successfully executed and the oracle test script is successfully executed, executing the hive test script, and inquiring the execution result of the scheduling task to obtain data to be tested;
In the process of executing the scheduling task corresponding to the scheduling task name and executing the oracle test script, the method further comprises the following steps:
monitoring the execution state of the scheduling task and the execution result of the oracle test script;
when the execution state of the scheduling task is in progress, if the oracle test script executes error reporting, terminating the execution process of the scheduling task, and outputting error reporting information through the data test platform;
when the execution state of the scheduling task is failure, inquiring log information of a linux server, outputting the log information through the data testing platform, and terminating an execution process of an oracle test script;
and comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data testing platform.
2. The data comparison method based on the Django framework as recited in claim 1, further comprising:
and if the database corresponding to the database name and/or the data table corresponding to the table name information in the data test request does not exist, generating first verification prompt information, and outputting the first verification prompt information through the data test platform.
3. The data comparison method based on the Django framework as recited in claim 1, further comprising:
and if the scheduling task name in the data test request does not exist on the azkaban scheduling platform, generating second checking prompt information, and outputting the second checking prompt information through the data test platform.
4. The data comparison method based on the Django framework of claim 1, further comprising, in the process of executing the hive test script:
and if the hive test script executes error reporting, outputting error reporting information through the data test platform.
5. The data comparison device based on the Django framework is characterized by being applicable to the data comparison method based on the Django framework as claimed in any one of claims 1 to 4, and comprising the following steps:
the request acquisition module is used for acquiring a data test request input by a user through a data test platform constructed based on a Django framework when the data test is executed, wherein the data test request comprises a scheduling task name, an oracle test script, a database name, table name information and a hive test script;
the verification module is used for verifying the data test request;
The task execution module is used for calling the azkaban scheduling platform to execute the scheduling task corresponding to the scheduling task name and storing the task execution result in the hive data warehouse if the data test request is successfully checked;
the first test module is used for executing the oracle test script through an object relation mapping technology and inquiring expected data from a data table appointed by the database name and table name information;
the second test module is used for executing the hive test script when the scheduled task is successfully executed and the oracle test script is successfully executed, inquiring the execution result of the scheduled task and obtaining data to be tested;
and the comparison module is used for comparing the data to be tested with the expected data to obtain a comparison result, and outputting the data to be tested, the expected data and the comparison result through the data test platform.
6. The Django framework-based data alignment device of claim 5, further comprising:
the monitoring module is used for monitoring the execution state of the scheduling task and the execution result of the oracle test script;
the oracle error reporting module is used for terminating the execution process of the scheduling task and outputting error reporting information through the data testing platform if the oracle test script executes error reporting when the execution state of the scheduling task is in progress;
And the failure prompting module is used for inquiring the log information of the linux server when the execution state of the scheduling task is failure, outputting the log information through the data testing platform and terminating the execution process of the oracle testing script.
7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data alignment method based on the Django framework as claimed in any of claims 1 to 4 when the computer program is executed.
8. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the data alignment method based on the Django framework as claimed in any one of claims 1 to 4.
CN201910842151.4A 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework Active CN110764998B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910842151.4A CN110764998B (en) 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910842151.4A CN110764998B (en) 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework

Publications (2)

Publication Number Publication Date
CN110764998A CN110764998A (en) 2020-02-07
CN110764998B true CN110764998B (en) 2024-04-02

Family

ID=69330368

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910842151.4A Active CN110764998B (en) 2019-09-06 2019-09-06 Data comparison method, device, equipment and storage medium based on Django framework

Country Status (1)

Country Link
CN (1) CN110764998B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782489A (en) * 2020-07-16 2020-10-16 上海乾臻信息科技有限公司 Data table monitoring method and device
CN112231181B (en) * 2020-12-08 2021-03-16 平安科技（深圳）有限公司 Data abnormal update detection method and device, computer equipment and storage medium
CN112559331A (en) * 2020-12-10 2021-03-26 京东数字科技控股股份有限公司 Test method and device
CN112463836A (en) * 2020-12-15 2021-03-09 广州品唯软件有限公司 Data query method and device based on Django framework, computer equipment and medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104239148A (en) * 2013-06-06 2014-12-24 腾讯科技（深圳）有限公司 Distributed task scheduling method and device
WO2017063470A1 (en) * 2015-10-16 2017-04-20 北京源创云网络科技有限公司 Method, device and system for verifying consistency of electronic data and certificate verification platform
CN107577599A (en) * 2017-08-21 2018-01-12 同程网络科技股份有限公司 A kind of automatic interface testing method and platform based on custom script
CN108549606A (en) * 2018-04-16 2018-09-18 成都医云科技有限公司 interface test method and device
CN109101415A (en) * 2018-06-25 2018-12-28 平安科技（深圳）有限公司 Interface test method, system, equipment and the storage medium compared based on database
CN109308251A (en) * 2017-07-27 2019-02-05 阿里巴巴集团控股有限公司 The method of calibration and device of test data
CN109542765A (en) * 2018-10-18 2019-03-29 深圳壹账通智能科技有限公司 Database script verification method, device, computer equipment and storage medium
CN110069572A (en) * 2019-03-19 2019-07-30 深圳壹账通智能科技有限公司 HIVE method for scheduling task, device, equipment and storage medium based on big data platform

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104239148A (en) * 2013-06-06 2014-12-24 腾讯科技（深圳）有限公司 Distributed task scheduling method and device
WO2017063470A1 (en) * 2015-10-16 2017-04-20 北京源创云网络科技有限公司 Method, device and system for verifying consistency of electronic data and certificate verification platform
CN109308251A (en) * 2017-07-27 2019-02-05 阿里巴巴集团控股有限公司 The method of calibration and device of test data
CN107577599A (en) * 2017-08-21 2018-01-12 同程网络科技股份有限公司 A kind of automatic interface testing method and platform based on custom script
CN108549606A (en) * 2018-04-16 2018-09-18 成都医云科技有限公司 interface test method and device
CN109101415A (en) * 2018-06-25 2018-12-28 平安科技（深圳）有限公司 Interface test method, system, equipment and the storage medium compared based on database
CN109542765A (en) * 2018-10-18 2019-03-29 深圳壹账通智能科技有限公司 Database script verification method, device, computer equipment and storage medium
CN110069572A (en) * 2019-03-19 2019-07-30 深圳壹账通智能科技有限公司 HIVE method for scheduling task, device, equipment and storage medium based on big data platform

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于Celery和Django的分布式自动化测试系统设计;李一风;;信息技术(05);全文 *

Also Published As

Publication number Publication date
CN110764998A (en) 2020-02-07

Similar Documents

Publication Publication Date Title
CN110764998B (en) Data comparison method, device, equipment and storage medium based on Django framework
CN110069572B (en) HIVE task scheduling method, device, equipment and storage medium based on big data platform
CN110941546A (en) Automatic test method, device, equipment and storage medium for WEB page case
CN107241315B (en) Access method and device of bank gateway interface and computer readable storage medium
CN110716878B (en) Automatic interface testing method, device and system
CN110704312B (en) Method, device, computer equipment and storage medium for pressure test
CN113127347B (en) Interface testing method, device, equipment and readable storage medium
CN113238924B (en) Chaotic engineering realization method and system in distributed graph database system
CN112035363A (en) Automatic interface testing method and device
CN112650676A (en) Software testing method, device, equipment and storage medium
WO2019148657A1 (en) Method for testing associated environments, electronic device and computer readable storage medium
CN111694734A (en) Software interface checking method and device and computer equipment
CN117215910A (en) Interface test frame generation method, test method, electronic device and storage medium
WO2020253045A1 (en) Configured supplementary processing method and device for data of which forwarding has abnormality, and readable storage medium
WO2020232883A1 (en) Script defect scanning method and apparatus, computer device and storage medium
CN115048257A (en) System service function verification method and device, computer equipment and storage medium
CN111694684A (en) Abnormal construction method and device of storage equipment, electronic equipment and storage medium
CN111522881B (en) Service data processing method, device, server and storage medium
CN112379913A (en) Software optimization method, device, equipment and storage medium based on risk identification
CN116841653A (en) Execution method and device of operation and maintenance job, processor and electronic equipment
CN116150020A (en) Test case conversion method and device
CN115982049A (en) Abnormity detection method and device in performance test and computer equipment
CN111324668B (en) Database data synchronous processing method, device and storage medium
CN112486838A (en) Interface testing method and device, computer equipment and storage medium
CN112699023B (en) Project interface testing method and device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant