CN116627785A - Bypass test method, bypass test device, electronic equipment and readable storage medium - Google Patents
Bypass test method, bypass test device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN116627785A CN116627785A CN202310602120.8A CN202310602120A CN116627785A CN 116627785 A CN116627785 A CN 116627785A CN 202310602120 A CN202310602120 A CN 202310602120A CN 116627785 A CN116627785 A CN 116627785A
- Authority
- CN
- China
- Prior art keywords
- output result
- target task
- storage space
- result
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 80
- 238000010998 test method Methods 0.000 title claims description 14
- 238000000034 method Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 24
- 238000013524 data verification Methods 0.000 claims abstract description 12
- 238000012795 verification Methods 0.000 claims description 61
- 238000004590 computer program Methods 0.000 claims description 15
- 230000002159 abnormal effect Effects 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 4
- 230000008859 change Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 238000013508 migration Methods 0.000 description 8
- 230000005012 migration Effects 0.000 description 8
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000011990 functional testing Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the field of off-line testing, and discloses a bypass testing method, a bypass testing device, electronic equipment and a readable storage medium. The method comprises the following steps: and acquiring a target task to be tested, analyzing the target task, and obtaining a storage space of a first output result of the target task. And changing the storage space of the first output result of the target task into a preset storage space. And processing the target task at the cloud to obtain a first output result, and storing the first output result into a preset storage space. And performing data verification according to the first output result and a second output result of the target task to obtain a bypass test result, wherein the second output result is output by the target task before migrating to the cloud. The method can be applied to various scenes without separate programming for different calculation engines, has better accuracy and can also prevent pollution to online data.
Description
Technical Field
The present application relates to the field of offline testing, and in particular, to a bypass testing method, a bypass testing device, an electronic device, and a readable storage medium.
Background
With new market revolution caused by digital economy, the big data storage and calculation capacities are gradually expanded to cloud services. For manufacturers who move from the traditional mode to the cloud service mode or carry out service migration between different cloud service modes, the problems of data consistency, version compatibility and the like before and after task migration need to be solved.
At present, after a business is cloud in the big data offline field, the problem that a mode aiming at task verification of a big data offline computing engine is single in scene, poor in verification effect, likely to pollute data and the like exists.
Disclosure of Invention
Based on the technical problems, the application provides a bypass test method, a bypass test device, electronic equipment and a readable storage medium, which can solve the problems of single mode scene, poor verification effect and possible data pollution of the task verification of a big data offline computing engine.
In order to solve the technical problems, the application adopts the following technical scheme:
in a first aspect, the present application provides a bypass test method, comprising:
and acquiring a target task to be tested, analyzing the target task, and obtaining a storage space of a first output result of the target task. And changing the storage space of the first output result of the target task into a preset storage space. And processing the target task at the cloud to obtain a first output result, and storing the first output result into a preset storage space. And performing data verification according to the first output result and a second output result of the target task to obtain a bypass test result, wherein the second output result is output by the target task before migrating to the cloud.
In some embodiments, changing the storage space of the first output result of the target task to a preset storage space includes: the target task is intercepted at the first module. And changing the storage space of the first output result into a preset storage space according to the type of the first output result.
In some implementations, the first output result is of a type of table and the first module is a structured query language (Structured Query Language, SQL) total interface module.
Changing the storage space of the first output result to a preset storage space according to the type of the first output result comprises the following steps: and intercepting the target task at the SQL master interface module. And replacing the write-in table of the first output result with the test table.
In some embodiments, the first output result is of a type of path and the first module is a distributed file system (Hadoop Distributed File System, HDFS) write module.
Changing the storage space of the first output result to a preset storage space according to the type of the first output result comprises the following steps: and intercepting the target task at the HDFS writing module. And replacing the output path of the first output result with the test path.
In some embodiments, parsing the target task includes: and classifying and disassembling the SQL of the target task according to a preset rule, wherein the classification comprises writing operation or reading operation. Analyzing the disassembled SQL of each section, wherein the set of output results of the disassembled SQL of each section is used as a first output result.
In some embodiments, after analyzing the disassembled SQL of each segment, the method further comprises: and carrying out identity verification on the target task, and releasing the read-write permission of the target task when the verification is passed.
In some embodiments, performing data verification according to the first output result and the second output result of the target task to obtain a bypass test result, including: and when the types of the first output result and the second output result are tables, comparing the data consistency of the first output result and the second output result, and outputting a bypass test result.
In some embodiments, performing data verification according to the first output result and the second output result of the target task to obtain a bypass test result, including: when the types of the first output result and the second output result are paths, the data indicated by the first output result is mapped to a first verification table, and the data indicated by the second output result is mapped to a second verification table. And comparing the data consistency of the first verification table and the second verification table, and outputting a bypass test result.
In some embodiments, the method further comprises: and when the bypass test result is abnormal, analyzing inconsistent data to obtain an abnormal analysis result. And displaying an abnormal analysis result.
In the first aspect, the target task is processed in the cloud after being analyzed, the storage space of the first output result of the target task is changed into a preset storage space, and then the output result is stored in the preset storage space. Because only the storage position of the output result of the target task is changed, the method can be applied to various scenes without separately programming different calculation engines. Meanwhile, when verification is performed, verification is performed according to output results before and after cloud on a target task, and accuracy is better. Finally, the verification result is stored in a separate storage space, so that the generation and the test can be effectively isolated, and the pollution of on-line data is prevented.
In a second aspect, the present application provides a bypass test device comprising:
the processing module is used for acquiring a target task to be tested, analyzing the target task and obtaining a storage space of a first output result of the target task; the changing module is used for changing the storage space of the first output result of the target task into a preset storage space; the processing module is further configured to process the target task at a cloud end to obtain the first output result, and store the first output result into the preset storage space; and the verification module is used for carrying out data verification according to the first output result and a second output result of the target task to obtain a bypass test result, wherein the second output result is output before the target task is migrated to the cloud.
In some embodiments, the modification module is specifically configured to intercept the target task at the first module. And changing the storage space of the first output result into a preset storage space according to the type of the first output result.
In some embodiments, the first output result is of a table type and the first module is an SQL total interface module.
The change module is specifically used for intercepting the target task in the SQL master interface module. And replacing the write-in table of the first output result with the test table.
In some embodiments, the first output result is of a path type, and the first module is an HDFS write module.
The change module is specifically used for intercepting the target task in the HDFS writing module. And replacing the output path of the first output result with the test path.
In some embodiments, the processing module is specifically configured to classify and disassemble the SQL of the target task according to a preset rule, where the classification includes a write operation or a read operation. Analyzing the disassembled SQL of each section, wherein the set of output results of the disassembled SQL of each section is used as a first output result.
In some embodiments, the processing module is further configured to perform identity verification on the target task, and release the read-write permission of the target task when the verification passes.
In some embodiments, the verification module is specifically configured to compare data consistency of the first output result and the second output result when the types of the first output result and the second output result are tables, and output a bypass test result.
In some embodiments, the verification module is specifically configured to map, when the types of the first output result and the second output result are paths, data indicated by the first output result to a first verification table, and data indicated by the second output result to a second verification table. And comparing the data consistency of the first verification table and the second verification table, and outputting a bypass test result.
In some embodiments, the verification module is further configured to analyze inconsistent data when the bypass test result is abnormal, so as to obtain an abnormal analysis result. And displaying an abnormal analysis result.
In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores instructions executable by the processor; the processor is configured to execute the instructions to cause the electronic device to implement the method of the first aspect described above.
In a fourth aspect, the present application provides a readable storage medium storing a computer program; the computer program, when run in an electronic device, causes the electronic device to implement the method of the first aspect described above.
In a fifth aspect, the present application provides a computer program product for, when run on an electronic device, causing the electronic device to perform the steps of the method related to the first aspect described above, to carry out the method of the first aspect described above.
The advantages of the second aspect to the fifth aspect described above may refer to the first aspect and the second aspect, and are not repeated.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 shows an application scenario schematic of a bypass test method;
FIG. 2 is a schematic flow chart of a bypass testing method according to an embodiment of the application;
fig. 3 is a schematic diagram illustrating a change of a storage space of a first output result to a preset storage space in a bypass test method according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating a change of a storage space of a first output result to a preset storage space in a bypass test method according to another embodiment of the present application;
FIG. 5 shows a block diagram of a bypass test device according to an embodiment of the present application;
fig. 6 shows a schematic block diagram of an electronic device 600 that may be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 shows a schematic diagram of an application scenario of a bypass test method.
Referring to fig. 1, the cloud 12, the local device 11, and the bypass test device 13 are included. The local device 11, the cloud 12 and the bypass test device 13 are connected through a network.
The local device 11 runs a big data offline computing engine, which is a computing platform used for computing, statistical analysis and other operations in the big data field. For example, common computing engines include Spark (Spark) engines, hive (Hive) engines, or rapid (Presto) engines, among others. The local device may be a server, personal computer, cluster of computers, or the like, having data processing capabilities.
And when the computing engine is not migrated to the cloud end 12, executing the big data offline computing task through the big data offline computing engine in the local device 11. The cloud 12 refers to a software platform adopting an application virtualization technology (Application Virtualization), and the cloud 12 may include a cloud server, a cloud database, etc., where the cloud server may provide enough computing power, and the cloud database may provide enough storage space.
When the big data offline computing engine in the local device 11 is migrated to the cloud 12, test verification can be performed on the before and after migration through the bypass test device 13, so that the task can be correctly processed in the cloud 12. The bypass test device 13 may be an electronic device such as a personal computer, a notebook computer, a tablet computer, a personal digital assistant (Personal Digital Assistant), a custom terminal, or a smart phone.
After migrating the big data offline computing engine in the local device 11 to the cloud 12, the big data offline computing engine in the cloud 12 may execute the big data offline computing task.
When the big data offline computing engine in the local device 11 is migrated to the cloud 12, there may be problems of data consistency, version compatibility and the like before and after migration, so that a certain risk is generated when the business is cloud-on.
In the prior art, task verification before and after cloud loading is mainly performed in a mode of simulating flow test based on verification data, functional test based on specific service scene, forced backtracking test based on online service, single task step-by-step test based on SQL and the like. The problems of single scene, good verification effect, possible data pollution and the like exist in the modes.
For this, the present application provides a bypass test method, comprising: and acquiring a target task to be tested, analyzing the target task, and obtaining a storage space of a first output result of the target task. And changing the storage space of the first output result of the target task into a preset storage space. And processing the target task at the cloud to obtain a first output result, and storing the first output result into a preset storage space. And performing data verification according to the first output result and a second output result of the target task to obtain a bypass test result, wherein the second output result is output by the target task before migrating to the cloud.
According to the method, the target task is processed in the cloud after being analyzed, the storage space of the first output result of the target task is changed into the preset storage space, and then the output result is stored in the preset storage space. Because only the storage position of the output result of the target task is changed, the method can be applied to various scenes without separately programming different calculation engines. Meanwhile, when verification is performed, verification is performed according to output results before and after cloud on a target task, and accuracy is better. Finally, the verification result is stored in a separate storage space, so that the generation and the test can be effectively isolated, and the pollution of on-line data is prevented.
Fig. 2 is a schematic flow chart of a bypass testing method according to an embodiment of the application.
Referring to fig. 2, a bypass test method includes:
s201, acquiring a target task to be tested.
In some embodiments, the target task to be tested may be presented by a different computing engine. The target tasks can be classified according to different products in the output result.
As an example, if the output result of the target task is a table, and the processed data is stored in the table, hive may start the target task through a direct connection (beeline) and Java database connection (Java Database Connectivity, JDBC) mode; spark can start the target task through three interfaces of a Spark-sql mode, a Spark-jdbc mode and a Spark-shell mode; prest may then initiate the target task through the client.
In other examples, the client may be used to launch the target task Hive, spark, presto if the target task output is a path, i.e., the received data is stored in the path.
The target task may be described by an SQL statement, but is not limited thereto.
S202, analyzing the target task to obtain a storage space of a first output result of the target task.
In some embodiments, the SQL of the target task is classified and disassembled according to preset rules, wherein the classification comprises a writing operation or a reading operation. Analyzing the disassembled SQL of each section, wherein the set of output results of the disassembled SQL of each section is used as a first output result.
As an example, the preset rule may be to classify SQL according to operation type, for example, may be classified into a read operation and a write operation. The SQL break down may then be to break down complex query statements into simple queries. For example, multi-tier nested queries are broken down into single-tier queries.
In some embodiments, when analyzing the SQL after each segment is disassembled, analysis such as word sense analysis, grammar tree construction and the like can be performed on the SQL segment through another tool (Another Tool for Language Recognition, ANTLR) for language recognition. Obtaining a final output product set, e.g., an output product set of a read operation includes obtaining a corresponding table, field, and an output product set of a write operation includes obtaining a corresponding output table, path. The scheme of the application is mainly applied to writing operation.
And S203, authenticating the target task, and executing S204 after the authentication is passed.
In some embodiments, identity verification can be performed on the target task, and when verification passes, the read-write permission of the target task is released.
As an example, the authority interface of the data security authentication system can be called to transmit authentication information to the server to complete the authentication and authorization operation.
For example, when the data security authentication system recognizes the agreed identity of the bypass verification, all read rights are automatically verified and released. If the authentication is legal and the table, field and path comparison pass, the data authentication system returns a success flag 'success', and the task is allowed to be submitted to the cloud for execution. If one of the table, the field and the path comparison is abnormal in the authentication, a failure sign 'fail' is returned, and the task submission is refused.
S204, changing the storage space of the first output result of the target task into a preset storage space.
In some implementations, the target task may be intercepted at the first module. And changing the storage space of the first output result into a preset storage space according to the type of the first output result.
In some implementations, the first output result is of a type of table and the first module is a structured query language (Structured Query Language, SQL) total interface module.
Fig. 3 is a schematic diagram illustrating a change of a storage space of a first output result to a preset storage space in a bypass test method according to an embodiment of the application.
Referring to fig. 3, changing the storage space of the first output result to a preset storage space according to the type of the first output result includes: and intercepting the target task at the SQL master interface module. And replacing the write-in table of the first output result with the test table.
With reference to fig. 3, whichever computing engine initiates the target task in what manner, it is ultimately required to go through the SQL total interface module. Therefore, the first module is set as an SQL total interface module. The write actions may be intercepted at the interface location by the output artifact change module. The output product changing module is a virtual functional module and is composed of a plurality of codes and used for changing the output product.
For example, after the output product changing module detects an instruction for executing the target task, the writing table of the first output result of the target task is obtained, and the writing table is replaced by the test table. No processing is done for the query data type operation.
The write-in table is a table which is actually written in the cloud after the target task is executed. The test table may be a table stored in the bypass test equipment for testing. The test result is written into the test table, so that data separation can be realized, and online data is ensured not to be polluted.
Fig. 4 is a schematic diagram illustrating a change of a storage space of a first output result to a preset storage space in a bypass test method according to another embodiment of the present application.
Referring to fig. 4, the type of the first output result is a path, and the first module is an HDFS writing module.
Changing the storage space of the first output result to a preset storage space according to the type of the first output result comprises the following steps: and intercepting the target task at the HDFS writing module. And replacing the output path of the first output result with the test path.
With reference to fig. 4, whichever computing engine starts the target task eventually needs to go through the HDFS write module. Therefore, the first module is set as an HDFS write module. At this location, the incoming path may be intercepted.
For example, when it is monitored that the HDFS write module is started, the write path of the module is acquired, and the path is changed to the test path. No processing is done for the query data type operation.
The write-in path is an output path of data generated by the target task in the cloud, and the test path is a local storage path. And writing the data generated by the target task into the test path, so that data separation can be realized, and the online data is ensured not to be polluted.
When the write table or the output path is changed, if no abnormality occurs, a success flag "success" may be returned, and if any abnormality occurs, a failure flag "fail" may be returned. When a success flag is returned, the next step may be continued, and when a failure flag is returned, the time failure flag is revealed and the user is instructed to check.
S205, processing the target task at the cloud to obtain a first output result, and storing the first output result into a preset storage space.
In some embodiments, after receiving the success flag shown in S204, the target task may be transmitted to the cloud, executed by another resource coordinator (Yet Another Resource Negotiator, YARN) resource scheduler or cluster, and the first output result written to a corresponding test table or test path.
S206, obtaining a second output result of the target task.
In some embodiments, the second output result of the target task may be that the target task is executed in the local device shown in fig. 1 with the same data. The second output result stored after the execution in the local device can also be obtained, and the method for obtaining the second output result is not limited.
S207, performing data verification according to the first output result and the second output result of the target task to obtain a bypass test result.
In some embodiments, the consistency check can be performed by the data check module mainly on the first output result (table or path) and the second output result of the target task. The number, format, content, etc. of data before and after migration are compared. The bypass test results mainly comprise the results of the consistency check.
As an example, when the types of the first output result and the second output result are tables, the data consistency of the first output result and the second output result is compared, and the bypass test result is output.
In other examples, when the types of the first output result and the second output result are paths, the data of the target task execution output is directly stored in the test path. Since multiple types of data may be stored in the test path, the data may also be stored in different formats, for example, there may be file formats such as an optimized columnar record (Optimized Record Columnar, ORC) format, text (txt) format, binary format, and the like.
In addition, the number of data files output by the target task before and after migration and the data storage mode are not completely consistent, so that the data under the path is difficult to verify directly. The data under the path may first be mapped into a table by the Spark engine to facilitate data comparison. That is, the data indicated by the first output result is mapped to the first verification table, and the data indicated by the second output result is mapped to the second verification table. And comparing the data consistency of the first verification table and the second verification table, and outputting a bypass test result.
In some embodiments, the data consistency check may use two tables before and after the SQL association migration to compare the data, and if the output result after the comparison is "data consistency", directly record the comparison state.
As an example, when the types of the first output result and the second output result are tables, the test table of the first output result and the write table of the second output result can be compared through SQL association, whether the data in the two tables are consistent is verified, and if so, a bypass verification result of "data consistency" can be obtained.
When the types of the first output result and the second output result are paths, mapping the data in the paths into a first verification table and a second verification table through a Spark engine. And comparing the first verification table with the second verification table through SQL association, verifying whether the data in the two tables are consistent, and if so, obtaining a bypass verification result of 'data consistency'.
In still other embodiments, when the bypass test result is abnormal, that is, if there is inconsistent data when verifying whether the data in the two tables are consistent, the bypass test result is "data inconsistent".
In this case, inconsistent data may be analyzed to obtain an anomaly analysis result. And displaying an abnormal analysis result.
As an example, two tables before and after SQL association migration can be used first to uniformly store data which is not checked, inconsistent data is compared again, and a preliminary conclusion of data abnormality is given through a preset error data rule.
For example, if it is determined that the problem of double-precision floating point (double) precision results in inconsistent data contrast according to a preset error data rule, a "conclusion" is returned: the doule precision results ", and the returned conclusions are shown.
If it is determined that the association (unit) associated field problem causes inconsistent data comparison according to the preset error data rule, a conclusion is returned: the unit association field results in ", and the returned conclusion is presented.
If it cannot be determined what cause causes the data contrast to be inconsistent according to the preset error data rule, a conclusion is returned: the specific reason cannot be given temporarily, and a returned conclusion is displayed, and the analysis of the manual intervention is waited.
In an exemplary embodiment, the embodiment of the present disclosure further provides a bypass test device, which may be used to implement the bypass test method provided in the foregoing embodiment.
FIG. 5 shows a block diagram of a bypass test device according to an embodiment of the present application.
Referring to fig. 5, the bypass test device includes:
the processing module 51 is configured to obtain a target task to be tested, analyze the target task, and obtain a storage space of a first output result of the target task.
And a changing module 52, configured to change the storage space of the first output result of the target task to a preset storage space.
The processing module 51 is further configured to process the target task at the cloud end, obtain a first output result, and store the first output result in a preset storage space.
And the verification module 53 is configured to perform data verification according to the first output result and a second output result of the target task, so as to obtain a bypass test result, where the second output result is output by the target task before migrating to the cloud.
In some embodiments, the modification module 52 is specifically configured to intercept the target task at the first module. And changing the storage space of the first output result into a preset storage space according to the type of the first output result.
In some embodiments, the first output result is of a table type and the first module is an SQL total interface module.
The change module 52 is specifically configured to intercept the target task at the SQL total interface module. And replacing the write-in table of the first output result with the test table.
In some embodiments, the first output result is of a path type, and the first module is an HDFS write module.
The change module 52 is specifically configured to intercept the target task at the HDFS write module. And replacing the output path of the first output result with the test path.
In some embodiments, the processing module 51 is specifically configured to classify and disassemble the SQL of the target task according to a preset rule, where the classification includes a write operation or a read operation. Analyzing the disassembled SQL of each section, wherein the set of output results of the disassembled SQL of each section is used as a first output result.
In some embodiments, the processing module 51 is further configured to perform identity verification on the target task, and release the read-write permission of the target task when the verification passes.
In some embodiments, the verification module 53 is specifically configured to compare the data consistency of the first output result and the second output result when the types of the first output result and the second output result are tables, and output a bypass test result.
In some embodiments, the verification module 53 is specifically configured to map, when the types of the first output result and the second output result are paths, data indicated by the first output result to a first verification table, and data indicated by the second output result to a second verification table. And comparing the data consistency of the first verification table and the second verification table, and outputting a bypass test result.
In some embodiments, the verification module 53 is further configured to analyze inconsistent data to obtain an exception analysis result when the bypass test result is abnormal. And displaying an abnormal analysis result.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as provided in the above embodiments.
In an exemplary embodiment, the readable storage medium may be a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method according to the embodiments provided above.
In an exemplary embodiment, the computer program product comprises a computer program which, when executed by a processor, implements a method according to the embodiments provided above.
Fig. 6 illustrates a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure.
Electronic devices are intended to represent various forms of devices with computing capabilities, such as personal computers, laptop computers, tablet computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the first device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 may also be stored. The computing unit 601, ROM 602, and RAM 603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the respective methods and processes described above, such as a page rendering method. For example, in some embodiments, the page rendering method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 605. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the page rendering method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the bypass verification method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (12)
1. A bypass test method, comprising:
acquiring a target task to be tested, and analyzing the target task to obtain a storage space of a first output result of the target task;
changing the storage space of the first output result of the target task into a preset storage space;
processing the target task at a cloud to obtain the first output result, and storing the first output result into the preset storage space;
and performing data verification according to the first output result and a second output result of the target task to obtain a bypass test result, wherein the second output result is output by the target task before migrating to the cloud.
2. The method of claim 1, wherein changing the storage space of the first output result of the target task to a preset storage space comprises:
intercepting the target task at a first module;
and changing the storage space of the first output result into the preset storage space according to the type of the first output result.
3. The method of claim 2, wherein the type of the first output result is a table and the first module is a structured query language, SQL, total interface module;
the changing the storage space of the first output result to the preset storage space according to the type of the first output result includes:
intercepting the target task at an SQL master interface module;
and replacing the write-in table of the first output result with a test table.
4. The method of claim 2, wherein the type of the first output result is a path and the first module is a distributed file system HDFS write module;
the changing the storage space of the first output result to the preset storage space according to the type of the first output result includes:
intercepting the target task at an HDFS write module;
and replacing the output path of the first output result with a test path.
5. The method of any one of claims 1-4, wherein the parsing the target task comprises:
classifying and disassembling the Structured Query Language (SQL) of the target task according to a preset rule, wherein the classification comprises writing operation or reading operation;
analyzing each section of the disassembled SQL, wherein a set of output results of each section of the disassembled SQL is used as the first output result.
6. The method of claim 5, further comprising, after said analyzing each of said disassembled SQL segments:
and carrying out identity verification on the target task, and releasing the read-write permission of the target task when the verification is passed.
7. The method of claim 1, wherein the performing data verification according to the first output result and the second output result of the target task to obtain a bypass test result includes:
and when the types of the first output result and the second output result are tables, comparing the data consistency of the first output result and the second output result, and outputting a bypass test result.
8. The method of claim 1, wherein the performing data verification according to the first output result and the second output result of the target task to obtain a bypass test result includes:
when the types of the first output result and the second output result are paths, mapping the data indicated by the first output result into a first verification table, and mapping the data indicated by the second output result into a second verification table;
and comparing the data consistency of the first verification table and the second verification table, and outputting a bypass test result.
9. The method of claim 7 or 8, wherein the method further comprises:
when the bypass test result is abnormal, analyzing inconsistent data to obtain an abnormal analysis result;
and displaying the abnormal analysis result.
10. A bypass test device, comprising:
the processing module is used for acquiring a target task to be tested, analyzing the target task and obtaining a storage space of a first output result of the target task;
the changing module is used for changing the storage space of the first output result of the target task into a preset storage space;
the processing module is further configured to process the target task at a cloud end to obtain the first output result, and store the first output result into the preset storage space;
and the verification module is used for carrying out data verification according to the first output result and a second output result of the target task to obtain a bypass test result, wherein the second output result is output before the target task is migrated to the cloud.
11. An electronic device, the electronic device comprising: a processor and a memory, the memory having executable instructions stored therein;
the processor is configured to, when executing the instructions, cause the electronic device to implement the method of any one of claims 1-9.
12. A readable storage medium, wherein the readable storage medium stores a computer program;
the computer program, when run in an electronic device as provided in claim 11, causes the electronic device to implement the method as claimed in any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310602120.8A CN116627785A (en) | 2023-05-25 | 2023-05-25 | Bypass test method, bypass test device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310602120.8A CN116627785A (en) | 2023-05-25 | 2023-05-25 | Bypass test method, bypass test device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116627785A true CN116627785A (en) | 2023-08-22 |
Family
ID=87616663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310602120.8A Pending CN116627785A (en) | 2023-05-25 | 2023-05-25 | Bypass test method, bypass test device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116627785A (en) |
-
2023
- 2023-05-25 CN CN202310602120.8A patent/CN116627785A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209650B (en) | Data normalization and migration method and device, computer equipment and storage medium | |
| WO2021217846A1 (en) | Interface data processing method and apparatus, and computer device and storage medium | |
| US11036491B1 (en) | Identifying and resolving firmware component dependencies | |
| US11720825B2 (en) | Framework for multi-tenant data science experiments at-scale | |
| CN111177113A (en) | Data migration method and device, computer equipment and storage medium | |
| CN111597118A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN112363814A (en) | Task scheduling method and device, computer equipment and storage medium | |
| CN112528067A (en) | Graph database storage method, graph database reading method, graph database storage device, graph database reading device and graph database reading equipment | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN108959508B (en) | SQL data generation method and device | |
| US11593511B2 (en) | Dynamically identifying and redacting data from diagnostic operations via runtime monitoring of data sources | |
| CN114816772B (en) | Debugging method, debugging system and computing device for application running based on compatible layer | |
| US9201937B2 (en) | Rapid provisioning of information for business analytics | |
| US20200327364A1 (en) | Clustering simulation failures for triage and debugging | |
| CN113824717B (en) | Configuration checking method and device | |
| CN116627785A (en) | Bypass test method, bypass test device, electronic equipment and readable storage medium | |
| CN115203339A (en) | Multi-data source integration method and device, computer equipment and storage medium | |
| CN114003497A (en) | Method, device and equipment for testing service system and storage medium | |
| CN112559547A (en) | Method and device for determining consistency among multiple storage object copies | |
| CN113760765B (en) | Code testing method and device, electronic equipment and storage medium | |
| WO2019100646A1 (en) | Method, electronic device, and storage medium for splitting and calling rule engine file | |
| CN113239296B (en) | Method, device, equipment and medium for displaying small program | |
| US11430534B2 (en) | Method for testing storage systems, electronic device, and computer program product | |
| US20230106490A1 (en) | Automatically improving data annotations by processing annotation properties and user feedback | |
| CN112101571B (en) | Method and apparatus for monitoring operator compatibility under deep learning framework |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |