CN113722223A - Automatic testing method based on neural network - Google Patents

Abstract

The invention provides an automatic testing method based on a neural network aiming at the problem of automatic testing in the field of software testing, can compile a testing software capable of automatically testing software by classifying the existing testing video and tracking the target of a cursor, has a self-optimization function, realizes a correct automatic testing function, and can quickly recover to the last relevant testing part for subsequent operation when accidental blockage occurs in the testing process.

Description

Automatic testing method based on neural network

Technical Field

The invention belongs to the technical field of neural networks, and particularly relates to equipment for realizing video classification and software automated testing through a CNN + RNN neural network.

Background

In the current software testing field, automated testing has become the mainstream of system testing development. Nowadays, many computer companies at home and abroad develop automatic testing software for automatically testing products. However, the existing automatic test software can only perform automatic test on a certain type or a certain types of special products or functions, and if other products or functions need to be tested, a programmer is required to modify the software. Particularly, when the correct design test of the same software system in different departments is faced, because the programming level and the design concept of different departments are different, the design of the same software system is also different, and at this time, if a test script is written by a programmer or a manual test mode is adopted, the efficiency is very low, and the cost is higher.

Video can be viewed as a sequence of frames consisting of a large number of images arranged in a temporal order. For the classification of images, the classification is mainly performed by adopting a CNN convolutional neural network, and because the images are arranged according to a time sequence, the RNN convolutional neural network can be added to extract the characteristics of the time sequence. Videos can be classified in a CNN + RNN manner.

The invention patent CN201710928075.X software testing method, device, equipment and computer storage medium discloses a software testing method, which comprises the following steps: when a software testing request is detected, displaying a software testing task list according to the software testing request so that a tester can input testing information; determining a test task according to the test information, calling test atoms in a preset test atom library according to the test task, and configuring to form a test atom set; and when an execution instruction of the software test is detected, calling and executing the test atoms in the test atom set according to the execution parameters in the test information so as to complete the atomic operation of the software test. The invention also discloses a software testing device, equipment and a computer storage medium. The invention aims to reduce the cost of software testing by utilizing atomic operation and improve the efficiency of the software testing to avoid errors caused by manual operation. The invention mainly depends on manual testing, and has low efficiency and high cost.

Currently, in the field of software testing, software interfaces and functions are mainly tested by compiling different test scripts for different test items or directly performing manual testing. As software functionality increases or new software items increase, testing becomes very cumbersome and difficult. In order to avoid reducing the working strength of testing personnel and improving the testing efficiency, it is very necessary to design an automatic testing method. The neural network plays an important role in the field of artificial intelligence as a core of deep learning. By constructing the neural network model, a computer can complete work which can be completed only by human beings. In the field of software testing, from the aspects of video classification and target tracking, testing software and a neural network model are compiled, so that the testing software tracks the testing video of the existing testing project, and the software testing is automatically completed.

Disclosure of Invention

An automated testing method based on a neural network, the method comprising:

according to a video which is recorded in advance and demonstrated by manual testing software and used as a deep learning training corpus, acquiring a software interface which needs to be opened during each test and a software testing sequence; putting the recorded video into a neural network for training, and acquiring a handle according to an opened interface; according to the rewriting of the source code at the bottom layer of the SelenimumIDE, the SelenimumIDE has the basic function of testing software; acquiring position parameters of a mouse needing to be automatically clicked according to the video, and automatically inputting test sentences according to the position automatically clicked by the mouse; when the information acquired by the input test statement is wrong, searching the subsequent test statement for testing; when all the test sentences acquire information errors, changing the automatic click positions and inputting again; when the test process is blocked accidentally, recording the tested content, automatically testing the content which is tested successfully at the next test, recovering to the last related test part, and continuing to perform subsequent tests; and optimizing and updating the video classification model according to the comparison between the video test item label and the test item name.

Further optionally, the obtaining a software interface and a software testing sequence that need to be opened during each test according to a video that is recorded in advance and demonstrated by the manual testing software as a deep learning corpus includes:

before testing, recording the name of a testing project; the test video of the same software is named by the number and the name and is stored in the folder named by the software.

Further optionally, in the method as described above, the training of the recorded video in the neural network and the obtaining of the handle according to the opened interface include:

putting the video into a download-trained neural network, and acquiring a picture image of each frame for subsequent operation; and when the test software acquires the information error, the software interface is returned to the interface before change according to the software interface handle, and the software operation mode is used for storing the video element.

Further optionally, in the method as described above, the adapting, according to the rewriting of the SeleniumIDE underlying source code, to have the basic function of the test software, includes:

the method realizes the rewriting of the software-driven carex test script by rewriting the source code at the bottom layer of the SelenimumIDE, and has the operation of simulating the movement of a mouse, performing click and double-click operations on the mouse and inputting a text by a keyboard.

Further optionally, in the method, the obtaining, according to the video, a position parameter that the mouse needs to be automatically clicked, and automatically inputting, according to the position that the mouse automatically clicks, a test statement includes:

the method comprises the steps of adjusting the HSV threshold range to be white by converting an image color space into an HSV color space, and finding the position of a mouse cursor; and automatically simulating the basic functions of the test software in sequence according to the position of the mouse cursor.

Further optionally, in the method as described above, when the information obtained by the input test statement is incorrect, searching for a subsequent test statement for testing includes:

according to the found mouse cursor position, sequentially performing mouse clicking and double clicking and text input operation by a keyboard; when the text is identified by the video, the text is identified word by word, and the operation is carried out in sequence until the software interface is similar to the video picture.

Further optionally, in the method as described above, when all the test statements obtain an information error, changing the automatic click position and re-inputting includes:

when all the test sentences are invalid, judging that the positions of the buttons of the software part in the video are inconsistent with the positions of the buttons of the current test software; and carrying out operation of the test sentence by one pair of buttons until the software interface is similar to the video picture.

Further optionally, in the method as described above, when the test process is accidentally blocked, the content that has been tested is recorded, and at the next test, the content that has been tested successfully is automatically tested, and the last associated test part is recovered, and the subsequent tests are continued, including:

the recording of the tested content comprises: mouse position, evaluation mode and test statement; by the recorded content, the machine can be quickly operated and recovered to the last relevant test part.

Further optionally, in the method as described above, the optimizing and updating the video classification model according to the comparison between the video test item label and the test item name includes:

and comparing the video item labels with the test item names, and retraining the videos with the labels inconsistent with the names through a neural network to obtain the really matched video classification labels.

The technical scheme provided by the implementation of the invention can have the following beneficial effects:

the invention provides an automatic testing method based on a neural network aiming at the problem of automatic testing in the field of software testing, can compile a testing software capable of automatically testing software by classifying the existing testing video and tracking the target of a cursor, has a self-optimization function, realizes a correct automatic testing function, and can quickly recover to the last relevant testing part for subsequent operation when accidental blockage occurs in the testing process.

Drawings

FIG. 1 is a schematic flow chart of a neural network-based automated testing method according to an embodiment of the present invention;

Detailed Description

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required to be used in the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention;

recording a video demonstrated by manual testing software in advance as a deep learning training corpus, and acquiring a software interface required to be opened during each test and a software testing sequence.

The requirement for recording the video should be as follows: a specific software testing process; before a specific test is carried out, recording the name of the test item; storing the test video of the same software in a folder named by the software by taking the serial number and the name as names;

and (4) utilizing screen recording software, opening the software to be tested, performing different test items according to the test standard document, and recording a test video. After the recording is finished, the recorded video is stored in a folder named by the test items by the number names.

For example: and opening software 'test, exe' to be tested, after the software is opened and enters an initial interface, checking a test standard document, determining whether a user login function of the test software is normal for a next test item according to the test standard document, and starting screen recording by using a screen recording function of the Windows system. After the project to be tested is completed, screen recording is stopped, and the recorded video is stored in a folder named 'user login' in a format of '01. mp 4'.

And step two, taking the video recorded in the step one as input, taking the recorded test item name as a label, and constructing a neural network model combining the CNN and the RNN for training.

Firstly, a neural network model is constructed, a trained inclusion ConvNet CNN neural network model is downloaded and directly used, the GoogleNet model traverses all frames of a video, the output of the final pooling layer of the network is input to a single-layer LSTM RNN cyclic neural network model, and a video classification model is obtained. And (4) inputting the videos of different test items stored in the step one, wherein the name of the folder in which the videos of different test items are stored is a label, and the label can automatically carry out coding and training on the video classification model.

For example: after a video classification model is built, dividing the total 40 test videos under the folders of 'user login' and 'drawing of graphics' obtained in the step one into a training set and a test set, wherein the number of the training sets is 30, the number of the test sets is 10, inputting the videos in the training set into the video classification model, after 50 epochs are trained, finding that the accuracy rate is maintained at 93.34% and does not increase any more, stopping the training at the moment, inputting the videos in the test set, finding that the accuracy rate is 90.48%, and using the videos in the test set normally.

Step three: downloading SelenimumIDE, rewriting the bottom layer source code, and compiling the basic function part of the test software to make it have the basic function required by automatic test software.

The method comprises the steps of downloading the Selenium IDE, wherein the Selenium IDE is originally browser-driven software, so that the bottom layer source code of the Selenium IDE needs to be rewritten, software driving and test script rewriting are realized, and basic functions of software testing are realized, wherein the basic functions comprise the functions of opening a software program in a windows system, performing keyword search on a software interface, simulating a mouse and a keyboard, acquiring handles of the software interface, and performing screenshot and recording.

For example: after the source code at the bottom layer of the Selenium IDE is rewritten, the method can be operated to automatically open the software to be tested, namely test exe, simulate the functions of a mouse and a keyboard in the software, capture handles of different interfaces when the software interfaces are switched, and realize the switching of the software interfaces according to the handles.

Step four, writing a test algorithm part for the test software in the step three.

The test algorithm comprises the following steps: and acquiring the content of the test standard document, finding out the video which is classified correspondingly in the fourth step according to the test project name in the document, and acquiring the image picture of one frame in the video by downloading the trained neural network model. And simultaneously, capturing the current interface in real time by using the screen capture function of the test software, comparing the similarity of the software program which is opened and the initial software interface in the video by using a perceptual hash algorithm, and if the similarity is not similar, opening the next-stage or previous-stage interface of the interface by using the simulated mouse and keyboard functions to compare until the next-stage or previous-stage interface is the same as the initial software interface in the video.

The steps of the perceptual hashing algorithm include: reduce the picture to a size of 8x 8; converting the reduced picture into 64-level gray scale; calculating transformed DCT (discrete cosine transform) values that decompose the picture into frequency aggregation and a ladder, where the DCT values are a 32 x 32 matrix; only the 8x8 matrix in the top left corner of the DCT is retained; calculating the average value of all 64 values in the matrix; setting a hash value of 0 or 1 according to the DCT average value, setting the hash value to be 1 when the value is more than or equal to the DCT average value, and setting the hash value to be 0 when the value is less than the DCT average value; combining the comparison results of the previous step together to form a 64-bit integer, wherein the combination order must be consistent, namely, from left to right and from top to bottom; comparing the 64-bit integers, if the data bits which are not identical do not exceed 5, the two pictures are very similar.

And step five, predicting the position of the mouse needing to be automatically clicked according to the recorded video and the algorithm in the step four, and automatically inputting a test statement.

And inputting the recorded video serving as a parameter into a trained neural network model, traversing the picture of the video by each frame by the GoogleNet network, simultaneously comparing the similarity by using a perceptual Hash algorithm, and if the judgment result of the current frame number picture is not similar to the previous frame picture, greatly changing the current frame number picture. And converting the image color space of the previous frame of picture into an HSV color space, adjusting the HSV threshold range to be white, and finding the position of the mouse cursor. And after rewriting the Selenimum IDE source code, the test software moves the simulated mouse to the same position, respectively simulates clicking a left key, double clicking or keyboard input, and if the changed software interface is similar, finds the position of the mouse needing to be automatically clicked.

Further, if the video and the icon or button of the software test are changed in position, that is: and after the operation is carried out according to the position of the mouse cursor of the video, the transformed interface image is not similar to the picture in the video. And clicking each button in the software one by simulating the function of the mouse, temporarily storing the position of the mouse, and comparing the similarity of the converted pictures until the position of the mouse which is temporarily stored is the position which needs to be automatically clicked when a similar image is found.

For example: after the test software opens the test, exe, the test software can obtain the content of the test standard document, find the test item as the user login according to the test item name in the document, and find the corresponding video according to the test item name. And putting the video into a neural network to obtain the picture of each frame. And when the video content is a mouse click button and then an interface change occurs, acquiring a previous frame of picture of the current frame. Adjusting the HSV threshold range, finding the position of the mouse, testing sentences at the same position in the testing software according to the mouse simulating function in the step three, and comparing the video picture with the Hash algorithm of the testing software interface once every operation until the similarity is judged.

And step six, searching subsequent test sentences for testing when the information acquired by inputting the test sentences is wrong, and changing the automatic click position and inputting again when all the test sentences acquire the information which is wrong.

And acquiring the picture of each frame in the video according to the GoogleNet deep learning network, performing mouse clicking, double clicking or keyboard input operation according to the acquired mouse clicking position, and continuing to operate when the software interface is changed and is similar to the picture presented by the video. And if the evaluation statements are different, backing the previous interface to test the subsequent evaluation statements.

Further, if the evaluation statement is a mouse double click, the mouse double click is tried after rollback.

Further, if the information obtained by single and double clicking of the mouse is wrong, that is, the evaluation sentence is input by the keyboard, the frame picture of the sentence input beginning and the frame picture after the input (that is, the previous frame picture with large interface change) are marked. And in the marking range, carrying out perceptual hash algorithm comparison on the first frame and all subsequent frames in sequence, wherein a threshold value is set for comparing 64-bit integers in DCT, so that circulation is stopped when only two frames have a character difference, and a continuous label is made. Identifying the characters currently in the input box, inputting the same characters in the evaluation software, and if the interface is inconsistent with the pictures after the label segment, performing hash comparison on the pictures of the subsequent frames in sequence from the continuation of the label, wherein the steps are consistent with the steps.

Further, if all the test statements have information errors, that is, it is determined that the positions of the icons or buttons tested by the video and the software have changed, according to the special condition mentioned in the step five, the operation of clicking each button in the software one by one is performed until the mouse position is finally found, and the test statements are input again.

This has the advantage that when the video evaluation sentence is an input text, and many texts are input. The method can reduce the error of simultaneously recognizing a plurality of characters, and can judge the same picture when a shorter key word is input.

For example: the video is shot for a long time, so that the positions of the buttons of the software part and the buttons in the video are changed, and the test item at the moment needs to input a long string of characters in the text box. And finding the position of the text box appearing in the video according to the steps, and synchronously performing in the test software, wherein all the test statement acquisition information is completely invalid when the software is tested because the positions are inconsistent. Each button in the software is clicked one by one, and the mouse click and the double click are performed in sequence to input the text by the keyboard (wherein, when the text is input, the text is searched one by one). When the first five texts are searched in the button at a random position, the picture of the video is similar to the interface of the software, and the next operation is carried out.

And step seven, when the test process is blocked accidentally, recording the tested contents, automatically testing the contents which are tested successfully at the next test, recovering to the last associated test part, and continuing to perform subsequent tests.

And when the software interface changes and is similar to the video picture after the statement is tested, automatically acquiring the video elements in the video. The video elements comprise a picture image of a current frame, video progress bar numbers, a mouse click position, an evaluation mode and a sentence. When the test process is blocked accidentally, the numbers of the video progress bars are sequenced from small to large, the mouse positions, the evaluation modes and the sentences of the video elements are rapidly acquired in sequence, automatic operation is carried out, and the relevant test part is recovered.

Further, since the saved video elements are not saved until the interface is similar to the video after the operation, the saved video elements are available and can be quickly restored to the associated test part according to the saved video elements

Further, according to the method, the automatic mouse click position is obtained, and the position parameters are stored.

Further, the evaluation modes include mouse click, double click, keyboard input, which are respectively stored by the numbers 123.

Further, when the operation is successful, the text input by the keyboard is stored.

For example: and when the software is tested to the step five, clicking the buttons one by one, identifying the text and testing when the positions of the buttons are inconsistent, wherein the step is a step with a larger occupied specific time. And if the operation is successful, automatically storing the video element, namely the new mouse position, and inputting the content of the text. When the test process is blocked accidentally, the operation is rapidly performed according to the progress bar numerical sequence and the stored video elements in sequence, so that the process of re-traversal is reduced, and the time is shortened extremely. Meanwhile, for the user, the accidental blocking can be quickly recovered, and the user experience is greatly improved.

Step eight: and optimizing and updating the video classification model.

And (4) running test software to obtain a test video, classifying the test video by using the video classification model obtained in the step two to obtain a test item label to which the video belongs, comparing the test item label with the test item name in the step four, if the test item label is different from the test item name in the step four, taking the video as input, taking the test item name in the step four as a label, and retraining the neural network model.

For example: and when the test software carries out a user login test item on the software, obtaining a screen recorded video, inputting the video into the video classification model obtained in the step two for classification, obtaining that the video belongs to a name inquiry test item and is not in accordance with the actual situation, and then inputting the video as input, inputting the user login as a label into the video classification model for retraining.

Programs for implementing the information governance of the present invention may be written in computer program code for carrying out operations of the present invention in one or more programming languages, including an object oriented programming language such as Java, python, C + +, or a combination thereof, as well as conventional procedural programming languages, such as the C language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit. The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention.

And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (9)

1. An automated testing method based on a neural network, the method comprising:

according to a video which is recorded in advance and demonstrated by manual testing software and used as a deep learning training corpus, acquiring a software interface which needs to be opened during each test and a software testing sequence; putting the recorded video into a neural network for training, and acquiring a handle according to an opened interface; according to the rewriting of the source code at the bottom layer of the SelenimumIDE, the SelenimumIDE has the basic function of testing software; acquiring position parameters of a mouse needing to be automatically clicked according to the video, and automatically inputting test sentences according to the position automatically clicked by the mouse; when the information acquired by the input test statement is wrong, searching the subsequent test statement for testing; when all the test sentences acquire information errors, changing the automatic click positions and inputting again; when the test process is blocked accidentally, recording the tested content, automatically testing the content which is tested successfully at the next test, recovering to the last related test part, and continuing to perform subsequent tests; and optimizing and updating the video classification model according to the comparison between the video test item label and the test item name.

2. The method according to claim 1, wherein the obtaining of the software interface to be opened and the software testing sequence for each test according to the video of the pre-recorded manual testing software demonstration as the deep learning corpus comprises:

before testing, recording the name of a testing project; the test video of the same software is named by the number and the name and is stored in the folder named by the software.

3. The method of claim 1, wherein the training of the recorded video in the neural network and the obtaining of the handle according to the opened interface comprises:

putting the video into a download-trained neural network, and acquiring a picture image of each frame for subsequent operation; and when the test software acquires the information error, the software interface is returned to the interface before change according to the software interface handle, and the software operation mode is used for storing the video element.

4. The method of claim 1, wherein said adapting the SeleniumIDE underlying source code to provide the underlying functionality of the test software comprises:

the method realizes the rewriting of the software-driven carex test script by rewriting the source code at the bottom layer of the SelenimumIDE, and has the operation of simulating the movement of a mouse, performing click and double-click operations on the mouse and inputting a text by a keyboard.

5. The method of claim 1, wherein the obtaining of the position parameter of the mouse needing to be automatically clicked according to the video and the automatic input of the test sentence according to the position of the mouse needing to be automatically clicked comprise:

the method comprises the steps of adjusting the HSV threshold range to be white by converting an image color space into an HSV color space, and finding the position of a mouse cursor; and automatically simulating the basic functions of the test software in sequence according to the position of the mouse cursor.

6. The method of claim 1, wherein when the input test statement obtains an error in information, searching a subsequent test statement for testing comprises:

according to the found mouse cursor position, sequentially performing mouse clicking and double clicking and text input operation by a keyboard; when the text is identified by the video, the text is identified word by word, and the operation is carried out in sequence until the software interface is similar to the video picture.

7. The method of claim 1, wherein when all test sentences have acquired information errors, changing the automatic click position and re-inputting comprises:

when all the test sentences are invalid, judging that the positions of the buttons of the software part in the video are inconsistent with the positions of the buttons of the current test software; and carrying out operation of the test sentence by one pair of buttons until the software interface is similar to the video picture.

8. The method of claim 1, wherein the recording of the tested contents when the test process is blocked accidentally, automatically testing the contents which have been tested successfully at the next test, recovering to the last associated test part, and continuing to perform subsequent tests comprises:

the recording of the tested content comprises: mouse position, evaluation mode and test statement; by the recorded content, the machine can be quickly operated and recovered to the last relevant test part.

9. The method of claim 1, wherein said optimizing and updating the video classification model based on the video test item label versus the test item name comprises:

and comparing the video item labels with the test item names, and retraining the videos with the labels inconsistent with the names through a neural network to obtain the really matched video classification labels.

Images