KR20190135074A - Ui testing automation method using deep learning algorithem and tree and appratus for the same - Google Patents

Abstract

According to an embodiment of the present invention, provided is a UI testing automation method using a deep learning algorithm and a tree. The method comprises: a first step of extracting an object from a captured image based on a predetermined deep learning algorithm and generating a tree composed of a node generated by utilizing the captured image and the extracted object; and a second step of performing automation for user interface testing using the generated tree. As a result, efficient UI testing with minimal time is possible.

Description

UI TESTING AUTOMATION METHOD USING DEEP LEARNING ALGORITHEM AND TREE AND APPRATUS FOR THE SAME}

The present invention relates to a method and apparatus for automating UI testing using a deep learning algorithm and a tree, and more particularly, to extract a object from a captured image and to generate a tree related to the extracted object. The present invention relates to a method and apparatus for automating UI testing using a deep learning algorithm and a tree for automation.

Infotainment is a compound word of information and entertainment, and refers to a device that provides information, audio, and visual entertainment. Recent advances in telematics, connected cars, and autonomous vehicle technologies have made In-Vehicle Infotainment (IVI) more than simple multimedia devices to inform users of their vehicles or directly The role and importance have increased significantly. As the IVI evolved, the user interface (UI), which is responsible for the interaction between the vehicle and the user, has increased in complexity and the need for UI testing techniques for high quality UI development. UI testing is the process of verifying that the desired response and screen changes occur when the user operates the system through the interface. If this is done manually by the tester, a lot of manpower and costs are incurred. In particular, the amount of resources consumed to perform the test can be a significant burden on the tester, so automation of UI testing is essential.

It is an object of the present invention to implement efficient automated UI testing with minimal resources.

In the UI testing automation method using a deep learning algorithm and a tree according to an embodiment of the present invention, an object is extracted from a captured image based on a predetermined deep learning algorithm, and the node is generated by using the captured image and the extracted object. It may include a first step of generating a tree consisting of a second step of automating UI testing using the generated tree.

UI testing automation apparatus using a deep learning algorithm and a tree according to another embodiment of the present invention, an object extracting unit for extracting an object from a captured image based on a predetermined deep learning algorithm, utilizing the captured image and the extracted object Tree generation unit for generating a tree consisting of the nodes generated by; And a UI testing automation unit for automating user interface testing using the generated tree.

The computer-readable recording medium according to another embodiment of the present invention may record a computer program for executing the UI testing automation method using the deep learning algorithm and the tree according to the embodiment of the present invention.

According to an embodiment of the present invention, UI testing can be automated, and simple and repetitive tasks such as generating resources (correct image, click point coordinates, etc.) that can be used in a test case and moving to a screen requiring testing are performed. Instead, it allows testers to perform efficient UI testing with minimal time.

According to an embodiment of the present invention, it is possible to determine whether the captured image to be added to the node already exists on the path, and to perform the test by reducing the system error by performing the node addition only if it does not already exist. Become.

According to an embodiment of the present invention, even if the user receives only the predetermined path information to be tested without receiving the node information, the comparison with the pre-stored path corresponding to the corresponding path can be performed, thereby efficiently and quickly performing the test. This has the advantage of being possible.

According to an embodiment of the present invention, the source code script can be simplified, and even if the UI of the testing target system is changed, the result of performing the menu tree search can be repeatedly used, thereby improving the reusability of the script code. Will be.

1 is a block diagram of a UI testing automation apparatus using a deep learning algorithm and a tree according to an embodiment of the present invention.
2 to 4 are flowcharts of a method for automating UI testing using a deep learning algorithm and a tree according to an embodiment of the present invention.
5 to 9 are views referenced to describe a method for automating UI testing using a deep learning algorithm and a tree according to an embodiment of the present invention.

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

Hereinafter, a UI testing automation method and apparatus using a deep learning algorithm and a tree according to an embodiment of the present invention will be described with reference to the accompanying drawings.

1 is a block diagram of a UI testing automation apparatus using a deep learning algorithm and a tree according to an embodiment of the present invention.

As shown in FIG. 1, the UI testing automation apparatus using the deep learning algorithm and the tree according to an embodiment of the present invention includes an input unit 100, a control unit 200, a memory unit 300, and an output unit 400. It may include.

The input unit 100 may be an input path for allowing a user to operate the computing system and may include a keyboard, a mouse, a pointing device, a touch screen, and the like.

The control unit 200 controls an UI testing automation device using a deep learning algorithm and a tree in cooperation with the input unit 100, the memory unit 300, and the output unit 400, and extracts an object according to an embodiment of the present invention. The unit 210, the tree generator 220, and the UI testing automation unit 230 may be included. Each component in the block diagram will be described later in detail.

The memory unit 300 stores various information of the UI testing automation device using the deep learning algorithm and the tree. The memory unit 300 includes a plurality of application programs (applications or applications) driven by the UI testing automation device using the deep learning algorithm and the tree, and related data for performing the UI testing automation method using the deep learning algorithm and the tree. For example, you can store commands. At least some of these application programs may be downloaded from an external server via wireless communication.

The output unit 400 may include a touch screen, a monitor, and the like, and display the result processed by the controller 200.

2 to 4 are flowcharts of a method for automating UI testing using a deep learning algorithm and a tree according to an embodiment of the present invention.

As shown in FIG. 2, the object extractor 210 may extract an object from a captured image based on a predetermined deep learning algorithm, and the tree generator 220 may utilize the captured image and the extracted object. A tree composed of the generated nodes may be generated (S200). The UI testing automation unit 230 may automate UI testing using the generated tree (S300).

Step S200 of FIG. 2 is described in detail with reference to FIG. 3, and step S300 of FIG. 2 is described with reference to FIG. 4.

For reference, according to an embodiment of the present invention, an example in which the algorithm of the present invention is applied to a vehicle infotainment system testing automation tool (VISTA), which is one of the vehicle automation testing tools, has been described. Is not limited to this and is applicable to all systems capable of capturing screens. Vehicle automation testing tools can be written in C / C ++ based scripts.

As illustrated in FIG. 3, the tree generation unit 220 may add an Nth node corresponding to the Nth captured image on a predetermined path (S210). For example, referring to FIG. 5, the vehicle automation testing tool may output a menu tree C and user interface captured images A and B associated with each node of the menu tree.

A first node (node 1, root node) corresponding to the first captured image A of FIG. 5 may be generated. Here, the first capture image A may be an initial screen to be subjected to user interface testing based on an AVN (Audio Video Navigation) system. On the other hand, this is merely an embodiment, and the present technology may be applied to the same / similarly to other user interface screens as well as a user interface screen for an AVN system in a vehicle. As shown in FIG. 5, a predetermined first path is generated including the first node based on the generated first node. As shown in FIG. 5, the menu tree C is a collection of paths composed of nodes and edges (arrows). In the present invention, the node may include information about the screenshot, and the edge represents a transition between the screenshots. The transition from the first node to the second node means changing from the first capture screen to the second capture screen, and in particular, the second capture screen is related to the first capture screen that is the user interface testing target. 1 may be the next screen of the capture screen. For example, the second capture screen may be a sequential next screen of the first capture screen to be tested for user interface, in which content related to a specific object of the first capture screen is output.

The tree generating unit 220 may extract the N-th object from the N-th capture image based on a predetermined deep learning algorithm (S220), and output the N + 1th capture image related to the extracted N-th object. . For example, referring to FIG. 5, the first object [o] may be extracted from the first captured image A. FIG. In addition, the second capture image B including the content of the first object [o] may be output. In the present invention, object extraction from an image may be automatically extracted using an algorithm based on a deep learning neural network. For example, Fast R-CNN (Convolutional Neural Networks) algorithm can be used to receive the image and the boundary of the object to be extracted from the image, and to learn to find the object similar to the input object. have. In the present invention, in particular, by learning a selectable area (button, widget, etc.) for the user interface screen, it is possible to find selectable objects on a specific screen. Meanwhile, the above-described deep learning neural network based algorithm is just an example, and various other types of deep learning neural network based algorithms may be applied to the present invention to perform object extraction.

The tree generator 220 may determine whether the N + 1th captured image related to the extracted N-th object satisfies a predetermined condition (S230), and the tree generator 220 may extract the N-th object. When the N + 1th capture image associated with the object satisfies a predetermined condition, an N + 1th node corresponding to the N + 1th capture image may be added (S240). On the other hand, if the N + 1th capture image associated with the extracted N-th object does not satisfy a predetermined condition, generation of an additional node may be stopped (S270). For example, a first condition in which the second capture image B related to the extracted first object [o] of FIG. 5 does not exist on a predetermined path and the second capture image B are extracted It may be determined whether a second condition including a possible second object [o '] is satisfied.

Determining the first condition is necessary in order to prevent this, because when the captured image already existing on the predetermined path is included again on the path, the same screen is continuously generated, thereby falling into an infinite loop. Therefore, in the present invention, a second node corresponding to the second captured image B may be added only when the second captured image B does not exist in the predetermined path. In FIG. 5, since the second capture image B does not exist on a predetermined path, a second node is added after the first node. In addition, the predetermined first path includes a first node and a second node. According to another embodiment, it may be determined whether a captured image is already present on a predetermined path, and whether or not a wrong object is extracted or an inactive object is recognized in the object extraction process.

The judging of the second condition is that if the second capture image B does not include an extractable second object [o '], it is no longer selectable (or user input) on the user interface screen for testing. This means that it doesn't need to create an additional node anymore.

That is, the node addition may be performed only when both of the above-described first and second conditions are satisfied, and generation of the additional node may be stopped if any one of the conditions is not satisfied. Meanwhile, according to another exemplary embodiment, even when only one of two conditions is satisfied, the node addition may be performed in a manner of performing node addition.

In this case, the first node and the second node may be implemented to enable a click input. That is, the first node and the second node may each include coordinate information of the click. The first capture image A corresponding to the first node may be output by a click input, and the second capture image B corresponding to the second node may be output by another click input.

In the present invention, the N + 1th node may include coordinate information of the extracted N-th object and image information of the N-th object. Referring to FIG. 5, the second node (node 2) may include coordinate information of the first object [o] and image information of the first object [o]. That is, the child node may include the 2D coordinate information and the image information of the specific object on the capture screen of the parent node, so that the current node may refer to information related to the previous node, and the current node may be connected to the previous node. . The coordinate information of the first object [o] may include x, y coordinate (position) information located on the screen, and the image information may include information such as the size and shape of the object [o]. have. In the present invention, the coordinate information and the image information are implemented in two dimensions, but according to another embodiment, the object may be implemented in three dimensions and may include three-dimensional coordinate information and image information.

In addition, by setting the N + 1 index to N again (S250), the above-described steps S220 to S250 may be repeated. That is, according to FIG. 5, after the second node is further included in the first path following the tree, the second object [o '] is extracted from the second captured image B, and the extracted second object is extracted. It is determined whether the third capture image (not shown) associated with the second object [o '] satisfies the above-described predetermined condition, thereby determining the addition of the third node. By repeating these steps, the first path as shown in FIGS. 6 and 7 is completed. 6 and 7 show the completion of the first path by generating the last node (node 12) through the above repetition, and finally, the result of the result screen of the user interface to test the AUX cable connection state is output. have. As shown in FIG. 5, the menu tree C further includes other nodes and other paths in addition to the first path including the Nth node and the N + 1th node, and the above-described steps S220 to S250 for the other node. The other route may be generated by repeating the steps.

The tree generator 220 may store the menu tree generated by the method of FIG. 3 in the memory 300 (S260). When the menu tree is stored, each node, captured image, and object information may also be stored.

Up to now, the process of extracting an object from the captured image and generating and storing a menu tree composed of nodes has been described in detail with reference to step S200 of FIG. 2. In the following steps, it will be described in detail for step S300 to automate UI testing using the generated tree.

As shown in FIG. 4, the UI testing automation unit 230 may receive a play button input by a user (S310). The play button may be any object implemented on the vehicle automation testing tool, as shown in FIG. When receiving the play button input, the UI testing automation unit 230 may generate a menu tree (S320). The generated menu tree may be stored in the memory 300.

The UI testing automation unit 230 may receive predetermined path information from the user (S330). Even if the user receives only the predetermined path information to be tested without receiving the node information, the comparison with the previously stored path corresponding to the corresponding path can be performed, thereby making it possible to perform an efficient and fast test. In the present invention, path information corresponding to the first path previously stored in the generated tree may be input.

The UI testing automation unit 230 may compare the information included in the N + 1th node of the generated tree with the node information included in the predetermined path information in a one-to-one correspondence (S340) as illustrated in FIGS. 5 and 6. Each node (2, 10, 11, 12) included in the first path pre-stored in the generated tree and each node information included in the path information input from the user may be compared in a one-to-one correspondence. As described above, the N + 1th node of the generated tree may include the coordinate information of the extracted N-th object and the image information of the N-th object, and the coordinate information and the image information may be included in the path information input from the user. A process of comparing the coordinate information and the image information of the object included in each included node is performed. In this case, the UI testing automation unit 230 automatically performs testing for switching of each screen of the user interface.

The UI testing automation unit 230 may output the above-described comparison result (S350). As shown in FIG. 8, the output result screen is obtained from a node of a path input from a user in relation to a reference data item obtained from a node previously stored in a path of a generated tree and a device to be tested. As a result of comparing the device data item, reference data and device data, if there is a difference, the difference data item describing the difference part and the reference data ) And a result of comparing the device data (device data), and if the match, the test pass, and if there is a mismatch can include a result (result) for writing (or alarm) the test failure. According to an embodiment of the present invention, each item shown in FIG. 8 may be combined with one or more items and output on the result screen. In addition, information such as an event name, an event time, an elapsed time, and a log may also be output. Accordingly, if the test result passes, it can be seen that the device is implemented as the correct program. If the test result fails, the device can be known to be implemented as an error program. Based on this, the test can be easily performed.

9 is a source code (B) implemented according to the prior art and a source code (A) implemented according to the present invention for moving to a screen for testing. Conventionally, as in the source code (B), it was inevitable to move to the target screen over a plurality of lines, but according to the present invention, as in the source code (A), the structure previously identified through the menu tree search By simply declaring the specific screen you want (path input), you can move to the target screen quickly. That is, according to the conventional source code (B), the complexity of the script is increased and the reusability is low, but according to the source code (A) of the present invention, the source code script can be simplified, and even if the UI is changed, the menu tree is searched. The result of re-executing can be used repeatedly, thus improving the reusability of script code.

The above-described embodiments may be implemented in the form of program instructions that can be executed by various computer components, and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to carry out the process according to the invention, and vice versa.

Features, structures, effects, etc. described in the above embodiments are included in one embodiment of the present invention, and are not necessarily limited to one embodiment. Furthermore, the features, structures, effects, and the like illustrated in the embodiments may be combined or modified with respect to other embodiments by those skilled in the art to which the embodiments belong. Therefore, contents related to such combinations and modifications should be construed as being included in the scope of the present invention.

In addition, the above description has been made with reference to the embodiment, which is merely an example, and is not intended to limit the present invention. Those skilled in the art to which the present invention pertains will be illustrated as above without departing from the essential characteristics of the present embodiment. It will be appreciated that various modifications and applications are possible. For example, each component specifically shown in the embodiment can be modified. And differences relating to such modifications and applications will have to be construed as being included in the scope of the invention defined in the appended claims.

Claims (11)

Extracting an object from a captured image based on a predetermined deep learning algorithm, and generating a tree composed of the captured image and the nodes generated by using the extracted object; And
A second step of automating user interface testing using the generated tree,
UI testing automation method using deep learning algorithm and tree.
The method of claim 1,
The tree includes a predetermined path consisting of an Nth node and an N + 1th node,
The first step,
(A) adding the Nth node corresponding to the Nth captured image on the predetermined path;
(B) extracting an N-th object from the N-th captured image based on the predetermined deep learning algorithm;
(C) adding an N + 1 node corresponding to the N + 1th captured image when the N + 1th captured image related to the extracted Nth object satisfies a predetermined condition;
(D) setting the N + 1 index to N; And
Repeating steps (B) to (D)
Included,
UI testing automation method using deep learning algorithm and tree.
The method of claim 2,
The predetermined condition is
A first condition in which the N + 1 capture image does not exist on the predetermined path and a second condition including an N + 1 object in which the N + 1 capture image is extractable;
UI testing automation method using deep learning algorithm and tree.
The method of claim 2 or 3,
The tree further includes other nodes and other paths,
Repeating the steps (B) to (D) for the other node to generate the other path,
UI testing automation method using deep learning algorithm and tree.
The method according to claim 2 or 3,
The N + 1th node includes coordinate information of the extracted Nth object and image information of the Nth object.
UI testing automation method using deep learning algorithm and tree.
The method of claim 2 or 3,
The captured image includes information on a user interface screen based on an AVN (Audio Video Navigation) system.
UI testing automation method using deep learning algorithm and tree.
The method of claim 1,
The node is implemented to allow a click input,
Outputting the captured image associated with the node with the click input;
UI testing automation method using deep learning algorithm and tree.
The method of claim 2,
The second step,
Reading the generated tree;
Receiving predetermined path information from a user;
Comparing the information included in the N + 1th node of the generated tree with the node information included in the predetermined path information in a one-to-one correspondence; And
And outputting the comparison result.
Automated UI Testing Using Deep Learning Algorithm and Tree.
The method of claim 8,
The N + 1th node of the generated tree includes coordinate information of the extracted N-th object and image information of the N-th object.
UI testing automation method using deep learning algorithm and tree.
An object extracting unit extracting an object from a captured image based on a predetermined deep learning algorithm;
A tree generation unit generating a tree composed of nodes generated by using the captured image and the extracted object; And
It includes; UI testing automation unit for performing an automated user interface testing using the generated tree, including,
UI testing automation device using deep learning algorithm and tree.
A computer-readable recording medium for recording a computer program for executing the method according to any one of claims 1 to 9.

Images