TWI512594B - Graphic user interface testing method and testing apparatus - Google Patents

Description

Graphic user interface test method and test device

The invention relates to a test method and a test device, and in particular to a graphical user interface test method and test device.

The tools for automated graphical user interface (GUI) testing are designed to minimize the manpower required by testers to test the GUI. However, today's common automated GUI test tools are designed in the direction of testing computer applications, rather than being designed for handheld or handheld applications. Therefore, these common automated GUI testing tools are not suitable for handheld devices with heterogeneous nature.

However, even if a designer has designed a corresponding test mechanism and test code for different handheld devices, this method is less efficient and is particularly unsuitable for testing black application applications. In addition, these mechanisms are generally not designed for graphical user interfaces. And, these mechanisms can usually only be used in the development build, not in the product phase. (development build) use (ie, cannot be applied to real handheld devices). In this case, even if the functional test result of the handheld device is normal, there is no guarantee that the GUI of the handheld device can operate normally.

In view of this, the present invention provides a graphical user interface test method and test apparatus that can efficiently test a wide variety of handheld devices.

The present invention provides a graphical user interface test method suitable for testing device testing electronic devices, the method comprising the steps of: controlling a photographing device to capture a plurality of first images displayed by the electronic device in response to the plurality of first instructions, and Recording the plurality of first instructions to the storage unit, wherein a difference degree between the ith first picture and the i-1th first picture in the plurality of first pictures is greater than a first threshold And setting a check point corresponding to the ith first picture; inputting a plurality of second instructions that are the same as the plurality of first instructions to the electronic device; and controlling the photographic device to capture the electronic device to react to the plurality of a plurality of second pictures displayed by the second instruction; when the current picture of the plurality of second pictures corresponds to the check point, calculating a similarity between the current picture and the ith first picture; When the degree is lower than the second threshold, the test electronics is stopped and the test error result is reported.

In an embodiment of the invention, the plurality of first instructions are a plurality of scripting languages input by the testing device to the electronic device.

In an embodiment of the present invention, the step of controlling the photographing device to capture the plurality of first screens displayed by the electronic device in response to the plurality of first instructions Before, it further includes controlling the photographing device to capture the initial screen and setting an initial check point corresponding to the initial screen.

In an embodiment of the present invention, before the step of inputting the plurality of second instructions that are the same as the plurality of first instructions to the electronic device, the method further includes: controlling the capturing device to capture the end screen, and setting the corresponding end Check the end of the screen.

In an embodiment of the present invention, the check point is an index value of the ith first picture, and the current picture is one of the plurality of second pictures having an index value.

The invention provides a testing device comprising a storage unit and a processing unit. The storage unit stores a plurality of modules. The processing unit is coupled to the storage unit, and accesses the plurality of modules to perform the following steps: controlling the photographing device to capture the plurality of first screens displayed by the electronic device in response to the plurality of first instructions, and recording the plurality of a first instruction to the storage unit, wherein when the degree of difference between the ith first picture and the i-1th first picture in the plurality of first pictures is greater than the first threshold, the setting corresponds to a check point of the i-th first picture; inputting a plurality of second instructions identical to the plurality of first instructions to the electronic device; and controlling the photographing device to capture the electronic device to display in response to the plurality of second instructions a plurality of second pictures; when the current picture of the plurality of second pictures corresponds to the check point, calculating a similarity between the current picture and the ith first picture; when the similarity is lower than the second At the threshold, the test electronics is stopped and the test error results are reported.

In an embodiment of the invention, the plurality of first instructions are a plurality of scripting languages input by the testing device to the electronic device.

In an embodiment of the invention, the processing unit further controls the photographing device to capture the initial screen and set an initial checkpoint corresponding to the initial screen.

In an embodiment of the present invention, the processing unit further controls the photographing device to capture the end screen and sets an end check point corresponding to the end screen.

In an embodiment of the present invention, the check point is an index value of the ith first picture, and the current picture is one of the plurality of second pictures having an index value.

Based on the above, the graphical user interface test method and system thereof according to the embodiments of the present invention can capture electronic devices through two photographic devices (for example, a first instruction and a second instruction identical to the first instruction) through the photographic device. The displayed two screens (for example, the first screen and the second screen) determine whether an abnormality occurs in the graphical user interface of the electronic device according to the similarity between the two images.

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧Graphic User Interface Test System

110‧‧‧Testing device

112‧‧‧storage unit

114‧‧‧Processing unit

120‧‧‧Photographing device

130‧‧‧Electronic devices

S210~S250, S310~S370‧‧‧ steps

1 is a schematic diagram of a graphical user interface test system in accordance with an embodiment of the present invention.

2 is a flow chart of a graphical user interface test method according to an embodiment of the invention.

FIG. 3A is a graphical user interface test according to an embodiment of the invention. The recording phase of the method flow chart.

FIG. 3B is a flowchart of a replay phase of a graphical user interface test method according to an embodiment of the invention.

1 is a schematic diagram of a graphical user interface test system in accordance with an embodiment of the present invention. In the present embodiment, the graphical user interface test system 100 includes a test device 110 and a photographing device 120. The test device 110 is, for example, a personal computer, a notebook computer, a workstation, a server, or other computer device, but the embodiments of the present invention are not limited thereto. The test device 110 includes a storage unit 112 and a processing unit 114. The storage unit 112 is, for example, a memory, a hard disk, or any other component that can be used to store data, and can be used to record a plurality of code or modules. The processing unit 114 is connected to the storage unit 112. The processing unit 114 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, and a control , Microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), any other kind of integrated circuit, state machine, based on advanced reduced instructions Advanced RISC Machine (ARM) processors and similar products.

The photographing device 120 is, for example, a Charge Coupled Device (CCD) and/or a Complementary Metal Oxide (Complementary Metal Oxide). Image capture components implemented by Semiconductor, CMOS). In this embodiment, the photographing device 120 may be a camera connected to the test device 110 through a Universal Serial Bus (USB) interface. Also, the photographing device 120 can be controlled by the test device 110 to take a picture of the electronic device 130.

The electronic device 130 is, for example, a smart phone, a tablet computer, a personal digital assistant (PDA), or other device having a screen capable of displaying a graphical user interface, but embodiments of the present invention are not limited thereto. The electronic device 130 can also be connected to the testing device 110 via, for example, a USB interface.

In this embodiment, the processing unit 114 can access multiple modules stored in the storage unit 112 to perform various steps of the graphical user interface (GUI) test method proposed in FIG. 2, and then test the electronic The GUI of device 130. In summary, in the method of the embodiment, the testing device 110 may first control the photographing device 120 to capture a plurality of first screens displayed by the electronic device 130 in response to a plurality of commands. Then, when the test device 110 inputs the plurality of instructions to the electronic device 130 again, the test device 110 can determine the electronic device by determining whether the current screen displayed by the electronic device 130 is consistent with the first screen corresponding to the current screen. Is the GUI function of 130 normal?

2 is a flow chart of a graphical user interface test method according to an embodiment of the invention. The method proposed in this embodiment can be illustrated by the graphical user interface test system 100 of FIG. 1, and the following is a description of the various steps of the method.

In step S210, the processing unit 114 can control the photographing device 120 to capture The electronic device 130 reflects the plurality of first pictures displayed by the plurality of first instructions, and records the plurality of first instructions to the storage unit 112. In an embodiment, the plurality of first instructions are, for example, instructions input by the user to the electronic device 130 (eg, a sliding operation or a click operation, etc.).

In other embodiments, the plurality of first instructions may also be a plurality of scripts input by the testing device 110 to the electronic device. In detail, in order for the photographing device 120 to clearly capture the plurality of first screens displayed by the GUI of the electronic device 130, the testing device 110 may simulate an instruction input by the user (eg, a sliding operation) to generate a corresponding script. Language. Therefore, when the test device 110 inputs these scripting languages to the electronic device 130, the electronic device 130 can still generally display the plurality of first pictures as received by the user. In this way, the plurality of first images captured by the photographing device 120 are not interfered by the user's hand or an object such as a stylus.

In the present embodiment, in the process of capturing the plurality of first pictures by the photographing device 120, the processing unit 114 may calculate the degree of difference between the two consecutive first pictures. For example, when the photographing device 120 captures the i-th first screen (i is an arbitrary integer value), the processing unit 114 may calculate the difference between the i-th first screen and the i-th first screen. degree.

In an embodiment, processing unit 114 may calculate the degree of difference based on a Speeded Up Robust Features (SURF) algorithm. Specifically, the processing unit 114 may first calculate a first similarity between the i-th first picture and the i-1th first picture. The similarity can be characterized as (100 x α) / β, wherein α represents the number of features matched between the i-th first picture and the i-1th first picture, and β represents the detection on the i-th first picture and the i-th first picture The average number of features measured. After the first similarity is obtained, the processing unit 114 may calculate the degree of difference by 1-(100×α)/β. When the degree of difference between the ith first picture and the i-1th first picture is greater than the first threshold (eg, 80%), the processing unit 114 may set a check point corresponding to the ith first picture. (checkpoint). In other words, when the processing unit 114 determines that a significant change in the picture of the electronic device 130 occurs, the processing unit 114 may correspondingly set the check point. In other embodiments, the checkpoint may be characterized as an index value of the i-th first picture (ie, i), but embodiments of the present invention are not limited thereto.

In other embodiments, after step S220, the user can return the GUI of the electronic device 130 to the operating state before the processing unit 114 performs the step S210. Alternatively, the processing unit 114 may also record the current operating state of the electronic device 130 before performing step S210, and return the GUI of the electronic device 130 to the operating state before the step S210 is performed after the step S210 is performed, but the present invention The implementation manner is not limited to this.

Thereafter, in step S220, the processing unit 114 may input a plurality of second instructions identical to the plurality of first instructions to the electronic device 130. Moreover, in step S230, the processing unit 114 can control the photographing device 120 to capture a plurality of second screens that the electronic device 130 displays in response to the plurality of second instructions.

In step S240, when the current picture in the plurality of second pictures corresponds to the check point, the processing unit 114 may calculate the current picture and the ith first picture. The similarity between the two. In an embodiment, the current screen is, for example, an i-th second screen captured by the photographing device 120 after the processing unit 114 starts inputting the plurality of second commands to the electronic device 130. In other words, the current picture can be regarded as a second picture having the same index value (ie, i) as the i-th first picture.

In an embodiment, the processing unit 114 may also calculate the similarity between the current picture and the ith first picture based on the foregoing SURF algorithm, but the embodiments of the present invention are not limited thereto.

In step S250, when the similarity is lower than the second threshold (for example, 80%), the processing unit 114 stops the test electronic device 130 and reports the test error result. The test error result may, for example, represent a situation in which the GUI of the electronic device 130 has abnormality. In detail, since the plurality of second instructions input by the processing unit 114 to the electronic device 130 are the same as the plurality of first instructions, the plurality of second pictures displayed by the electronic device 130 should be the same as the plurality of The second picture. In other words, when the similarity between the current picture and the ith first picture is too low, that is, the GUI representing the electronic device 130 has an abnormality, the processing unit 113 may not continuously input the remaining second instruction to The electronic device 130 directly reports the test error result. In an embodiment, the processing unit 114 can control the user interface unit (not shown) of the testing device 110 to display information related to the test error result, but the embodiments of the present invention are not limited thereto.

In other embodiments, if the similarity between all the second pictures and the corresponding first picture is not lower than the second threshold, the processing unit 114 may report the test success result. In an embodiment, the processing unit 114 can control the testing device 110 The user interface unit (not shown) displays information associated with the successful result of the test, but embodiments of the present invention are not limited thereto.

In summary, the method proposed by the embodiment of the present invention can be regarded as including a record phase and a replay phase. In the recording phase (ie, step S210), the testing device 110 may record a plurality of first screens displayed by the electronic device 130 in response to the plurality of first instructions, and set a checkpoint at a specific timing. Then, in the replay phase (ie, steps S220-S250), the testing device 114 may input a plurality of second instructions identical to the plurality of first instructions to the electronic device 130 again, and observe that the electronic device 130 reacts Whether the plurality of second pictures displayed by the plurality of second instructions are consistent with the plurality of first pictures. In addition, since the test apparatus 110 can compare only the first picture and the second picture corresponding to the same check point without comparing all the first and second pictures one by one, the efficiency of the overall graphical interface test mechanism can be made. Effectively improve.

Moreover, since the embodiment of the present invention records the respective screens displayed by the electronic device 130 through the photographing device 114 outside the electronic device 130, no additional load is generated on the electronic device 130. In addition, since the embodiment of the present invention captures each screen displayed by the electronic device 130 through the photographing device 120, it has a high degree of reusability. In other words, the method proposed by the embodiment of the present invention can be used to test various electronic devices having a graphical user interface.

It should be noted that although FIG. 1 shows the test device 110 and the photographing device 120 as two devices, in other embodiments, the designer may also use an image capturing unit (such as a lens) built in the test device 110. To implement the embodiment of the present invention The various technical solutions associated with the photographic device 120 are mentioned.

FIG. 3A is a flow chart of a recording phase of a graphical user interface test method according to an embodiment of the invention. The method proposed in this embodiment can be illustrated by the graphical user interface test system 100 of FIG. 1, and the following is a description of the various steps of the method.

In step S310, the processing unit 114 may control the photographing device 120 to capture the initial screen and set an initial check point corresponding to the initial screen. In step S320, the processing unit 114 may control the photographing device 120 to capture a plurality of first screens displayed by the electronic device 130 in response to the plurality of first instructions, and record the plurality of first instructions to the storage unit 112. For details of step S320, refer to step S210 of FIG. 2, and details are not described herein again. In step S330, the processing unit 114 may control the photographing device 120 to capture the end screen and set an end check point corresponding to the end screen.

After step S330, the user can return the GUI of the electronic device 130 to the operating state before the processing unit 114 performs step S310. Alternatively, the processing unit 114 may also record the current operating state of the electronic device 130 before performing step S310, and return the GUI of the electronic device 130 to the operating state before the step S310 is performed after the step S330 is performed, but the present invention The implementation manner is not limited to this. Next, the testing device 110 can continue to perform the various steps shown in FIG. 3 below.

Please refer to FIG. 3B. FIG. 3B is a flowchart of a replay phase of a graphical user interface test method according to an embodiment of the invention. In step S340, when the screen currently displayed by the electronic device 130 is the same as the initial screen, the processing unit 114 may A plurality of second instructions identical to the plurality of first instructions are input to the electronic device 130. In other words, before the processing unit 114 inputs the plurality of second instructions to the electronic device 130, the processing unit 114 may first determine whether the screen currently displayed by the electronic device 130 is consistent with the initial screen recorded in step S310. If so, the processing unit 114 can continue to input the plurality of second instructions to the electronic device 130. If the currently displayed screen of the electronic device 130 does not match the initial screen, the processing unit 114 may determine that the GUI of the electronic device 130 does not return to the operating state before the processing unit 114 performs step S310, thereby reporting the test error result.

Next, in step S350, the processing unit 114 can control the photographing device 120 to capture a plurality of second screens that the electronic device 130 displays in response to the plurality of second instructions. In step S360, when the current picture of the plurality of second pictures corresponds to the check point, the processing unit 114 may calculate the similarity between the current picture and the ith first picture. In step S370, when the similarity is lower than the second threshold, the processing unit 114 may stop testing the electronic device 130 and report the test error result.

In other embodiments, if the similarity between all the second pictures and the corresponding first picture is not lower than the second threshold, the processing unit 114 may further determine the picture currently displayed by the electronic device 130 and step S330. Whether the end screen captured in the middle is consistent. If so, processing unit 114 can report the test success.

In summary, the graphical user interface test method and system thereof according to the embodiments of the present invention can capture electronic devices through two photographic devices (for example, a first instruction and a second instruction identical to the first instruction). The two screens displayed (for example, the first screen and the second screen), and according to the two screens The similarity is used to determine whether an abnormality occurs in the graphical user interface of the electronic device. In addition, since the test device can compare only the first picture and the second picture corresponding to the same check point without comparing all the first and second pictures one by one, the efficiency of the overall graphical interface test mechanism can be effectively Improve the ground. Since the embodiment of the present invention captures each picture displayed by the electronic device through the photographing device, it has high reusability and can be used to test various electronic devices having a graphical user interface.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S210~S250‧‧‧Steps

Claims (10)

A graphical user interface test method, which is suitable for testing a electronic device by a test device, the method comprising the steps of: controlling a camera device to capture a plurality of first images displayed by the electronic device in response to a plurality of first commands, Recording the first instructions to a storage unit, wherein a difference between the ith first picture and the i-1th first picture in the first pictures is greater than a first threshold And setting a check point corresponding to the ith first picture; inputting a plurality of second instructions that are the same as the first instructions to the electronic device; and controlling the photographic device to capture the electronic device to react to the a plurality of second pictures displayed by the second instruction; when a current picture of the second pictures corresponds to the check point, calculating a similarity between the current picture and the ith first picture; And when the similarity is lower than a second threshold, stopping testing the electronic device and reporting a test error result. The method of claim 1, wherein the first instructions are a plurality of scripting languages input by the testing device to the electronic device. The method of claim 1, wherein before the step of controlling the photographing device to capture the first screens displayed by the electronic device in response to the first instructions, the method further comprises: controlling the photographing device. An initial picture is taken and an initial check point corresponding to the initial picture is set. The method of claim 3, wherein before the step of inputting the second instructions that are the same as the first instructions to the electronic device, the method further comprises: controlling the camera to capture an end screen, and An end check point corresponding to the end screen is set. The method of claim 1, wherein the checkpoint is an index value of the i-th first picture, and the current picture is one of the second pictures having the index value. . A testing device includes a storage unit for storing a plurality of modules; and a processing unit coupled to the storage unit to access the modules to perform the following steps: controlling a camera device to capture an electronic device to react to multiple a plurality of first pictures displayed by the first instruction, and recording the first instructions to a storage unit, wherein, when the ith first picture and the i-1th first picture in the first pictures are When a difference degree is greater than a first threshold value, setting a check point corresponding to the ith first picture; inputting a plurality of second instructions identical to the first instructions to the electronic device; and controlling The photographing device captures a plurality of second screens displayed by the electronic device in response to the second commands; and when a current screen of the second screens corresponds to the checkpoint, calculating the current screen and the first a similarity between i first pictures; When the similarity is lower than a second threshold, the electronic device is stopped from being tested and a test error result is reported. The test device of claim 6, wherein the first instructions are a plurality of script languages input by the test device to the electronic device. The test apparatus of claim 6, wherein the processing unit further controls the photographing apparatus to capture an initial screen and set an initial check point corresponding to the initial screen. The test device of claim 8, wherein the processing unit further controls the photographing device to capture an end screen and set an end check point corresponding to the end screen. The test apparatus of claim 6, wherein the check point is an index value of the ith first picture, and the current picture is one of the second pictures having the index value. One.