WO2022145499A1

WO2022145499A1 - User interface autonomous test system and method

Info

Publication number: WO2022145499A1
Application number: PCT/KR2020/019220
Authority: WO
Inventors: 황재준; 박현진; 이주현
Original assignee: 주식회사 앱테스트에이아이
Priority date: 2020-12-28
Filing date: 2020-12-28
Publication date: 2022-07-07

Abstract

Disclosed are a system and method for automatically testing a user interface error of a website or application without intervention of a person. For example, disclosed is a user interface system for testing an error of a user interface of an application or website, the system comprising: a testable action recognizer for obtaining a screenshot of a screen of the application or website, and managing a layout and test action, on the basis of a UI configuration and text recognition information from the screenshot; and a test action generator for receiving the layout, selecting a test scenario corresponding to the layout, and performing a test action according to the test scenario, wherein the testable action recognizer manages progressing of a test for each layout of each screen, according to the test scenario.

Description

User Interface Autonomous Testing Systems and Methods

The present invention relates to a system and method capable of automatically testing a web site or an application for errors in a user interface without human intervention.

Companies that provide IT services are testing whether the functions included in the platform are operating normally while servicing the platform as a mobile application or website, before release or before updating functions.

In the past, a large number of people directly performed tests using the development version of the platform according to various scenarios. However, as the IT field becomes more competitive, the number and types of features that a platform must include has increased significantly, while the time it takes to launch or update has to be shortened. Therefore, the manpower, time, and cost required for testing the platform is greatly reduced. An increasing problem occurred.

Accordingly, IT companies have entrusted external testing to test them individually, or developed a separate test program to perform automatic testing. However, there is a problem that human testing not only causes deviations in accuracy and time depending on the skill level, but also increases the cost exponentially as the function and application environment of the platform become more complex. In the method of developing a test program, since the test program must be modified whenever a platform change occurs, the development period not only increases, but also reports non-error as an error due to errors or insufficient implementation of the test program itself, which makes it impractical. there is a problem.

The present invention provides a system and method capable of automatically testing an application or a web site for errors in a user interface without human intervention.

The user interface autonomous test system according to the present invention is a system for testing errors on the user interface of an application or website, and obtains a screen shot of the screen of the application or website, and UI configuration and text from the screen shot a test motion recognition unit for recognizing a layout based on the recognition information and converting the layout into a test motion; and an error recognizer configured to receive the test action, select a test scenario corresponding to the test action, and perform a test action according to the test scenario, and check the test result to recognize an error, It includes an external system linkage that verifies whether the test operation is accurately reflected in the external system.

Here, the test action recognition unit includes a test action manager, and the layout includes UI element objects such as buttons and test input boxes (EditText) as a common header between screens of a single application or website. .

In addition, the test operation manager may store a layout of a corresponding screen and a necessary test operation in the table.

In addition, the test action recognition unit may recognize the layout by combining the UI element object and text recognition information of the screen and the hierarchical structure information (UI Hierarchy) of the UI elements of the screen sent by the operating system.

In addition, the test action recognition unit may further include a UI element object detection unit that learns and stores parameters of a model capable of recognizing UI element objects through machine learning in advance, and detects the UI element objects based on the parameters. can

In addition, the test action generation unit may further include an intent/object detection unit that classifies and stores an intent (intent/intent) and an entity of the screen based on information obtained through the UI element object or text information. can

In addition, the test action generating unit may further include a test persona storage that stores previously set information on the virtual person.

In addition, the test action generating unit further includes an intent/object matching unit that retrieves data of a matching type from among information set for virtual person information from the test persona storage according to a context obtained through the UI element object or text recognition information. can do.

Also, the test persona storage may store at least one virtual person having e-mail, mobile phone number, and credit card information usable for an actual test.

Also, the test issue recognizer may detect whether an error has occurred in a system log or a message on the screen.

In addition, the external system linkage unit performs an operation in the test target system, such as sending a mail or inputting a SNS comment, but the result can detect an error by verifying whether the functions reflected in the external system are correctly performed.

In addition, the user interface autonomous test method according to the present invention is a method of testing an error on a user interface of an application or website, wherein the test operation generating unit is set based on the intent in advance according to the classification of the application or website. calling a test scenario; recognizing a layout through a screen shot of the application or website by a test motion recognition unit; detecting, by the test action generating unit, a list of testable actions based on layout information: selecting the test action based on the test scenario and possible test list; generating test data by the test operation generating unit; and performing, by the test operation generating unit, a test operation on the screen of the application or website.

Here, the test operation recognition unit includes a test operation manager, and the layout is grouped into a common part (header, footer) and a variable part (body) common between screens of a single application or website, and the test operation The manager may manage whether the test is in progress in the form of a table listing test operations. In addition, the test operation manager may store a layout of a corresponding screen and a necessary test operation in the table.

In addition, the test action recognition unit recognizes the layout by combining the UI element object (Element or Widget) and text recognition information of the screen and the hierarchical structure information (UI Hierarchy) of the UI elements of the screen sent by the operating system. can

In addition, the test action recognition unit may further include a UI element object detection unit that learns and stores parameters of a model capable of recognizing UI element objects through machine learning, and detects the UI element objects based on the parameters. .

In addition, the test action generation unit further includes an intent/entity detection unit that divides and stores the screen intent (Intent) and the entity (Entity) based on information obtained through the UI element object or text information. may include

In addition, when the user test operation affects an external system, such as sending a mail, an external service integrator may further include an external service integrator that accesses the external system and checks whether the test operation actually occurs.

The user interface autonomous test system and method according to the present invention recognizes UI element objects and texts that require testing based on a screenshot image, recognizes the layout of the screen by combining UI layer information provided by the operating system, and provides a list of necessary test actions. can figure out In addition, according to the classification of applications or websites such as shopping, news, messenger, etc., the set test scenario is abstracted based on the intent (intent/intent) in advance, and the test scenario is recognized in real time to match the intent of the screen, but text, When specific data such as numbers are required to be entered, the main functions of the application or website can be tested automatically without human intervention by allowing the information to be entered automatically using the information of the test persona set in advance. In addition, even if an appropriate scenario is not established, it can recognize the intent to be tested and the object information to be entered on the screen and input the information of the corresponding test persona, so it performs more effective exploration tests than general random tests. can do. Since the screen is managed as a test action list (table) that is abstracted into intents and objects, even when there is a small design change or content change, the screen is recognized as the same screen and duplicate tests are prevented, so more functions at the same time than the random test can be tested.

1 is a block diagram of a user interface autonomous test system according to an embodiment of the present invention.

2 is a flowchart illustrating a test operation of a user interface autonomous test system according to an embodiment of the present invention.

3 is a view for explaining the detailed configuration and operation of a test operation recognition unit of the user interface autonomous test system according to an embodiment of the present invention on an exemplary home screen.

4 illustrates a UI layer information adapter of a user interface autonomous test system according to an embodiment of the present invention on an exemplary home screen.

5 illustrates an example of recognizing a UI layout by a UI layer information adapter of a user interface autonomous test system according to an embodiment of the present invention on an exemplary home screen.

6A to 6C are diagrams for explaining the detailed configuration and operation of a test operation management unit of a user interface autonomous test system according to an embodiment of the present invention on an exemplary credit card information input screen.

7 is a flowchart illustrating an operation of a test operation manager of a user interface autonomous test system according to an embodiment of the present invention on an exemplary home screen.

8 illustrates a test action table of a test action manager of a user interface autonomous test system according to an embodiment of the present invention on an exemplary home screen.

9 illustrates a test operation table of a test operation manager of a user interface autonomous test system according to an embodiment of the present invention in an exemplary credit card information input screen.

10 is a flowchart illustrating an operation of determining a next test operation as a test proceeds in the user interface autonomous test system according to an embodiment of the present invention.

11A to 11C illustrate an operation of retrieving information of an exemplary persona used in the user interface autonomous test system according to an embodiment of the present invention from the persona storage.

12 illustrates information stored by the intent/object recognition unit used in the user interface autonomous test system according to an embodiment of the present invention.

13A and 13B are diagrams illustrating a process in which the intent/object matching unit calls up persona information in the user interface autonomous test system according to an embodiment of the present invention.

14 is an example illustrating a test scenario stored in a graph structure of intents in the user interface autonomous test system according to an embodiment of the present invention.

15 is an example of a user interface autonomous test system implemented in a single desktop or server according to an embodiment of the present invention.

16 is an example of cloud-based implementation of the user interface autonomous test system according to an embodiment of the present invention.

[Explanation of major reference signs]

10; User Interface Autonomous Test System 11; test motion recognition unit

111; UI element object detection unit 112; text recognition unit

113; UI layer information adapter 114; UI Layout Recognition Unit

115; test action manager 12; issue detection unit

13; test operation generation unit 131; Intent/object recognition unit

132; intent/object matching unit 133; test persona repository

134; test scenario storage unit 14; External Service Integration Department

A preferred embodiment of the present invention will be described in detail with reference to the drawings to the extent that a person of ordinary skill in the art to which the present invention pertains can easily carry out the present invention.

Hereinafter, the user interface autonomous test system 10 according to an embodiment of the present invention will be described.

First, referring to FIG. 1 , the user interface autonomous test system 10 according to an embodiment of the present invention includes a test action recognizer 11 , an issue detector 12 , and a test action generator (Test Action Generator, 13), may include an external service integrator (External Service Integrator, 14).

The testable action recognizer (11) may determine testable actions for an application or website as a test target. To this end, the test action recognition unit 11 determines a necessary test action on the current screen by combining the user interface (UI) element object, text, and UI layer information for the current screen. Specifically, the test action recognition unit 11 includes a UI element object detection unit (UI Element Detector, 111), a text recognition unit (Text Recognizer, 112), a UI hierarchy information adapter (UI Hierarchy Information Adapter, 113), a UI layout recognition unit 114 and a test operation manager 115 .

The UI element object detection unit 111 may recognize UI elements to be tested by performing object detection from a screen shot of a currently displayed screen. In particular, the UI element object detection unit 111 may perform machine learning, particularly deep learning, and accuracy may be increased according to learning. To this end, the UI element object detection unit 111 may learn the screen shot image of the application or website collected in advance, calculate and store model parameters for object detection, and use the learned model parameter for the test target screen. Based on the UI element object type (Class) and location (Bounding Box) can be recognized.

The text recognition unit 112 may recognize and extract text from a screen shot of the current screen. In particular, the text recognition unit 112 may use an optical character recognition (OCR) technique for text extraction. Through OCR, the text recognition unit 112 can extract all texts displaying information from the current screen, and as will be described later, the test motion recognition unit 11 utilizes the information of the text to obtain a corresponding area. The value and type of data to be input can be inferred.

The UI layer information adapter 113 may extract layout information of the corresponding screen from UI layer information provided by the operating system. This UI layer information is that the operating system collects and provides the ID value or text value given to the UI element by the application or website developer to control each UI element while programming the UI, and the explanatory data entered for the visually impaired. Yes, it comes in different formats depending on the Android, iOS) and development platform (Web, App). In the present invention, the UI layer information adapter 113 is provided to convert UI layer information that comes in a different form for each development platform into an internally standardized form. Through this, when the automatic user interface test system according to the present invention is applied to a third operating system or development platform, it is possible to add only an adapter without changing the remaining components. However, since this information collects and sends the information entered by the developer in advance, and the operating system performs in the background, omission or inconsistency of information frequently occurs. For example, if the developer does not set explanatory data or text for the image, blank data will come in. If the time when the screen shot is taken and the time when the operating system provides UI layer information are different, the consistency between the two information may not match. do. FIG. 5 shows that the layout information received from the UI layer information and the configuration of the screenshot are very different in this case. Therefore, in the present invention, the UI layout can be recognized by directly recognizing the screenshot image without relying on the UI layer information provided by the operating system to obtain information and text information of the UI element object.

The UI layout recognition unit 114 combines the UI element object detected by the UI element object detection unit 111, the text extracted from the text recognition unit 112, and the layer information identified from the UI layer information adapter 113, The layout of the screen being displayed can be determined. In addition, the UI layout recognition unit 114 divides the layout of the corresponding screen into a common part (header, footer) and a variable part (body) area shared by screens of the same application or website, and suppresses repeated testing of the common part. . In addition, the same unit of information for the variable body (for example, if it is a birthday), the year, month, and day are divided into one group, and the test action of entering the birthday is uniformly consistent for the year, month, and day 3 UI element objects. It is guaranteed to be performed through input of test data.

The test operation manager 115 calculates the necessary test operation in a table form based on information on the type (Class), text (Value), and location (Bounding Box) of each UI element included in the layout input by the UI layout recognition unit 114 . It is possible to organize into a list of , and record and manage which test data is generated and the test is performed by the test operation generating unit 13, which will be described later. The test action manager abstracts and manages the screen as a test action standard to prevent duplicate tests of the same test action on the same screen. For example, in the case of the initial screen of a shopping mall, the text values of UI elements are changed every time the initial screen is accessed as the products to be sold are changed, so that they are recognized as a different screen and the product in the same location is selected (e.g., click) ) can be repeated indefinitely. After configuring the test action table for each screen, the text action manager compares the previously recorded test action table in consideration of the type of each UI element to prevent duplicate creation and management of the test action table for the same screen. For example, when UI components of the same type are arranged in the form of a list or grid, the text information of each UI element is excluded and only the arrangement type is compared to determine whether it is the same screen. Due to the present test operation manager 114 , the user interface autonomous test system 10 according to an embodiment of the present invention can perform a test without omitting or repeating a test operation.

The Issue Detector 12 may detect errors or performance problems by monitoring log records, CPU, memory, network status, etc. of the device under test. For example, when an error display appears on the screen or when an application dies log (Log) is detected, the issue detection unit 12 reports screen information where an error occurs, input data used at that time, and error log and message information to the developer.

The test action generator 13 receives the test action list of the corresponding screen from the test action recognition unit 11 described above, and generates input data required to test the screen in an appropriate format. to provide. To this end, the test action generation unit 13 may include an intent/object recognition unit 131 , an intent/object matching unit 132 , a test persona storage 133 , and a test scenario storage 134 .

First, the intent/entity recognizer (Intent/Entity recognizer, 131) may recognize texts recognized from the screenshot of the corresponding screen as intents and entities again. Intent and entity are technical terms commonly used when modeling conversational systems such as chatbots. In general, intent is a unit of conversation in which words come and go or the speaker's intention to convey through conversation. can be explained, and the entity can be described as a specific requirement among the user's intentions. For example, in the sentence "Please select a credit card type", the intent is "Please select" and the entity is "credit card type". The intent/object recognition unit 131 may be based on a natural language processing (NLP) technology. Example For example, when there is an option selection UI element (Radio) of 'A card', 'B card', and 'C card' on the screen, if the text at the nearest position is 'Select payment method', the intent/object recognition unit The 131 may recognize 'select credit card' as an intent, and recognize 'A card', 'B card', and 'C card' as entities.

The intent/object matching unit 132 may convert the test actions of the corresponding screen into intents and objects and store them in the test action table. If the intent/object matching unit 132 is used, a test target screen can be defined based on an abstracted intent instead of a specific terminology system used by each application or website. Through this, it is possible to test while recognizing whether the defined test scenario and the current test screen match in real time based on the intent, so you can have a generalized test scenario designed based on intent without writing each test scenario for each application or website. You can test applications or websites of the same category.

On the other hand, the test action generating unit 13 selects the highest priority intent based on the test action table managed based on the intent of the intent/object matching unit 132 and performs an action for executing it. You can choose. For example, when a purchase scenario through bank transfer is in progress, there are two intents: 'Select credit card' and 'Input bank account number' on a single test screen. , the test action generation unit 13 selects the 'Deposit bank account input' intent instead of the 'Payment method selection' intent, and generates data related to this intent.

When the user interface autonomous test system 10 according to the embodiment of the present invention operates, the Test Persona Repository 133 may set and store information of a virtual person having the same conditions as ordinary people. For example, the test persona storage 133 virtually creates a 28-year-old person with the name 'John Doe', and includes information such as job, email, address, mobile phone number, credit card information, chatting app, and SNS can be preset. In particular, information on e-mail and mobile phone numbers can be used to authenticate the user in a specific application or website, and credit card information is frequently required for testing of a purchase process of a paid item or testing for online shopping. Therefore, in order to prevent errors due to input data during testing, the test persona storage 133 actually uses e-mail, mobile phone number and/or credit card information when creating/storing virtual person information in advance. It is desirable to set it as possible. Most modern applications or websites do not operate alone, but are utilized in connection with various external systems. Therefore, in order to test this linkage, there must be an external system account, and the test operation result must be connected to the system and checked. The test persona of the present invention is different from the existing methods in that it has not only data for testing the corresponding system, but also information and settings for testing this external system linkage.

The test scenario storage 134 may store a scenario to be tested through the user interface autonomous test system 10 according to an embodiment of the present invention, that is, a test path. The test scenario may be set according to the classification of the application or website. For example, all online shopping applications use the test scenario as a series of actions (intents) to search for and select an item from the main home screen, enter personal information and address to receive delivery, and proceed with payment through a credit card. configurable. The classification of the test target application or website can be obtained by scraping this site information if the application has already been published in the App Store, or can be specified by the developer directly before the start of the test, and can be inferred through the intent information recognized in the initial screen. may be

The external service integrator 14 is configured to verify whether the operation is normally performed in the external system when the test operation of the user interface autonomous test system 10 according to the embodiment of the present invention occurs in the external system. do. For example, when there is a 'post' button on the screen on social media (e.g. Facebook), after performing this test action, check whether a new post is made on the initial screen of the SNS owned by the test persona, and if there is no posting, an error can report.

As described above, the user interface autonomous test system 10 according to an embodiment of the present invention combines UI configuration, text recognition, and UI layer information based on a screen shot for an application or website for testing, and provides layout and You can understand the test behavior. In addition, according to the classification of the application or website, the test is conducted according to the predefined test scenario based on the intent, but if data input such as text rather than a simple click or touch is required, it can be entered using the information of the test persona. can In addition, each test path and test progress is organized into an abstract test action table from which the screen design or content information is removed, so that the test can be performed accurately and quickly while minimizing duplicate tests.

Hereinafter, a test method using the user interface autonomous test system 10 according to an embodiment of the present invention will be described in stages.

2 is a flowchart illustrating a test operation according to a user interface autonomous test system according to an embodiment of the present invention. Referring to FIG. 2 , the test method using the user interface autonomous test system 10 according to the embodiment of the present invention includes device setup (S1), application installation (S2), general-purpose test scenario call (S3), and application screenshots. It may include steps of recognition (S4), operation selection (S5), test data generation (S6), test operation execution (S7), and test operation result verification (S8). Hereinafter, each step of FIG. 2 will be described with reference to FIGS. 3 to 15 together.

First, in the device setup (S1) step, for the operation of the user interface autonomous test system 10 according to the embodiment of the present invention, a test environment of the test device may be configured. Specifically, an operation such as selecting an operating system version of the system 10 test device and installing an application that may conflict with a corresponding application or needs to be installed first may be performed in advance. For example, if you want to test the bank app, first install the anti-virus app, which is an essential app, and then install the Facebook app with the account the test persona has and log in to prepare for the Facebook integration test in the bank app.

In the application installation (S2) step, an application to be tested may be installed on the test device. In this step, it is also possible to receive the installation file before application distribution from the developer of the company requesting the test and install it on the test device. It is also possible for general users to download and install the installation file by accessing the App Store where they download applications from their smartphones or tablets. In the case of a website, it is used on the website like ActiveX, and data files necessary to run the website, such as programs and public certificates, are installed.

In the general-purpose test scenario loading (S3) step, from among the test scenarios stored in the test scenario storage 134, an appropriate one for the corresponding application or website may be selected and loaded. As described above, when the classification of the application has already been published in the App Store, a test scenario suitable for the classification can be easily selected. If necessary, it is possible to set an appropriate test scenario by receiving the details of the application from the developer, and it is also possible to call the scenario after inferring the classification from the intent of the initial screen after running the application.

In the step of recognizing an application screenshot (S4), starting from the home screen initially displayed when each application or website is executed, a screen shot may be acquired and recognized step by step according to the test scenario. In this step, the screenshot may be transmitted to the test motion recognition unit 11 together with UI layer information of the corresponding screen.

The operation selection (S5) step will be described with reference to FIG. 3 . In the case of the shopping application illustrated in FIG. 3 , it is a screen composed of a simple content selection (Click) test operation. As described above, the UI element object detection unit 111 detects the UI configuration based on the existing deep learning from the screenshot of the home screen, and the text recognition unit 112 may also recognize the text of each area from the screenshot. . Finally, the UI layer information adapter 113 creates a screen layout using UI layer information provided by the operating system. The three screen information is divided into a common part (Header, Footer) and a variable part (Body) after being integrated by the UI layout recognition unit 114 .

In the test data generation (S6) step, the test action generation unit 13 may retrieve information suitable for the test action from the test persona storage 133 . As described above, due to the characteristics of an application or web site, there are cases in which a specific operating system or actually operating e-mail, mobile phone number, and credit card information are requested. Accordingly, the test operation generating unit 13 may generate input data according to the test scenario by retrieving information corresponding thereto from the storage.

In the test operation execution (S7) step, the test operation generation unit 13 may perform operations, such as inputting data or clicking on the screen, by transferring the operations to the operating system.

Finally, in the step of verifying the test operation result (S8), the message or system log of the application or website is verified before or after the test operation is performed to detect whether an error has occurred and a report to be delivered to the developer is stored. When it is necessary to check the test operation execution result in an external system, such as sending an email or inputting an SNS comment, it can be verified by accessing the external system through the external service integrator 14 as described above.

Meanwhile, FIGS. 6A to 6C are diagrams for explaining the detailed configuration and operation of the test motion recognition unit of the user interface autonomous test system according to the embodiment of the present invention on the personal information input screen, unlike the relatively simple example of the shopping mall home screen of FIG. 3 . . Referring to FIG. 6A , as described above, as the test progresses, the UI layout recognition unit 114 forms a layout as shown in FIG. 6A by combining the authorized UI element object, text, and UI hierarchical structure, and the layout After first grouping each of the components into a common part (Header, Footer) and a variable part (Body), the UI components in the variable part can be grouped again as an information integrity unit. In this example, the variable part was again divided into four groups: Card Number, Name on Card, Expiration date, and Operation button (Add your Card).

At this time, the intent/object recognition unit 131 , as shown in FIG. 6B , based on the UI configuration detected by the UI element object detection unit 111 and the text recognized by the text recognition unit 112 , in advance Based on the learned deep learning-based machine learning parameters, the intent and object of the corresponding area can be recognized. For example, when four text input UI elements (EditText) are on the same horizontal line as shown in FIG. 6B, the intent/ The entity recognition unit recognizes the definition and format of an entity called a card number consisting of an intent of inputting a credit card number and four input values.

Using this, the test action generating unit 13 may query and call information of an actually inputable card from the test persona storage 133 as shown in FIG. 13A, and each UI element of the test action table as shown in 13B Allocate the card number to be input to so that the test is performed.

Hereinafter, the operation of the test operation manager in the user interface autonomous test system according to an embodiment of the present invention will be described in detail.

7 is a flowchart illustrating an operation of a test operation manager of a user interface autonomous test system according to an embodiment of the present invention on an exemplary home screen, and FIG. 8 is a user interface autonomy according to an embodiment of the present invention on an exemplary home screen It shows the test action table of the test action manager of the test system. In the execution of the test operation, the test operation manager 115 extracts a test operation table using information of the type (Class) and displayed value (Value) of the UI element object as shown in FIG. 12 ( 200 ), and the table By comparing the information with the previously tested test action table, it is determined whether actions have already been tested on the screen ( 202 ). In this case, a separate test session storage 201 may be provided. In this case, the test operation manager 115 may determine whether or not the test is in progress with reference to the test session storage 201 . In addition, when the screen is tested (Yes), the test action manager 115 may record that the screen has been tested (Yes) in the test action table as shown in FIG. 8 , and specific test action details ( For example, a touch, a horizontal scroll, a vertical scroll) may be recorded together ( 203 ). On the other hand, if the corresponding screen is not tested (No), the test action manager 115 records that it has not been tested (No) in the test action table as shown in FIG. can do. (204).

In this way, the test operation manager 115 organizes the test progress in the form of a table as shown in FIG. 8 or 9 when testing each screen according to the test scenario, and after recognizing each screen, it is created and stored in the past. It is possible to prevent repeated tests from being performed by comparing the existing tables and integrating them if they are the same.

Meanwhile, FIG. 8 illustrates a test operation table of the test operation manager of the user interface autonomous test system according to an embodiment of the present invention in an exemplary personal information input screen. As shown in FIG. 8 , in the test operation table, when a card input is accompanied, it is also possible to display that a card number is input in addition to an existing touch operation.

Meanwhile, FIG. 10 is a flowchart illustrating a process of determining a next test operation as a test proceeds in the user interface autonomous test system according to an embodiment of the present invention. As illustrated in FIG. 10 , when determining each intent to be tested, the test operation manager 115 may check whether there is an untested intent on the corresponding screen ( S71 ). If there is (Yes), the test operation manager 115 may check whether there is a scenario being tested (S72). Also, if the scenario exists (Yes), it is checked whether there is an intent matching the scenario (S73), and if there is (Yes), the operation of selecting an intent according to the test scenario is continuously performed. However, if there is no intent that matches the scenario, there is no way to proceed with the preset test scenario, so the information indicating that the test scenario is in progress is reset (Reset) and the next intent to be tested based on the location information in the screen of each intent to decide For example, if the payment scenario in the shopping mall has progressed to credit card input and no possible intent is found in the payment scenario such as payment or delivery address modification on the next screen, the payment scenario cannot be continued, so you can initialize all the scenario progress. Guaranteed to go through the payment scenario all over again.

Meanwhile, if there is no scenario being tested among the untested intents, it may be checked whether there is an untested scenario starting with the corresponding intent ( S74 ). If it exists (Yes), start the scenario started with that intent. For example, if an intent such as add to cart is found on the screen while the payment scenario has not been performed, select add to cart as the next test action and indicate that the testing of the payment scenario has started. If an appropriate scenario does not exist (No), another intent may be selected based on the location in the screen as well.

On the other hand, when there is no untested intent on the screen, it is checked whether there is another test path performed with intents in the screen (S75), and if there is (Yes), the corresponding intent can be selected again. On the other hand, if there is no other test path (No), check if it is the home screen (S76), and if it is not the home screen (No), restart and return to the home screen to proceed with the test. As the test for the path is completed, the test ends.

Hereinafter, an example of data stored in a test persona storage and a method of efficiently expanding persona data among the user interface autonomous test system according to an embodiment of the present invention will be described.

As shown in FIG. 11A , in the test persona storage 133, for example, a virtual person named 'John Doe' may be stored, and the person has a job, email, address, mobile phone number, credit card information, chatting. Information such as apps and SNS may be set. In addition, the test persona storage 133 may store at least one such virtual person, preferably a plurality, and appropriate virtual person information may be utilized according to a required test scenario.

11B , in the test persona repository 133 , an RDF repository (RDF Repository, 133a) and an RDF inference engine (RDF Inference Engine, 133b) may be connected in the background. Here, RDF (Resource Description Framework) is a method of managing information in three tuples such as <subject, predicate (relationship), object>, and can be replaced with a similar information management method. The RDF repository (RDF Repository, 133a) refers to a repository that collects and stores virtual person data and external ontology for testers written by itself, and the RDF inference engine (RDF Inference Engine, 133b) performs final testing of RDF. It can perform the role of converting data of the format such as <Key, Value> that is required for <Name, 'John Doe'>. In this process, the reasoning engine plays the role of information inference to create a new relationship by integrating external and internal data, and uses synonyms or language mapping information to generate more information than input data and relationships through data transformation. can For example, by combining internal information of <'John Doe', 'Company Name', 'Facebook'> with information collected from the external Internet of <'Facebook', 'address', 'S California Avenue, Palo Alto'>, You can create information such as <'John Doe', 'Company Address', 'S California Avenue, Palo Alto'>.

As shown in FIG. 11C , the test persona storage 133 stores data converted from the RFD inference engine 133b in the form of a table. Accordingly, data required according to the test scenario may be provided to the test operation generating unit 13 in the form of a corresponding table.

Hereinafter, the storage table of the intent/object recognition unit will be described as an example.

12 illustrates a table in which the intent/object recognition unit stores information of persona used in the user interface autonomous test system according to an embodiment of the present invention.

As shown in FIG. 12 , the intent/object recognition unit 131 may classify intents/objects based on the UI configuration or OCR information identified from the screenshot, and may organize them as a table. Also, since the corresponding table may be shared with the test operation manager 115, whether an intent is tested through the corresponding table may also be recorded.

Hereinafter, the process of calling the persona information by the intent/object matching unit will be described.

As shown in FIG. 13A , according to a test scenario, the intent/object matching unit 132 may communicate with the test persona storage 133 and retrieve persona data that meets a required format. At this time, in the test persona storage 133, for example, in the case of a card number, it may be composed of a combination of 16 numbers of the card number and 3 digits of cvv, but in order to fit the input format of various applications, each one-digit block, 4 It may be stored in various forms such as a digit block, a 2-digit block, a 5-digit block, and the like, and the intent/object matching unit 132 retrieves data in a form suitable for the application.

Hereinafter, an exemplary test scenario management method will be described.

14 is an example illustrating an intent-based test scenario managed by the user interface autonomous test system according to an embodiment of the present invention in a graph structure.

Referring to FIG. 14 , it can be understood that there are various paths from the initial 'home' screen to the 'review the order' screen where the payment test is completed. For example, in case of payment by credit card, the necessary test actions are sequentially 'View content' -> 'Place an order' -> 'Enter credit card' -> 'Place an Order'. Test Scenario A general-purpose test scenario independent of an application can be defined by managing the sequence of intents required to switch to the next screen. In a general-purpose test scenario, a test scenario is determined and executed by recognizing the intent of the screen to be tested in real time during testing.

Hereinafter, an implementation example of a user interface system according to an embodiment of the present invention will be described.

15 is an implementation example of a user interface autonomous test system according to an embodiment of the present invention. 16 is another implementation example of a user interface autonomous test system according to an embodiment of the present invention.

As shown in FIG. 15 , the user interface system 10 may be provided in the form of a separately provided and combined device in the test device. In this case, the user interface system 10 may include a configuration of CPU, GPU, storage, and memory, and may include an I/O device interface for connection with a test device.

Here, the logic and all data referenced by the logic may be stored in the storage, but the logic necessary for the operation may be loaded back into the memory. In addition, operations such as object detection, OCR, and object recognition may be processed by the GPU, but in some cases, the CPU may be responsible for most of the logic including this.

Meanwhile, as shown in FIG. 16 , the user interface autonomous test system may be implemented in the form of a cloud.

As shown in FIG. 16 , there is a test device to install an application at the bottom of the configuration, but it is possible to substitute an emulator or a VM instead of an actual device.

In addition, in the cloud-type implementation, each agent that is directly connected to the device through the device connector and performs a test is called a testbot. The testbot installs the application, collects the screen shot of the application, accessibility information, etc., sends it to the AI TestBot Engine, and then decides the AI TestBot Engine and then executes the test action. In addition, when a user's test request is received, the testbot controller can allocate a testbot and monitor the test execution.

In addition, in the cloud form, multiple users share a device, so the same type is usually distributed and deployed on multiple servers. You can select fewer and assign test tasks to them. In this case, the user can select a model and request a test regardless of where the device he is using is actually connected.

On the other hand, the AI TestBot Engine can receive the screen shot information and log information of the application sent by each testbot, determine the next test action, and perform all actions to detect errors.

What has been described above is only one embodiment for implementing the user interface autonomous test system and method according to the present invention, and the present invention is not limited to the above embodiment, and as claimed in the claims below, the present invention Without departing from the gist of the present invention, it will be said that the technical spirit of the present invention exists to the extent that various modifications can be made by anyone with ordinary knowledge in the field to which the invention belongs.

The present invention provides a system and method capable of automatically testing a web site or an application for errors in a user interface without human intervention.

Claims

In the system for testing errors on the user interface of the application or website,

a test operation recognition unit that acquires a screenshot of the screen of the application or website, recognizes a layout based on UI configuration and text recognition information from the screenshot, and converts it into a test operation and manages; and

and a test action generator configured to receive the test action, select a test scenario corresponding to an intent of the test action, and perform a test action according to the test scenario;

The test operation recognition unit manages whether or not a test is performed according to the layout of each screen according to the test scenario.
The method of claim 1,

The test action recognition unit includes a test action manager,

The layout is grouped into a common part (Header, Footer) and a variable part (Body) shared by screens of an application or website,

The test operation manager manages whether the test is in progress through a table.
3. The method of claim 2,

The test operation manager converts a corresponding layout into a test operation in the table and stores it together.
The method of claim 1,

The test motion recognition unit

A system for recognizing the layout by combining the UI configuration and text recognition information with UI layer information provided by an operating system.
The method of claim 1,

The test operation recognition unit further comprises a UI element object detection unit for learning and storing a model parameter for recognizing a UI element object through machine learning, and detecting the UI element object based on the parameter.
The method of claim 1,

The test operation generating unit

The system further comprising an intent/object detection unit for storing the text acquired through the UI configuration or text recognition information by dividing the text into intents and objects.
The method of claim 1,

The test operation generating unit

The system further comprising a test persona repository for pre-stored information of the preset virtual person.
8. The method of claim 7,

The test operation generating unit

The system further comprising an intent/object matching unit for retrieving information of a virtual person in which data in a matching format is set from the test persona storage according to the text acquired through the UI configuration or text recognition information.
8. The method of claim 7,

The test persona repository is

A system for storing at least one virtual person with email, mobile phone number and credit card information usable for real-world testing.
The method of claim 1,

A system for verifying the result of the user interface test operation by accessing an external service using the test persona account information and detecting an error.
A method of testing an error for a user interface of an application or website, the method comprising:

calling a test scenario for the application or web site by the test operation generating unit;

recognizing a layout through a screen shot of the application or website by a test motion recognition unit;

selecting, by the test operation generation unit, the test operation according to the test scenario and layout;

generating test data by the test operation generating unit; and

and performing, by the test operation generating unit, a test operation on the screen of the application or website.
12. The method of claim 11,

The test action recognition unit includes a test action manager,

The layout is grouped into a common part (Header, Footer) and a variable part (Body),

The test operation manager manages whether the test is in progress through a table.
13. The method of claim 12,

The test action manager stores the corresponding layout, test action, and intent together in the table, and prevents duplicate testing even when there is a change in the design or value of the screen based on this abstract test action information.
12. The method of claim 11,

The test motion recognition unit

A method of recognizing the layout and test operation by combining the UI element object and text recognition information with UI layer information provided by an operating system.
12. The method of claim 11,

The test motion recognition unit

The system further comprising a UI element object detection unit for learning and storing a model parameter for recognizing a UI element object through machine learning, and detecting the UI element object based on the parameter.
12. The method of claim 11,

The test operation generating unit

The method further comprising an intent/object detection unit for storing the text acquired through the UI configuration or text recognition information by dividing the text into an intent and an object.
12. The method of claim 11,

The test operation generating unit

A method further comprising a test persona repository for pre-storing information of a preset virtual person.
18. The method of claim 17,

The test operation generating unit

The method further comprising an intent/object matching unit for retrieving information of a virtual person in which data in a matching form is set from the test persona storage according to the text acquired through the UI configuration or text recognition information.
18. The method of claim 17,

The test persona repository is

A method of storing at least one fictitious person with email, mobile phone number and credit card information usable for real-world testing.
12. The method of claim 11,

A method of verifying a result of the user interface test operation by accessing an external service using test persona account information and detecting an error.