KR102153811B1 - Method and apparatus for prioritizing web browser environment for testing web application - Google Patents

Abstract

A method and apparatus for determining the priority of a web browser environment for web application testing are presented. The method for determining the priority of a web browser environment for testing a web application according to an embodiment uses information on a failure history related to a web browser environment in the web application when performing a regression test on a client-side web application, and And determining the priority of the web browser environment by using information on failure of at least one of the information on the recently failed web browser environment and information on the frequently failed web browser environment.

Description

Method and device for determining the priority of web browser environment for web application testing {METHOD AND APPARATUS FOR PRIORITIZING WEB BROWSER ENVIRONMENT FOR TESTING WEB APPLICATION}

The following embodiments relate to a method and apparatus for determining the priority of a web browser environment for testing a web application.

Modern client-side web applications are becoming increasingly complex and error prone. Failure of these applications may appear differently in various combinations of web browsers, their versions, and base operating systems, and occurs because some combinations of browsers may not support specific functions required by client-side web applications. Can be.

Appropriate testing is required to ensure that such configuration-related failures do not affect the use of the web application. Since unexpected failures may occur in previously tested functions due to new or changed code, more testing is required, especially as web applications evolve. For this reason, web application regression test technology has been actively studied. Such technologies include test cases that are automatically generated from previous versions of the web application or approaches for Oracle, remediation of invalid test cases after web application updates, selection or prioritization of subsets of regression test suites, and augmentation test suites. . However, none of these tasks take into account the additional problems that arise when regression tests are performed in different web browser environments (hereinafter referred to as browser environments), such as environments in which different web browsers and operating systems are used.

When a developer modifies a web application that is intended to function in different browser environments, a lot of time and computing resources may be required for regression testing of the web application. To date, there are over 100 different web browsers. Each web browser can have many different versions simultaneously in the field, and each version can run on a wide range of different operating systems on different computing devices. This causes a problem of scale for regression testing of web applications. Prolonging the regression test time delays the feedback that can be provided to the developer about the web browser's mating failure, delaying the debugging of the failure and the release of a new version.

Mootools 2018. Mootools. http://mootools.net. (2018). Mootools-core in Travis CI 2018. https://travisci. org/mootools/mootools-core. (2018). Sauce Labs 2018. Sauce Labs. https://saucelabs.com. (2018). Wikipedia. 2017.Jaccard index--Wikipedia, The Free Encyclopedia. (2017).

The embodiments describe a method and apparatus for determining the priority of a web browser environment for testing a web application, and more specifically, provide feedback on failure to a developer when regression testing a client-side web application in various web browser environments. It provides the technology to prioritize the web browser environment that can be done.

The method for determining the priority of a web browser environment for testing a web application according to an embodiment uses information on a failure history related to a web browser environment in the web application when performing a regression test on a client-side web application, and And determining the priority of the web browser environment by using information on failure of at least one of the information on the recently failed web browser environment and information on the frequently failed web browser environment.

The step of determining the priority of the web browser environment includes information on the recently failed web browser environment at least one of a type of a web browser, a name of a web browser, a version of a web browser, and a type of an operating system stored in a cache. The web browser environment, which is scheduled according to the information in the cache, and has recent failure information, may be assigned a higher priority than other web browser environments.

The step of determining the priority of the web browser environment, wherein the cache lists information on the web browser environment that failed recently, and a higher priority when the web browser environment of the test target build is included in the cache. Allocating; And scheduling the plurality of web browser environments in the same order or randomly.

In the step of determining the priority of the web browser environment, the cache lists information on the web browser environment that has recently failed, and between each web browser environment in the test target build and each web browser environment in the cache. Calculating similarity; And scheduling each web browser environment in the test target build in descending order according to the calculated similarity.

The determining of the priority of the web browser environment may include: acquiring a failure frequency of a web browser environment in which the regression test frequently fails; And scheduling in descending order according to the failure frequency of the web browser environment.

The step of determining the priority of the web browser environment is to automatically learn the failure pattern of the web browser environment in which the regression test frequently fails based on machine learning (ML), and the web browser environment fails in subsequent builds. Providing a probability; And scheduling the web browser environment in descending order according to a probability of failure.

The step of determining the priority of the web browser environment, wherein the cache lists information on the web browser environment that failed recently, and a higher priority when the web browser environment of the test target build is included in the cache. Allocating; Obtaining a failure frequency of a web browser environment in which the regression test frequently fails when the plurality of web browser environments have the same priority; And scheduling in descending order according to the failure frequency of the web browser environment.

The step of determining the priority of the web browser environment, wherein the cache lists information on the web browser environment that failed recently, and a higher priority when the web browser environment of the test target build is included in the cache. Allocating; If the priority of multiple web browser environments is the same, it automatically learns the failure pattern of the web browser environment that frequently failed the regression test based on machine learning (ML), and the probability of failure in the web browser environment in subsequent builds. Providing a; And scheduling the web browser environment in descending order according to a probability of failure.

Further comprising the step of performing the regression test on the web browser environment according to the determined method of determining the priority of the web browser environment, wherein the step of performing the regression test is a regression test result of the first failed build Building a data structure and a machine learning (ML) model for a cache, a failure frequency based on the data; Determining a priority of the web browser environment of the test target build according to the priority generated by the determined priority of the web browser environment; Measuring a regression test result value for the test target build; And updating the cache, the data structure for the failure frequency, and a machine learning (ML) model based on the regression test result of the test target build when the regression test fails in some builds other than all web browser environments. can do.

Determining the priority of the web browser environment of the test target build according to the generated priority may include determining a random priority when a plurality of web browser environments have the same priority.

The apparatus for determining the priority of a web browser environment for testing a web application according to another embodiment uses information on a failure history related to a web browser environment in the web application when performing a regression test on a client-side web application, and And a priority determining unit that determines the priority of the web browser environment by using information on failure of at least one of the recently failed web browser environment information and the frequently failed web browser environment information.

And a cache for storing information on the recently failed web browser environment at least one of a type of a web browser, a name of a web browser, a version of a web browser, and a type of an operating system, and the priority determining unit includes: A web browser environment scheduled according to information and having recent failure information may be assigned a higher priority than other web browser environments.

The cache further includes a cache hit check unit for listing information on the recently failed web browser environment, and assigning a higher priority when the web browser environment of the test target build is included in the cache, and the priority When the priorities of the plurality of web browser environments are the same, the priority determining unit may schedule an original order or randomly.

The cache further includes a similarity calculation unit for listing information on the recently failed web browser environment, and calculating a similarity between each web browser environment in the test target build and each web browser environment in the cache, and the priority The ranking unit may schedule each web browser environment in a test target build in descending order according to the calculated similarity.

A failure frequency obtaining unit for obtaining a failure frequency of a web browser environment in which the regression test frequently fails, and the priority determining unit may schedule in a descending order according to the failure frequency of the web browser environment.

Further comprising a machine learning model unit that automatically learns a failure pattern of a web browser environment that frequently fails the regression test based on machine learning (ML), and provides a probability of failure to the web browser environment in a subsequent build, the The priority determining unit may schedule the web browser environment in descending order according to a probability of failure.

According to embodiments, a method and apparatus for prioritizing a web browser environment for testing a web application that can provide feedback on failure to a developer more quickly when performing a regression test on a client-side web application in various web browser environments are provided. I can.

According to the embodiments, the cost is reduced because code coverage information is not required, and the time interval at which the regression test is executed in a continuous integration (CI) environment is short, so it is effective to support execution. to be.

1 is a diagram for explaining the effect of determining the priority of a web browser environment of a web application according to an exemplary embodiment.
2 is a flowchart illustrating a method of determining a priority of a web browser environment for testing a web application according to an embodiment.
3 is a block diagram illustrating an apparatus for determining priority of a web browser environment for testing a web application according to an exemplary embodiment.
4 is a diagram for describing a method of determining an accurate matching-based priority according to an embodiment.
5 is a diagram for describing a method for determining a similarity matching-based priority according to an embodiment.
6 is a diagram illustrating a method of determining a failure-frequency-based priority according to an embodiment.
7 is a diagram for describing a method of determining a machine learning-based priority according to an embodiment.
8 is a diagram for describing a method of determining a hybrid priority according to an exemplary embodiment.
9 is a flowchart illustrating a method of performing a regression test according to an exemplary embodiment.
10 shows data on an APFD _C value according to a priority determination method according to an embodiment,

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more completely explain the present invention to those of ordinary skill in the art. In the drawings, the shapes and sizes of elements may be exaggerated for clearer explanation.

The following embodiments may provide a technology for prioritizing a web browser environment capable of providing feedback on failure to a developer more quickly when performing a regression test on a client-side web application in various web browser environments.

Embodiments may use information about a failure history related to a browser environment in an existing application of a regression test. Two technologies according to embodiments utilize recent failure information such as the type of web browser, browser version, and type of operating system stored in the cache, and the test execution environment may be scheduled according to the cached information. Browser environments with recent failure information have a higher priority than others and can be tested quickly. The other two techniques are based on browser environments that generally fail (common failure), and in prior builds, failure-frequency-based, which gives a greater priority to browser environments where regression tests fail multiple times. Approaches and machine learning-based approaches that automatically learn failure patterns and predict browser environments that are likely to fail web application tests can be proposed. Finally, another two techniques can take into account information about both the recent failure and the environment of general failure. The technology first schedules browser environments based on cached information, and can apply a failure-frequency-based or machine learning-based approach to ordering in the case of browser environments with the same priority.

There are several advantages to the prioritization technique according to the embodiments. First, the embodiments can be applied to modern web applications. Since modern web applications are written in a mixture of languages such as HTML, JavaScript, and CSS, it is difficult and expensive to obtain code coverage for test cases, which is generally required by the existing test case prioritization method. It costs a lot. However, the priority determination technique according to the embodiments does not require code coverage information. Second, the embodiments are particularly effective in supporting continuous integration (CI) execution. In a continuous integration (CI) environment, the time interval at which regression tests are executed is short. Developers frequently check their code in the mainline codebase, and regression tests related to the code must be performed in the browser environment. Code coverage calculation techniques cannot keep up with the pace of change occurring in this process.

Here, we have empirically studied techniques according to embodiments for four popular open source web applications that are non-trivial. The results show that the technology according to the embodiments can be cost-effective. The approach according to the embodiments can detect more failures generally faster than the two criteria approach in which the browser environment is not prioritized (no ordering) and random ordering (random ordering). In addition, we compared the six priority techniques and analyzed which technique is more cost-effective than the other techniques for each experimental object. This analysis can suggest prioritization techniques that developers should first apply in web application testing. In addition, some practical issues have been addressed with the use of techniques according to embodiments in industries such as prioritization time overhead and parallel test execution.

Below, we first investigate the process of determining the browser environment prioritization of the web application, and propose four new techniques to determine the priority of the web browser environment for the regression test. It depends on the test results of the previous regression test session. A number of actual web applications are used to report the result of empirical evaluation of the technology according to the embodiments, and as a result, the cost efficiency of the technology according to the embodiment is shown.

To motivate more of this work, we present the results of an initial investigation into whether scheduling the web browser environment affects the cost-effectiveness of regression testing for web applications. For this analysis, we considered Mootools (Non-Patent Document 1), a popular open source web application that manipulates DOM (Document Object Model) elements and handles events. In order to secure the web browser environment and measure the time required to test the application in each environment, the previous Mootools build history was searched in Travis CI (Non-Patent Document 2), which stores test results for Mootools. The build history used started on March 16, 2014 and ended on February 9, 2015. Exclude build history not directly related to this investigation. For example, some build records do not include test results due to compilation failures, and some build records do not provide information about the failed web browser environment. Excluding those build histories, 10 are left.

For each build, the number of tested web browser environments was calculated. The median, minimum, and maximum values of the entire build were 23.5, 22.0, and 24.0, respectively. Each web browser environment contains several attributes, which are browser type and version, operating system and version, and build type ("default" and "nocompat"). Here, the terms "default" and "nocompat" indicate whether the application is compatible with the previous version. There are 5 browser types, but the number of environments can be increased to 24 by changing other properties of the web browser environment.

One common strategy to shorten the regression test time across the entire execution environment is to run the test suite in each environment simultaneously. Popular cloud-based testing services such as Sauce Labs (Non-Patent Document 3) and BrowserStack support this approach, and Mootools uses Sauce Labs. In this case, the test suite can be run simultaneously, but the total test time required can vary depending on the number of environments to be tested. If the number of test environments is greater than the number of resources available in parallel, significant time may be required to detect failures that can only be found in a particular environment. Increasing the number of available resources can solve this problem, but it does incur additional costs. If there are many web applications to manage, it will become more important to use computing resources efficiently by preventing each web application from consuming excessive resources. For similar reasons, existing research on test case prioritization highlights the importance of scheduling test cases in a concurrent environment to improve the cost effectiveness of regression testing.

1 is a diagram for explaining the effect of determining the priority of a web browser environment of a web application according to an exemplary embodiment.

Referring to FIG. 1, two different schedules (original schedule and optimal schedule) of Mootools' web browser environment are compared, and it is measured whether the optimal schedule has improved the cost efficiency of the regression test. can do. The original schedule included using the test schedule recorded in Mootools' build history, while the optimal schedule would allow all web browser environments with observed failures to be deployed ahead of those with no observed failures. . You can then extract the execution time ("job time" from Sauce Labs) for each browser environment from the build history. Next, the time when the web browser environment including at least one failure test case is reported may be calculated. You can then compare the optimal schedule and the calculated time ("feedback time") reported against the original schedule.

In all 10 builds, the optimal schedule can provide feedback on average 315.12 seconds faster than the original schedule. As shown in FIG. 1, when the browser environment used for the build ID 200 is scheduled, the feedback time obtained for each schedule may be displayed. The optimal schedule is indicated by a solid line, and the schedule for which no ranking has been determined is indicated by a dotted line. Five failures occurred in 24 browser environments. The x-axis represents each failed browser environment, and the y-axis represents the feedback time associated with each failed browser environment. The optimal schedule was 153 seconds, giving feedback 472 seconds faster than the original schedule (68% to 86% faster). This shows that the technique of scheduling the web browser environment has the potential to improve feedback.

2 is a flowchart illustrating a method of determining a priority of a web browser environment for testing a web application according to an embodiment.

Referring to FIG. 2, in a method of determining the priority of a web browser environment for testing a web application according to an embodiment, when a client-side web application is subjected to a regression test, the failure history related to the web browser environment is determined in the web application. And determining the priority of the web browser environment by using information on the failure of at least one of the recently failed web browser environment information and the frequently failed web browser environment information (S110). I can.

In addition, the step S120 of performing a regression test on the web browser environment according to a method for determining the priority of the web browser environment determined according to the embodiment may be further included.

The embodiments provide six techniques for prioritizing a web browser environment that can provide developers with feedback on failure faster when performing a regression test on a client-side web application in a different web browser environment. can do. Here, we focus on the browser environment under the assumption that the existing test cases will be executed in each applicable environment. The embodiments use the method used to prioritize test cases, but the priority of the web browser environment may be determined based on information on recent and frequent failures.

As a result of evaluating the approach according to the embodiments using four not less popular open source web applications, the technology according to the embodiments was found to be two standard methods in terms of cost efficiency: no ordering and It shows that it performs better than random ordering. For example, in the case of no ordering, the improvement rate ranges from -12.24% to 39.05%, and in the case of random ordering, the improvement rate ranges from -0.04% to 45.85%. It will be described in more detail below.

The method for determining the priority of a web browser environment for testing a web application according to an embodiment may be described in more detail with an example of an apparatus for determining the priority of a web browser environment for testing a web application according to an embodiment.

3 is a block diagram illustrating an apparatus for determining priority of a web browser environment for testing a web application according to an exemplary embodiment.

Referring to FIG. 3, the apparatus 300 for determining a priority of a web browser environment for testing a web application according to an embodiment may include a priority determining unit 310, and according to an embodiment, a cache 320, A cache hit check unit 331, a similarity calculation unit 332, a failure frequency acquisition unit 333, and a machine learning model unit 334 may be further included. In addition, a regression test unit 340 may be further included.

In step S110, the priority determination unit 310 uses information on the failure history related to the web browser environment in the web application when performing a regression test on the client-side web application, and Priority of the web browser environment may be determined by using information on failure of at least one or more of the information and information on the frequently failed web browser environment.

It is possible to provide a technology for determining the priority of the web browser environment for use in regression testing of web applications. Here, in order to define the prioritization problem of the web browser environment, the definition of the formula of the test case prioritization problem can be expressed as follows.

Definition 1: Test-Execution Environment Prioritization Problem

Condition: Web browser environment list E, permutation set of E, PE, and function of real in PE, f.

Problem:

In definition 1, PE means all possible sequences of E, and f can represent an objective function used to quantify the goal of prioritization. In this task, the goal is to increase the test suite's failure detection rate for web applications according to a specific sequence of web browser environments. Different objective functions can be defined and used in different prioritization techniques.

In this embodiment, eight different techniques are considered to set the overall order of the web browser environment. The first two techniques may indicate a case where heuristics (empirical methods) are not applied. This technique can serve as a baseline sequence from which results can be compared as follows:

M ₁ : No prioritization. The web browser environment is not prioritized, which allows developers to maintain the order they first used.

M ₂ : Random prioritization. Web browser environments can be randomly ordered.

The other six technologies presented here can be classified into three categories: a technology that considers a recently failed web browser environment, a technology that considers an environment that frequently fails, and a technology that considers both. This is explained below.

First, the cache 320 may store information on a recently failed web browser environment at least one of a type of a web browser, a name of a web browser, a version of a web browser, and a type of an operating system. Accordingly, the priority determining unit 310 is scheduled according to the information in the cache, and the web browser environment in which the recent failure information is present may be assigned a higher priority than other web browser environments.

M ₃ : Exact matching-based prioritization.

The cache 320 lists information on a web browser environment that has recently failed, and the cache hit check unit 331 may allocate a higher priority when the web browser environment of the test target build is included in the cache. . In addition, when the priority of the plurality of web browser environments is the same, the priority determining unit 310 may schedule an original order or randomly.

Accurate matching-based prioritization technology can first consider the recently failed web browser environment. This technique can be based on the assumption that a recently failed web browser environment may fail again in the build of future web applications.

Therefore, this technique is based on prior research that considers recent failure information. A cache can be used to utilize the latest failure information to prioritize the web browser environment. The cache can list environments where previous builds failed. If the current build's web browser environment is included in the cache, that environment may be assigned a higher priority.

The cache-hit ratio can be defined as follows.

[Equation 1]

Here, TE is a set of web browser environments that the developer is considering for regression testing, and Build _i may represent a subset of TE in Build i . Cache _it is a subset of TEs that fail when moving from build i -1- t to build i -1. t is a threshold specifying the build range, and the browser environment in that range can be cached. If i-1-t is less than 1, the value may be set to 1. Each browser environment ( C _m ) of TE may be a vector containing properties of the browser environment.

4 is a diagram for describing a method of determining an accurate matching-based priority according to an embodiment.

Referring to Fig. 4, for example, it shows how a cache hit occurs. The web browser environment can include four attributes: build type, browser name, browser version, and OS. Each attribute in the vector can get a value from a set of possible values, and Cache_hit _{i It} can represent the intersection of Build _i and Cache _it .

A single browser environment can hit the cache. The cache hit browser environment can be given a higher priority than other environments. Browser environments can be scheduled based on priority scores in descending order. In this case, when the priorities of the two browser environments are the same, they may be scheduled in the original order or randomly. Here, the priority of the last browser environment is 1, and the priority of the other environments is 0. Therefore, if the scheduling result is shown in the order of ID, it can be "4, 1, 2, 3".

The advantage of this cache-based approach is that it is computationally lightweight, making it suitable for fast development environments such as continuous integration (CI) environments. However, the downside of the cache-based approach is that if the web application does not meet the assumption that the browser environment that recently failed will fail again in the near future, the performance of the approach can be degraded. The second potential drawback is that, while tests from previous builds fail in most browser environments, the approach may not be cost effective in situations where tests fail in subsequent builds in some environments. Since additional priorities are not taken into account in the cache hit browser environment, the cost effectiveness of this approach can be compromised if the failed environment is not detected before testing most of the cache hit browser environment. A third potential drawback is that only some attributes of the browser environment may be involved in predicting the failing environment in the next build, and the cache-based approach may not take that case into account. Instead, it should match all attributes of the browser environment.

M ₄ : Similarity matching-based prioritization.

The cache 320 lists information on a web browser environment that has recently failed, and the similarity calculator 332 may calculate a similarity between each web browser environment in the test target build and each web browser environment in the cache. Accordingly, the priority determining unit 310 may schedule each web browser environment in the test target build in descending order according to the calculated similarity.

The similarity matching-based prioritization technique attempts to address the second and third disadvantages of the above-described exact matching technique (M ₃ ). Instead of assigning the same priority to the cache hit web browser environment, this technique can calculate the similarity between each environment in the build under test and each environment in the cache. In the test target build, the priority of the browser environment is average similarity, which can be calculated by dividing the similarity value between each environment in the cache and the corresponding environment by the number of cached browser environments and adding them. Then, the browser environment can be scheduled in descending order according to the similarity score. To measure the similarity in this context, the Jaccard similarity coefficient can be used (Non-Patent Document 4). It can assign a float number between 0 and 1 in the browser environment. Compared to the exact matching technique, this can reduce the number of cases where the browser environment has the same priority. It can also give a positive priority to browser environments that don't exactly match the environment in the cache. For example, even if the browser version of the browser environment is not present in the cache, the environment may receive positive priority if other features such as browser name and operating system are in the cache.

You can use the following equation to measure the similarity between the browser environment of the build under test and the environment of the cache.

[Equation 2]

The definition is the same as that used in [Equation 1]. Sim _ij can represent the similarity score of the j- th browser environment in Build _i . In addition, the vector of the vector and the Cache Build _i for each vector of the Cache _it divided by him (norm) of a combination thereof, to determine their intersection he can calculate the Jaccard similarity coefficient. Then, the Jaccard similarity coefficient can be averaged.

5 is a diagram for describing a method for determining a similarity matching-based priority according to an embodiment.

Referring to FIG. 5, the similarity scores calculated for the browser environment using the approach according to the exemplary embodiments are 0/8 = 0, 3/5 = 0.6, 1/7 = 0.14, and 4/4 = It is 1. When using the correct matching method, browser environments 2 and 3 have priority 0. However, in the case of using the similarity matching method, since there are priority values of 0.6 and 0.14, respectively, the priority determination result may be "4, 2, 3, 1". If this added level of discrimination is useful, it can make similarity matching techniques more cost-effective than exact matching techniques.

M ₅ : Failure-frequency-based prioritization.

6 is a diagram illustrating a method of determining a failure-frequency-based priority according to an embodiment.

Referring to FIG. 6, the failure frequency obtaining unit 333 may obtain a failure frequency of a web browser environment in which the regression test frequently fails. Accordingly, the priority determining unit 310 may schedule in descending order according to the frequency of failure of the web browser environment.

One way to consider frequently failing web browser environments is to count the number of previous failures for each browser environment. This method is based on the assumption that the web browser environment is likely to fail frequently in future builds as well. If tests on a web application fail more frequently in certain web browser environments than in other environments, this method can be effective in predicting which environments will fail in future builds. Therefore, this method can give a higher priority to the browser environment where the number of previous failures was high. To schedule a browser environment using this method, you must have a key-value data structure where the key is the browser environment vector C _i , which is the number of previous builds where the environment failed.

One problem with this technique is that using all attributes of the browser environment to calculate the frequency of failures can reduce the cost effectiveness of scheduling the browser environment. For example, if the browser environment consists of the browser name, browser version, and OS name, a subset of those attributes such as browser name and OS name are used as keys, and the number of failed browser environments containing the key as this value is cost-effective. Can increase. However, it is difficult for a developer to manually determine which of a subset of attributes can be effective in rendering more cost-effective scheduling. As the number of attributes in the browser environment increases, this problem can become more important.

M ₆ : Machine-learning-based prioritization.

7 is a diagram for describing a method of determining a machine learning-based priority according to an embodiment.

Referring to FIG. 7, the machine learning model unit 334 automatically learns the failure pattern of the web browser environment where the regression test frequently fails based on machine learning (ML), and fails the web browser environment in subsequent builds. Can provide probability. In addition, the priority determining unit 310 may schedule the web browser environment in descending order according to the probability of failure.

To solve the problem of the failure frequency method, machine learning (ML) methods can be useful. Machine learning (ML) methods can automatically learn failure patterns from previous build history and provide a probability of failure in the browser environment in subsequent builds. Here we can use a Bernouli Naive Bayes classifier that calculates the probability of failure based on Bayes' theorem. Here the attribute of the browser environment is nominal or ordinal data, and the Bernouli Naive Bayes classifier is suitable for such data.

Classifiers can be built by learning the test history from previous builds where some browser environments failed. The classifier can return the probability of a test failure in a given browser environment. The classifier can adaptively update the training model after the build has been tested and failed in some browser environments. The priority score of each browser environment may be the same as the output of the ML model, which is the probability of test failure in the environment. Browser environments can be classified according to priority scores in descending order.

The method of calculating the test failure probability in a given browser environment can be expressed as the following equation.

[Equation 3]

Here, D denotes a series of browser environments, and k may denote the number of possible browser environments. T may be a set of test results that are "Pass" or "Fail".

는 주어진 브라우저 환경 하에서 테스트 실패의 확률이 될 수 있다. 베이즈(Bayes)의 정리를 이용하여 사후의

는 우도값(likelihood)인

에 P(fail)을 곱하여 증거인 P(D _i )로 나누어 계산할 수 있다. P(Fail) 또는 P(True)는 균일 분포 또는 도메인 전문가로부터 얻을 수 있으며,

또는

는 이전 빌드로부터의 테스트 결과로부터 계산될 수 있으며, P(D _i )는 이전 결과와 우도 값(likelihood)으로부터 계산할 수 있다. 이 기술은 사후 확률(posterior probability)에 기초하여 브라우저 환경의 우선순위를 정하므로 사후 확률의 임계값이 필요하지 않다.

Can be the probability of a test failure under a given browser environment. Post-mortem using Bayes' theorem

Is the likelihood

It can be calculated by multiplying by P(fail) and dividing by the evidence P(D _i ). P(Fail) or P(True) can be obtained from uniform distribution or domain experts,

or

Can be calculated from the test result from the previous build, and P(D _i ) can be calculated from the previous result and likelihood. Since this technique prioritizes the browser environment based on a posterior probability, there is no need for a threshold of the posterior probability.

A potential advantage of machine learning (ML) based methods is that they provide more sophisticated scheduling capabilities than cache-based or failure-frequency-based methods. Therefore, each attribute of the browser environment can be used in an adaptive way to predict the environment prone to failure. The second advantage is that this approach is suitable for fast development environments, since the developer does not have to write a new learning model every time a new build is performed. Instead, they simply update the existing model based on the new test results generated from the current build. A potential drawback of a machine learning (ML)-based approach is that it assumes there are browser environments or properties that fail frequently. If these assumptions are not met, the performance of the approach can be degraded.

M ₇ and M ₈ : Hybrid prioritization.

Hybrid prioritization techniques perform differently for each application, so it can be difficult for developers to decide which technique to use. It is possible to combine technologies to close these differences. Because hybrid prioritization technology uses two prioritization criteria, browser environments are ordered according to the first criterion, and if two or more environments have the same priority, hybrid prioritization uses the second criterion to determine these environments. Can be sorted.

Since the priority score of the exact match-based technique is 0 or 1, the correct match-based prioritization technique was selected as the first criterion. Therefore, it is more likely that this technology will have the same browser environment with the same priority than other technologies. The second criterion may be a failure-frequency-based or machine learning (ML) based technique. Here, M ₇ is defined as a priority determination technology that combines an accurate matching-based technology with a failure-frequency-based technology, and M ₈ is a priority determination in which a machine learning (ML) based technology is combined with an accurate matching-based-technology. It can be defined by technology. One advantage of hybrid prioritization technology is that it can take into account both recently failed and frequently failed browser environments.

8 is a diagram for describing a method of determining a hybrid priority according to an exemplary embodiment.

Referring to FIG. 8, in M ₇ , the cache lists information on the web browser environment that has recently failed, and the cache hit check unit 331 is higher when the web browser environment of the test target build is included in the cache. Priority can be assigned. Thereafter, when the priorities of the plurality of web browser environments are the same, the failure frequency acquisition unit 333 may obtain the failure frequency of the web browser environment in which the regression test frequently fails. Accordingly, the priority determining unit 310 may schedule in descending order according to the frequency of failure of the web browser environment.

In addition, in M ₈ , the cache lists information on the web browser environment that has recently failed, and the cache hit check unit 331 assigns a higher priority when the web browser environment of the test target build is included in the cache. can do. Thereafter, if the priorities of the plurality of web browser environments are the same, the machine learning model unit 334 automatically learns the failure pattern of the web browser environment in which the regression test frequently fails based on machine learning (ML), and follows. You can give the web browser environment the probability of failure in the build. Accordingly, the priority determining unit 310 may schedule the web browser environment in descending order according to the probability of failure.

Table 1 shows the code, operation symbol code (Mnemonic) and description of the above-described priority determination technique.

[Table 1]

Priority determination methods can be divided into independent variables and dependent variables.

Independent variables may include prioritization methods. Eight methods were evaluated here. These are the two basic techniques (M ₁ and M ₂ ) and six heuristic techniques (M ₃ -M ₈ ) described above. The order in which the browser environment was actually tested for each web application can be obtained from the Travis CI log data, and this order can be regarded as the original schedule.

As a dependent variable, the APFD _C (Average Percentage Faults Detected) metric can be used to measure the effectiveness of the prioritization method that improves the speed of failure detection. This measure can be used to measure the cost effectiveness of test case prioritization, typically when the cost of test cases is different (run time). You can use this metric because you can consider the browser environment as a test case. Therefore, scheduling the browser environment can have similar effects as when determining test case priorities.

The equation of APFD _C can be expressed as follows.

[Equation 4]

Here, n is the number of browser environments, and m may be the number of failures. C _i may indicate the location of the browser environment that detects the i- th failure in the browser environment list. t _j represents the cost of performing the regression test on C _j , and f _i can represent the severity of the i- th failure. The more effective the prioritization method is, the closer the APFD _C value can be to 100%.

Through the log data of each application program, it is possible to determine when to start and end the regression test in each browser environment, which can measure the working time. However, in the case of Lodash, the build work time was not recorded in the log data. So we can set t _j to 1. This means that the build work time of each expansion is the same. For all apps, the severity of the failure is not known from the log data, so you can set f _i to 1 for each failure.

9 is a flowchart illustrating a method of performing a regression test according to an exemplary embodiment.

In step S120, the regression test unit 340 may perform a regression test on the web browser environment according to a method for determining the priority of the determined web browser environment.

Referring to FIG. 9, the step of performing a regression test (S120) is a step of building a cache, a data structure for failure frequency, and a machine learning (ML) model based on the regression test result of the first failed build. (S121), determining the priority of the web browser environment of the test target build according to the priority generated by the method for determining the priority of the determined web browser environment (S122), the regression test result value for the test target build The step of measuring (S123), and when the regression test fails in some builds rather than all web browser environments, updates the cache, data structure for failure frequency, and machine learning (ML) model based on the results of the regression test in the build under test It may include a step (S124).

In addition, the step of determining the priority to the web browser environment of the test target build according to the generated priority (S122) may include determining a random priority when a plurality of web browser environments have the same priority. have.

For example, log data from Travis CI and Sauce Labs can be used to simulate regression tests in different browser environments and in different sequences of environments.

In step S121, a cache, a data structure for a failure frequency, and a machine learning (ML) model may be built based on the test result of the first failed build.

In step S122, the priority may be determined in the browser environment of the current build according to the priority generated by each technology. At this time, in step S122, when two or more browser environments have the same priority, a random priority may be determined.

In step S123, the APFD _C value of the regression test result for the current build may be measured. In this case, steps S122 and S123 are repeated about 30 times to obtain an average APFD _C value.

In step S124, when the regression test fails in some builds other than all browser environments, the cache, the data structure for the failure frequency, and the machine learning (ML) model may be updated based on the test result of the current build.

Thereafter, steps S122-S124 may be repeated for the next failed build.

With two exceptions, you can perform the previously described process for each of the eight priority methods. In the case of random priority determination, step (S122) is performed through simple randomization using a different order for every 30 executions, and in the case of no priority determination, the random priority determination and step (S124) of step (S122) are You can skip it. Since the random priority determination is used whenever there is a tie in the random priority determination in step S122, a technique other than no priority determination may be executed 30 times in step S124. No prioritization can be used instead of random prioritization, but there is no evidence that the original ordering is meaningful.

10 shows data on an APFD _C value according to a priority determination method according to an embodiment,

Referring to FIG. 10, data on the APFD _C value obtained using a method of determining a priority considered for each web application and a basic technique are shown. The y-axis represents the APFD _C value, and the x-axis can represent the technology used. The illustrated box may represent the distribution of APFD _C values obtained in all executions related to the technology and application.

Since web applications can fail in various environments (browser, operating system, and device), they must all be tested in the target environment. Testing all of the target environments can be time consuming and expensive, and it can take a long time to inform the developer of the environment that is causing the failure. As a result, the next stage of development (debugging, product launch, etc.) may be delayed. Accordingly, embodiments may suggest various techniques for determining the priority of the test environment. By implementing this technique, it can be seen that the feedback on test failures can be given to the developer faster than when priorities were not given when experimenting with four open source web applications.

Embodiments can be used for web application testing. As the usage environment of web applications is becoming more and more diverse, testing in various environments is becoming more important. This technique can quickly give feedback on failure to the web application developer, so it can be debugged quickly. As a result, it is possible to quickly respond to failures and reduce damage to clients. As described above, since there are many web application test target environments, the technology according to the embodiments will be helpful to the developer.

Recently, many software is developed in a continuous integration (CI) environment, and in a continuous integration (CI) environment, the test interval is much narrower than in the past development environment. That is, tests are often performed. If the test environment scheduling technique is complex (eg, when using code analysis information), cost-effectiveness may be improved, but scheduling may not be completed within the interval between test executions. A simple scheduling method is required to perform scheduling within the interval between test executions, but it is difficult to have a high cost effect.

Examples have proposed a method that is lightweight but cost effective using test history. In particular, it was confirmed that the hybrid method using machine learning technology generally recorded higher cost-effectiveness compared to other methods.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in an order different from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (16)

In the method of determining the priority of a web browser environment for web application testing,
When performing a regression test on a client-side web application, information on the failure history related to the web browser environment is used in the web application, and at least one of information on a web browser environment that has recently failed and information on a web browser environment that has frequently failed The step of determining the priority of the web browser environment using the information on the above failure
Including,
The step of determining the priority of the web browser environment,
At least one of the web browser type, web browser name, web browser version, and operating system type stored in the cache is used, and is scheduled according to the information of the cache, and The web browser environment with the failure information is assigned a higher priority than other web browser environments.
Characterized in, a method for determining the priority of a web browser environment. delete The method of claim 1,
The step of determining the priority of the web browser environment,
The cache listing information on the recently failed web browser environment, and assigning a higher priority when the web browser environment of the test target build is included in the cache; And
If multiple web browser environments have the same priority, scheduling in the original order or randomly
Including, the method of determining the priority of the web browser environment. The method of claim 1,
The step of determining the priority of the web browser environment,
The cache listing information on the recently failed web browser environment, and calculating a similarity between each web browser environment in a test target build and each web browser environment in the cache; And
Scheduling each web browser environment in the test target build in descending order according to the calculated similarity
Including, the method of determining the priority of the web browser environment. The method of claim 1,
The step of determining the priority of the web browser environment,
Acquiring a failure frequency of a web browser environment in which the regression test frequently fails; And
Scheduling in descending order according to the failure frequency of the web browser environment
Including, the method of determining the priority of the web browser environment. In the method of determining the priority of a web browser environment for web application testing,
When performing a regression test on a client-side web application, information on the failure history related to the web browser environment is used in the web application, and at least one of information on a web browser environment that has recently failed and information on a web browser environment that has frequently failed The step of determining the priority of the web browser environment using the information on the above failure
Including,
The step of determining the priority of the web browser environment,
Automatically learning a failure pattern of a web browser environment in which the regression test frequently fails based on machine learning (ML), and providing a failure probability to a web browser environment in a subsequent build; And
Scheduling the web browser environment in descending order according to the probability of failure
Including, the method of determining the priority of the web browser environment. The method of claim 1,
The step of determining the priority of the web browser environment,
The cache listing information on the recently failed web browser environment, and assigning a higher priority when the web browser environment of the test target build is included in the cache;
Obtaining a failure frequency of a web browser environment in which the regression test frequently fails when the plurality of web browser environments have the same priority; And
Scheduling in descending order according to the failure frequency of the web browser environment
Including, the method of determining the priority of the web browser environment. The method of claim 1,
The step of determining the priority of the web browser environment,
The cache listing information on the recently failed web browser environment, and assigning a higher priority when the web browser environment of the test target build is included in the cache;
If the priority of multiple web browser environments is the same, it automatically learns the failure pattern of the web browser environment that frequently failed the regression test based on machine learning (ML), and the probability of failure in the web browser environment in subsequent builds. Providing a; And
Scheduling the web browser environment in descending order according to the probability of failure
Including, the method of determining the priority of the web browser environment. In the method of determining the priority of a web browser environment for web application testing,
When performing a regression test on a client-side web application, information on the failure history related to the web browser environment is used in the web application, and at least one of information on a web browser environment that has recently failed and information on a web browser environment that has frequently failed The step of determining the priority of the web browser environment using the information on the above failure
Including,
Performing the regression test on the web browser environment according to the determined method of determining the priority of the web browser environment
Including more,
The step of performing the regression test,
Building a cache, a data structure for failure frequency, and a machine learning (ML) model based on a regression test result of the first failed build;
Determining a priority of the web browser environment of the test target build according to the priority generated by the determined priority of the web browser environment;
Measuring a regression test result value for the test target build; And
When the regression test fails in some builds other than all web browser environments, updating the cache, the data structure for the failure frequency, and a machine learning (ML) model based on the regression test result of the test target build
Including, the method of determining the priority of the web browser environment. The method of claim 9,
The step of determining the priority in the web browser environment of the test target build according to the generated priority,
Step of randomly prioritizing when multiple web browser environments have the same priority
Including, the method of determining the priority of the web browser environment. delete In the prioritization apparatus of a web browser environment for web application testing,
When performing a regression test on a client-side web application, information on the failure history related to the web browser environment is used in the web application, and at least one of information on a web browser environment that has recently failed and information on a web browser environment that has frequently failed Priority decision unit that determines the priority of the web browser environment using information on the above failures
Including,
A cache for storing information on the recently failed web browser environment at least one of a type of web browser, a web browser name, a web browser version, and an operating system type
Including,
The priority determining unit,
The web browser environment, which is scheduled according to the information in the cache, and has recent failure information, is assigned a higher priority than other web browser environments.
A device for determining the priority of a web browser environment, characterized in that. The method of claim 12,
The cache lists information on the recently failed web browser environment, and a cache hit check unit that allocates a higher priority when the web browser environment of the test target build is included in the cache
Including more,
The priority determining unit,
If multiple web browser environments have the same priority, scheduling in the original order or randomly
A device for determining the priority of a web browser environment, characterized in that. The method of claim 12,
The cache lists information on the recently failed web browser environment, and a similarity calculator that calculates the similarity between each web browser environment in the test target build and each web browser environment in the cache
Including more,
The priority determining unit,
Scheduling each web browser environment in the test target build in descending order according to the calculated similarity
A device for determining the priority of a web browser environment, characterized in that. The method of claim 12,
A failure frequency acquisition unit that obtains the failure frequency of a web browser environment that frequently fails the regression test
Including more,
The priority determining unit,
Scheduling in descending order according to the failure frequency of the web browser environment
A device for determining the priority of a web browser environment, characterized in that. In the prioritization apparatus of a web browser environment for web application testing,
When performing a regression test on a client-side web application, information on the failure history related to the web browser environment is used in the web application, and at least one of information on a web browser environment that has recently failed and information on a web browser environment that has frequently failed Priority decision unit that determines the priority of the web browser environment using information on the above failures
Including,
Machine learning model unit that automatically learns the failure pattern of the web browser environment that frequently failed the regression test based on machine learning (ML) and provides the probability of failure to the web browser environment in subsequent builds
Including more,
The priority determining unit,
Scheduling in descending order according to the probability of failure in the web browser environment
A device for determining the priority of a web browser environment, characterized in that.

Images