JP2017138715A - Screen difference pass/fail determination device, screen difference pass/fail determination method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a screen difference pass/fail determination device, screen difference pass/fail determination method, and program for extracting a position of difference between screens and automating determination of pass/fail as to the difference.SOLUTION: A screen difference pass/fail determination device 10 comprises: a size matching part 12 that changes the size of either one of a first image showing a first display result of a screen and a second image showing a second display result of the screen, on the basis of the ratio of the size related to screen elements of the first image and screen elements of the second image; a screen difference extraction part 13 that extracts partial areas respectively from the first image and the second image after the size change, compares the partial area extracted from the first image with the partial area extracted from the second image, and extracts the difference between a first display result and a second display result in a unit of the partial area; and a pass/fail determination rule application part 15 that applies a set rule to the extracted difference and determines pass/fail as to the difference.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a screen difference acceptance / rejection determination device, a screen difference acceptance / rejection determination method, and a program.

  In recent years, a screen such as a Web page has been browsed in various environments (a combination of a terminal model, an OS (Operating System), a browser, and the like), but the screen layout may be different in each environment. For example, in a specific environment, layout disruption that is not intended by the creator of the screen may occur. Therefore, the creator of the screen needs to perform a test to confirm that the created screen is displayed as intended in each environment.

  Conventionally, such a test is generally performed by visually confirming a display result in each environment and a display result as intended, but the type of environment (that is, terminal model, Since there are many types of combinations of OS, browsers, etc., the workload of the tester is heavy.

  Thus, for example, it is conceivable to improve the efficiency of the test by using a technique as described in Non-Patent Document 1.

Akihiro Hori, Shingo Takada, Haruto Tanno and Morihide Oinuma, `` An Oracle based on Image Comparison for Regression Testing of Web Applications '', Proceedings of 27th International Conference on Software Engineering and Knowledge Engineering (SEKE 2015), ACM (USA), pp. 639-645 (July 2015). Sonal Mahajan and William G.J.Halfond, `` Finding html presentation failures using image comparison techniques '', Proceedings of the 29th ACM / IEEE international conference on Automated software engineering (ASE 2014), ACM (USA), pp.91-96 (2014).

  However, since the technique described in Non-Patent Document 1 requires information of a DOM (Document Object Model) tree of HTML (HyperText Markup Language) that is a source code of a Web page as input, Dependent. For example, it is difficult to use the technology in an environment that is not based on Web standard specifications, such as an Android (registered trademark) native application.

  Further, in Non-Patent Document 2, since the entire image is compared with the pixel granularity, it can be determined whether there is a difference, but it is difficult for the user to specify a portion having a difference. In addition, there is a problem in that images having different screen sizes cannot be compared properly because the images to be compared cannot be compared unless the resolutions are set in advance.

  In addition, it may be difficult to determine pass / fail for each difference by visual observation of the user. In particular, regarding the difference near the pass / fail boundary, if the determination is made by visual observation, the pass / fail may differ for each user.

  The present invention has been made in view of the above points, and an object of the present invention is to make it possible to extract a portion of a difference between screens and to automate a pass / fail determination for the difference.

  Therefore, in order to solve the above-described problem, the screen difference acceptance / rejection determination device has a size of either the first image indicating the first display result of the screen or the second image indicating the second display result of the screen. Is changed based on the ratio of the sizes of the screen element of the first image and the screen element of the second image, and after the change of the size, the first image and the second image A partial region is extracted from each of the first region and the partial region extracted from the first image is compared with the partial region extracted from the second image. An extraction unit that extracts a difference between a display result and the second display result; and an application unit that applies a set rule to the difference extracted by the extraction unit and determines whether the difference is acceptable or not Have.

  Differences between screens can be extracted, and pass / fail judgments for differences can be automated.

It is a figure which shows the hardware structural example of the screen difference acceptance / rejection determination apparatus in embodiment of this invention. It is a figure which shows the function structural example of the screen difference acceptance / rejection determination apparatus in embodiment of this invention. It is a flowchart for demonstrating an example of the process sequence which a screen difference acceptance / rejection determination apparatus performs. It is a figure which shows the example of a definition of the data structure in embodiment of this invention. It is a figure which shows an example of a folder structure. It is a figure which shows an example of the layout of a correct answer screen and a comparison screen. It is a figure which shows the example of a display of the link screen to a result report. It is a figure which shows the example of a display of a result report. It is a flowchart for demonstrating an example of the process sequence of the input process of a screen image. It is a flowchart for demonstrating an example of the process procedure of size adjustment. It is a flowchart for demonstrating an example of the process sequence of the extraction process of screen difference information. It is a flowchart for demonstrating an example of the process sequence of the application process of the pass / fail determination rule to screen difference information. It is a figure which shows the structural example of a pass / fail determination rule.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a hardware configuration example of a screen difference acceptance / rejection determination apparatus according to an embodiment of the present invention. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, an input device 107, and the like that are connected to each other via a bus B. Have.

  A program for realizing the processing in the screen difference acceptance / rejection determination apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 realizes functions related to the screen difference acceptance / rejection determination device 10 according to a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a GUI (Graphical User Interface) or the like by a program. The input device 107 includes a keyboard and a mouse, and is used for inputting various operation instructions.

  FIG. 2 is a diagram illustrating a functional configuration example of the screen difference acceptance / rejection determination device according to the embodiment of the present invention. In FIG. 2, the screen difference acceptance / rejection determination apparatus 10 includes an image input unit 11, a size matching unit 12, a screen difference extraction unit 13, a report output unit 14, and a pass / fail determination rule application unit 15. Each of these units is realized by processing that one or more programs installed in the screen difference acceptance / rejection determination apparatus 10 cause the CPU 104 to execute. The screen difference acceptance / rejection determination apparatus 10 also uses the acceptance / rejection determination rule storage unit 16. The pass / fail judgment rule storage unit 16 can be realized using, for example, a storage device that can be connected to the auxiliary storage device 102 or the screen difference pass / fail judgment device 10 via a network.

  The image input unit 11 reads the correct screen and one or more comparison screens for each screen to be tested from the folder specified by the user. The image input unit 11 also generates pair information including a pair with the comparison screen for each correct screen. The correct screen is referred to as image data (hereinafter referred to as “screen image”) indicating the result of displaying the screen in an environment where the screen is displayed as intended by the creator (hereinafter referred to as “correct environment”). ). The comparison screen is a screen image showing a result displayed in an environment different from the correct answer environment (hereinafter referred to as “comparison environment”). The environment is distinguished by, for example, a combination of a terminal model that displays a screen, an OS (Operating System), a browser, and the like. Furthermore, the environment may be distinguished according to the display size of the browser. It can be said that the environment is an example of a display condition of the screen. Note that the correct answer screen and the comparison screen may be acquired by screen capture or the like, for example.

  The screen refers to a screen that the program displays on the computer. The screen may be a screen such as an application program, or may be a screen described in HTML (HyperText Markup Language) or the like.

  For each pair of the correct screen and the comparison screen, the size matching unit 12 absorbs the difference in resolution between the screen elements included in the two screen images related to the pair, so that the screen elements of the correct screen and the comparison screen The size of one screen image is changed based on the size ratio with the screen element. A screen element refers to a drawing element of a screen, and is, for example, a unit constituting a screen layout. Note that the size of the screen image is expressed by the number of pixels (number of pixels). The size change is realized by changing the resolution.

  The screen difference extraction unit 13 extracts, for each pair, a difference between two screen images related to the pair. Specifically, the screen difference extraction unit 13 extracts a rectangular area with approximately the granularity of the screen elements for each of the two screen images related to the pair, and performs matching in units of rectangular areas. As a result of matching, the screen difference extraction unit 13 detects differences such as disappearance, addition, and movement with the granularity of the rectangular area, and generates screen difference information indicating the contents of the differences for each difference. The screen difference extraction unit 13 sets the generated screen difference information for the pair information.

  For each pair, the pass / fail judgment rule application unit 15 applies each pass / fail judgment rule stored in the pass / fail judgment rule storage unit 16 for the difference related to each screen difference information generated by the screen difference extraction unit 13. The presence or absence of is determined. The pass / fail judgment rule is information indicating an allowable range for the difference, and is set in advance by the user, for example. The pass / fail judgment rule application unit 15 determines that a comparison screen related to a pair having no problem (within an allowable range) is acceptable for all differences, and is rejected for a comparison screen related to a pair that is not. judge. The determination result is set in the pair information of each pair.

  The report output unit 14 outputs each pair information including the screen difference information generated by the screen difference extraction unit 13 and set with the pass / fail judgment result by the pass / fail judgment rule application unit 15 as a result report. Present to the user.

  Hereinafter, a processing procedure executed by the screen difference acceptance / rejection determination device 10 will be described. FIG. 3 is a flowchart for explaining an example of a processing procedure executed by the screen difference acceptance / rejection determination device.

  In step S110, the image input unit 11 reads the correct screen and the comparison screen group stored for each screen to be tested under the folder specified by the user, and the comparison screen related to the same screen for each correct screen. A list of pair information with is generated.

  FIG. 4 is a diagram showing a definition example of the data structure in the embodiment of the present invention. In FIG. 4, the data structures of various data used in the present embodiment are defined by BNF (Backus-Naur Form).

  Regarding step S110, the definition in “■ Image group to be input data” is related. Here, it is shown that the environment root folder includes one correct environment folder and one or more comparison environment folders. Further, it is shown that each of the correct answer environment folder and each comparison environment folder includes an image file storing a screen image (correct answer screen or comparison screen). The image file included in the correct environment folder is an image file storing the correct answer screen, and the image file included in the comparison environment folder is an image file storing the comparison screen.

  FIG. 5 is a diagram illustrating an example of a folder configuration. FIG. 5 shows an example in which one correct environment folder and two comparison environment folders (comparison environment folders (a) and (b)) are included under the environment root folder. An image file X and an image file Y are stored in each of the correct answer environment folder and each comparison environment folder. The image file X is a file that stores a screen image generated by screen capture of the display result of the screen X. The image file Y is a file that stores a screen image generated by screen capture of the display result of the screen Y. However, the image files X and Y under the correct answer environment folder are image files that store the correct answer screens related to the screen X or the screen Y. The image files X and Y under the comparison environment folder (a) are image files storing a comparison screen indicating the display result of the screen X or the screen Y in the environment (a). The image files X and Y under the comparison environment folder (b) are image files storing a comparison screen showing the display result of the screen X or the screen Y in the environment (b). The folders and files shown in FIG. 5 are stored in the auxiliary storage device 102, for example.

  According to the example of FIG. 5, in step S110, a pair of a correct screen related to the image file X under the correct environment folder and a correct screen related to the image file X under the comparison environment folder (a), and the correct environment Comparison between the correct screen related to the image file Y under the folder and the correct screen related to the image file Y under the comparison environment folder (a), and the correct screen related to the image file X under the correct environment folder Pair with correct screen related to image file X under environment folder (b), correct screen related to image file Y under correct environment folder, and correct answer related to image file Y under comparison environment folder (b) Pair information is generated for the four pairs of the screen and the pair.

  The pair information includes correct screen information, comparison screen information, a screen difference list, and a pass / fail judgment result as described in the column “■ Report as output data” in FIG. Correct screen information is screen information of a correct screen. The comparison screen information is screen information of the comparison screen. The screen information is information including the image file of the screen and the path name of the image file. The screen difference list will be described later. The pass / fail determination result is information indicating the pass / fail automatically determined for the comparison screen related to the pair information.

  Hereinafter, for the sake of convenience, the correct answer screen related to the screen X is referred to as “correct answer screen X”. The comparison screen in the environment (a) related to the screen X is referred to as “comparison screen X (a)”. Further, the comparison screen in the environment (a) related to the screen X is referred to as “comparison screen X (b)”.

  FIG. 6 is a diagram illustrating an example of the layout of the correct answer screen and the comparison screen. FIG. 6 shows an example of the layout of the correct screen X, the comparison screen X (a), and the comparison screen X (b) with respect to the screen X. The size of each screen is based on the difference in the size (resolution) of each screen. As described above, the same screen X (for example, the same HTML (HyperText Markup Language) or the like) may have a difference in size or layout depending on the environment.

  Subsequently, the image input unit 11 generates an empty data area for the result report (step S120).

  Subsequent steps S130 to S160 are executed for each pair information generated in step S110. Hereinafter, the pair information to be processed is referred to as “target pair information”.

  In step S130, the size matching unit 12 matches the sizes of the screen elements of the two screen images by changing the size (resolution) of one of the two screen images related to the target pair information. For example, in FIG. 6, the correct screen X and the comparison screen X (a) are different in size. Accordingly, the size of either one of the two screen image pairs is changed and adjusted so that the sizes of both screen images coincide with each other at the screen element level.

  Subsequently, the screen difference extraction unit 13 extracts a rectangular area from two screen images related to the target pair information and specifies a difference in units of the rectangular area (step S140). As a result, a screen difference list which is a list of screen difference information indicating the specified differences is set for the target pair information. For example, in the case of the correct answer screen X and the comparison screen X (a) shown in FIG. 6, the rectangular areas of “Title”, “Advertisement”, “Sentence”, “Photo”, and “Link” are extracted. In the comparison screen (b), the rectangular areas of “title”, “sentence”, “photograph”, “link”, and “link that does not exist in the specification” are extracted. Hereinafter, the extracted rectangular area is referred to as a “screen element rectangle”.

  As will be described later, since the extraction of the screen element rectangle is performed by analyzing the image, the screen element rectangle extracted in step S140 is not necessarily a screen element in a meaningful unit as viewed from a human. It is not necessarily rectangular. For example, the “title” area may be divided into a plurality of screen element rectangles and extracted. In addition, there is a possibility that a screen element rectangle extending over the “title” and “advertisement” areas is extracted.

  Subsequently, the pass / fail determination rule application unit 15 stores the pass / fail determination rule stored in the pass / fail determination rule storage unit 16 for each piece of screen difference information included in the screen difference list set for the target pair information. Is applied to determine whether the comparison screen related to the target pair information is acceptable (step S150). The pass / fail determination rule application unit 15 sets the determination result in the target pair information.

  Subsequently, the report output unit 14 adds target pair information to the result report (step S160).

  When Steps S130 to S160 are executed for all pairs, the report output unit 14 outputs a result report to the display device 106 (Step S170).

  FIG. 7 is a diagram illustrating a display example of a link screen to a result report. As shown in FIG. 7, the link screen has a pass / fail judgment rule application unit 15 for each combination of screen type (screen X, screen Y) and environment type (environment (a), environment (b)). The result of pass / fail judgment by and a link to the result report are included. When any link is selected by the user, a result report as shown in FIG. 8 is displayed.

  FIG. 8 is a diagram illustrating a display example of a result report. The result report R1 is a result report for the combination of the screen X and the environment (b).

  The result report R1 includes a correct answer screen X and a comparison screen X (b), and a table showing a list of differences between the two screen images. In the table, for each difference, an item number, a difference type, a mutation value, a determination result based on a pass / fail determination rule, and an applied pass / fail determination rule are displayed. The item number is a number assigned to each difference, and the item number related to the difference is given to the screen element rectangle corresponding to the difference on the correct answer screen and the comparison screen. The difference type is a type of difference, and the value is “no difference”, “movement in the X-axis direction”, “movement in the Y-axis direction”, “enlargement / reduction”, “disappearance”, or “addition”. The variation value is a variation value of the difference according to the difference type. The determination result based on the pass / fail determination rule is information indicating whether or not there is a problem obtained by applying the pass / fail determination rule to the difference. “No problem” means that the pass / fail judgment rule is not violated, and “confirmation required” means that the pass / fail judgment rule is violated. The applied pass / fail judgment rule is the content of the pass / fail judgment rule applied to the difference.

  The user can easily confirm the pass / fail of the comparison screen by referring to such a result report.

  Next, details of step S110 in FIG. 3 will be described. FIG. 9 is a flowchart for explaining an example of the processing procedure of the screen image input processing.

  In step S210, the image input unit 11 reads the environment root folder designated by the user. Subsequently, the image input unit 11 acquires a correct answer environment folder under the environment root folder (step S220). For example, the folder name of the correct answer environment folder may be determined in advance, and the folder related to the folder name may be acquired.

  Subsequently, the image input unit 11 generates an empty list (hereinafter referred to as “pair list”) for storing pair information to be generated (step S230). Subsequently, the image input unit 11 executes step S240 and subsequent steps for each correct answer screen stored in the correct answer environment folder. The correct answer screen that is the processing target after step S240 is referred to as a “target correct answer screen”.

  In step S240, the image input unit 11 acquires all comparison environment folders (hereinafter referred to as “comparison environment folder group”) under the environment root folder.

  Subsequently, the image input unit 11 executes steps S250 and S270 for each acquired comparison environment folder. The comparison environment folder to be processed in steps S250 and S270 is referred to as “target comparison environment folder”.

  In step S250, the image input unit 11 acquires a comparison screen corresponding to the target correct answer screen from the target comparison environment folder, and generates pair information for the target correct answer screen and the comparison screen. The comparison screen corresponding to the target correct answer screen is a comparison screen corresponding to the same screen as the screen corresponding to the target correct answer screen. For example, screen images with common screens may be stored in files with the same file name. By doing so, the comparison screen corresponding to the correct answer screen can be easily specified.

  Subsequently, the image input unit 11 adds the generated pair information to the pair list (step S270).

  Next, details of step S130 in FIG. 3 will be described. FIG. 10 is a flowchart for explaining an example of the size adjustment processing procedure.

  In step S310, the size matching unit 12 reads the target pair information into its working memory area.

  Subsequently, the size matching unit 12 extracts a feature point from each of the correct screen and the comparison screen related to the target pair information, performs feature point matching in the direction of the correct screen → the comparison screen, and between the correct screen and the comparison screen. A feature point pair group that is a set of corresponding feature point pairs is generated (step S320). The feature point matching can be performed using a known algorithm such as the Harris method or the Shi-Toshi method.

  The data structure of one feature point pair is as described in the column “■ Intermediate data” in FIG. That is, the feature point pair is composed of the coordinate point of the feature point on the correct answer screen and the coordinate point of the feature point corresponding to the feature point on the comparison screen.

  Subsequently, the size matching unit 12 selects two feature point pairs that are sufficiently separated from each other from the feature point pair group (step S330). For example, among all combinations of two feature point pairs, two feature point pairs related to the combination having the maximum sum of the distance on the correct screen and the distance on the comparison screen may be selected. Alternatively, two feature point pairs related to a combination in which the sum of the distance on the correct answer screen and the distance on the comparison screen is greater than or equal to the threshold value may be selected. The threshold value may be obtained based on the width size of the correct answer screen or the comparison screen, or may be obtained by another method.

  Subsequently, the size matching unit 12 calculates a size ratio between the screen element rectangle of the correct answer screen and the screen element rectangle of the comparison screen based on the two selected feature point pairs (step S340). Specifically, the coordinate point on the correct screen of one feature point pair (hereinafter referred to as “feature point pair 1”) and the correct screen of the other feature point pair (hereinafter referred to as “feature point pair 2”). The distance of the coordinate point at is calculated and set as the correct screen distance. Further, the distance between the coordinate point on the comparison screen of the feature point pair 1 and the coordinate point on the comparison screen of the feature point pair 2 is calculated and set as the comparison screen distance. Then, the distance within the correct screen / the distance within the comparison screen is calculated, and this is used as the size ratio. That is, the size ratio is not necessarily the ratio between the size of the entire correct screen and the size of the entire comparison screen. In the present embodiment, since the comparison is performed in units of screen element rectangles, it is important to match the size of the screen element rectangles between the correct answer screen and the comparison screen.

  The size ratio may be calculated by a method other than the above. For example, the width of the correct screen / the width of the comparison screen may be calculated as the size ratio. Alternatively, the distance between screen element rectangles on the correct screen / the distance between screen element rectangles on the comparison screen may be calculated as the size ratio. Also, the width of the correct screen / the width of the screen in the comparison environment may be calculated as the size ratio. Here, the width of the screen in the comparison environment is the width of the display of the terminal in the environment corresponding to the comparison screen, and may be stored in advance in the auxiliary storage device 102 or the like for each environment, for example. Such a calculation method is effective for an environment where the screen is displayed in accordance with the horizontal width of the display.

  Subsequently, the size matching unit 12 changes the size of the comparison screen based on the size ratio (step S350). Specifically, resolution conversion is performed on the comparison screen so that the vertical and horizontal sizes (number of pixels) of the comparison screen are values obtained by multiplying the size ratios, respectively. In the target pair information, the comparison screen before the size change is replaced with the comparison screen after the size change. Note that the size of the correct answer screen may be changed.

  Next, details of step S140 in FIG. 3 will be described. FIG. 11 is a flowchart for explaining an example of a processing procedure of screen difference information extraction processing.

  In step S410, the screen difference extraction unit 13 reads the target pair information into its own working memory area.

  Subsequently, the screen difference extraction unit 13 generates an empty data area for storing the screen difference list (step S415). The screen difference list is composed of zero or more pieces of screen difference information as described in the column “Report as output data” in FIG.

  Subsequently, the screen difference extraction unit 13 extracts a feature point pair group from the correct screen and the comparison screen related to the target pair information by the same method as in step S320 of FIG. 10 (step S420).

  Subsequently, the screen difference extraction unit 13 extracts a screen element rectangle (hereinafter referred to as “correct answer screen element rectangle”) from the correct answer screen related to the target pair information, and a screen in which the extracted correct answer screen element rectangles are listed. An element rectangle list is generated (step S425). The screen element rectangle is represented by the upper left coordinates and upper right coordinates of the rectangle as described in the column “■ Report as output data” in FIG. The extraction of the screen element rectangle can be realized by combining known techniques. For example, a screen element rectangle is extracted by executing procedures such as gray scale, canny processing, expansion processing, contour extraction, and rectangle conversion (obtaining a circumscribed rectangle of the contour and extracting the rectangle therefrom) on the image. May be.

  Note that the screen element rectangle is not necessarily a “rectangle”. It suffices if the partial areas constituting the screen are extracted. Therefore, L-shaped, convex, concave, or other polygonal partial areas may be extracted as screen element rectangles.

  Subsequently, the screen difference extraction unit 13 executes Steps S430 to S475 for each correct screen element rectangle included in the generated screen element rectangle list. The correct screen element rectangle to be processed in steps S430 to S475 is referred to as “target correct screen element rectangle”.

  In step S430, the screen difference extraction unit 13 has two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle in the comparison screen (hereinafter referred to as “target comparison screen”) related to the target pair information. It is determined whether or not to do. The feature point group related to the target correct screen element rectangle refers to, for example, feature points included in the range of the target correct screen element rectangle. However, feature points included in a predetermined range outside from the periphery of the target correct screen element rectangle may be included in the feature point group related to the target correct screen element rectangle. Further, whether or not there are two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle on the target comparison screen is a feature point pair including the feature point group related to the target correct answer screen element rectangle (hereinafter, “ It can be determined based on whether or not two or more “feature feature point pairs” are included in the feature point pair group generated in step S420.

  When two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle exist on the target comparison screen (Yes in step S430), the screen difference extraction unit 13 selects from two or more target feature point pairs that exist. Then, two feature point pairs that are sufficiently apart from each other are selected, and the target correct screen element rectangle and the screen element corresponding to the target correct screen element rectangle in the target comparison screen are selected in the same manner as in step S340 of FIG. An expansion / contraction ratio with the rectangle is calculated (step S435).

  Subsequently, the screen difference extraction unit 13 searches the comparison screen for a screen element rectangle corresponding to the target correct screen element rectangle, and the corresponding screen element rectangle (hereinafter referred to as “target comparison screen element rectangle”) is searched. In this case, the target correct screen element rectangle and the target comparison screen element rectangle are set in the screen difference information (step S440). Specifically, the screen difference extraction unit 13 calculates the positional relationship between the target correct screen element rectangle and the feature points related to the correct screen in the target feature point pair (for example, a vector from each vertex of the target correct screen element rectangle). A candidate rectangle of the target correct screen element rectangle is cut out from the comparison screen based on the feature point and the positional relationship related to the comparison screen in the target feature point pair. The screen difference extraction unit 13 compares the pixel values of the target correct screen element rectangle and the candidate rectangle, and the pixel value matching rate (pixel unit matching rate) is equal to or greater than a predetermined value (for example, 100% or 95%). If so, the candidate rectangle is set as a target comparison screen element rectangle. However, when the enlargement / reduction ratio is not 1, the size of the candidate rectangle is enlarged or reduced based on the enlargement / reduction ratio, and the pixel value is compared after the size of the candidate rectangle matches the target correct screen element rectangle. .

  The target comparison screen element rectangle may be cut out by pattern matching between the target correct screen element rectangle and the target comparison screen. For example, after changing the size of the target comparison screen based on the enlargement / reduction ratio, the target correct screen element rectangle is a rectangular area having the same size as the target correct screen element rectangle and all pixels (or a predetermined ratio (for example, The target comparison screen element rectangle may be cut out by searching for a rectangular region with a matching pixel value with respect to (95%) or more pixels).

  The screen difference extraction unit 13 generates screen difference information, and sets the target correct screen element rectangle as the screen difference information (hereinafter referred to as “target screen difference information”). As shown in the “■ Report to be output data” column in FIG. 4, the screen difference information is an item number, a difference type, 0 or more mutation values, 0 or more correct screen element rectangles, and 0 or more comparisons. This is information including a screen element rectangle and a determination result based on one or more pass / fail determination rules. The correct screen element rectangle and the comparison screen element rectangle are expressed by the coordinates of the upper left vertex and the lower right vertex, respectively. The determination result based on the pass / fail determination rule is information indicating the pass / fail automatically determined based on the pass / fail determination rule with respect to the difference related to the screen difference information.

  When the target comparison screen element rectangle is searched, the screen difference extraction unit 13 sets the target comparison screen element rectangle in the target screen difference information. When the target comparison screen element rectangle is not searched, the target comparison screen element rectangle is not set in the target screen difference information.

  Note that “no difference” is set as the initial value for the difference type of the target screen difference information.

  Subsequently, when the target comparison screen element rectangle exists (Yes in step S445), the screen difference extraction unit 13 determines whether or not the positions of the target correct screen element rectangle and the target comparison screen element rectangle are different (S450). ). The difference between the positions may be determined based on whether or not the coordinates of the upper left vertices of the target correct screen element rectangle and the target comparison screen element rectangle are the same.

  When the positions of the target correct screen element rectangle and the target comparison screen element rectangle are different (Yes in step S450), the screen difference extraction unit 13 selects “X-axis direction movement” or “Y-axis direction” as the difference type of the target screen difference information. “Move” is set, and the difference (movement amount) of the position is set as the variation value of the target screen difference information (step S455). The difference between the positions is calculated as, for example, the absolute value of the difference between the X coordinate or the Y coordinate of the upper left vertex between the target correct screen element rectangle and the target comparison screen element rectangle. Specifically, when the difference in the X-axis direction is not 0, “X-axis direction movement” is set as the difference type of the target screen difference information, and the movement amount in the X-axis direction is set as the variation value of the target screen difference information. Is done. When the difference in the Y-axis direction is not 0, “Y-axis direction movement” is set as the difference type of the target screen difference information, and the movement amount in the Y-axis direction is set as the variation value of the target screen difference information. If both the difference in the X-axis direction and the difference in the Y-axis direction are not 0, a copy of the target screen difference information is generated, and the difference type and the mutation value of one target screen difference information are set to “X-axis direction movement”, X The movement amount in the axial direction is set, and “movement in the Y-axis direction” and the movement amount in the Y-axis direction are set in the difference type and the variation value of the other target screen difference information.

  Subsequently, the screen difference extraction unit 13 determines whether or not the enlargement / reduction ratio calculated in step S435 is 1 (step S460). If the enlargement / reduction ratio is not 1 (No in step S460), the screen difference extraction unit 13 sets “enlargement / reduction” as the difference type of the target screen difference information, and sets the enlargement / reduction ratio as the mutation value of the target screen difference information (step). S465). When a value is set in step S455 for the target screen difference information, the screen difference extraction unit 13 generates a copy of the target screen difference information, and for the difference type and mutation value of the copy, “ Set the scaling ratio.

  On the other hand, when two or more feature points corresponding to the feature point group related to the target correct screen element rectangle do not exist in the target comparison screen (No in step S430), or when no target comparison screen element rectangle exists (No in step S445). The screen difference extraction unit 13 sets “disappear” as the difference type of the target screen difference information (step S470).

  Subsequent to step S460, S465, or S470, the screen difference extraction unit 13 adds the generated target screen difference information to the screen difference list (step S475). When a plurality of target screen difference information is generated, the plurality of target screen difference information is added to the screen difference list. Further, when adding the target screen difference information to the screen difference list, the screen difference extraction unit 13 sets a numerical value corresponding to the order of the target screen difference information in the screen difference list as the item number of the target screen difference information. To do.

  When steps S430 to S475 are executed for all correct screen element rectangles included in the screen element rectangle list, the screen difference extraction unit 13 executes steps S420 to S475 by exchanging the correct screen and the comparison screen. (Step S480). In this case, in step S470, “addition” is set for the difference type of the target screen difference information. This is because, when matching is performed in the direction of the comparison screen → the correct answer screen, when step S470 is executed, the screen element rectangle existing on the comparison screen does not exist on the correct answer screen.

  Subsequently, the screen difference extraction unit 13 sets the screen difference list as target pair information (step S485).

  Next, details of step S150 will be described. FIG. 12 is a flowchart for explaining an example of the processing procedure of the application process of the pass / fail judgment rule to the screen difference information.

  In step S510, the pass / fail judgment rule application unit 15 reads the target pair information into its own working memory area. Subsequently, the pass / fail judgment rule application unit 15 acquires a screen difference list set in the target pair information (step S520). Subsequently, the pass / fail determination rule application unit 15 reads a list of pass / fail determination rules (pass / fail determination rule list) stored in the pass / fail determination rule storage unit 16 (step S530).

  FIG. 13 is a diagram illustrating a configuration example of a pass / fail judgment rule. In FIG. 13, each row is an example of a pass / fail judgment rule. The pass / fail judgment rule includes a rule target, a difference type, an allowable range, and the like. The rule target is information indicating a comparison environment to which the pass / fail judgment rule is applied. The whole indicates that all comparison environments (in this embodiment, environment (a) and environment (b)) are applicable. The difference type indicates a difference type to which the pass / fail judgment rule is applied. The tolerance indicates the tolerance for the difference.

  Subsequently, the pass / fail determination rule application unit 15 executes steps S540 to S570 for each pass / fail determination rule included in the pass / fail determination rule list for each screen difference information included in the screen difference list. The screen difference information to be processed in steps S540 to S570 is referred to as “target screen difference information”. In addition, the pass / fail judgment rule that is the processing target in steps S540 to S570 is referred to as a “target rule”.

  In step S540, the pass / fail determination rule application unit 15 determines whether the environment of the comparison screen related to the target pair information and the difference type of the target screen difference information match the rule target and the difference type of the target rule. . That is, it is determined whether the target screen difference information is an application target of the target rule. When at least one of the comparison screen environment related to the target pair information and the difference type of the target screen difference information does not match the rule target or the difference type of the target rule (No in step S540), in the pass / fail judgment rule list The next pass / fail decision rule is set as a processing target, and the steps after step S540 are repeated.

  On the other hand, when the environment of the comparison screen related to the target pair information and the difference type of the target screen difference information match the rule target and the difference type of the target rule (Yes in step S540), the pass / fail judgment rule application unit 15 It is determined whether or not the variation value of the target screen difference information is within the allowable range of the target rule (step S550). When the variation value is within the permissible range (Yes in step S550), the pass / fail judgment rule application unit 15 sets “no problem” in the target screen difference information as a judgment result based on the pass / fail judgment rule (step S560). On the other hand, when the mutation value is outside the allowable range (No in step S550), the pass / fail determination rule application unit 15 sets “confirmation required” in the target screen difference information as a determination result based on the pass / fail determination rule. (Step S570).

  When Steps S540 to S570 are executed for all the pass / fail determination rules included in the pass / fail determination rule list for all the screen difference information included in the screen difference list, the pass / fail determination rule application unit 15 sets the target pair. Based on the screen difference list of information, the pass / fail of the comparison screen related to the target pair information is determined (step S580). Specifically, if the determination result based on the pass / fail determination rule of all screen difference information included in the screen difference list of the target pair information is “no problem”, the pass / fail determination result of the target pair information is “ “Pass” is set. If the judgment result based on the pass / fail judgment rule of any screen difference information included in the screen difference list of the target pair information is “required confirmation”, “fail” is indicated for the pass / fail judgment result of the target pair information. Is set.

  Based on the target pair information generated in this way, the link screen shown in FIG. 7 and the result report shown in FIG. 8 are output.

  As described above, according to the present embodiment, when a test is executed in n test environments, the screen image of the test result in one test environment is set as a correct answer screen, and the other n-1 environments. The difference is automatically extracted by comparing the screen image (comparison screen) of the test result with the correct screen. At this time, the difference is extracted in units of screen element rectangles. Therefore, it is possible to easily detect the screen layout collapse and to easily identify the differences between the screens. As a result, the user can efficiently perform pass / fail determination on the comparison screen.

  In the present embodiment, based on the size ratio between the screen element of the correct screen and the screen element of the comparison screen, the difference is extracted after one image size is changed. As a result, it is possible to avoid erroneous detection caused by the difference in resolution between the correct answer screen and the comparison screen, and to detect the difference with higher accuracy. In addition, the screen element rectangles can be compared more appropriately by using the size ratio in consideration of the positional relationship of the screen elements grasped through the feature point pairs instead of simply the resolution ratio. For example, it is assumed that the size of the correct screen (vertical × horizontal) is 1000 × 500 and the size of the comparison screen is 1500 × 600. However, in the comparison screen, the range in which the screen is drawn is a range of (0 to 1000) × (0 to 500), and the other ranges are blanks. That is, it is assumed that the screen element rectangle of the correct answer screen is the same as the screen element rectangle of the comparison screen. In this case, simply, if one size is changed based on the ratio between the size of the correct answer screen and the size of the comparison screen, the aspect ratio of the screen element rectangle included in the one is destroyed. On the other hand, according to the size ratio based on the feature point pair, the possibility of avoiding such inconvenience increases.

  Further, in the present embodiment, screen element rectangles are extracted from each of the correct answer screen and the comparison screen, and then the screen element is divided into two directions: a direction from the correct answer screen to the comparison screen and a direction from the comparison screen to the correct answer screen. Rectangle matching is performed. Therefore, various differences such as movement, enlargement / reduction, disappearance, and addition of the screen element rectangle can be detected with an easily understandable granularity, and it is possible to easily determine whether the screen layout is broken.

  Moreover, according to this Embodiment, the pass / fail determination rule set beforehand is applied about the extracted difference, and a pass / fail can be determined automatically. Therefore, the pass / fail determination result can be made uniform.

  The pass / fail judgment rule can be set according to the environment. Therefore, by applying a pass / fail determination rule corresponding to the environment, it is possible to automatically perform the pass / fail determination according to the characteristics of each environment.

  In the present embodiment, the size matching unit 12 is an example of a changing unit. The screen difference extraction unit 13 is an example of an extraction unit. The pass / fail judgment rule application unit 15 is an example of an application unit. The pass / fail judgment rule is an example of a rule. The screen element rectangle is an example of a partial area. The correct answer screen is an example of a first image showing the first display result. The comparison screen is an example of a second image showing the second display result.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

DESCRIPTION OF SYMBOLS 10 Screen difference acceptance / rejection determination apparatus 11 Image input part 12 Size matching part 13 Screen difference extraction part 14 Report output part 15 Pass / fail judgment rule application part 16 Pass / fail judgment rule storage part 100 Drive apparatus 101 Recording medium 102 Auxiliary storage apparatus 103 Memory apparatus 104 CPU
105 interface device 106 display device 107 input device B bus

Claims (5)

The size of any one of the first image indicating the first display result of the screen and the second image indicating the second display result of the screen is set to the screen element of the first image and the second image. A change unit that changes based on a ratio of sizes with respect to screen elements of the image;
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image And an extraction unit that extracts a difference between the first display result and the second display result in units of the partial area,
An application unit that applies a set rule to the difference extracted by the extraction unit and determines whether the difference is acceptable;
A screen difference acceptance / rejection determination device characterized by comprising: The rules include tolerances for differences that indicate movement in the X-axis direction, movement in the Y-axis direction, enlargement, or reduction.
The screen difference acceptance / rejection determination device according to claim 1. The rules are set according to the display conditions of the screen,
The application unit applies the rule corresponding to the display condition of the second display result, and determines pass / fail for the difference;
The screen difference acceptance / rejection determination device according to claim 1 or 2, characterized in that The size of any one of the first image indicating the first display result of the screen and the second image indicating the second display result of the screen is set to the screen element of the first image and the second image. A change procedure that changes based on the ratio of the size to the screen element of the image,
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image And an extraction procedure for extracting a difference between the first display result and the second display result in the unit of the partial area;
An application procedure for applying a set rule to the difference extracted by the extraction procedure and determining pass / fail for the difference;
Is executed by a computer. The size of any one of the first image indicating the first display result of the screen and the second image indicating the second display result of the screen is set to the screen element of the first image and the second image. A change procedure that changes based on the ratio of the size to the screen element of the image,
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image And an extraction procedure for extracting a difference between the first display result and the second display result in the unit of the partial area;
An application procedure for applying a set rule to the difference extracted by the extraction procedure and determining pass / fail for the difference;
A program that causes a computer to execute.

Images