JP6608719B2 - Screen difference extraction apparatus, screen difference extraction method, and program - Google Patents

Description

  The present invention relates to a screen difference extraction device, a screen difference extraction method, and a program.

  In recent years, a screen such as a Web page has been browsed in various environments (a combination of a terminal model, an OS (Operating System), a browser, and the like), but the screen layout may be different in each environment. For example, in a specific environment, layout disruption that is not intended by the creator of the screen may occur. Therefore, the creator of the screen needs to perform a test to confirm that the created screen is displayed as intended in each environment.

  Conventionally, such a test is generally performed by visually confirming a display result in each environment and a display result as intended, but the type of environment (that is, terminal model, Since there are many types of combinations of OS, browsers, etc., the workload of the tester is heavy.

  Therefore, for example, it is conceivable to improve the efficiency of the test by using a technique described in Non-Patent Document 1 or Non-Patent Document 2.

Akihiro Hori, Shingo Takada, Haruto Tanno and Morihide Oinuma, `` An Oracle based on Image Comparison for Regression Testing of Web Applications '', Proceedings of 27th International Conference on Software Engineering and Knowledge Engineering (SEKE 2015), ACM (USA), pp. 639-645 (July 2015). Sonal Mahajan and William G.J.Halfond, `` Finding html presentation failures using image comparison techniques '', Proceedings of the 29th ACM / IEEE international conference on Automated software engineering (ASE 2014), ACM (USA), pp.91-96 (2014).

  However, since the technique described in Non-Patent Document 1 requires information of a DOM (Document Object Model) tree of HTML (HyperText Markup Language) that is a source code of a Web page as input, Dependent. For example, it is difficult to use the technology in an environment that is not based on Web standard specifications, such as an Android (registered trademark) native application.

  Further, in Non-Patent Document 2, since the entire image is compared with the pixel granularity, it can be determined whether there is a difference, but it is difficult for the user to specify a portion having a difference. In addition, there is a problem in that images having different screen sizes cannot be compared properly because the images to be compared cannot be compared unless the resolutions are set in advance.

  The present invention has been made in view of the above points, and it is an object of the present invention to make it possible to easily identify the location of a difference between screens.

Therefore, in order to solve the above-described problem, the screen difference extraction device displays a first image indicating the first display result of the screen and a second display result in a display environment different from the first display result for the screen. An input unit for inputting a second image; and a size of one of the first image and the second image, a screen element of the first image, and a screen element of the second image A change unit that changes based on a ratio of sizes, and a partial region that is extracted from each of the first image and the second image after the size change, and that is extracted from the first image And an extraction unit that compares the partial area extracted from the second image and extracts a difference between the first display result and the second display result in units of the partial area. Have.

  It is possible to easily identify the location of the difference between the screens.

It is a figure which shows the hardware structural example of the screen difference extraction apparatus in embodiment of this invention. It is a figure which shows the function structural example of the screen difference extraction apparatus in embodiment of this invention. It is a flowchart for demonstrating an example of the process sequence which a screen difference extraction apparatus performs. It is a figure which shows the example of a definition of the data structure in embodiment of this invention. It is a figure which shows an example of a folder structure. It is a figure which shows an example of the layout of a correct answer screen and a comparison screen. It is a figure which shows the example of a display of the link screen to a result report. It is a figure which shows the example of a display of a result report. It is a flowchart for demonstrating an example of the process sequence of the input process of a screen image. It is a flowchart for demonstrating an example of the process procedure of size adjustment. It is a flowchart for demonstrating an example of the process sequence of the extraction process of screen difference information.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a hardware configuration example of a screen difference extraction device according to an embodiment of the present invention. The screen difference extraction device 10 in FIG. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, an input device 107, and the like that are mutually connected by a bus B. .

  A program for realizing processing in the screen difference extraction apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 realizes functions related to the screen difference extraction device 10 in accordance with a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a GUI (Graphical User Interface) or the like by a program. The input device 107 includes a keyboard and a mouse, and is used for inputting various operation instructions.

  FIG. 2 is a diagram illustrating a functional configuration example of the screen difference extraction device according to the embodiment of the present invention. In FIG. 2, the screen difference extraction apparatus 10 includes an image input unit 11, a size matching unit 12, a screen difference extraction unit 13, a report output unit 14, and the like. Each of these units is realized by processing that one or more programs installed in the screen difference extraction apparatus 10 cause the CPU 104 to execute.

  The image input unit 11 reads the correct screen and one or more comparison screens for each screen to be tested from the folder specified by the user. The image input unit 11 also generates pair information including a pair with the comparison screen for each correct screen. The correct screen is referred to as image data (hereinafter referred to as “screen image”) indicating the result of displaying the screen in an environment where the screen is displayed as intended by the creator (hereinafter referred to as “correct environment”). ). The comparison screen is a screen image showing a result displayed in an environment different from the correct answer environment (hereinafter referred to as “comparison environment”). The environment is distinguished by, for example, a combination of a terminal model that displays a screen, an OS (Operating System), a browser, and the like. Furthermore, the environment may be distinguished according to the display size of the browser. It can be said that the environment is an example of a display condition of the screen. Note that the correct answer screen and the comparison screen may be acquired by screen capture or the like, for example.

  The screen refers to a screen that the program displays on the computer. The screen may be a screen such as an application program, or may be a screen described in HTML (HyperText Markup Language) or the like.

  For each pair of the correct screen and the comparison screen, the size matching unit 12 absorbs the difference in resolution between the screen elements included in the two screen images related to the pair, so that the screen elements of the correct screen and the comparison screen The size of one screen image is changed based on the size ratio with the screen element. A screen element refers to a drawing element of a screen, and is, for example, a unit constituting a screen layout. Note that the size of the screen image is expressed by the number of pixels (number of pixels). The size change is realized by changing the resolution.

  The screen difference extraction unit 13 extracts, for each pair, a difference between two screen images related to the pair. Specifically, the screen difference extraction unit 13 extracts a rectangular area with approximately the granularity of the screen elements for each of the two screen images related to the pair, and performs matching in units of rectangular areas. As a result of matching, the screen difference extraction unit 13 detects differences such as disappearance, addition, and movement with the granularity of the rectangular area, and generates screen difference information indicating the contents of the differences for each difference. The screen difference extraction unit 13 sets the generated screen difference information for the pair information.

  The report output unit 14 outputs each pair information including the screen difference information generated by the screen difference extraction unit 13 as a result report, and presents the result report to the user.

  Hereinafter, a processing procedure executed by the screen difference extraction device 10 will be described. FIG. 3 is a flowchart for explaining an example of a processing procedure executed by the screen difference extraction apparatus.

  In step S110, the image input unit 11 reads the correct screen and the comparison screen group stored for each screen to be tested under the folder specified by the user, and the comparison screen related to the same screen for each correct screen. A list of pair information with is generated.

  FIG. 4 is a diagram showing a definition example of the data structure in the embodiment of the present invention. In FIG. 4, the data structures of various data used in the present embodiment are defined by BNF (Backus-Naur Form).

  Regarding step S110, the definition in “■ Image group to be input data” is related. Here, it is shown that the environment root folder includes one correct environment folder and one or more comparison environment folders. Further, it is shown that each of the correct answer environment folder and each comparison environment folder includes an image file storing a screen image (correct answer screen or comparison screen). The image file included in the correct environment folder is an image file storing the correct answer screen, and the image file included in the comparison environment folder is an image file storing the comparison screen.

  FIG. 5 is a diagram illustrating an example of a folder configuration. FIG. 5 shows an example in which one correct environment folder and two comparison environment folders (comparison environment folders (a) and (b)) are included under the environment root folder. An image file X and an image file Y are stored in each of the correct answer environment folder and each comparison environment folder. The image file X is a file that stores a screen image generated by screen capture of the display result of the screen X. The image file Y is a file that stores a screen image generated by screen capture of the display result of the screen Y. However, the image files X and Y under the correct answer environment folder are image files that store the correct answer screens related to the screen X or the screen Y. The image files X and Y under the comparison environment folder (a) are image files storing a comparison screen indicating the display result of the screen X or the screen Y in the environment (a). The image files X and Y under the comparison environment folder (b) are image files storing a comparison screen showing the display result of the screen X or the screen Y in the environment (b). The folders and files shown in FIG. 5 are stored in the auxiliary storage device 102, for example.

  According to the example of FIG. 5, in step S110, a pair of a correct screen related to the image file X under the correct environment folder and a correct screen related to the image file X under the comparison environment folder (a), and the correct environment Comparison between the correct screen related to the image file Y under the folder and the correct screen related to the image file Y under the comparison environment folder (a), and the correct screen related to the image file X under the correct environment folder Pair with correct screen related to image file X under environment folder (b), correct screen related to image file Y under correct environment folder, and correct answer related to image file Y under comparison environment folder (b) Pair information is generated for the four pairs of the screen and the pair.

  The pair information includes correct screen information, comparison screen information, and a screen difference list as described in the column “■ Report as output data” in FIG. Correct screen information is screen information of a correct screen. The comparison screen information is screen information of the comparison screen. The screen information is information including the image file of the screen and the path name of the image file. The screen difference list will be described later.

  Hereinafter, for the sake of convenience, the correct answer screen related to the screen X is referred to as “correct answer screen X”. The comparison screen in the environment (a) related to the screen X is referred to as “comparison screen X (a)”. Further, the comparison screen in the environment (a) related to the screen X is referred to as “comparison screen X (b)”.

  FIG. 6 is a diagram illustrating an example of the layout of the correct answer screen and the comparison screen. FIG. 6 shows an example of the layout of the correct screen X, the comparison screen X (a), and the comparison screen X (b) with respect to the screen X. The size of each screen is based on the difference in the size (resolution) of each screen. As described above, the same screen X (for example, the same HTML (HyperText Markup Language) or the like) may have a difference in size or layout depending on the environment.

  Subsequently, the image input unit 11 generates an empty data area for the result report (step S120).

  Subsequent steps S130 to S160 are executed for each pair information generated in step S110. Hereinafter, the pair information to be processed is referred to as “target pair information”.

  In step S130, the size matching unit 12 matches the sizes of the screen elements of the two screen images by changing the size (resolution) of one of the two screen images related to the target pair information. For example, in FIG. 6, the correct screen X and the comparison screen X (a) are different in size. Accordingly, the size of either one of the two screen image pairs is changed and adjusted so that the sizes of both screen images coincide with each other at the screen element level.

  Subsequently, the screen difference extraction unit 13 extracts a rectangular area from two screen images related to the target pair information and specifies a difference in units of the rectangular area (step S140). As a result, a screen difference list which is a list of screen difference information indicating the specified differences is set for the target pair information. For example, in the case of the correct answer screen X and the comparison screen X (a) shown in FIG. 6, the rectangular areas of “Title”, “Advertisement”, “Sentence”, “Photo”, and “Link” are extracted. In the comparison screen (b), the rectangular areas of “title”, “sentence”, “photograph”, “link”, and “link that does not exist in the specification” are extracted. Hereinafter, the extracted rectangular area is referred to as a “screen element rectangle”.

  As will be described later, since the extraction of the screen element rectangle is performed by analyzing the image, the screen element rectangle extracted in step S140 is not necessarily a screen element in a meaningful unit as viewed from a human. It is not necessarily rectangular. For example, the “title” area may be divided into a plurality of screen element rectangles and extracted. In addition, there is a possibility that a screen element rectangle extending over the “title” and “advertisement” areas is extracted.

  Subsequently, the report output unit 14 adds target pair information to the result report (step S160).

  When Steps S130 to S160 are executed for all pairs, the report output unit 14 outputs a result report to the display device 106 (Step S170).

  FIG. 7 is a diagram illustrating a display example of a link screen to a result report. As shown in FIG. 7, the link screen displays a link to the result report for each combination of screen type (screen X, screen Y) and environment type (environment (a), environment (b)). Including. When any link is selected by the user, a result report as shown in FIG. 8 is displayed.

  FIG. 8 is a diagram illustrating a display example of a result report. FIG. 8 shows a display example of two result reports R1 and R2.

  The result report R1 is a result report for a combination of the screen X and the environment (a) (a pair of the correct screen X and the comparison screen X (a)). The result report R2 is a result report for a combination of the screen X and the environment (b) (a pair of the correct answer screen X and the comparison screen X (b)).

  Each result report includes a correct answer screen, a comparison screen, and a table showing a list of differences between the two screen images. In the table, for each difference, the item number, the difference type, and the mutation value are displayed. The item number is a number assigned to each difference, and the item number related to the difference is given to the screen element rectangle corresponding to the difference on the correct answer screen and the comparison screen. The difference type is a type of difference, and the value is “no difference”, “movement in the X-axis direction”, “movement in the Y-axis direction”, “enlargement / reduction”, “disappearance”, or “addition”. The variation value is a value indicating the magnitude of the difference related to the difference type.

  Next, details of step S110 in FIG. 3 will be described. FIG. 9 is a flowchart for explaining an example of the processing procedure of the screen image input processing.

  In step S210, the image input unit 11 reads the environment root folder designated by the user. Subsequently, the image input unit 11 acquires a correct answer environment folder under the environment root folder (step S220). For example, the folder name of the correct answer environment folder may be determined in advance, and the folder related to the folder name may be acquired.

  Subsequently, the image input unit 11 generates an empty list (hereinafter referred to as “pair list”) for storing pair information to be generated (step S230). Subsequently, the image input unit 11 executes step S240 and subsequent steps for each correct answer screen stored in the correct answer environment folder. The correct answer screen that is the processing target after step S240 is referred to as a “target correct answer screen”.

  In step S240, the image input unit 11 acquires all comparison environment folders (hereinafter referred to as “comparison environment folder group”) under the environment root folder.

  Subsequently, the image input unit 11 executes steps S250 and S270 for each acquired comparison environment folder. The comparison environment folder to be processed in steps S250 and S270 is referred to as “target comparison environment folder”.

  In step S250, the image input unit 11 acquires a comparison screen corresponding to the target correct answer screen from the target comparison environment folder, and generates pair information for the target correct answer screen and the comparison screen. The comparison screen corresponding to the target correct answer screen is a comparison screen corresponding to the same screen as the screen corresponding to the target correct answer screen. For example, screen images with common screens may be stored in files with the same file name. By doing so, the comparison screen corresponding to the correct answer screen can be easily specified.

  Subsequently, the image input unit 11 adds the generated pair information to the pair list (step S270).

  Next, details of step S130 in FIG. 3 will be described. FIG. 10 is a flowchart for explaining an example of the size adjustment processing procedure.

  In step S310, the size matching unit 12 reads the target pair information into its working memory area.

  Subsequently, the size matching unit 12 extracts a feature point from each of the correct screen and the comparison screen related to the target pair information, performs feature point matching in the direction of the correct screen → the comparison screen, and between the correct screen and the comparison screen. A feature point pair group that is a set of corresponding feature point pairs is generated (step S320). The feature point matching can be performed using a known algorithm such as the Harris method or the Shi-Toshi method.

  The data structure of one feature point pair is as described in the column “■ Intermediate data” in FIG. That is, the feature point pair is composed of the coordinate point of the feature point on the correct answer screen and the coordinate point of the feature point corresponding to the feature point on the comparison screen.

  Subsequently, the size matching unit 12 selects two feature point pairs that are sufficiently separated from each other from the feature point pair group (step S330). For example, among all combinations of two feature point pairs, two feature point pairs related to the combination having the maximum sum of the distance on the correct screen and the distance on the comparison screen may be selected. Alternatively, two feature point pairs related to a combination in which the sum of the distance on the correct answer screen and the distance on the comparison screen is greater than or equal to the threshold may be selected. The threshold value may be obtained based on the width size of the correct answer screen or the comparison screen, or may be obtained by another method.

  Subsequently, the size matching unit 12 calculates a size ratio between the screen element rectangle of the correct answer screen and the screen element rectangle of the comparison screen based on the two selected feature point pairs (step S340). Specifically, the coordinate point on the correct screen of one feature point pair (hereinafter referred to as “feature point pair 1”) and the correct screen of the other feature point pair (hereinafter referred to as “feature point pair 2”). The distance of the coordinate point at is calculated and set as the correct screen distance. Further, the distance between the coordinate point on the comparison screen of the feature point pair 1 and the coordinate point on the comparison screen of the feature point pair 2 is calculated and set as the comparison screen distance. Then, the distance within the correct screen / the distance within the comparison screen is calculated, and this is used as the size ratio. That is, the size ratio is not necessarily the ratio between the size of the entire correct screen and the size of the entire comparison screen. In the present embodiment, since the comparison is performed in units of screen element rectangles, it is important to match the size of the screen element rectangles between the correct answer screen and the comparison screen.

  The size ratio may be calculated by a method other than the above. For example, the width of the correct screen / the width of the comparison screen may be calculated as the size ratio. Alternatively, the distance between screen element rectangles on the correct screen / the distance between screen element rectangles on the comparison screen may be calculated as the size ratio. Also, the width of the correct screen / the width of the screen in the comparison environment may be calculated as the size ratio. Here, the width of the screen in the comparison environment is the width of the display of the terminal in the environment corresponding to the comparison screen, and may be stored in advance in the auxiliary storage device 102 or the like for each environment, for example. Such a calculation method is effective for an environment where the screen is displayed in accordance with the horizontal width of the display.

  Subsequently, the size matching unit 12 changes the size of the comparison screen based on the size ratio (step S350). Specifically, resolution conversion is performed on the comparison screen so that the vertical and horizontal sizes (number of pixels) of the comparison screen are values obtained by multiplying the size ratios, respectively. In the target pair information, the comparison screen before the size change is replaced with the comparison screen after the size change. Note that the size of the correct answer screen may be changed.

  Next, details of step S140 in FIG. 3 will be described. FIG. 11 is a flowchart for explaining an example of a processing procedure of screen difference information extraction processing.

  In step S410, the screen difference extraction unit 13 reads the target pair information into its own working memory area.

  Subsequently, the screen difference extraction unit 13 generates an empty data area for storing the screen difference list (step S415). The screen difference list is composed of zero or more pieces of screen difference information as described in the column “Report as output data” in FIG.

  Subsequently, the screen difference extraction unit 13 extracts a feature point pair group from the correct screen and the comparison screen related to the target pair information by the same method as in step S320 of FIG. 10 (step S420).

  Subsequently, the screen difference extraction unit 13 extracts a screen element rectangle (hereinafter referred to as “correct answer screen element rectangle”) from the correct answer screen related to the target pair information, and a screen in which the extracted correct answer screen element rectangles are listed. An element rectangle list is generated (step S425). The screen element rectangle is represented by the upper left coordinates and upper right coordinates of the rectangle as described in the column “■ Report as output data” in FIG. The extraction of the screen element rectangle can be realized by combining known techniques. For example, a screen element rectangle is extracted by executing procedures such as gray scale, canny processing, expansion processing, contour extraction, and rectangle conversion (obtaining a circumscribed rectangle of the contour and extracting the rectangle therefrom) on the image. May be.

  Note that the screen element rectangle is not necessarily a “rectangle”. It suffices if the partial areas constituting the screen are extracted. Therefore, L-shaped, convex, concave, or other polygonal partial areas may be extracted as screen element rectangles.

  Subsequently, the screen difference extraction unit 13 executes Steps S430 to S475 for each correct screen element rectangle included in the generated screen element rectangle list. The correct screen element rectangle to be processed in steps S430 to S475 is referred to as “target correct screen element rectangle”.

  In step S430, the screen difference extraction unit 13 has two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle in the comparison screen (hereinafter referred to as “target comparison screen”) related to the target pair information. It is determined whether or not to do. The feature point group related to the target correct screen element rectangle refers to, for example, feature points included in the range of the target correct screen element rectangle. However, feature points included in a predetermined range outside from the periphery of the target correct screen element rectangle may be included in the feature point group related to the target correct screen element rectangle. Further, whether or not there are two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle on the target comparison screen is a feature point pair including the feature point group related to the target correct answer screen element rectangle (hereinafter, “ It can be determined based on whether or not two or more “feature feature point pairs” are included in the feature point pair group generated in step S420.

  When two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle exist on the target comparison screen (Yes in step S430), the screen difference extraction unit 13 selects from two or more target feature point pairs that exist. Then, two feature point pairs that are sufficiently apart from each other are selected, and the target correct screen element rectangle and the screen element corresponding to the target correct screen element rectangle in the target comparison screen are selected in the same manner as in step S340 of FIG. An expansion / contraction ratio with the rectangle is calculated (step S435).

  Subsequently, the screen difference extraction unit 13 searches the comparison screen for a screen element rectangle corresponding to the target correct screen element rectangle, and the corresponding screen element rectangle (hereinafter referred to as “target comparison screen element rectangle”) is searched. In this case, the target correct screen element rectangle and the target comparison screen element rectangle are set in the screen difference information (step S440). Specifically, the screen difference extraction unit 13 calculates the positional relationship between the target correct screen element rectangle and the feature points related to the correct screen in the target feature point pair (for example, a vector from each vertex of the target correct screen element rectangle). A candidate rectangle of the target correct screen element rectangle is cut out from the comparison screen based on the feature point and the positional relationship related to the comparison screen in the target feature point pair. The screen difference extraction unit 13 compares the pixel values of the target correct screen element rectangle and the candidate rectangle, and the pixel value matching rate (pixel unit matching rate) is equal to or greater than a predetermined value (for example, 100% or 95%). If so, the candidate rectangle is set as a target comparison screen element rectangle. However, when the enlargement / reduction ratio is not 1, the size of the candidate rectangle is enlarged or reduced based on the enlargement / reduction ratio, and the pixel value is compared after the size of the candidate rectangle matches the target correct screen element rectangle. .

  The target comparison screen element rectangle may be cut out by pattern matching between the target correct screen element rectangle and the target comparison screen. For example, after changing the size of the target comparison screen based on the enlargement / reduction ratio, the target correct screen element rectangle is a rectangular area having the same size as the target correct screen element rectangle and all pixels (or a predetermined ratio (for example, The target comparison screen element rectangle may be cut out by searching for a rectangular region with a matching pixel value with respect to (95%) or more pixels).

  The screen difference extraction unit 13 generates screen difference information, and sets the target correct screen element rectangle as the screen difference information (hereinafter referred to as “target screen difference information”). As shown in the column “■ Report as output data” in FIG. 4, the screen difference information includes an item number, a difference type, zero or more variation values, zero or more correct screen element rectangles, and zero or more Information including a comparison screen element rectangle. The correct screen element rectangle and the comparison screen element rectangle are expressed by the coordinates of the upper left vertex and the lower right vertex, respectively.

  When the target comparison screen element rectangle is searched, the screen difference extraction unit 13 sets the target comparison screen element rectangle in the target screen difference information. When the target comparison screen element rectangle is not searched, the target comparison screen element rectangle is not set in the target screen difference information.

  Note that “no difference” is set as the initial value for the difference type of the target screen difference information.

  Subsequently, when the target comparison screen element rectangle exists (Yes in step S445), the screen difference extraction unit 13 determines whether or not the positions of the target correct screen element rectangle and the target comparison screen element rectangle are different (S450). ). The difference between the positions may be determined based on whether or not the coordinates of the upper left vertices of the target correct screen element rectangle and the target comparison screen element rectangle are the same.

  When the positions of the target correct screen element rectangle and the target comparison screen element rectangle are different (Yes in step S450), the screen difference extraction unit 13 selects “X-axis direction movement” or “Y-axis direction” as the difference type of the target screen difference information. “Move” is set, and the difference (movement amount) of the position is set as the variation value of the target screen difference information (step S455). The difference between the positions is calculated as, for example, the absolute value of the difference between the X coordinate or the Y coordinate of the upper left vertex between the target correct screen element rectangle and the target comparison screen element rectangle. Specifically, when the difference in the X-axis direction is not 0, “X-axis direction movement” is set as the difference type of the target screen difference information, and the movement amount in the X-axis direction is set as the variation value of the target screen difference information. Is done. When the difference in the Y-axis direction is not 0, “Y-axis direction movement” is set as the difference type of the target screen difference information, and the movement amount in the Y-axis direction is set as the variation value of the target screen difference information. If both the difference in the X-axis direction and the difference in the Y-axis direction are not 0, a copy of the target screen difference information is generated, and the difference type and the mutation value of one target screen difference information are set to “X-axis direction movement”, X The movement amount in the axial direction is set, and “movement in the Y-axis direction” and the movement amount in the Y-axis direction are set in the difference type and the variation value of the other target screen difference information.

  Subsequently, the screen difference extraction unit 13 determines whether or not the enlargement / reduction ratio calculated in step S435 is 1 (step S460). If the enlargement / reduction ratio is not 1 (No in step S460), the screen difference extraction unit 13 sets “enlargement / reduction” as the difference type of the target screen difference information, and sets the enlargement / reduction ratio as the mutation value of the target screen difference information (step). S465). When a value is set in step S455 for the target screen difference information, the screen difference extraction unit 13 generates a copy of the target screen difference information, and for the difference type and mutation value of the copy, “ Set the scaling ratio.

  On the other hand, when two or more feature points corresponding to the feature point group related to the target correct screen element rectangle do not exist in the target comparison screen (No in step S430), or when no target comparison screen element rectangle exists (No in step S445). The screen difference extraction unit 13 sets “disappear” as the difference type of the target screen difference information (step S470).

  Subsequent to step S460, S465, or S470, the screen difference extraction unit 13 adds the generated target screen difference information to the screen difference list (step S475). When a plurality of target screen difference information is generated, the plurality of target screen difference information is added to the screen difference list. Further, when adding the target screen difference information to the screen difference list, the screen difference extraction unit 13 sets a numerical value corresponding to the order of the target screen difference information in the screen difference list as the item number of the target screen difference information. To do.

  When steps S430 to S475 are executed for all correct screen element rectangles included in the screen element rectangle list, the screen difference extraction unit 13 executes steps S420 to S475 by exchanging the correct screen and the comparison screen. (Step S480). In this case, in step S470, “addition” is set for the difference type of the target screen difference information. This is because, when matching is performed in the direction of the comparison screen → the correct answer screen, when step S470 is executed, the screen element rectangle existing on the comparison screen does not exist on the correct answer screen.

  Subsequently, the screen difference extraction unit 13 sets the screen difference list as target pair information (step S485).

  Based on the target pair information generated in this way, the link screen shown in FIG. 7 and the result report shown in FIG. 8 are output.

  As described above, according to the present embodiment, when a test is executed in n test environments, the screen image of the test result in one test environment is set as a correct answer screen, and the other n-1 environments. The difference is automatically extracted by comparing the screen image (comparison screen) of the test result with the correct screen. At this time, the difference is extracted in units of screen element rectangles. Therefore, it is possible to easily detect the screen layout collapse and to easily identify the differences between the screens. As a result, the user can efficiently perform pass / fail determination on the comparison screen.

  In the present embodiment, based on the size ratio between the screen element of the correct screen and the screen element of the comparison screen, the difference is extracted after one image size is changed. As a result, it is possible to avoid erroneous detection caused by the difference in resolution between the correct answer screen and the comparison screen, and to detect the difference with higher accuracy. In addition, the screen element rectangles can be compared more appropriately by using the size ratio in consideration of the positional relationship of the screen elements grasped through the feature point pairs instead of simply the resolution ratio. For example, it is assumed that the size of the correct screen (vertical × horizontal) is 1000 × 500, and the size of the comparison screen is 1500 × 600. However, in the comparison screen, the range in which the screen is drawn is a range of (0 to 1000) × (0 to 500), and the other ranges are blanks. That is, it is assumed that the screen element rectangle of the correct answer screen is the same as the screen element rectangle of the comparison screen. In this case, simply, if one size is changed based on the ratio between the size of the correct answer screen and the size of the comparison screen, the aspect ratio of the screen element rectangle included in the one is destroyed. On the other hand, according to the size ratio based on the feature point pair, the possibility of avoiding such inconvenience increases.

  Further, in the present embodiment, screen element rectangles are extracted from each of the correct answer screen and the comparison screen, and then the screen element is divided into two directions: a direction from the correct answer screen to the comparison screen and a direction from the comparison screen to the correct answer screen. Rectangle matching is performed. Therefore, various differences such as movement, expansion / contraction, disappearance, and addition of the screen element rectangle can be detected with an easy-to-understand granularity, and the screen layout can be easily determined.

  In the present embodiment, the size matching unit 12 is an example of a changing unit. The screen difference extraction unit 13 is an example of an extraction unit. The report output unit 14 is an example of an output unit. The screen element rectangle is an example of a partial area. The correct answer screen is an example of a first image showing the first display result. The comparison screen is an example of a second image showing the second display result.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

DESCRIPTION OF SYMBOLS 10 Screen difference extraction apparatus 11 Image input part 12 Size matching part 13 Screen difference extraction part 14 Report output part 100 Drive apparatus 101 Recording medium 102 Auxiliary storage apparatus 103 Memory apparatus 104 CPU
105 interface device 106 display device 107 input device B bus

Claims (7)

An input unit for inputting a first image showing a first display result of a screen and a second image showing a second display result in a display environment different from the first display result for the screen;
A changing unit that changes the size of one of the first image and the second image based on a ratio of the sizes of the screen element of the first image and the screen element of the second image; ,
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image Are compared in two directions, the direction from the first image to the second image and the direction from the second image to the first image, and the first display result an extraction unit to extract at a unit of the movement of the second display results, scaling, the difference comprising a loss or addition, the partial region,
A screen difference extraction apparatus characterized by comprising: The extraction unit is extracted from the first image based on a correspondence relationship between a plurality of feature points extracted from the first image and a plurality of feature points extracted from the second image. Identifying a correspondence between the partial region and the partial region extracted from the second image, comparing the partial regions having the correspondence, and extracting the difference;
The screen difference extraction apparatus according to claim 1, wherein: The extraction unit compares the partial areas having the correspondence relationship with each other and compares the partial areas having a pixel unit matching rate equal to or higher than a predetermined value, and extracts the difference.
The screen difference extraction apparatus according to claim 2, wherein: The changing unit includes a distance between two feature points extracted from the first image and a distance between two feature points corresponding to the two feature points extracted from the second image. Or the ratio between the width of the first image and the width of the second image, or the distance between the partial areas extracted from the first image and the portion extracted from the second image Changing the size of one of the first image and the second image based on a ratio to a distance between regions;
The screen difference extraction device according to claim 1, wherein the screen difference extraction device is a screen difference extraction device. A list of partial area unit differences extracted by the extraction unit, the first image showing the partial area unit differences, and the second area showing the partial area unit differences. An output unit for outputting the image of
The screen difference extraction device according to claim 1, wherein the screen difference extraction device is a screen difference extraction device. An input procedure for inputting a first image showing a first display result of a screen and a second image showing a second display result in a display environment different from the first display result for the screen;
A change procedure for changing the size of one of the first image and the second image based on a ratio of the sizes of the screen element of the first image and the screen element of the second image; ,
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image Are compared in two directions, the direction from the first image to the second image and the direction from the second image to the first image, and the first display result movement of the second display results, scaled, the differences involve erasure or additions, the extraction procedure to extract in units of the partial region,
A screen difference extraction method characterized in that the computer executes the above. An input procedure for inputting a first image showing a first display result of a screen and a second image showing a second display result in a display environment different from the first display result for the screen;
A change procedure for changing the size of one of the first image and the second image based on a ratio of the sizes of the screen element of the first image and the screen element of the second image; After the size change, a partial region is extracted from each of the first image and the second image, and the partial region extracted from the first image and the portion extracted from the second image The first display result is obtained by comparing the area in two directions: a direction from the first image to the second image and a direction from the second image to the first image. an extraction procedure to extract between the movement of the second display results, scaled, the differences involve erasure or additional, the unit of the partial region,
A program that causes a computer to execute.

Images