JP2017138714A - Screen difference confirmation support device, screen difference confirmation support method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a screen difference confirmation support device, screen difference confirmation support method, and program for supporting confirmation of a position of difference between screens.SOLUTION: A screen difference confirmation support device 10 comprises: a size matching part 12 that changes the size of either one of a first image showing a first display result of a screen and a second image showing a second display result, on the basis of the ratio of the size related to screen elements of the first image and screen elements of the second image; a screen difference extraction part 13 that extracts partial areas respectively from the first image and the second image after the size change, compares the partial area extracted from the first image with the partial area extracted from the second image, and extracts the difference between a first display result and a second display result in a unit of the partial area; and a manual determination result providing part 15 that displays the first image and second image and highlights the partial area related to the extracted difference in the first image and second image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a screen difference confirmation support device, a screen difference confirmation support method, and a program.

  In recent years, a screen such as a Web page has been browsed in various environments (a combination of a terminal model, an OS (Operating System), a browser, and the like), but the screen layout may be different in each environment. For example, in a specific environment, layout disruption that is not intended by the creator of the screen may occur. Therefore, the creator of the screen needs to perform a test to confirm that the created screen is displayed as intended in each environment.

  Conventionally, such a test is generally performed by visually confirming a display result in each environment and a display result as intended, but the type of environment (that is, terminal model, Since there are many types of combinations of OS, browsers, etc., the workload of the tester is heavy.

  Thus, for example, it is conceivable to improve the efficiency of the test by using a technique as described in Non-Patent Document 1.

Akihiro Hori, Shingo Takada, Haruto Tanno and Morihide Oinuma, `` An Oracle based on Image Comparison for Regression Testing of Web Applications '', Proceedings of 27th International Conference on Software Engineering and Knowledge Engineering (SEKE 2015), ACM (USA), pp. 639-645 (July 2015). Sonal Mahajan and William G.J.Halfond, `` Finding html presentation failures using image comparison techniques '', Proceedings of the 29th ACM / IEEE international conference on Automated software engineering (ASE 2014), ACM (USA), pp.91-96 (2014).

  However, since the technique described in Non-Patent Document 1 requires information of a DOM (Document Object Model) tree of HTML (HyperText Markup Language) that is a source code of a Web page as input, Dependent. For example, it is difficult to use the technology in an environment that is not based on Web standard specifications, such as an Android (registered trademark) native application.

  Further, in Non-Patent Document 2, since the entire image is compared with the pixel granularity, it can be determined whether there is a difference, but it is difficult for the user to specify a portion having a difference. In addition, there is a problem in that images having different screen sizes cannot be compared properly because the images to be compared cannot be compared unless the resolutions are set in advance.

  Even if the difference can be specified between the screens, it is difficult for the user to efficiently check the difference portion if the difference presentation method is not appropriate.

  The present invention has been made in view of the above points, and an object of the present invention is to be able to support confirmation of differences between screens.

  Therefore, in order to solve the above-described problem, the screen difference confirmation support device has a size of either the first image indicating the first display result of the screen or the second image indicating the second display result of the screen. Is changed based on the ratio of the sizes of the screen element of the first image and the screen element of the second image, and after the change of the size, the first image and the second image A partial region is extracted from each of the first region and the partial region extracted from the first image is compared with the partial region extracted from the second image. An extraction unit that extracts a difference between a display result and the second display result, and the first image and the second image are displayed and extracted in the first image and the second image. A display control unit that highlights a partial area related to the difference. That.

  It is possible to support the confirmation of the difference between the screens.

It is a figure which shows the hardware structural example of the screen difference confirmation assistance apparatus in embodiment of this invention. It is a figure which shows the function structural example of the screen difference confirmation assistance apparatus in embodiment of this invention. It is a flowchart for demonstrating an example of the process sequence which a screen difference confirmation assistance apparatus performs. It is a figure which shows the example of a definition of the data structure in embodiment of this invention. It is a figure which shows an example of a folder structure. It is a figure which shows an example of the layout of a correct answer screen and a comparison screen. It is a figure which shows the example of a display of the link screen to a result report. It is a figure which shows the example of a display of a result report. It is a flowchart for demonstrating an example of the process sequence of the input process of a screen image. It is a flowchart for demonstrating an example of the process procedure of size adjustment. It is a flowchart for demonstrating an example of the process sequence of the extraction process of screen difference information. It is a flowchart for demonstrating an example of the process sequence of the provision process of the user determination with respect to the difference between screens. It is a figure which shows the 1st example of a display of a user determination input screen. It is a figure which shows the 2nd example of a display of a user determination input screen.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a hardware configuration example of a screen difference confirmation support apparatus according to an embodiment of the present invention. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, an input device 107, and the like that are mutually connected by a bus B. Have.

  A program that realizes processing in the screen difference confirmation support apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 realizes functions related to the screen difference confirmation support device 10 according to a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a GUI (Graphical User Interface) or the like by a program. The input device 107 includes a keyboard and a mouse, and is used for inputting various operation instructions.

  FIG. 2 is a diagram illustrating a functional configuration example of the screen difference confirmation support device according to the embodiment of the present invention. 2, the screen difference confirmation support apparatus 10 includes an image input unit 11, a size matching unit 12, a screen difference extraction unit 13, a report output unit 14, a manual determination result adding unit 15, and the like. Each of these units is realized by a process that the CPU 104 causes one or more programs installed in the screen difference confirmation support apparatus 10 to execute.

  The image input unit 11 reads the correct screen and one or more comparison screens for each screen to be tested from the folder specified by the user. The image input unit 11 also generates pair information including a pair with the comparison screen for each correct screen. The correct screen is referred to as image data (hereinafter referred to as “screen image”) indicating the result of displaying the screen in an environment where the screen is displayed as intended by the creator (hereinafter referred to as “correct environment”). ). The comparison screen is a screen image showing a result displayed in an environment different from the correct answer environment (hereinafter referred to as “comparison environment”). The environment is distinguished by, for example, a combination of a terminal model that displays a screen, an OS (Operating System), a browser, and the like. Furthermore, the environment may be distinguished according to the display size of the browser. It can be said that the environment is an example of a display condition of the screen. Note that the correct answer screen and the comparison screen may be acquired by screen capture or the like, for example.

  The screen refers to a screen that the program displays on the computer. The screen may be a screen such as an application program, or may be a screen described in HTML (HyperText Markup Language) or the like.

  For each pair of the correct screen and the comparison screen, the size matching unit 12 absorbs the difference in resolution between the screen elements included in the two screen images related to the pair, so that the screen elements of the correct screen and the comparison screen The size of one screen image is changed based on the size ratio with the screen element. A screen element refers to a drawing element of a screen, and is, for example, a unit constituting a screen layout. Note that the size of the screen image is expressed by the number of pixels (number of pixels). The size change is realized by changing the resolution.

  The screen difference extraction unit 13 extracts, for each pair, a difference between two screen images related to the pair. Specifically, the screen difference extraction unit 13 extracts a rectangular area with approximately the granularity of the screen elements for each of the two screen images related to the pair, and performs matching in units of rectangular areas. As a result of matching, the screen difference extraction unit 13 detects differences such as disappearance, addition, and movement with the granularity of the rectangular area, and generates screen difference information indicating the contents of the differences for each difference. The screen difference extraction unit 13 sets the generated screen difference information for the pair information.

  For each pair, the manual determination result assigning unit 15 highlights (highlights) a portion where there is a difference between the correct answer screen and the comparison screen based on the screen difference information extracted by the screen difference extracting unit 13. The determination result of pass or fail for each difference is received from the user. The manual determination result adding unit 15 adds the received determination result to each screen difference information.

  The report output unit 14 outputs, for each pair, each pair information including the screen difference information generated by the screen difference extraction unit 13 and given the pass / fail determination result by the user as a result report, and outputs the result report to the user. Present.

  The report output unit 14 outputs each pair information including the screen difference information generated by the screen difference extraction unit 13 as a result report, and presents the result report to the user.

  Hereinafter, a processing procedure executed by the screen difference confirmation support device 10 will be described. FIG. 3 is a flowchart for explaining an example of a processing procedure executed by the screen difference confirmation support device.

  In step S110, the image input unit 11 reads the correct screen and the comparison screen group stored for each screen to be tested under the folder specified by the user, and the comparison screen related to the same screen for each correct screen. A list of pair information with is generated.

  FIG. 4 is a diagram showing a definition example of the data structure in the embodiment of the present invention. In FIG. 4, the data structures of various data used in the present embodiment are defined by BNF (Backus-Naur Form).

  Regarding step S110, the definition in “■ Image group to be input data” is related. Here, it is shown that the environment root folder includes one correct environment folder and one or more comparison environment folders. Further, it is shown that each of the correct answer environment folder and each comparison environment folder includes an image file storing a screen image (correct answer screen or comparison screen). The image file included in the correct environment folder is an image file storing the correct answer screen, and the image file included in the comparison environment folder is an image file storing the comparison screen.

  FIG. 5 is a diagram illustrating an example of a folder configuration. FIG. 5 shows an example in which one correct environment folder and two comparison environment folders (comparison environment folders (a) and (b)) are included under the environment root folder. An image file X and an image file Y are stored in each of the correct answer environment folder and each comparison environment folder. The image file X is a file that stores a screen image generated by screen capture of the display result of the screen X. The image file Y is a file that stores a screen image generated by screen capture of the display result of the screen Y. However, the image files X and Y under the correct answer environment folder are image files that store the correct answer screens related to the screen X or the screen Y. The image files X and Y under the comparison environment folder (a) are image files storing a comparison screen indicating the display result of the screen X or the screen Y in the environment (a). The image files X and Y under the comparison environment folder (b) are image files storing a comparison screen showing the display result of the screen X or the screen Y in the environment (b). The folders and files shown in FIG. 5 are stored in the auxiliary storage device 102, for example.

  According to the example of FIG. 5, in step S110, a pair of a correct screen related to the image file X under the correct environment folder and a correct screen related to the image file X under the comparison environment folder (a), and the correct environment Comparison between the correct screen related to the image file Y under the folder and the correct screen related to the image file Y under the comparison environment folder (a), and the correct screen related to the image file X under the correct environment folder Pair with correct screen related to image file X under environment folder (b), correct screen related to image file Y under correct environment folder, and correct answer related to image file Y under comparison environment folder (b) Pair information is generated for the four pairs of the screen and the pair.

  The pair information includes correct screen information, comparison screen information, a screen difference list, and a pass / fail judgment result as described in the column “■ Report as output data” in FIG. Correct screen information is screen information of a correct screen. The comparison screen information is screen information of the comparison screen. The screen information is information including the image file of the screen and the path name of the image file. The screen difference list will be described later. The pass / fail determination result is information indicating the pass / fail automatically determined for the comparison screen related to the pair information, and can take a value of “pass” or “fail”.

  Hereinafter, for the sake of convenience, the correct answer screen related to the screen X is referred to as “correct answer screen X”. The comparison screen in the environment (a) related to the screen X is referred to as “comparison screen X (a)”. Further, the comparison screen in the environment (a) related to the screen X is referred to as “comparison screen X (b)”.

  FIG. 6 is a diagram illustrating an example of the layout of the correct answer screen and the comparison screen. FIG. 6 shows an example of the layout of the correct screen X, the comparison screen X (a), and the comparison screen X (b) with respect to the screen X. The size of each screen is based on the difference in the size (resolution) of each screen. As described above, the same screen X (for example, the same HTML (HyperText Markup Language) or the like) may have a difference in size or layout depending on the environment.

  Subsequently, the image input unit 11 generates an empty data area for the result report (step S120).

  Subsequent steps S130 to S160 are executed for each pair information generated in step S110. Hereinafter, the pair information to be processed is referred to as “target pair information”.

  In step S130, the size matching unit 12 matches the sizes of the screen elements of the two screen images by changing the size (resolution) of one of the two screen images related to the target pair information. For example, in FIG. 6, the correct screen X and the comparison screen X (a) are different in size. Accordingly, the size of either one of the two screen image pairs is changed and adjusted so that the sizes of both screen images coincide with each other at the screen element level.

  Subsequently, the screen difference extraction unit 13 extracts a rectangular area from two screen images related to the target pair information and specifies a difference in units of the rectangular area (step S140). As a result, a screen difference list which is a list of screen difference information indicating the specified differences is set for the target pair information. For example, in the case of the correct answer screen X and the comparison screen X (a) shown in FIG. 6, the rectangular areas of “Title”, “Advertisement”, “Sentence”, “Photo”, and “Link” are extracted. In the comparison screen (b), the rectangular areas of “title”, “sentence”, “photograph”, “link”, and “link that does not exist in the specification” are extracted. Hereinafter, the extracted rectangular area is referred to as a “screen element rectangle”.

  As will be described later, since the extraction of the screen element rectangle is performed by analyzing the image, the screen element rectangle extracted in step S140 is not necessarily a screen element in a meaningful unit as viewed from a human. It is not necessarily rectangular. For example, the “title” area may be divided into a plurality of screen element rectangles and extracted. In addition, there is a possibility that a screen element rectangle extending over the “title” and “advertisement” areas is extracted.

  Subsequently, based on the screen difference information extracted by the screen difference extraction unit 13, the manual determination result assigning unit 15 highlights (highlights) a portion having a difference between the correct answer screen and the comparison screen, For each difference, a pass or fail determination result is received from the user, and the determination result is assigned to each screen difference information (step S150).

  Subsequently, the report output unit 14 adds target pair information to the result report (step S160).

  When Steps S130 to S160 are executed for all pairs, the report output unit 14 outputs a result report to the display device 106 (Step S170).

  FIG. 7 is a diagram illustrating a display example of a link screen to a result report. As shown in FIG. 7, the link screen is a pass / fail determination result by the user for each combination of the screen type (screen X, screen Y) and the environment type (environment (a), environment (b)). And a link to the results report. When any link is selected by the user, a result report as shown in FIG. 8 is displayed.

  FIG. 8 is a diagram illustrating a display example of a result report. The result report R1 is a result report for the combination of the screen X and the environment (b).

  The result report R1 includes a correct answer screen X and a comparison screen X (b), and a table showing a list of differences between the two screen images. In the table, for each difference, an item number, a difference type, a mutation value, a user determination, and a comment are displayed. The item number is a number assigned to each difference, and the item number related to the difference is given to the screen element rectangle corresponding to the difference on the correct answer screen and the comparison screen. The difference type is a type of difference, and the value is “no difference”, “movement in the X-axis direction”, “movement in the Y-axis direction”, “enlargement / reduction”, “disappearance”, or “addition”. The variation value is a value indicating the magnitude of the difference related to the difference type. The user determination is a pass / fail determination result for the difference input by the user. “No problem” means pass and “has problem” means fail. The comment is a comment input by the user regarding the difference.

  The user can easily determine whether or not the comparison screen is acceptable by referring to such a result report.

  Next, details of step S110 in FIG. 3 will be described. FIG. 9 is a flowchart for explaining an example of the processing procedure of the screen image input processing.

  In step S210, the image input unit 11 reads the environment root folder designated by the user. Subsequently, the image input unit 11 acquires a correct answer environment folder under the environment root folder (step S220). For example, the folder name of the correct answer environment folder may be determined in advance, and the folder related to the folder name may be acquired.

  Subsequently, the image input unit 11 generates an empty list (hereinafter referred to as “pair list”) for storing pair information to be generated (step S230). Subsequently, the image input unit 11 executes step S240 and subsequent steps for each correct answer screen stored in the correct answer environment folder. The correct answer screen that is the processing target after step S240 is referred to as a “target correct answer screen”.

  In step S240, the image input unit 11 acquires all comparison environment folders (hereinafter referred to as “comparison environment folder group”) under the environment root folder.

  Subsequently, the image input unit 11 executes steps S250 and S270 for each acquired comparison environment folder. The comparison environment folder to be processed in steps S250 and S270 is referred to as “target comparison environment folder”.

  In step S250, the image input unit 11 acquires a comparison screen corresponding to the target correct answer screen from the target comparison environment folder, and generates pair information for the target correct answer screen and the comparison screen. The comparison screen corresponding to the target correct answer screen is a comparison screen corresponding to the same screen as the screen corresponding to the target correct answer screen. For example, screen images with common screens may be stored in files with the same file name. By doing so, the comparison screen corresponding to the correct answer screen can be easily specified.

  Subsequently, the image input unit 11 adds the generated pair information to the pair list (step S270).

  Next, details of step S130 in FIG. 3 will be described. FIG. 10 is a flowchart for explaining an example of the size adjustment processing procedure.

  In step S310, the size matching unit 12 reads the target pair information into its working memory area.

  Subsequently, the size matching unit 12 extracts a feature point from each of the correct screen and the comparison screen related to the target pair information, performs feature point matching in the direction of the correct screen → the comparison screen, and between the correct screen and the comparison screen. A feature point pair group that is a set of corresponding feature point pairs is generated (step S320). The feature point matching can be performed using a known algorithm such as the Harris method or the Shi-Toshi method.

  The data structure of one feature point pair is as described in the column “■ Intermediate data” in FIG. That is, the feature point pair is composed of the coordinate point of the feature point on the correct answer screen and the coordinate point of the feature point corresponding to the feature point on the comparison screen.

  Subsequently, the size matching unit 12 selects two feature point pairs that are sufficiently separated from each other from the feature point pair group (step S330). For example, among all combinations of two feature point pairs, two feature point pairs related to the combination having the maximum sum of the distance on the correct screen and the distance on the comparison screen may be selected. Alternatively, two feature point pairs related to a combination in which the sum of the distance on the correct answer screen and the distance on the comparison screen is greater than or equal to the threshold value may be selected. The threshold value may be obtained based on the width size of the correct answer screen or the comparison screen, or may be obtained by another method.

  Subsequently, the size matching unit 12 calculates a size ratio between the screen element rectangle of the correct answer screen and the screen element rectangle of the comparison screen based on the two selected feature point pairs (step S340). Specifically, the coordinate point on the correct screen of one feature point pair (hereinafter referred to as “feature point pair 1”) and the correct screen of the other feature point pair (hereinafter referred to as “feature point pair 2”). The distance of the coordinate point at is calculated and set as the correct screen distance. Further, the distance between the coordinate point on the comparison screen of the feature point pair 1 and the coordinate point on the comparison screen of the feature point pair 2 is calculated and set as the comparison screen distance. Then, the distance within the correct screen / the distance within the comparison screen is calculated, and this is used as the size ratio. That is, the size ratio is not necessarily the ratio between the size of the entire correct screen and the size of the entire comparison screen. In the present embodiment, since the comparison is performed in units of screen element rectangles, it is important to match the size of the screen element rectangles between the correct answer screen and the comparison screen.

  The size ratio may be calculated by a method other than the above. For example, the width of the correct screen / the width of the comparison screen may be calculated as the size ratio. Alternatively, the distance between screen element rectangles on the correct screen / the distance between screen element rectangles on the comparison screen may be calculated as the size ratio. Also, the width of the correct screen / the width of the screen in the comparison environment may be calculated as the size ratio. Here, the width of the screen in the comparison environment is the width of the display of the terminal in the environment corresponding to the comparison screen, and may be stored in advance in the auxiliary storage device 102 or the like for each environment, for example. Such a calculation method is effective for an environment where the screen is displayed in accordance with the horizontal width of the display.

  Subsequently, the size matching unit 12 changes the size of the comparison screen based on the size ratio (step S350). Specifically, resolution conversion is performed on the comparison screen so that the vertical and horizontal sizes (number of pixels) of the comparison screen are values obtained by multiplying the size ratios, respectively. In the target pair information, the comparison screen before the size change is replaced with the comparison screen after the size change. Note that the size of the correct answer screen may be changed.

  Next, details of step S140 in FIG. 3 will be described. FIG. 11 is a flowchart for explaining an example of a processing procedure of screen difference information extraction processing.

  In step S410, the screen difference extraction unit 13 reads the target pair information into its own working memory area.

  Subsequently, the screen difference extraction unit 13 generates an empty data area for storing the screen difference list (step S415). The screen difference list is composed of zero or more pieces of screen difference information as described in the column “Report as output data” in FIG.

  Subsequently, the screen difference extraction unit 13 extracts a feature point pair group from the correct screen and the comparison screen related to the target pair information by the same method as in step S320 of FIG. 10 (step S420).

  Subsequently, the screen difference extraction unit 13 extracts a screen element rectangle (hereinafter referred to as “correct answer screen element rectangle”) from the correct answer screen related to the target pair information, and a screen in which the extracted correct answer screen element rectangles are listed. An element rectangle list is generated (step S425). The screen element rectangle is represented by the upper left coordinates and upper right coordinates of the rectangle as described in the column “■ Report as output data” in FIG. The extraction of the screen element rectangle can be realized by combining known techniques. For example, a screen element rectangle is extracted by executing procedures such as gray scale, canny processing, expansion processing, contour extraction, and rectangle conversion (obtaining a circumscribed rectangle of the contour and extracting the rectangle therefrom) on the image. May be.

  Note that the screen element rectangle is not necessarily a “rectangle”. It suffices if the partial areas constituting the screen are extracted. Therefore, L-shaped, convex, concave, or other polygonal partial areas may be extracted as screen element rectangles.

  Subsequently, the screen difference extraction unit 13 executes Steps S430 to S475 for each correct screen element rectangle included in the generated screen element rectangle list. The correct screen element rectangle to be processed in steps S430 to S475 is referred to as “target correct screen element rectangle”.

  In step S430, the screen difference extraction unit 13 has two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle in the comparison screen (hereinafter referred to as “target comparison screen”) related to the target pair information. It is determined whether or not to do. The feature point group related to the target correct screen element rectangle refers to, for example, feature points included in the range of the target correct screen element rectangle. However, feature points included in a predetermined range outside from the periphery of the target correct screen element rectangle may be included in the feature point group related to the target correct screen element rectangle. Further, whether or not there are two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle on the target comparison screen is a feature point pair including the feature point group related to the target correct answer screen element rectangle (hereinafter, “ It can be determined based on whether or not two or more “feature feature point pairs” are included in the feature point pair group generated in step S420.

  When two or more feature points corresponding to the feature point group related to the target correct answer screen element rectangle exist on the target comparison screen (Yes in step S430), the screen difference extraction unit 13 selects from two or more target feature point pairs that exist. Then, two feature point pairs that are sufficiently apart from each other are selected, and the target correct screen element rectangle and the screen element corresponding to the target correct screen element rectangle in the target comparison screen are selected in the same manner as in step S340 of FIG. An expansion / contraction ratio with the rectangle is calculated (step S435).

  Subsequently, the screen difference extraction unit 13 searches the comparison screen for a screen element rectangle corresponding to the target correct screen element rectangle, and the corresponding screen element rectangle (hereinafter referred to as “target comparison screen element rectangle”) is searched. In this case, the target correct screen element rectangle and the target comparison screen element rectangle are set in the screen difference information (step S440). Specifically, the screen difference extraction unit 13 calculates the positional relationship between the target correct screen element rectangle and the feature points related to the correct screen in the target feature point pair (for example, a vector from each vertex of the target correct screen element rectangle). A candidate rectangle of the target correct screen element rectangle is cut out from the comparison screen based on the feature point and the positional relationship related to the comparison screen in the target feature point pair. The screen difference extraction unit 13 compares the pixel values of the target correct screen element rectangle and the candidate rectangle, and the pixel value matching rate (pixel unit matching rate) is equal to or greater than a predetermined value (for example, 100% or 95%). If so, the candidate rectangle is set as a target comparison screen element rectangle. However, when the enlargement / reduction ratio is not 1, the size of the candidate rectangle is enlarged or reduced based on the enlargement / reduction ratio, and the pixel value is compared after the size of the candidate rectangle matches the target correct screen element rectangle. .

  The target comparison screen element rectangle may be cut out by pattern matching between the target correct screen element rectangle and the target comparison screen. For example, after changing the size of the target comparison screen based on the enlargement / reduction ratio, the target correct screen element rectangle is a rectangular area having the same size as the target correct screen element rectangle and all pixels (or a predetermined ratio (for example, The target comparison screen element rectangle may be cut out by searching for a rectangular region with a matching pixel value with respect to (95%) or more pixels).

  The screen difference extraction unit 13 generates screen difference information, and sets the target correct screen element rectangle as the screen difference information (hereinafter referred to as “target screen difference information”). As shown in the “■ Report to be output data” column in FIG. 4, the screen difference information is an item number, a difference type, 0 or more mutation values, 0 or more correct screen element rectangles, and 0 or more comparisons. This is information including a screen element rectangle and whether or not the difference is accepted. The correct screen element rectangle and the comparison screen element rectangle are expressed by the coordinates of the upper left vertex and the lower right vertex, respectively. The pass / fail to the difference is information indicating the presence / absence (pass / fail) of the problem input by the user regarding the difference related to the screen difference information.

  When the target comparison screen element rectangle is searched, the screen difference extraction unit 13 sets the target comparison screen element rectangle in the target screen difference information. When the target comparison screen element rectangle is not searched, the target comparison screen element rectangle is not set in the target screen difference information.

  Note that “no difference” is set as the initial value for the difference type of the target screen difference information.

  Subsequently, when the target comparison screen element rectangle exists (Yes in step S445), the screen difference extraction unit 13 determines whether or not the positions of the target correct screen element rectangle and the target comparison screen element rectangle are different (S450). ). The difference between the positions may be determined based on whether or not the coordinates of the upper left vertices of the target correct screen element rectangle and the target comparison screen element rectangle are the same.

  When the positions of the target correct screen element rectangle and the target comparison screen element rectangle are different (Yes in step S450), the screen difference extraction unit 13 selects “X-axis direction movement” or “Y-axis direction” as the difference type of the target screen difference information. “Move” is set, and the difference (movement amount) of the position is set as the variation value of the target screen difference information (step S455). The difference between the positions is calculated as, for example, the absolute value of the difference between the X coordinate or the Y coordinate of the upper left vertex between the target correct screen element rectangle and the target comparison screen element rectangle. Specifically, when the difference in the X-axis direction is not 0, “X-axis direction movement” is set as the difference type of the target screen difference information, and the movement amount in the X-axis direction is set as the variation value of the target screen difference information. Is done. When the difference in the Y-axis direction is not 0, “Y-axis direction movement” is set as the difference type of the target screen difference information, and the movement amount in the Y-axis direction is set as the variation value of the target screen difference information. If both the difference in the X-axis direction and the difference in the Y-axis direction are not 0, a copy of the target screen difference information is generated, and the difference type and the mutation value of one target screen difference information are set to “X-axis direction movement”, X The movement amount in the axial direction is set, and “movement in the Y-axis direction” and the movement amount in the Y-axis direction are set in the difference type and the variation value of the other target screen difference information.

  Subsequently, the screen difference extraction unit 13 determines whether or not the enlargement / reduction ratio calculated in step S435 is 1 (step S460). If the enlargement / reduction ratio is not 1 (No in step S460), the screen difference extraction unit 13 sets “enlargement / reduction” as the difference type of the target screen difference information, and sets the enlargement / reduction ratio as the mutation value of the target screen difference information (step). S465). When a value is set in step S455 for the target screen difference information, the screen difference extraction unit 13 generates a copy of the target screen difference information, and for the difference type and mutation value of the copy, “ Set the scaling ratio.

  On the other hand, when two or more feature points corresponding to the feature point group related to the target correct screen element rectangle do not exist in the target comparison screen (No in step S430), or when no target comparison screen element rectangle exists (No in step S445). The screen difference extraction unit 13 sets “disappear” as the difference type of the target screen difference information (step S470).

  Subsequent to step S460, S465, or S470, the screen difference extraction unit 13 adds the generated target screen difference information to the screen difference list (step S475). When a plurality of target screen difference information is generated, the plurality of target screen difference information is added to the screen difference list. Further, when adding the target screen difference information to the screen difference list, the screen difference extraction unit 13 sets a numerical value corresponding to the order of the target screen difference information in the screen difference list as the item number of the target screen difference information. To do.

  When steps S430 to S475 are executed for all correct screen element rectangles included in the screen element rectangle list, the screen difference extraction unit 13 executes steps S420 to S475 by exchanging the correct screen and the comparison screen. (Step S480). In this case, in step S470, “addition” is set for the difference type of the target screen difference information. This is because, when matching is performed in the direction of the comparison screen → the correct answer screen, when step S470 is executed, the screen element rectangle existing on the comparison screen does not exist on the correct answer screen.

  Subsequently, the screen difference extraction unit 13 sets the screen difference list as target pair information (step S485).

  Based on the target pair information generated in this way, the link screen shown in FIG. 7 and the result report shown in FIG. 8 are output.

  Next, details of step S150 will be described. FIG. 12 is a flowchart for explaining an example of a processing procedure of a user determination giving process for a difference between screens.

  In step S510, the manual determination result assigning unit 15 reads the target pair information into its working memory area. Subsequently, the manual determination result assigning unit 15 acquires the screen difference list set in the target pair information (step S515). Subsequently, the manual determination result giving unit 15 displays a user determination input screen on the display device 106 based on the target pair information and the screen difference list (step S520). The layout of the user determination input screen is the same as that of the result report R1 shown in FIG. However, in the table including the list of differences, the user determination and comment columns are blank and input is possible.

  Subsequently, the manual determination result giving unit 15 executes steps S525 to S555 for each screen difference information included in the screen difference list. The screen difference information to be processed in steps S525 to S555 is referred to as “target screen difference information”.

  In step S525, the manual determination result assigning unit 15 determines whether or not the difference type of the target screen difference information is “no difference”. When the difference type is “no difference” (Yes in step S525), the next screen difference information is set as a processing target in the screen difference list, and step S525 and subsequent steps are repeated.

  When the difference type is not “no difference” (No in step S525), the manual determination result assigning unit 15 highlights the line corresponding to the target screen difference information in the table included in the user determination screen (step S530). ).

  Subsequently, the manual determination result assigning unit 15 determines whether or not the difference type of the target screen difference information is “addition” or “disappearance” (step S535). When the difference type is “addition” or “disappearance” (Yes in step S535), the manual determination result assignment unit 15 displays the target screen difference information on the correct answer screen or the comparison screen displayed on the user determination input screen. The portion corresponding to the correct screen element rectangle or the comparison screen element rectangle is highlighted (step S540). That is, if the difference type is “addition”, a portion corresponding to the comparison screen element rectangle is highlighted on the comparison screen. On the other hand, if the difference type is “disappearance”, a portion corresponding to the correct screen element rectangle is highlighted on the correct screen.

  Subsequently, the manual determination result assigning unit 15 determines the location corresponding to the screen element rectangle that is relatively close to the correct screen element rectangle or the comparison screen element rectangle highlighted in step S540 as a user determination. Highlight display is performed on both the correct answer screen and the comparison screen of the input screen (step S545). For example, the manual determination result assigning unit 15 adds the highlighted correct screen element rectangle or comparison screen element rectangle among the screen difference information included in the screen difference information list and having the difference type “no difference”. On the other hand, the screen difference information related to the nearest correct screen element rectangle (arranged at the shortest distance) or the comparison screen element rectangle is selected. The manual determination result assigning unit 15 includes the portion corresponding to the correct screen element rectangle of the selected screen difference information on the correct screen of the user determination input screen and the screen difference information selected on the comparison screen of the user determination input screen. The portion corresponding to the comparison screen element rectangle is highlighted and displayed with a color different from the color used for the highlight display in step S540. The distance between the two screen element rectangles may be evaluated based on, for example, the distance between the respective upper right vertices, or may be evaluated based on the minimum distance between the respective screen element rectangles. The screen element rectangle to be selected (highlighted) may not be limited to the nearest screen element rectangle. For example, a plurality of screen element rectangles adjacent to the correct screen element rectangle or the comparison screen element rectangle highlighted in step S540 may be highlighted. Further, a screen element rectangle other than the difference type “no difference” may be a highlight display target.

  FIG. 13 is a diagram illustrating a first display example of the user determination input screen. The user determination input screen 500 shown in FIG. 13 includes a correct answer screen X, a comparison screen X (b), and a table T1.

  FIG. 13 shows a display example in the case where the screen difference information corresponding to the row whose item number is “5” is the target screen difference information in the table T1. Therefore, the row is highlighted. The highlight display is an effect of step S530. As a result, the user can grasp the difference currently targeted.

  In the comparison screen X (b), the comparison screen element rectangle r2b set in the target screen difference information is highlighted. The highlight display is an effect of step S540. As a result, the user can grasp the part corresponding to the difference currently targeted.

  Further, in the comparison screen X (b), the comparison screen element rectangle r1b located closest to the comparison screen element rectangle r2b among the comparison screen element rectangles that are not different from the correct screen X is compared with the correct screen X. The correct screen element rectangle r1 corresponding to the screen element rectangle r1b is highlighted. The highlight display is an effect of step S545. As a result, the user can easily grasp at which position the part has been added on the correct answer screen X where there is no part corresponding to the currently targeted difference.

  On the other hand, when the difference type of the target screen difference information is not “addition” or “disappearance”, that is, when both the correct screen element rectangle and the comparison screen element rectangle are set in the target screen difference information (No in step S535). ) The manual determination result assigning unit 15 highlights a position corresponding to the correct screen element rectangle or the comparison screen element rectangle set in the target screen difference information on the correct screen or the comparison screen in the user determination input screen 500. (Step S550).

  FIG. 14 is a diagram illustrating a second display example of the user determination input screen. 14, the same parts as those in FIG. 13 are denoted by the same reference numerals.

  FIG. 14 shows a display example when the screen difference information corresponding to the row whose item number is “2” is the target screen difference information in the table T1. Therefore, the row is highlighted. The highlight display is an effect of step S530.

  In the correct screen X, the correct screen element rectangle r3 set in the target screen difference information is highlighted. Furthermore, in the comparison screen X (b), the comparison screen element rectangle r3b set in the target screen difference information is highlighted. These highlight displays are the effect of step S540. As a result, the user can grasp the part corresponding to the difference currently targeted.

  Subsequent to step S545 or S550, the manual determination result assigning unit 15 accepts input of values for the user determination and the comment column of the highlighted row in the table T1 of the user determination input screen 500 (step S555). Specifically, if the difference related to the row is within the allowable range, the user manually inputs “no problem” to the column of the user determination, and the difference related to the row If it is out of range, enter “problem” manually. Further, the user can input a comment (arbitrary character string) regarding the difference with respect to the comment column of the row. The manual determination result giving unit 15 sets the user determination and comment input by the user for the manual determination result and comment of the target screen difference information.

  Steps S525 to S555 are sequentially executed for each screen difference information included in the screen difference list, so that the user can continuously calculate each difference between the correct screen and the comparison screen related to the target pair information. The presence / absence (pass / fail) of the problem can be entered.

  When steps S525 to S555 are executed for all the screen difference information included in the screen difference list, the manual determination result assigning unit 15 sets the user set to each screen difference information included in the screen difference list. Based on the determination value, pass / fail is determined for the comparison screen related to the target pair information (step S560). Specifically, if the value of the manual determination result of all screen difference information included in the screen difference information list is “no problem”, the manual determination result adding unit 15 determines whether the target pair information is acceptable or not. Set “pass” to. If the value of the manual determination result of any of the screen difference information included in the screen difference information list is “problem”, the manual determination result adding unit 15 determines whether or not the pass / fail determination result of the target pair information is “ Set “Fail”.

  The report output unit 14 displays the user determination and comments input as described above in the result report shown in FIG. In addition, the report output unit 14 displays the pass / fail determination result set in the target pair information in step S560 in a link corresponding to the target pair information on the link screen illustrated in FIG.

  As described above, according to the present embodiment, when a test is executed in n test environments, the screen image of the test result in one test environment is set as a correct answer screen, and the other n-1 environments. The difference is automatically extracted by comparing the screen image (comparison screen) of the test result with the correct screen. At this time, the difference is extracted in units of screen element rectangles. Therefore, it is possible to easily detect the screen layout collapse and to easily identify the differences between the screens. As a result, the user can efficiently perform pass / fail determination on the comparison screen.

  In the present embodiment, based on the size ratio between the screen element of the correct screen and the screen element of the comparison screen, the difference is extracted after one image size is changed. As a result, it is possible to avoid erroneous detection caused by the difference in resolution between the correct answer screen and the comparison screen, and to detect the difference with higher accuracy. In addition, the screen element rectangles can be compared more appropriately by using the size ratio in consideration of the positional relationship of the screen elements grasped through the feature point pairs instead of simply the resolution ratio. For example, it is assumed that the size of the correct screen (vertical × horizontal) is 1000 × 500 and the size of the comparison screen is 1500 × 600. However, in the comparison screen, the range in which the screen is drawn is a range of (0 to 1000) × (0 to 500), and the other ranges are blanks. That is, it is assumed that the screen element rectangle of the correct answer screen is the same as the screen element rectangle of the comparison screen. In this case, simply, if one size is changed based on the ratio between the size of the correct answer screen and the size of the comparison screen, the aspect ratio of the screen element rectangle included in the one is destroyed. On the other hand, according to the size ratio based on the feature point pair, the possibility of avoiding such inconvenience increases.

  Further, in the present embodiment, screen element rectangles are extracted from each of the correct answer screen and the comparison screen, and then the screen element is divided into two directions: a direction from the correct answer screen to the comparison screen and a direction from the comparison screen to the correct answer screen. Rectangle matching is performed. Therefore, various differences such as movement, enlargement / reduction, disappearance, and addition of the screen element rectangle can be detected with an easily understandable granularity, and it is possible to easily determine whether the screen layout is broken.

  Further, according to the present embodiment, the screen element rectangle corresponding to each difference extracted between the correct answer screen and the comparison screen is overlapped with the correct answer screen or the comparison screen and continuously highlighted. And pass / fail can be input. Therefore, it becomes easy to discriminate easily the difference part made into object. As a result, the user does not need to compare the correct answer screen and the entire comparison screen, and can determine pass / fail by visually observing the highlighted screen element rectangle (or its vicinity). Therefore, it is possible to efficiently input pass / fail for each difference.

  In addition, the user can input a comment if there is a problem or the like when making a pass / fail determination for each difference. By making it possible to record the comment in association with each difference, it is possible to finally output a bug report with good visibility.

  In the present embodiment, the size matching unit 12 is an example of a changing unit. The screen difference extraction unit 13 is an example of an extraction unit. The manual determination result giving unit 15 is an example of a display control unit. The correct answer screen is an example of a first image showing the first display result. The comparison screen is an example of a second image showing the second display result.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

DESCRIPTION OF SYMBOLS 10 Image difference confirmation assistance apparatus 11 Image input part 12 Size matching part 13 Screen difference extraction part 14 Report output part 15 Manual determination result provision part 100 Drive apparatus 101 Recording medium 102 Auxiliary storage apparatus 103 Memory apparatus 104 CPU
105 interface device 106 display device 107 input device B bus

Claims (7)

The size of any one of the first image indicating the first display result of the screen and the second image indicating the second display result of the screen is set to the screen element of the first image and the second image. A change unit that changes based on a ratio of sizes with respect to screen elements of the image;
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image And an extraction unit that extracts a difference between the first display result and the second display result in units of the partial area,
A display control unit that displays the first image and the second image, and highlights a partial region related to the extracted difference in the first image and the second image;
A screen difference confirmation support apparatus characterized by comprising: The display control unit highlights the partial areas related to the extracted plurality of differences in order according to the input by the user in the displayed first image and second image.
The screen difference confirmation support apparatus according to claim 1, wherein: When there is a partial region that is included in one of the first image and the second image and not included in the other, the display control unit is relatively close to the partial region in the one. Highlighting the partial region located and the partial region corresponding to the partial region on the other side,
The screen difference confirmation support apparatus according to claim 1 or 2, characterized in that The display control unit accepts input of information from the user for each extracted difference,
The screen difference confirmation support device according to any one of claims 1 to 3. The display control unit determines pass / fail regarding the comparison between the first image and the second image based on information input for each extracted difference.
The screen difference confirmation support device according to any one of claims 1 to 3. The size of any one of the first image indicating the first display result of the screen and the second image indicating the second display result of the screen is set to the screen element of the first image and the second image. A change procedure that changes based on the ratio of the size to the screen element of the image,
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image And an extraction procedure for extracting a difference between the first display result and the second display result in the unit of the partial area;
A display control procedure for displaying the first image and the second image, and highlighting a partial region relating to the extracted difference in the first image and the second image;
A screen difference confirmation support method characterized in that the computer executes. The size of any one of the first image indicating the first display result of the screen and the second image indicating the second display result of the screen is set to the screen element of the first image and the second image. A change procedure that changes based on the ratio of the size to the screen element of the image,
After changing the size, a partial area is extracted from each of the first image and the second image, the partial area extracted from the first image, and the partial area extracted from the second image And an extraction procedure for extracting a difference between the first display result and the second display result in the unit of the partial area;
A display control procedure for displaying the first image and the second image, and highlighting a partial region relating to the extracted difference in the first image and the second image;
A program that causes a computer to execute.

Images