JP2019102042A - Screen difference extraction device and program - Google Patents

Abstract

To provide a screen difference extraction device and a program that can contribute to increase of efficiency of comparing a plurality of screens.SOLUTION: A screen difference extraction device 10 includes: an extraction unit for extracting pairs of a screen element of a first screen image and the screen element of a second screen image corresponding to the screen element of the first screen image, from the first screen image and a plurality of different second screen images; a classification unit for classifying the extracted pairs into a plurality of first groups according to the similarity of the screen elements in the first screen image between the pairs and the similarity of the screen elements in the second screen images; and a provision unit for providing a second pair, not a first pair of any one of the pairs belonging to the first group with the reference information for the first pair, on a first-group-by-first-group basis.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a screen difference extraction apparatus and program.

  For the purpose of detection of UI layout collapse when operating an application in multiple environments, degradation detection in regression test, etc., the screen displayed as the execution result of the application is an image of the correctly displayed screen (correct screen) There is a tool for automatically detecting an error location by comparing the images of the test target screen (comparison screen). Pixel-wise difference extraction tools (eg DiffImage (https://en.osdn.net/projects/sfnet_diffimg/)) and pixel-wise difference extraction techniques (Non-Patent Document 1) The unit can be compared and the difference can be displayed.

JP, 2017-138713, A

2014. Finding HTML presentation failures using image comparison technology. In Proceedings of the 29th ACM / IEEE international conference on Automated software engineering (ASE '14). ACM, New York, NY, USA, 91-96 (University of Southern California, USA) Osamu Tanno: "Proposal of UI Layout Failure Detection Support Method Using Image Processing", IPSJ SIG Technical Report, vol. 2016-SE-194, no. 9, pp. 1-8, November 2016

  When detecting a UI layout breakdown when operating an application in multiple environments, it is necessary to compare the same screen in multiple environments and confirm the result. In addition, even for screen comparison in a specific environment, such as regression test, menus and headers that make up the screen of the application often exist in common regardless of the application of the screen, and each screen that makes up the application When performing comparisons, it is necessary to compare the same area many times and check the result.

  Therefore, in the techniques of Non-Patent Documents 1 and 2, when the number of environments and the number of screens increase, it is necessary to check many same results (different points) common to a plurality of screens many times, and confirmation operation is very There is a problem that

  The present invention has been made in view of the above-described points, and an object of the present invention is to contribute to the efficiency of comparison work of a plurality of screens.

  Then, in order to solve the above-mentioned subject, a screen difference extraction device makes a pair of a screen element of the 1st screen image, and a screen element of the 2nd screen image corresponding to the screen element with the 1st screen image. An extracting unit for extracting from a plurality of different second screen images, similarity between screen elements of the first screen image between the pairs, and similarity between screen elements of the second screen image A classification unit that classifies the plurality of extracted pairs into a plurality of first groups, and any one of the pairs belonging to the first group for each of the first groups The second pair that is not the first pair has an assigning unit that assigns reference information to the first pair.

  It can contribute to the efficiency improvement of comparison work of a plurality of screens.

It is a figure which shows the example of hardware constitutions of the screen difference extraction apparatus 10 in 1st Embodiment. It is a figure which shows the function structural example of the screen difference extraction apparatus 10 in 1st Embodiment. It is a flowchart for demonstrating an example of the process sequence which the screen difference extraction apparatus 10 performs. It is a figure which shows an example of a folder structure. It is a figure which shows an example of the extraction result of a rectangular area. It is a figure which shows an example of the determination result of the correspondence of a rectangular pair, and a matching. It is a flowchart for demonstrating an example of the process sequence of the input process of a screen image. It is a flowchart for demonstrating an example of the process sequence of a production | generation process of a rectangular pair group group. It is a figure which shows an example of the classification result to a rectangular pair group. It is a figure which shows an example of the assignment | providing result of a representation flag and reference information. It is a figure which shows the 1st output example of a result report. It is a figure which shows the 2nd output example of a result report. It is a flowchart for demonstrating an example of a processing procedure of classification processing to a rectangular pair group of a rectangular pair group in a 2nd embodiment. It is a figure which shows the example of classification | category to the area | region group of a correct screen rectangle.

  Hereinafter, a first embodiment will be described based on the drawings. FIG. 1 is a diagram showing an example of the hardware configuration of the screen difference extraction apparatus 10 according to the first embodiment. The screen difference extraction apparatus 10 of FIG. 1 has a drive unit 100, an auxiliary storage unit 102, a memory unit 103, a CPU 104, an interface unit 105, a display unit 106, an input unit 107, etc. .

  A program for realizing the processing in the screen difference extraction apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive apparatus 100, the program is installed from the recording medium 101 to the auxiliary storage apparatus 102 via the drive apparatus 100. However, the installation of the program does not necessarily have to be performed from the recording medium 101, and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads out the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 implements the function of the screen difference extraction apparatus 10 in accordance with a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a graphical user interface (GUI) according to a program. The input device 107 includes a keyboard, a mouse, and the like, and is used to input various operation instructions.

  FIG. 2 is a diagram showing an example of a functional configuration of the screen difference extraction device 10 according to the first embodiment. In FIG. 2, the screen difference extraction device 10 includes an image input unit 11, a rectangle extraction unit 12, a rectangle pair extraction unit 13, a matching determination unit 14, a rectangle pair group generation unit 15, a report output unit 16 and the like. These units are realized by processing that one or more programs installed in the screen difference extraction device 10 cause the CPU 104 to execute.

  The processing procedure executed by the screen difference extraction device 10 will be described below. FIG. 3 is a flowchart for explaining an example of the processing procedure executed by the screen difference extraction device 10.

  In step S110, the image input unit 11 reads the correct answer screen and the comparison screen group stored for each screen to be tested under the folder specified by the user, and for each correct answer screen, the comparison screen pertaining to the same screen Generate a list of pair information with

  The correct answer screen is image data (hereinafter referred to as a “screen image”) indicating the result of displaying the screen in an environment where the screen is displayed according to the intention of the creator (hereinafter referred to as “correct environment”). Say). The comparison screen refers to a screen image showing a result displayed in an environment (hereinafter referred to as “comparison environment”) different from the correct answer environment. The environment is distinguished by, for example, a combination of a model of a terminal displaying a screen, an OS (Operating System), and a browser. Furthermore, environments may be distinguished according to browser display sizes. The environment can be said to be an example of the display condition of the screen. The correct answer screen and the comparison screen may be acquired by, for example, screen capture.

  Here, the screen refers to a screen that the program causes a computer to display. The screen may be a screen of an application program or the like, or may be a screen described in HTML (Hyper Text Markup Language) or the like.

  FIG. 4 is a diagram showing an example of the folder configuration. FIG. 4 shows an example in which one solution environment folder and two comparison environment folders (comparison environment folders (a) and (b)) are included under the environment root folder. An image file X and an image file Y are stored in each of the correct answer environment folder and each comparison environment folder. The image file X is a file storing a screen image generated by screen capture or the like of the display result of the screen X. The image file Y is a file in which a screen image generated by screen capture of the display result of the screen Y is stored. However, the image files X and Y under the correct answer environment folder are image files storing the correct answer screen regarding the screen X or the screen Y. Image files X and Y under the comparison environment folder (a) are image files storing a comparison screen indicating the display result of the screen X or the screen Y in the environment (a). The image files X and Y under the comparison environment folder (b) are image files storing a comparison screen indicating the display results of the screen X or the screen Y in the environment (b). The folder and file shown in FIG. 4 are stored in the auxiliary storage device 102, for example.

  According to the example of FIG. 4, in step S110, a correct answer screen pertaining to image file X under the correct answer environment folder, a pair of a correct answer screen pertaining to image file X under the comparison environment folder (a), and a correct answer environment A pair of the correct answer screen concerning the image file Y under the folder and the correct answer screen concerning the image file Y under the comparison environment folder (a) and the correct answer screen concerning the image file X under the correct answer environment folder Correct pair screen with image file X under environment folder (b), correct screen with image file Y under correct environment folder, correct answer with image file Y under comparison environment folder (b) Pair information is generated for four pairs with the screen and the pair.

  The pair information includes correct screen information, comparison screen information, a rectangular pair group, and a comparison result group of rectangular pairs. The correct answer screen information is screen information of the correct answer screen. Comparison screen information is screen information of a comparison screen. The screen information is information including an image file of the screen and a path name of the image file. The rectangular pair group and thereafter will be described later.

  Subsequently, the image input unit 11 generates an empty data area for the result report (step S120). The result report is data including one or more pair information.

  The following steps S130 to S160 are performed for each pair of information generated in step S110. Hereinafter, the pair information to be processed is referred to as “target pair information”.

  In step S130, the rectangle extracting unit 12 uses the technique of Non-Patent Document 2 or Patent Document 1 etc., in units of screen elements etc., for each of the correct screen related to the target pair information and the comparison screen related to the target pair information. Extract a rectangular area (ie extract the area of each screen element).

  FIG. 5 is a diagram showing an example of the extraction result of the rectangular area. In FIG. 5, in the first line, rectangular regions of screen elements extracted for each of the correct screen (x), the comparison screen (x1) and the comparison screen (x2) related to the screen X are shown. The second line shows rectangular areas of the screen elements extracted for each of the correct answer screen (y), the comparison screen (y1), and the comparison screen (y2) related to the screen Y.

  In FIG. 5, for convenience, extraction results of rectangular areas of the correct answer screen and the comparison screen related to the plurality of pair information are shown, but the correct answer screen and the comparison screen related to the target pair information are the correct answer screen (x) And in the case of the comparison screen (x1), in one execution of step S130, rectangular regions are extracted only for each of the correct answer screen (x) and the comparison screen (x1).

  In addition, although the rectangular area here refers to the rectangular area in a screen, if it is a nonpatent literature 2 or patent document 1, it becomes an area | region of the size of the screen element (a button, an advertisement image, etc.) on a screen. Other techniques may be used to extract larger regions than screen elements.

  Subsequently, the rectangular pair extraction unit 13 has a rectangular area extracted from the correct answer screen (hereinafter referred to as "correct answer screen rectangle") and a rectangular area extracted from the comparison screen (hereinafter referred to as "comparison screen rectangle"). Then, all pairs of rectangular regions in correspondence (hereinafter, referred to as "rectangular pairs") are extracted (S140). That is, one rectangular pair includes one correct screen rectangle and one comparison screen rectangle. The correspondence relationship is, for example, a relationship determined to have the highest degree of similarity based on a predetermined pairing determination rule. The similarity may be evaluated, for example, by the matching rate of pixels. Note that a set of all extracted rectangular pairs is called "rectangular pair group". The rectangular pair group is attached to the target pair information.

  Subsequently, the matching determination unit 14 performs matching determination on the correct screen rectangle and the comparison screen rectangle (that is, between the screen elements of the corresponding rectangular pair) for each rectangular pair included in the rectangular pair group of the target pair information (S150). ). Matching determination refers to comparing the correct screen rectangle contained in the rectangular pair with the comparison screen rectangle based on a predetermined matching determination rule, and determining the presence or absence of a difference. The presence or absence of the difference means the presence or absence of the difference based on the matching determination rule, and it may not necessarily coincide with whether there is an actual difference. For example, if the matching determination rule is a rule in which there is no difference if the images of 95% or more match, it may be determined that there is no difference even if it is not a perfect match. A set of determination results (with / without difference) of the matching determination is attached to the object pair information as a comparison result group. Steps S130 to S150 may be performed using, for example, the method disclosed in Non-Patent Document 2 or Patent Document 1.

  FIG. 6 is a diagram showing an example of the correspondence between rectangular pairs and the determination result of matching. In FIG. 6, (1) indicates the correspondence of the rectangular area for each of the pair of the correct answer screen (x) and the comparison screen (x1) and the pair of the correct answer screen (x) and the comparison screen (x2) regarding the screen X And show the determination result of the matching. (2) is the determination of the correspondence relationship and the matching of the rectangular region for each of the pair of the correct answer screen (y) and the comparison screen (y1) and the pair of the correct answer screen (y) and the comparison screen (y2) Show the results.

  In FIG. 6, rectangular regions connected by lines have a correspondence. That is, rectangular regions connected by lines constitute a rectangular pair. Moreover, the code | symbol (alphabet) provided to each rectangular area shows the similarity of the rectangular area which concerns on a rectangular pair. The rectangular areas (for example, the rectangular area C of the correct screen (x) and the rectangular area C of the comparison screen (x1 etc.) to which the same code is added indicate the rectangular areas determined to have no difference in the matching determination. .

  Subsequently, the report output unit 16 adds target pair information to the result report (S160).

  When steps S130 to S160 are executed for all the pair information, the rectangle pair group generation unit 15 executes the rectangle pair group group generation processing for the rectangle pair group obtained by integrating the rectangle pair group included in each pair information. (S170). The rectangular pair group will be described later.

  Subsequently, the report output unit 16 adds the generated rectangular pair group group to the result report (S180). Subsequently, the report output unit 16 outputs a result report (S190).

  Subsequently, the details of step S110 will be described. FIG. 7 is a flowchart for describing an example of a processing procedure of screen image input processing.

  In step S210, the image input unit 11 reads an environment root folder specified by the user. Subsequently, the image input unit 11 acquires a correct environment folder under the environment root folder (step S220). For example, the folder name of the correct solution environment folder may be determined in advance, and the folder related to the folder name may be acquired.

  Subsequently, the image input unit 11 generates an empty list (hereinafter, referred to as a “pair list”) for storing pair information to be generated from now (step S230). Subsequently, the image input unit 11 executes step S240 and subsequent steps for each of the correct answer screens stored in the correct answer environment folder. The correct answer screen to be processed in step S240 and subsequent steps is referred to as a “target correct answer screen”.

  In step S240, the image input unit 11 acquires all comparison environment folders (hereinafter referred to as "comparison environment folder group") under the environment root folder.

  Subsequently, the image input unit 11 executes steps S250 and S270 for each acquired comparison environment folder. The comparison environment folder to be processed in steps S250 and S270 is referred to as a "target comparison environment folder".

  In step S250, the image input unit 11 acquires a comparison screen corresponding to the target correct screen from within the target comparison environment folder, and generates pair information for the target correct screen and the comparison screen. The comparison screen corresponding to the target correct answer screen means a comparison screen corresponding to the same screen as the screen to which the target correct answer screen corresponds. For example, screen images having a common screen may be stored in a file having the same file name. By doing so, the comparison screen corresponding to the correct answer screen can be easily specified.

  Subsequently, the image input unit 11 adds the generated pair information to the pair list (step S270).

  Subsequently, details of step S170 in FIG. 3 will be described. FIG. 8 is a flowchart for describing an example of a processing procedure of a process of generating a rectangular pair group group.

  In step S310, the rectangle pair group generation unit 15 compares the rectangle pair group (rectangle pair group integrating the rectangle pair group related to each pair information) with the similarity between the correct solution screen rectangles between the rectangle pairs and the comparison screen rectangles. Are classified into a plurality of groups (hereinafter referred to as "rectangular pair groups") based on a set of similarities of That is, a plurality of rectangular pairs in which the correct screen rectangles are similar and the comparison screen rectangles are similar are classified into the same group. Classification of each rectangular pair into rectangular pair groups is performed across pair information (in other words, across the screen X and the screen Y in FIG. 5).

  For example, the rectangular pair group generation unit 15 calculates, for each rectangular pair, the feature value of the correct screen rectangle and the feature value of the comparison screen rectangle. As a result, each rectangular pair is converted to a feature value pair. For calculation of the feature value, dHash or the like may be used. The rectangle pair group generation unit 15 classifies the rectangle pair group such that the rectangle pairs whose feature value pairs match are in the same rectangle pair group.

  FIG. 9 is a diagram showing an example of the classification result into the rectangular pair group. In FIG. 9, the “rectangle” column indicates an identifier of each rectangular area. The identifier has a format of “type of screen (name of correct screen or comparison screen) rectangular area”. For example, “X (x) A” indicates the rectangular area A of the correct screen (x) of the screen X. Further, in FIG. 9, “feature value” indicates the feature value of the corresponding rectangular area for convenience as a symbol, and the same symbol indicates the same feature value.

  In FIG. 9, a group 1 is a group in which a pair of feature values is (d1, d2). Group 2 is a group in which the pair of feature values is (d3, d4). Group 3 is a group in which a pair of feature values is (d5, d6).

  Note that FIG. 9 does not include rectangular pairs determined to be identical in rectangular regions in the matching determination. This is because such rectangular pairs are less likely to be output targets of screen differences. Therefore, the rectangular pair group generation unit 15 may execute step S310 after removing the rectangular pairs related to the rectangular regions determined to be identical in the matching determination from the rectangular pair group.

  Subsequently, the rectangular pair group generation unit 15 executes steps S320 and S330 for each rectangular group included in the rectangular pair group group. Hereinafter, the rectangular pair group to be processed is referred to as “target rectangular pair group”.

  In step S320, the rectangular pair group generation unit 15 selects any one of rectangular pairs included in the target rectangular pair group as a representative pair. A representative flag is given to the rectangular pair selected as the representative pair. Note that the method of selecting representative pairs is not limited to a predetermined method.

  Subsequently, the rectangular pair group generation unit 15 assigns a reference link (reference information or link information) to the representative pair for rectangular pairs other than the representative pair among the rectangular pairs included in the target rectangular pair group (S330). The reference link may be any information that can identify and identify a representative pair.

  When steps S320 and S330 are performed for all the rectangular pair groups, the process of FIG. 8 ends. As a result, for each rectangular pair group, a representative flag is given to one of the rectangular pairs, and a reference link for the representative pair is given to the rectangular pair other than the representative pair.

  FIG. 10 is a diagram showing an example of the assignment result of the representative flag and the reference link. In FIG. 10, columns of “representative” and “reference link” are added to FIG. In the "representative" column, "1" is set for the representative pair. In the column "reference link", reference links to representative pairs are set for rectangular pairs other than representative pairs.

  For example, in group 1, a rectangular pair of the rectangular area A of the correct answer screen (x) and the rectangular area A ′ of the comparison screen (x1) is selected as a representative pair, and other rectangular pairs are selected to the representative pair A reference link is given. In group 2, a rectangular pair of rectangular area B of the correct answer screen (x) and rectangular area B 'of the comparison screen (x1) is selected as a representative pair, and other rectangular pairs are selected to the representative pair. A reference link is given. Furthermore, in group 3, a rectangular pair of the rectangular area C of the correct answer screen (x) and the rectangular area C 'of the comparison screen (x2) is selected as a representative pair, and other rectangular pairs are selected to the representative pair. A reference link is given.

  Note that, as shown in group 2 and group 3, reference links to representative pairs of screen X may be added to rectangular pairs of screen Y as well.

  In step S180 described above, information as shown in FIG. 10 is added to the result report, and in step S190, the report output unit 16 outputs the result report, for example, as follows.

  FIG. 11 is a diagram illustrating a first output example of the result report. FIG. 11 shows an example of a result report for group 2. In FIG. 11, the determination result of the matching determination is shown only for the representative pair, and for the pairs other than the representative pair, the representative pair and the determination result are the same, and therefore, it is indicated that they are not checked. . Also, reference links to representative pairs are shown.

  FIG. 12 is a diagram showing a second output example of the result report. FIG. 12 illustrates an example in which the determination result of the matching determination regarding each rectangular pair related to the pair information is output as “comment” for each pair information. However, for rectangular pairs other than the representative pair, “comment” indicates that the representative pair should be referred to, together with the reference link indicating the representative pair.

  The result report may be displayed on the display device 106, may be output from a printer, or may be stored as a file in the auxiliary storage device 102 or the like.

  According to the result report as shown in FIG. 11 or FIG. 12 or the like, it is possible to reduce the work of repeatedly checking the common differences in the plurality of screens repeatedly. Therefore, according to the present embodiment, it is possible to contribute to the efficiency of comparison work of a plurality of screens. According to the present embodiment, as the number of screens including screen elements having the same content of difference between environments or screens increases, it is possible to expect an increase in the reduction effect of the confirmation operation.

  Next, a second embodiment will be described. In the second embodiment, points different from the first embodiment will be described. The points not specifically mentioned in the second embodiment may be the same as in the first embodiment.

  In the first embodiment, when utilizing dHash, which is a representative method for comparing the similarity between images, in the process of classifying the rectangular pair group into the rectangular pair group (S310 in FIG. 8), the rectangles between the screens are used. Since it is necessary to compare pairs in a round-robin manner, the number of comparisons increases as the number of environments and the number of screens increase.

Therefore, in the second embodiment, in view of the tendency that the layout (display position) of the header, menu, etc. constituting the screen of the application tends to be the same, each of the correct screen and the comparison screen is divided into a plurality of areas (hereinafter Groups (hereinafter referred to as "area group") which are divided into "divided areas" and each rectangular area (correct screen rectangle or comparison screen rectangle) extracted from the correct screen or the comparison screen is based on the area to which the rectangular area belongs To be classified). When generating a rectangular pair group, the number of comparisons can be reduced by narrowing down targets of comparison with rectangular regions included in each rectangular pair to rectangular regions of the rectangular pair belonging to the same region group. Note that at least one of the plurality of divided regions includes a plurality of rectangular regions (correct screen rectangles or comparison screen rectangles). Moreover, as long as the size of each correct answer screen and each comparison screen is the same, the way of dividing each correct answer screen and each comparison screen may be completely the same. In this case, the position and range of each divided area may be designated by an absolute value. On the other hand, when the size of each correct answer screen and each comparison screen is slightly different, the position and range of each divided area may be designated by a ratio. Specifically, in the second embodiment, the figure is a diagram. The processing content of step S310 of 8 is replaced with the processing procedure shown in FIG.

  FIG. 13 is a flowchart for describing an example of a processing procedure of classification processing of rectangular pair groups into rectangular pair groups in the second embodiment.

  First, the rectangular pair group generation unit 15 executes steps S311 and S312 for each rectangular pair included in the rectangular pair group (a rectangular pair group obtained by integrating the rectangular pair group related to each pair information). Hereinafter, the rectangular pair to be processed is referred to as a “target rectangular pair”.

  In step S311, the rectangular pair group generation unit 15 sets the correct screen rectangle of the target rectangular pair to the area group related to the divided area completely including the correct screen rectangle among the divided areas related to the correct screen to which the correct screen rectangle belongs. Classification

  FIG. 14 is a view showing an example of classification of correct screen rectangles into area groups. FIG. 14 shows a classification example in which the correct answer screen V or the correct answer screen W is divided into four divided areas. In FIG. 14, “rectangle x” (x indicates a to p) indicates a correct screen rectangle of the correct screen V or a correct screen rectangle (that is, a rectangular area) of the correct screen W. The area A corresponds to the upper divided area, the area B corresponds to the lower divided area, the area C corresponds to the left divided area, and the area D corresponds to the right divided area.

  In the correct answer screen V of FIG. 14, each of the rectangles a, b and d is completely included in the area A. Therefore, rectangles a, b and d are classified into the area group related to the area A. In addition, each of the rectangles g, h and i is completely included in the region B. Therefore, rectangles g, h and i are classified into the area group related to the area B. Also, in the region C, each of the rectangles e and c is completely included. Therefore, rectangles e and c are classified into the area group related to the area C. Although the rectangle b is completely included in the region C, the rectangle b is not classified into the region group related to the region C because the rectangle b is already classified into the region A. That is, one rectangular area does not belong to a plurality of area groups. However, one rectangular area may be permitted to belong to a plurality of area groups. By allowing such a state, it can be avoided that the rectangular areas are determined not to be common by the rectangular areas expected to be common belonging to different area groups. In addition, in the region D, the rectangle f is completely included. Therefore, the rectangle f is classified into the area group related to the area D.

  On the other hand, in the correct answer screen W, the area A completely includes each of the rectangles j and k. Therefore, rectangles j and k are classified into the area group related to the area A. Also, in the region B, each of the rectangles o and p is completely included. Therefore, rectangles o and p are classified into the area group related to the area B. In addition, in the region C, each of the rectangles l and m is completely included. Therefore, rectangles l and m are classified into the area group related to the area C. In addition, in the region D, the rectangle n is completely included. Therefore, the rectangle n is classified into the area group related to the area D.

  When the correct answer screen rectangle of the target rectangle pair is classified into any area group, identification information (hereinafter referred to as "area name") of the area group to which the correct answer screen rectangle is classified is added.

  According to the classification method described with reference to FIG. 14, there is a possibility that a correct screen rectangle which is not completely included in any area and is not classified in any area group may occur. Such possibilities are acceptable. However, in order to eliminate such a possibility, the shape of each divided area may not be limited to a rectangle while allowing a difference in the shape of the divided area of each screen. That is, the area formed by the boundary of the rectangular area may be taken as the divided area.

  Also, the position and size of each divided area may be changed each time a test is repeated by utilizing machine learning, for example, learning an area in which a common rectangular area is displayed as the number of tests increases. By doing this, the number, position, and size of divided areas suitable for each application may be defined.

  In step S312, the rectangle pair group generation unit 15 sets the comparison screen rectangle of the target rectangle pair to the area group pertaining to the divided area completely including the comparison screen rectangle among the divided areas pertaining to the comparison screen to which the comparison screen rectangle belongs. Classification As a result, the area name of the area group of the classification destination of the comparison screen rectangle is given to the comparison screen rectangle of the target rectangle pair.

  Subsequently, the rectangular pair group generation unit 15 executes step S313 for each area group. Hereinafter, the area group to be processed is referred to as “target area group” below. In step S313, the rectangular pair group generation unit 15 generates a rectangular pair group in which the correct screen rectangle belongs to the target area group (the area name of the target area group is added) based on the similarity between the correct screen rectangles. Classify into groups. For example, the rectangular pair group generation unit 15 calculates the feature value of the correct screen rectangle for each rectangular pair. For calculation of the feature value, dHash or the like may be used. The rectangular pair group generation unit 15 classifies rectangular pairs to which the correct screen rectangles having matching feature values belong to the same rectangular pair group. At this time, comparative symmetry for classification is limited to the correct screen rectangle group belonging to the same area group (target area group). Therefore, according to the example of FIG. 14, the number of comparisons for region A is 3 × 2 = 6, the number of comparisons for region B is 3 × 2 = 6, and the number of comparisons for region C is 2 The number of comparisons for the region D is 1 × 1 = 1. Therefore, the total number of comparisons is 6 + 6 + 4 + 1 = 17 times. On the other hand, in the first embodiment, 9 × 7 = 63 comparisons are required.

  Subsequently, the rectangular pair group generation unit 15 executes step S314 for each rectangular pair group generated by the execution of step S313 for each area group. Hereinafter, the rectangular pair group to be processed is referred to as a “target pair group”. In step S314, the rectangle pair group generation unit 15 sets the rectangle pair group belonging to the target pair group to the rectangle pair group based on the similarity between comparison screen rectangles belonging to the same region group (the same region name is added). Reclassify. As a result of the reclassification, a rectangular pair group group in which the correct screen rectangles and the comparison screen rectangles are similar to each other is generated. Note that the classification based on the similarity between comparison screen rectangles may be performed in the same manner as the classification based on the similarity between correct screen rectangles. At this time, comparison targets for reclassification are limited to comparison screen rectangles belonging to the same area group.

  In addition, in steps S313 and S314, before comparing the similarity of the image content of the rectangular area by dHash, not only narrowing down the comparison object with each correct screen rectangle or each comparison screen rectangle by the belonging divided area ( Alternatively, instead of such narrowing down, it may be classified according to the rectangular size, and rectangular areas which are obviously different in size may be excluded from comparison objects. That is, size-based grouping may be performed. In addition, the rectangular areas with different roles are compared by classifying each rectangular area according to the role (application, function, etc.) (button, text box, etc.) of the rectangular area (each correct screen rectangle or each comparison screen rectangle) It may be excluded from the target. That is, role-based grouping may be performed.

  As described above, according to the second embodiment, when grouping a rectangular pair group into a rectangular pair group, classification is made into divided regions based on the display position of each rectangular region (correct screen rectangle or comparison screen rectangle) By doing this, the comparison target can be limited within each area group without comparing all rectangular areas of the correct screens (correct screen rectangles) and all rectangular areas of the comparison screens (comparative screen rectangles) in a round robin manner. it can. As a result, the number of comparisons can be reduced, which in turn can shorten the test execution time.

  When the screen layout is significantly changed, there is a possibility that the rectangular pairs to be grouped may leak. Therefore, in the case where there is no rectangular pair belonging to the area group, for example, the comparison of the rectangular pair may be performed round by round.

  In each of the above-described embodiments, the rectangular pair extraction unit 13 is an example of an extraction unit. The matching determination unit 14 is an example of a determination unit. The rectangular pair group generation unit 15 is an example of a classification unit and an assignment unit. The report output unit 16 is an example of an output unit. The correct answer screen is an example of the first screen image. The comparison screen is an example of a second screen image. The representative pair is an example of the first pair. The rectangular pair other than the representative pair is an example of the second pair. The rectangular pair group is an example of a first group. The area group is an example of a second group.

  Although the embodiments of the present invention have been described above in detail, the present invention is not limited to such specific embodiments, and various modifications may be made within the scope of the present invention as set forth in the claims. Modifications and changes are possible.

DESCRIPTION OF SYMBOLS 10 screen difference extraction apparatus 11 image input unit 12 rectangle extraction unit 13 rectangle pair extraction unit 14 matching determination unit 15 rectangle pair group generation unit 16 report output unit 100 drive device 101 recording medium 102 auxiliary storage device 103 memory device 104 CPU
105 interface device 106 display device 107 input device B bus

Claims (6)

Extracting a pair of a screen element of a first screen image and a screen element of a second screen image corresponding to the screen element from the first screen image and a plurality of different second screen images Department,
Based on the similarity between the screen elements of the first screen image between the pairs and the similarity between the screen elements of the second screen image, a plurality of extracted pairs are divided into a plurality of first groups A classification unit that classifies into
Assigning reference information for the first pair to the second pair other than the first pair, which is any one of the pairs belonging to the first group, for each first group Department,
The screen difference extraction apparatus characterized by having. A determination unit that determines the presence or absence of a difference between screen elements included in each of the plurality of pairs extracted by the extraction unit;
An output unit that outputs the determination result by the determination unit for the first pair, and outputs reference information given to the second pair for the second pair;
The screen difference extraction apparatus according to claim 1, comprising: The extraction unit generates a pair of screen elements from the first screen image and a plurality of second screen images corresponding to the first screen image for each of the plurality of different first screen images. Extract,
The screen difference extraction apparatus according to claim 1 or 2, characterized in that After the classification unit classifies each pair into a second group based on display positions, sizes, or roles of screen elements of the first screen image or the second screen image included in the pair. And, in each of the second groups, classifying the pairs classified into the second group into the first group.
The screen difference extraction apparatus according to any one of claims 1 to 3, characterized in that: The classification unit is configured such that the screen elements of the first screen image or the second screen image included in the pair are a plurality of areas of the first screen image or the second screen image. Are classified into a second group based on which one of a plurality of regions obtained by dividing into a second group, and the pairs classified into the second group are included in each of the second groups. Classify into the first group,
The screen difference extraction apparatus according to any one of claims 1 to 3, characterized in that: Extracting a pair of a screen element of a first screen image and a screen element of a second screen image corresponding to the screen element from the first screen image and a plurality of different second screen images Steps and
Based on the similarity between the screen elements of the first screen image between the pairs and the similarity between the screen elements of the second screen image, a plurality of extracted pairs are divided into a plurality of first groups Classification procedure to be classified into
Assigning reference information for the first pair to the second pair other than the first pair, which is any one of the pairs belonging to the first group, for each first group Steps and
A program characterized by causing a computer to execute.

Images