JP6603194B2 - Screen recognition device, screen recognition method, and screen recognition program - Google Patents

Description

  The present invention relates to a screen recognition device, a screen recognition method, and a screen recognition program.

  The operator can input / register necessary business information or refer to it by operating the business application screen (screen). In many cases, since the operator performs work through the screen, the operation state of the operator can be mechanically estimated by acquiring the screen and operation operated by the operator. The operation of the screen can be acquired by recognizing both information relating to the screen and information relating to operations by the operator. Information about the screen, for example, information that summarizes the screen (window title, window ID, title bar, etc.), information about GUI components in the screen (class, ID, text name, etc. of each GUI component), and obtain these information Date and time. Information relating to the operation of the operator includes, for example, mouse operation and keyboard operation, and information on the screen or GUI component that is the operation target. By acquiring these pieces of information, it is possible to acquire one by one what value the operator has input to which GUI component on which screen at what timing. Hereinafter, information relating to the screen is referred to as “screen information”, and information relating to the operation of the operator is referred to as “operation information”. Also, GUI parts are called “objects”. There are various APIs for each application and operation system to properly recognize operator screen information. For example, (1) API that uses an ID assigned to an object, (2) API that specifies an object from position information such as the mouse cursor position, keyboard input position, and relative position on the screen, and (3) screen There is an API for specifying an object using a parent-child relationship (object tree) between objects defined from the above arrangement relationship.

Japanese Patent Laying-Open No. 2015-5243 JP 2014-13490 A

  However, there is a possibility that the above-described conventional technology cannot appropriately acquire screen information of various applications. For example, in the case of (1), there is a problem that there is no means for acquiring object information when the object does not have a unique ID. In the case of (2) above, since the calculation is performed based on the relative position starting from one point on the terminal screen, for example, terminals with different displays have different window size and resolution prescribed values, and thus cannot acquire objects correctly. In addition, the flow text type GUI (HTML) or the like has a problem that the object cannot be acquired correctly because the size and position of the object change dynamically. Furthermore, in the case of (3) above, it is necessary to recognize the object tree structure in advance, but the object tree structure tends to be created depending on each application. In that case, even if the screen looks the same, the depth of the tree, the structure of the path, and the number of objects differ little by little, so there is a problem that an object cannot be acquired, for example, in an application in which the object tree changes dynamically.

  Further, in response to the problem that the position information on the screen depends on the object recognition of (2) above, the user defines a relative relationship between objects as a rule, and obtains a corresponding solution candidate using the information as a constraint condition. A technique for performing appropriate masking according to the image content is proposed (see Patent Document 1 and Patent Document 2). According to this technique, a rule for automatically ordering the arrangement relationship of each object from the absolute coordinates of each object is automatically generated for each of the x-axis and the y-axis, and each object can be arranged based on the rule. Find the area. Thereby, the screen can be appropriately divided according to the arrangement position and size of each object. However, this technique mainly aims to divide the screen, and does not hold information as to whether the object is relatively in view of other objects. For this reason, for example, it is not possible to cope with a use of searching for another object two-dimensionally with a certain object as an axis, such as “an object immediately below the object two to the right of object X”.

  Accordingly, an object of the present invention is to make it possible to specify an object relative to other objects by causing each object on the screen displayed by various applications to recognize the relative relationship between the objects. . For example, if object X is on the left of object Y but object X cannot be obtained directly using the method described above, after acquiring object Y, object X can be specified starting from object Y using the relative relationship. Let's do that.

  In order to solve the above-described problem, the present invention is a screen recognition device that recognizes a relative relationship between objects displayed on a screen, and the screen information of the screen displays each object displayed on the screen. An object information acquisition unit that acquires absolute coordinate information; a relative position information acquisition unit that records a relative position between any two objects on the screen as relative position information based on the absolute coordinate information of each object; A grid row generation unit that generates one or more grid row groups in which objects are arranged from left to right starting from the leftmost object on the screen based on the relative position information, and the grid row generated by the grid row generation unit A grid matrix in which groups are arranged according to the upper and lower relative positions of objects included in the grid row based on the relative position information. Form, characterized in that it comprises a grid arrangement section for outputting.

  ADVANTAGE OF THE INVENTION According to this invention, the relative relationship of each object on the screen displayed by various applications can be recognized. As a result, an object can be specified relative to other objects.

FIG. 1 is a diagram for explaining the outline of the screen recognition apparatus. FIG. 2 is a diagram illustrating an example of object search processing on the screen. FIG. 3 is a functional block diagram of the screen recognition apparatus. FIG. 4 is a diagram for explaining the processing of the object information acquisition unit. FIG. 5 is a diagram for explaining processing of the relative position information acquisition unit. FIG. 6 is a diagram for explaining the nearest object. FIG. 7 is a diagram illustrating an example of a grid row. FIG. 8 is a diagram illustrating an example of a grid row. FIG. 9 is a diagram illustrating an example of a grid row arrangement procedure. FIG. 10 is a diagram illustrating an example of a grid row arrangement procedure. FIG. 11 is a diagram illustrating an example of a grid row arrangement procedure. FIG. 12A is a flowchart illustrating a processing procedure of the screen recognition apparatus. FIG. 12B is a flowchart showing details of the object information acquisition processing of FIG. 12A. FIG. 13 is a flowchart showing details of the relative position information acquisition processing of FIG. 12A. FIG. 14 is a flowchart showing details of the left-right relative relationship acquisition processing of FIG. FIG. 15 is a flowchart showing details of the vertical relative relationship acquisition process of FIG. FIG. 16 is a flowchart showing details of the nearest neighbor object acquisition process of FIG. FIG. 17 is a flowchart showing details of the horizontal nearest neighbor object acquisition processing of FIG. FIG. 18 is a flowchart showing details of the vertical nearest neighbor object acquisition process of FIG. FIG. 19 is a flowchart showing details of the grid row generation processing of FIG. 12A. FIG. 20 is a flowchart showing details of the grid arrangement processing of FIG. 12A. FIG. 21 is a flowchart showing details of the processing for inserting the column of FIG. FIG. 22 is a diagram for explaining an application example of screen recognition information recognized by the screen recognition device. FIG. 23 is a diagram for describing an application example of screen recognition information recognized by the screen recognition device. FIG. 24 is a diagram illustrating a computer that executes a screen recognition program.

  Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment.

  First, the outline of the screen recognition apparatus 100 of the present embodiment will be described with reference to FIG. The screen recognition apparatus 100 recognizes the relative positions of objects on the screen displayed by the application, and outputs information (screen recognition information) indicating the recognition result. For example, as shown in FIG. 1, the screen recognition information is information in which the relative positions of the objects on the screen are arranged in a minimum grid matrix.

  For example, as shown in FIG. 1, when the screen recognition apparatus 100 acquires screen information (information indicating the absolute position coordinates of each object on the screen), the screen recognition apparatus 100 indicates the relative position of each object from the screen information. Generate and output a grid matrix.

  In the following description, a grid-like rectangle for expressing the relative position (relative relationship) of objects on the screen is referred to as a grid, and a grid arranged in a predetermined number of matrices is referred to as a grid matrix. In addition, the attributes on each grid matrix (for example, images, buttons, labels, etc.) and text information (for example, “use another account”) written on the object are also written. Also good.

  In this way, the screen recognition apparatus 100 abstractly recognizes the relative positional relationship between the objects on the screen, so that, for example, it becomes easy to search for an object necessary for operation support.

  For example, when the device that supports the operation as shown in FIG. 2 receives an inquiry (query) including the arrangement pattern of the object to be operated and the variable name definition, the arrangement pattern indicated by the query is displayed on the screen. It is possible to search for objects that match the above (GUI parts on the window surrounded by the thick line in FIG. 2). And said apparatus outputs the searched object (object used as operation object). For example, the above-described device is the 23rd object (“set” button) on the window where the button_set to be operated is surrounded by a thick line, and the 24th object (“ “OK” button) is output.

  Next, the configuration of the screen recognition apparatus 100 will be described with reference to FIG. The screen recognition apparatus 100 includes an object information acquisition unit 111, a relative position information acquisition unit 112, a grid row generation unit 113, and a grid arrangement unit 114.

  Further, the screen recognition device 100 includes an object information holding unit 121, a relative position information holding unit 122, and a grid row holding unit 123 in a storage unit (not shown). The object information holding unit 121 stores the object information output from the object information acquisition unit 111. The relative position information holding unit 122 stores the relative position information output from the relative position information acquisition unit 112. The grid row holding unit 123 stores the grid row output from the grid row generation unit 113.

  The object information acquisition unit 111 acquires information (object information) for recognizing the relative position of each object displayed on the screen from the screen information of the screen displayed by the application. For example, the object information is information such as class (attribute), text (text information), area (absolute coordinate information), and the like.

  For example, the object information acquisition unit 111 attaches an index (for example, 1 to 4) to each object on the screen from the screen information shown in FIG. 4, and class (attribute), text (text information), area ( (Absolute coordinate information) is acquired, and the object information shown in FIG. 4 is generated. Then, the object information acquisition unit 111 outputs the generated object information to the object information holding unit 121.

  The relative position information acquisition unit 112 acquires relative position information indicating the relative position between any two objects on the screen from the absolute coordinate information of each object indicated in the object information. That is, the relative position information acquisition unit 112 generates relative position information indicating the relative position between any two objects on the screen based on the absolute coordinate information of each object, and outputs the relative position information to the relative position information holding unit 122.

  For example, as shown in FIG. 5, the relative position information acquisition unit 112, for any two objects on the screen indicated by the object information, one object is relatively in the right / left / lateral direction of the other object. Or objects that are peers in the lower / up / vertical direction (relative relationship), or objects that are next to each other in the vertical / horizontal direction (nearest) Object). Then, the determination result is recorded in the relative position information.

  For example, for the two objects (obj1 and obj2), the relative position information acquisition unit 112 determines whether obj2 is positioned relative to obj1 or relative to the left of obj1, as indicated by reference numeral 501, obj1 And the horizontal direction (that is, at least part of the x coordinate overlaps obj1 and obj2). Further, as indicated by reference numeral 502, the relative position information acquisition unit 112 determines whether obj2 is positioned below obj1 or positioned above obj1, or is in the same direction as obj1 (that is, obj1 And obj2 at least part of the y-coordinate overlaps). Furthermore, as indicated by reference numeral 503, the relative position information acquisition unit 112 determines whether obj2 is adjacent to obj1 (may be separated) or not adjacent to obj1. Note that obj2 is not adjacent to obj1 when there is another object (obj3) between obj2 and obj1, or obj2 is located diagonally (not vertically or horizontally) This is the case.

  For example, when obj2 and obj1 are the positions shown in (1) of FIG. 6, the relative position information acquisition unit 112 determines that the object obj2 is adjacent to obj1 in the vertical / horizontal direction (nearest neighbor). Judge. On the other hand, when obj2 and obj1 are the positions shown in (2) of FIG. 6, the relative position information acquisition unit 112 determines that obj2 is not closest to obj1. Note that the relative position information acquisition unit 112 does not handle a case where obj2 and obj1 overlap as shown in FIG.

  The grid row generation unit 113 in FIG. 3 generates a grid row in which objects are arranged from left to right starting from the upper left object on the screen based on the relative position information of the objects. Then, the grid row generation unit 113 outputs the generated grid rows to the grid row holding unit 123 in the order in which the grid rows are generated, for example, starting from the upper left object.

  For example, for the eight objects on the screen shown on the left in FIG. 7, the grid row generation unit 113 arranges objects that are the same in the vertical direction in the relative position information in the same grid row in the order of left → right, and FIG. The five grid lines shown on the right are generated. In addition, the number attached | subjected to the left of the grid line shown on the right of FIG. 7 shows the acquisition order of a grid line, or the line number by which the said grid line is arrange | positioned.

  Further, for example, the grid row generation unit 113 arranges the objects that are the same in the vertical direction in the relative position information in the same grid row in the order of left to right for the 14 objects on the screen shown on the left in FIG. 9 grid lines shown to the right of 8 are generated.

  As described above, the grid row generation unit 113 generates a grid row by arranging, from left to right, a group of objects that are relatively similar to each other in the vertical direction starting from the upper left object on the screen based on the relative position information. The grid row group is generated by sequentially executing the above processing until there is no object under the grid row.

  The grid arrangement unit 114 in FIG. 3 generates a grid matrix by arranging the grid row group generated by the grid row generation unit 113 according to the upper and lower relative positions of the objects included in the grid row based on the relative position information. ,Output.

  For example, the grid arrangement unit 114 arranges the grid row group stored in the grid row holding unit 123 so that the upper and lower relative positions of the objects included in each grid row are maintained in the grid row processing order (row number order). To do. Specifically, the grid placement unit 114 refers to the relative position information, and keeps the relative position between the objects of the same level (see FIG. 5) in the horizontal direction among the objects included in each grid row. Arrange so that.

  For example, when the object B is positioned relatively on the same side and relatively above the object A, the grid placement unit 114 may cause the object B to be positioned on the same row and in the same column in the grid matrix. The grid row including the object A and the grid row including the object B are arranged. Thereby, the grid arrangement | positioning part 114 can produce | generate the grid matrix which has arrange | positioned the objects located in the same direction and relatively lower / upper in the horizontal direction in the same column. The arrangement of the grid rows will be described later using a specific example.

  Also, if the grid placement unit 114 is placed with the objects included in the grid rows as they are, the vertical relative positions of the objects included in the grid rows relative to each other in the horizontal direction (see FIG. 5) are relatively long. If it cannot be maintained, increase the part of the object included in the grid row and place it. The increase in objects here will also be described later using a specific example.

  Here, a specific example of grid row arrangement will be described. In the following description, the “higher object” refers to an object that is in the horizontal direction and relatively above the currently focused object. In the description shown in FIGS. 9, 10, and 11, for example, the objects 1, 2,... Are abbreviated as “1, 2,.

  For example, the grid placement unit 114 places the grid rows as shown in FIG. 9 for the grid rows shown in FIG. 7, and generates a grid matrix.

  First, the grid arrangement unit 114 extracts the first grid row from the grid row group shown in FIG. 9 (S1 in FIG. 9). Next, when the grid placement unit 114 takes out the second grid line, the object 1 is placed above the object 3 in the second grid line, so the object is placed on the object 3 in the first grid line. 1 is added (S2). Then, when the grid placement unit 114 takes out the third grid row, the upper and lower relative positions of the objects in the respective grid rows are maintained, so that the placement is not changed (S3), and the process proceeds to S4.

  After S3, when the grid placement unit 114 takes out the fourth grid row, the upper and lower relative positions of the objects in the respective grid rows are maintained, so that there is no change in the placement (S4) and the process proceeds to S5. . Thereafter, the grid placement unit 114 takes out the fifth grid row. Here, since the object 7 in the fifth grid row has the object 6 at the upper level, the grid placement unit 114 places the object 7 below the object 6. Further, since 1 is placed above the object 8, the grid placement unit 114 adds the object 1 so that the object 8 is below the object 1 (S5). In this way, the grid placement unit 114 completes the placement of the grid row group shown in FIG.

  For example, the grid placement unit 114 places the grid rows as shown in FIGS. 10 and 11 for the grid rows shown in FIG.

  That is, when the grid placement unit 114 takes out the first grid row from the grid row group shown in FIG. 8 (S11 in FIG. 10) and then takes out the second grid row, Since the object 1 is included, the object 1 is added to the second column of the first grid row (S12), and the process proceeds to S13. Subsequently, when the grid placement unit 114 takes out the third grid row, it has the objects 1, 2, and 3 above (relatively above) the object 4, and therefore the object in the second column of the third grid row. 4 is added (S13).

  After S13, when the grid arrangement unit 114 takes out the fourth grid line, the object 5 in the fourth grid line has the objects 1, 2, 3, and 4 in the upper layer, so the fourth grid line The object 5 is added to the second column (S14).

  After S14, the grid placement unit 114 takes out the fifth grid row. Here, since the object 6 in the fifth grid row has the objects 1, 3, and 4 at the upper level but does not have the objects 2 and 5, the grid placement unit 114 sets 5 in the new column (third column). Arrangement is made so that the top of the grid line (object 6) comes, and objects 1, 3, and 4 are added to the column (third column) (S15). That is, the grid placement unit 114 places a fourth grid row and then adds a column made up of the object indicated by reference numeral 1001.

  After S15, the grid placement unit 114 takes out the sixth grid row. Here, since the object 7 of the sixth grid row has the objects 1 to 5 at the upper level, the grid placement unit 114 adds the object 7 to the second column of the sixth grid row (S16). Here, after the processing of S16, the object 8 in the sixth grid row can be arranged only in the third and subsequent columns. Furthermore, since the object 8 has the objects 1, 3, 4, 5, and 6 at the upper level, the grid placement unit 114 adds the objects 1, 3, 4, 5, and 8 before the third column (S17). That is, after arranging the sixth grid row, the grid placement unit 114 inserts a column composed of the object indicated by reference numeral 1002 into the third column.

  After S17, the grid placement unit 114 takes out the seventh grid row. Here, since the object 9 in the seventh grid line has the objects 1 to 8 in the upper layer, the grid placement unit 114 adds the object 9 to the second to fourth columns of the seventh grid line (S18). .

  Turning to the description of S19 in FIG. After S18, since the object 10 in the seventh grid row has the objects 1, 3, 4, 6, and 8 in the upper level, the grid placement unit 114 displays them (objects 1, 3, 4, and 4) in the fifth column. 6, 8) and the object 10 are arranged (S19).

  After S19, the grid placement unit 114 takes out the eighth grid row. Here, since the object 11 of the eighth grid row has the objects 1 to 10 at the upper level, the grid placement unit 114 adds the object 11 to the second to fourth columns of the eighth grid row (S20). .

  After S20, the grid placement unit 114 takes out the ninth grid row. Here, since the object 12 in the ninth grid row has the objects 1, 3, 4, 5, 6, 8, 9, and 11 at the upper level, the object 12 is arranged in the third and fourth columns (S21).

  After S21, the object 13 in the ninth grid row has the objects 1, 3, 4, 6, 8, 9, 10, and 11 in the upper layer, but the object 9 is insufficient and the object 13 is in the fifth column and thereafter. Can only be placed in Therefore, the grid arrangement unit 114 inserts columns between the fourth to fifth columns and adds columns of objects 1, 3, 4, 6, 8, 9, 11, and 13 (S22).

  After S22, the object 14 in the ninth grid row has the objects 1, 3, 4, 6, 8, and 10 at the upper level, but can be arranged only in the seventh and subsequent columns. Therefore, the grid arrangement unit 114 adds the objects 1, 3, 4, 6, 8, and 10 to the seventh column (higher than the object 14) (S23).

  By doing in this way, the grid arrangement | positioning part 114 can produce | generate the grid matrix which reflected the relative position of the up-down and left-right between objects. In addition, when generating the grid matrix, the grid placement unit 114 places objects that are relatively similar to each other in the vertical direction and relatively left / right in the same row, and that is relatively similar to each other in the horizontal direction. Objects that are relatively below / above are placed in the same row.

  Thereby, the grid arrangement unit 114 arranges the objects arranged in the same position in the horizontal direction on the screen in the same row, and arranges the objects arranged in the same position in the vertical direction on the screen in the same column. A grid matrix can be obtained. That is, the grid arrangement unit 114 can generate screen recognition information (grid matrix) obtained by rounding the object even if there is an object arrangement deviation on the screen. In other words, the grid placement unit 114 can robustly recognize the positional relationship of objects on the screen.

(Processing procedure)
Next, the outline of the processing procedure of the screen recognition apparatus 100 will be described with reference to FIG. 12A. First, the object information acquisition unit 111 of the screen recognition device 100 acquires object information based on the screen information (S121: acquisition of object information), and the relative position information acquisition unit 112 based on the absolute coordinate information of each object in the object information. Then, the relative position information between the objects is acquired (S122: acquisition of relative position information). Then, the grid row generation unit 113 generates a grid row based on the relative position information (S123: grid row generation), and the grid arrangement unit 114 arranges the generated grid row (S124: grid arrangement).

  Next, the object information acquisition process of S121 of FIG. 12A will be described in detail using FIG. 12B. First, the object information acquisition unit 111 reads screen information (S1211), acquires the class (class) and absolute coordinates (area) of each object from the screen information, and if there is text (text) (S1212). An index is indexed (S1213). Then, the object information acquisition unit 111 sends information (object information) in which the class (class), absolute coordinates (area) of each object, text (text), and index (index) are associated with each other to the object information holding unit 121. It outputs (S1214).

  Note that the object information may be generated as a class for each object, and the information may be held as a property, or may be held in a matrix.

  Next, the relative position information acquisition process in S122 of FIG. 12A will be described in detail with reference to FIG. First, when the relative position information acquisition unit 112 reads the object information of the object information holding unit 121 (S1221), the relative position information acquisition unit 112 acquires the left-right relative relationship of each object based on the absolute coordinates of each object (S1222: acquisition of the left-right relative relationship). The vertical relative relationship is acquired (S1223: vertical relative relationship acquisition), and the nearest object is acquired (S1224: nearest neighbor object acquisition). Then, the relative position information acquisition unit 112 records the acquired information as relative position information and outputs it to the relative position information holding unit 122 (S1225).

  The relative position information may be held in a list for each object or in a matrix. Also, a function for calculating the relative position may be prepared, and the relative position information acquisition unit 112 may calculate the relative position of the object each time.

  Next, the left-right relative relationship acquisition process of S1222 of FIG. 13 will be described in detail with reference to FIG. Here, a case where the left-right relative relationship (relative position) between two objects (obj1 and obj2) is determined will be described as an example.

  In the following explanation, the upper left coordinate of obj1 is (px1, py1), the lower right coordinate is (qx1, qy1), the upper left coordinate of obj2 is (px2, py2), and the lower right coordinate is (qx2 , Qy2).

  The relative position information acquisition unit 112 reads information on the coordinate positions of the objects (obj1 and obj2) from the object information (S131), and the coordinate positions of obj1 and obj2 are px2 ≧ qx1 in the x-axis direction (“px2 ≧ qx1” in S132). In other words, if the maximum x-axis value of obj1 is less than or equal to the minimum x-axis value of obj2, it is determined that obj2 is relative to obj1 (val = 1) (S133).

  On the other hand, if the coordinate position of obj1 and obj2 is qx2 ≦ px1 in the x-axis direction (“qx2 ≦ px1” in S132), that is, if the minimum x-axis value of obj1 is greater than or equal to the maximum x-axis value of obj2, the relative position information acquisition unit 112 determines that obj2 is relative to the left of obj1 (val = −1) (S134).

  Further, when neither px2 ≧ qx1 nor qx2 ≦ px1 (“else” in S132), that is, when px1 ≦ px2 ≦ qx1 or px1 ≦ qx2 ≦ qx1, the relative position information acquisition unit 112 determines that obj2 is in the horizontal direction with obj1 It is determined that they are the same (val = 0) (S135).

  Then, the relative position information acquisition unit 112 outputs the determination result (val) of S133 to S135 as information related to the left-right relative relationship between the objects (S136).

  Next, with reference to FIG. 15, the vertical relative relationship acquisition process in S1223 of FIG. 13 will be described in detail. Here again, a case where the relative positions of the two objects (obj1 and obj2) are determined will be described as an example. As shown in FIG. 15, it is assumed that the y-axis value of an object becomes higher as the object at the bottom of the screen.

  The relative position information acquisition unit 112 reads information on the coordinate positions of the objects (obj1 and obj2) from the object information (S141), and the coordinate positions of obj1 and obj2 are py2 ≧ qy1 in the y-axis direction (“py2 ≧ qy1” in S142) That is, if the maximum y-axis value of obj1 is less than or equal to the minimum y-axis value of obj2, it is determined that obj2 is below obj1 (val = 1) (S143).

  On the other hand, if the coordinate positions of obj1 and obj2 are qy2 ≦ py1 in the y-axis direction (“qy2 ≦ py1” in S142), that is, if the minimum y-axis value of obj1 is greater than or equal to the maximum y-axis value of obj2, the relative position information acquisition unit 112 determines that obj2 is relative to obj1 (val = −1) (S144).

  Further, when neither py2 ≧ qy1 nor qy2 ≦ py1 (“else” in S142), the relative position information acquisition unit 112 determines that obj2 is a peer (val = 0) in the vertical direction with obj1 (S145).

  Then, the relative position information acquisition unit 112 outputs the determination results (val) of S143 to S145 as information related to the vertical relative relationship of the objects (S146).

  Next, the nearest object acquisition process in S1224 of FIG. 13 will be described in detail with reference to FIG. For example, the relative position information acquisition unit 112 reads information on the left-right relative relationship and the vertical relative relationship of the object (obj1) (S152), acquires the nearest object in the horizontal direction of the object (obj1) (S153), When the vertical direction nearest object of (obj1) is acquired (S154), these are collectively output as the nearest object of the object (obj1) (S155). For example, the relative position information acquisition unit 112 outputs the object (obj2) surrounded by a thick line among the objects (obj2) indicated by reference numeral 1601 as the nearest object of the object (obj1).

  Next, the horizontal direction nearest neighbor object acquisition process of S153 in FIG. 16 will be described in detail with reference to FIG. For example, the relative position information acquisition unit 112 reads information related to the left-right relative relationship of the object (obj1) (S161), and sets the horizontal nearest neighbor group neighbor = {} (S162). In step S162, neighbor = {} is empty.

  After S162, the relative position information acquisition unit 112 acquires an object group (M) that is the same as obj1 in the vertical direction from the information related to the left-right relative relationship of obj1 (S163). For example, the relative position information acquisition unit 112 acquires the obj2 group indicated by the balloon 1701.

  After S163, the relative position information acquisition unit 112 executes the processes shown in S164 to S170 for each of obj2 belonging to the above M. That is, the relative position information acquisition unit 112 determines that the relative position of obj2 as viewed from obj1 is relative left (“relative left” in S164), and there is no object arranged between obj2 and obj1 in S165 (S165). No), the obj2 is determined as the nearest neighbor of obj1, and added to the neighbor (S167).

  On the other hand, if there is an object arranged between obj2 and obj1 in M (Yes in S165), the relative position information acquisition unit 112 does not determine that obj2 is the nearest neighbor of obj1 (S168). Similarly, when the relative position of obj2 as viewed from obj1 is relative right (“relative right” in S164), the relative position information acquisition unit 112 also determines that the object placed between obj2 and obj1 is M. If not (No in S166), it is determined that obj2 is the nearest neighbor of obj1, and is added to the neighbor (S169).

  On the other hand, if there is an object arranged between obj2 and obj1 in M (Yes in S166), the relative position information acquisition unit 112 does not determine that obj2 is the nearest neighbor of obj1 (S170).

  Then, the relative position information acquisition unit 112 outputs the obj2 group determined to be the nearest neighbor of obj1 by the processes of S167 to S170 as the laterally nearest neighbor object group neighbor (S171).

  For example, the relative position information acquisition unit 112 outputs the obj2 group indicated by ○ among the obj2 groups indicated by the balloon 1701 as the nearest neighbor object group neighbor of obj1. On the other hand, among the obj2 groups shown in the balloon 1701, the obj2 group marked with x is not output as the nearest neighbor object group neighbor in the horizontal direction because other objects are arranged between obj1 and obj1.

  Next, with reference to FIG. 18, the vertical nearest neighbor object acquisition processing in S154 of FIG. 16 will be described in detail. For example, the relative position information acquisition unit 112 reads information related to the vertical relative relationship of the object (obj1) (S172), and sets the vertical nearest neighbor object group = {} (S173). In step S173, neighbor = {} is empty.

  After S173, the relative position information acquisition unit 112 acquires an object group (M) that is the same as obj1 in the horizontal direction from the information about the vertical relative relationship of obj1 (S174). For example, the relative position information acquisition unit 112 acquires the obj2 group indicated by the balloon 1801.

  Then, the relative position information acquisition unit 112 executes the processes shown in S175 to S181 for each of obj2 belonging to M. That is, the relative position information acquisition unit 112 has the relative position of obj2 as viewed from obj1 relatively above (“relatively up” in S175), and if there is no object arranged between obj2 and obj1 in S175 (S176). No), obj2 is determined as the nearest neighbor of obj1 and added to the neighbor (S178).

  On the other hand, if there is an object arranged in M between obj2 and obj1 (Yes in S176), the relative position information acquisition unit 112 does not determine that obj2 is the nearest neighbor of obj1 (S179). Similarly, when the relative position of obj2 as viewed from obj1 is relatively lower (“relatively lower” in S175), the relative position information acquisition unit 112 similarly determines that an object placed between obj2 and obj1 is in M. If not (No in S177), it is determined that obj2 is the nearest neighbor of obj1, and is added to the neighbor (S180). On the other hand, if there is an object arranged between obj2 and obj1 in M (Yes in S177), the relative position information acquisition unit 112 does not determine that obj2 is the nearest neighbor of obj1 (S181).

  Then, the relative position information acquisition unit 112 outputs the obj2 group determined to be the nearest neighbor of obj1 by the processes of S178 to S181 as the vertical nearest neighbor object group neighbor (S182). For example, the relative position information acquisition unit 112 outputs the obj2 group marked with ◯ among the obj2 groups indicated by the balloon 1801 as the nearest neighbor object group neighbor of obj1. Of the obj2 groups shown in the balloon 1801, the obj2 group marked with x is not included in the nearest neighbor object group neighbor in the vertical direction because other objects are arranged between obj1 and obj1.

  With the above processing, the relative position information acquisition unit 112 can create relative position information indicating the relative left / right and relative lower / upper positional relationships of each object and the nearest object.

  Next, the grid row generation process in S123 of FIG. 12A will be described in detail with reference to FIG.

  First, the grid row generation unit 113 reads the relative position information of the object group (S191). Next, the grid row generation unit 113 selects the upper left object as obj (object to be processed) with reference to the relative position information of the object group (obj ← upper left object), and sets the grid row set S. = {}, Searched object set CS = {}, unsearched object set US = {} are set (S192). In step S192, an object group that is the target of the relative position information read in step S191 is set in the unsearched object set US.

  And the grid line production | generation part 113 performs the process of S193-S199 until it processes all the objects.

  First, the grid row generation unit 113 refers to the relative position information of the object group, acquires the object group that is the same as obj in the vertical direction from left to right, and sets this as a grid row (S193). Next, the grid row generation unit 113 adds the grid row obtained in S193 to the grid row set G (S194), adds the grid row to CS (searched object set), and starts from US (unsearched object set). The grid row is deleted (S195).

  After S195, when the grid row generation unit 113 refers to the relative position information and determines that there is an object nearest to obj and the nearest object (Yes in S196), the nearest object group The uppermost object group is acquired (S197). Then, the grid row generation unit 113 sets the leftmost object in the acquired object group as the next obj (S199). For example, as shown in a balloon 1901, if there is a nearest object group under obj, the grid row generation unit 113 selects the object with the smallest y-axis value (the object indicated by the bold line) as the next Let obj be. Then, the grid row generation unit 113 performs the processing after S193 for the next obj.

  On the other hand, even if the grid row generation unit 113 refers to the relative position information and no object exists below obj or an object exists below obj, the object is the nearest neighbor of obj. When it is determined that the object is not the object (No in S196), the grid row generation unit 113 acquires the highest-level object group from the US (unsearched object set) (S198), and proceeds to S199.

  The grid row generation unit 113 executes the above processing until the unsearched object set US = {} becomes empty, and outputs the generated grid row set (S200).

  Through the above processing, the grid row generation unit 113 can generate a grid row in which the same object group is arranged horizontally from the left in the horizontal direction. For example, the grid row generation unit 113 generates the grid row group illustrated in FIG. 8 from the object group on the screen illustrated in FIG.

  Next, the grid arrangement process in S124 of FIG. 12A will be described in detail with reference to FIG. For example, the grid placement unit 114 reads a grid row set (S201), and sorts the read grid row set S in ascending order of the y-axis (from the highest grid row position to the lowest) (S202). Next, the grid placement unit 114 creates an initial grid G and places the first row of the grid row set S on the first row (S203). For example, when generating the initial grid G indicated by reference numeral 2001, the grid placement unit 114 places the first row of the grid row set S (grid matrix including the object 1) on the first row.

  And the grid arrangement | positioning part 114 performs the process shown to S204-S209 until it processes all the grid rows of the grid row set S. FIG.

  First, the grid arrangement unit 114 sorts the grid row set S in ascending order of the x-axis (S204), and executes the processing of S205 to S209 for each object (obj) on the grid row. That is, the grid placement unit 114 acquires the minimum column number i in which obj can be placed from the grid row set S (S205). Next, the grid arrangement unit 114 acquires a column group having an object group higher than obj in the grid row set S (S206), and determines whether or not a column group satisfying the following condition exists after the i column. Judgment is made (S207).

  This condition is that the upper (relatively upper and laterally equal) object group is u, the grid column (column group acquired in S206) is q = {q1, q2,..., Qn}, and xi is the grid column i. When it is determined whether or not the object is to be arranged (1/0), q is a condition that q is arranged so as to be a subset (q⊆u) of u and Σqi xi = | u |.

  That is, an object arranged in the same column as an object i is one of the objects above the object i, and the number of types of objects arranged in the same column as the object i is higher than the object i ( The condition is that the number of objects that are relatively on the side of the object i and the same on the side is the same. In other words, the condition is that the object group arranged at the top of the same column as the object i covers all the objects existing above the object i in the relative position information.

  In S207, when the grid placement unit 114 determines that there are column groups satisfying the above condition after i columns (Yes in S207), G (grid) obj is placed in a row satisfying the above conditions (S208). . On the other hand, when the grid arrangement unit 114 determines that there is no column group satisfying the above condition after the i column (No in S207), the grid arrangement unit 114 inserts the column into G (grid) (S209). This row insertion processing will be described later with reference to FIG.

  And if the grid arrangement | positioning part 114 performs said process with respect to all the grid rows of a grid row set, it will output the produced | generated grid (grid matrix) (S210).

  Through the above processing, the grid arrangement unit 114 can generate a grid matrix in which the same object groups are arranged in the same column in the vertical direction.

  Next, with reference to FIG. 21, the column insertion processing in S209 of FIG. 20 will be described in detail. The grid placement unit 114 sets the object placed in the rightmost column of the search row (grid row where the object group higher than obj exists) as obj2 (S2091). If the grid placement unit 114 determines that obj2 is relative to obj1 (Yes in S2092), an empty column is added to the leftmost left side of the row where obj2 is placed (S2093).

  After S2093, the grid placement unit 114 places the object higher than obj and obj so that no contradiction occurs in the arrangement order of the other rows when the object group higher than obj2 is added (S2094). That is, the grid arrangement unit 114 refers to the relative position information with respect to the row where the object to be added is arranged, and checks whether other objects have a correct relative position relationship. Then, the grid placement unit 114 adds a column to the left or right of the object so that the contradiction is resolved when the contradiction occurs in the relative positional relationship.

  On the other hand, when the grid placement unit 114 determines that obj2 is not relative to the right of obj1 (No in S2092) and is relatively left (Yes in S2095), the grid placement unit 114 is empty in the rightmost column + 1 column. A column is added, and an object higher than obj and obj are arranged so that no contradiction occurs in the arrangement order of each row (S2096). If the grid placement unit 114 determines that obj2 is not relative to the right of obj1 (No in S2092) and is not relative to the left (No in S2095), the process of S2096 is not performed.

  Through the above processing, the grid placement unit 114 can reliably generate a grid matrix in which the peer object groups are placed in the same column in the vertical direction.

(Application example of this embodiment)
According to the screen recognition apparatus 100 described above, the relative relationship between objects can be recognized for each object on the screen displayed by various applications.

  For example, the screen recognition apparatus 100 recognizes the arrangement of objects on the screen of the travel reservation application indicated by reference numeral 2200 in FIG. 22 and obtains screen recognition information (grid matrix) indicated by reference numeral 2201.

  Thereby, the information retrieval system on the screen of the application concerned deletes the Image and Button objects in the screen information denoted by reference numeral 2201 and defines the item name (for example, the label name as the item name and the Label class as the item name) Screen recognition information indicated by reference numeral 2202 is obtained. As a result, the search system can perform advanced searches using item names such as “collect and extract information related to the outbound route”.

  Further, by using the screen recognition technology by the screen recognition apparatus 100 described above, an inquiry search based on the relative position of the object on the screen of the application can be performed.

  For example, as shown in (1) of FIG. 23, a query (inquiry) for checking whether the image on the right of “decision status” on the application screen is the same as the image (icon) of the comparison source is accepted. In this case, it is possible to return a result as to whether or not the image on the right of the “decision status” on the screen of the application is the same as the comparison source image (icon).

  Further, for example, as shown in (2) of FIG. 23, when a query (inquiry) for searching for “95-600727” in the image group under “equipment number” on the application screen is received. The search result “95-600727” in the image group under the “equipment number” on the application screen can be returned.

  In addition, by combining the screen recognition technology by the screen recognition device 100 with an existing image matching technology, each image element that has been identified is further matched using the relative position information, and more It is also possible to achieve highly accurate matching, such as selecting the correct element that meets the conditions.

(program)
The screen recognition apparatus 100 described in the above embodiment can be implemented by installing a screen recognition program for realizing the functions of the screen recognition apparatus 100 in a desired information processing apparatus (computer). For example, the information processing apparatus can function as the screen recognition apparatus 100 by causing the information processing apparatus to execute the screen recognition program provided as package software or online software. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, the information processing apparatus includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), PDA (Personal Digital Assistants), and the like. Further, the screen recognition device 100 may be mounted on a cloud server.

  An example of a computer that executes the above screen recognition program will be described below. FIG. 24 is a diagram illustrating a computer that executes a screen recognition program. As shown in FIG. 24, the computer 1000 includes, for example, a memory 1010, a CPU (Central Processing Unit) 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network. Interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. For example, a mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050. For example, a display 1130 is connected to the video adapter 1060.

  Here, as shown in FIG. 24, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Various data and information described in the above embodiment are stored in, for example, the hard disk drive 1090 or the memory 1010.

  Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1090 to the RAM 1012 as necessary, and executes the above-described procedures.

  The program module 1093 and the program data 1094 related to the program are not limited to being stored in the hard disk drive 1090. For example, the program module 1093 and the program data 1094 are stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. May be. Alternatively, the program module 1093 and the program data 1094 related to the program are stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and the CPU 1020 via the network interface 1070. May be read.

DESCRIPTION OF SYMBOLS 100 Screen recognition apparatus 111 Object information acquisition part 112 Relative position information acquisition part 113 Grid row production | generation part 114 Grid arrangement | positioning part 121 Object information holding part 122 Relative position information holding part 123 Grid line holding part

Claims (9)

A screen recognition device for recognizing a relative relationship between objects displayed on a screen,
From the screen information of the screen, an object information acquisition unit that acquires absolute coordinate information of each object displayed on the screen;
Based on the absolute coordinate information of each object, a relative position information acquisition unit that records the relative position between any two objects on the screen as relative position information;
A grid row generation unit that generates a grid row group including one or more grid rows in which the objects on the screen are arranged according to the left and right relative positions based on the relative position information;
A grid in which the grid row group generated by the grid row generation unit is arranged according to the upper and lower relative positions of objects included in the grid row for each grid row constituting the grid row group based on the relative position information. A grid layout unit for generating and outputting a matrix;
A screen recognition apparatus comprising:   A screen recognition device for recognizing a relative relationship between objects displayed on a screen,
  From the screen information of the screen, an object information acquisition unit that acquires absolute coordinate information of each object displayed on the screen;
  Based on the absolute coordinate information of each object, a relative position information acquisition unit that records the relative position between any two objects on the screen as relative position information;
  Based on the relative position information, a grid row generation unit that generates a grid row group including one or more grid rows in which objects are arranged from left to right starting from an object at the upper left of the screen;
  A grid in which the grid row group generated by the grid row generation unit is arranged according to the upper and lower relative positions of objects included in the grid row for each grid row constituting the grid row group based on the relative position information. A grid layout unit for generating and outputting a matrix;
  A screen recognition apparatus comprising: The grid row generation unit
Based on the relative position information, a process of generating a grid row by arranging a group of objects that are peers relative to each other in the vertical direction relative to the upper left object on the screen from the left to the right. screen recognition apparatus according to claim 1 or 2, characterized in that to generate the grid line group by sequentially executed until the object disappears. The grid placement unit is
When the grid row group generated by the grid row generation unit is arranged according to the relative vertical position of the objects included in the grid row based on the relative position information, the relative among the objects included in the grid row screen recognition apparatus according to claim 1 or 2 laterally object isotopically characterized in that arranged to keep the relative position of the upper and lower. The grid placement unit is
In the grid row group generated by the grid row generation unit, when the objects included in the grid row cannot be arranged so as to keep the relative positions of the top and bottom, a part of the objects included in the grid row is increased. in screen recognition apparatus according to claim 1 or 2, characterized in that disposed. A screen recognition device that recognizes the relative relationship between objects displayed on the screen,
From the screen information of the screen, an object information acquisition step of acquiring absolute coordinate information of each object displayed on the screen;
A relative position information acquisition step for recording a relative position between any two objects on the screen as relative position information based on the absolute coordinate information of each object;
A grid row generation step for generating a grid row group including one or more grid rows in which objects on the screen are arranged according to the left and right relative positions based on the relative position information;
A grid in which the grid row group generated by the grid row generation step is arranged according to the upper and lower relative positions of objects included in the grid row for each grid row constituting the grid row group based on the relative position information. A grid placement step for generating and outputting a matrix;
The screen recognition method characterized by including.   A screen recognition device that recognizes the relative relationship between objects displayed on the screen,
  From the screen information of the screen, an object information acquisition step of acquiring absolute coordinate information of each object displayed on the screen;
  A relative position information acquisition step for recording a relative position between any two objects on the screen as relative position information based on the absolute coordinate information of each object;
  A grid row generation step for generating a grid row group including one or more grid rows in which an object is arranged from left to right starting from the leftmost object on the screen based on the relative position information;
  A grid in which the grid row group generated by the grid row generation step is arranged according to the relative vertical position of objects included in the grid row for each grid row constituting the grid row group based on the relative position information. A grid placement step for generating and outputting a matrix;
  The screen recognition method characterized by including. A screen recognition program for recognizing a relative relationship between objects displayed on a screen,
From the screen information of the screen, an object information acquisition step of acquiring absolute coordinate information of each object displayed on the screen;
A relative position information acquisition step for recording a relative position between any two objects on the screen as relative position information based on the absolute coordinate information of each object;
A grid row generation step for generating a grid row group including one or more grid rows in which objects on the screen are arranged according to the left and right relative positions based on the relative position information;
A grid in which the grid row group generated by the grid row generation step is arranged according to the upper and lower relative positions of objects included in the grid row for each grid row constituting the grid row group based on the relative position information. A grid placement step for generating and outputting a matrix;
A screen recognition program for causing a computer to execute.   A screen recognition program for recognizing a relative relationship between objects displayed on a screen,
  From the screen information of the screen, an object information acquisition step of acquiring absolute coordinate information of each object displayed on the screen;
  A relative position information acquisition step for recording a relative position between any two objects on the screen as relative position information based on the absolute coordinate information of each object;
  A grid row generation step for generating a grid row group including one or more grid rows in which an object is arranged from left to right starting from the leftmost object on the screen based on the relative position information;
  A grid in which the grid row group generated by the grid row generation step is arranged according to the relative vertical position of objects included in the grid row for each grid row constituting the grid row group based on the relative position information. A grid placement step for generating and outputting a matrix;
  A screen recognition program for causing a computer to execute.

Images