JP2011159018A - Information processing device, information processing program, and information processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify other display areas in which display contents are changed in subordination to one display area among a plurality of the display areas in a screen. <P>SOLUTION: (1) In the present method, one display area is detected which includes a point arbitrarily designated from among a plurality of display areas A-G. (2) In the method, the image data Da and image data Db of a screen P at times before and after point designation are compared with each other. (3) In the method, the other display areas in which display contents are changed at the times before and after point designation are detected from among the plurality of display areas A-G according to difference data SD between the image data Da and the image data Db. (4) In the method, the steps (1)-(3) are repeated a plurality of times to detect a plurality of combinations of the one display area and the other display areas. (5) In the method, the other display areas in which display contents are changed in subordination to the one display area are identified among the plurality of display areas A-G in the screen P according to an occurrence frequency of each of the combinations. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, an information processing program, and an information processing method for identifying a specific display area in a screen having a plurality of display areas.

  Conventionally, for work analysis, a user's working time is measured using a log issued by an application. At this time, if the target application does not issue a log, the log is indirectly acquired by some method.

  Some applications have a plurality of display areas (for example, panes, toolbars, scroll bars) in the screen. When targeting such an application, it is necessary to identify the display area by some method in order to determine which display area the user is operating.

  Conventionally, there is a technique for an operator to grasp the transition state of a plurality of display screens (see, for example, Patent Document 1 below). For example, a conventional display device displays a tree of associations between display screens based on each display screen displayed in the past and the transition status of each display screen. At this time, the conventional display device displays each display screen at a higher density as the display frequency is higher, and as a lower density as the display frequency is lower.

  In addition, there is a technique related to frame display in software for browsing content (for example, see Patent Document 2 below). For example, the conventional display method determines whether a change that occurs for a frame in a page different from the frame that is being displayed is based on an action that satisfies a condition that the user intends to see a different frame explicitly. Here, in the case of a change based on an operation that satisfies the conditions, the screen is displayed as a whole, and the screen shifts to a display on which a frame in which a change has occurred is notified.

  There is also a technique for easily selecting an arbitrary item from among a plurality of items (see, for example, Patent Document 3 below). For example, when the position of the representative character on the touch panel is designated, the conventional mobile phone displays characters belonging to the character group represented by the representative character in the input area. Thereafter, when the drag position is moved to a certain character and the drag is released, the conventional mobile phone uses the character at the position where the drag is released as an input character.

JP 2008-204296 A International Publication No. 2005/106628 Pamphlet JP 2007-316760 A

  However, according to the above-described conventional technology, there is a problem that it is difficult to accurately identify a specific display area in a screen having a plurality of display areas in a situation where information for identifying individual display areas cannot be obtained. . For example, when targeting mail software, it is difficult to identify a list area and a body area from a plurality of display areas on the screen. This makes it difficult to obtain a log for judging the work contents such as whether the user was referring to the mail list or the body of the mail, which in turn reduced the analysis accuracy of the business analysis. There is a problem of inviting.

  Note that it is conceivable that the class name of each pane in the screen is acquired and the pane is identified by collating with a predefined class name. However, there is a problem that an application using a unique class name cannot be used without using a predefined class name as the class name of each pane.

  Further, it is conceivable to identify a pane by comparing the size and arrangement position of each pane on the screen with a predetermined size and arrangement pattern of the pane. However, there is a problem that it is not possible to cope with the case where the layout position of the pane on the screen does not become a predefined layout pattern, or when the layout position of each pane on the screen is variable.

  In order to solve the above-described problems caused by the prior art, the present invention can specify another display area whose display contents change depending on one display area among a plurality of display areas in the screen. An object is to provide an apparatus, an information processing program, and an information processing method.

  In order to solve the above-described problems and achieve the object, the disclosed information processing apparatus, information processing program, and information processing method include a coordinate position of an arbitrary point specified on a screen including a plurality of elements and the arbitrary The time at which the point is specified is acquired, and the image data of the screen at a time before the acquired time among the series of image data of the screen obtained at a predetermined time interval, and the time at the time after the time Comparing with the image data of the screen, based on the coordinate position of each element on the screen accumulated in the database, detecting one element including the coordinate position of the arbitrary point from the plurality of elements, Based on the coordinate position of each element on the screen and the compared result of comparison, other elements whose display contents have changed before and after the time at which the arbitrary point is designated are detected from the plurality of elements. And before As a result of detecting a plurality of combinations of one element and the other element, the other element is identified as an element whose display content changes depending on the one element based on the appearance frequency of each combination. It is a requirement to do.

  According to the information processing apparatus, the information processing program, and the information processing method, it is possible to specify another display area whose display contents change depending on one display area among a plurality of display areas in the screen. There is an effect.

It is explanatory drawing which shows one Example of the area | region identification method concerning this Embodiment. It is a block diagram which shows an example of the hardware constitutions of information processing apparatus. It is a block diagram which shows the functional structure of information processing apparatus. It is explanatory drawing which shows an example of the memory content of event DB. It is explanatory drawing which shows the specific example of a window structure. It is explanatory drawing which shows an example of the memory content of screen DB. It is explanatory drawing which shows an example of the memory content of window structure DB. It is explanatory drawing which shows the outline | summary of the example of an element detection. It is explanatory drawing which shows an example of the memory content of click frequency DB. It is explanatory drawing which shows an example of the memory content of simultaneous screen change frequency DB. It is explanatory drawing which shows an example of the memory content of a combination table. It is explanatory drawing which shows an example of the memory content of a cumulative frequency table. It is explanatory drawing which shows an example of the memory content of pane area | region DB. It is a flowchart which shows an example of the pane area | region identification process procedure of information processing apparatus. It is a flowchart which shows an example of the specific process sequence of a log acquisition process. It is a flowchart which shows an example of the specific process sequence of an image acquisition process. It is a flowchart which shows an example of the specific process sequence of an element detection process. It is a flowchart which shows an example of the specific process sequence of a screen change element detection process. It is a flowchart (the 1) which shows an example of the specific process sequence of a pane area | region specific process. It is a flowchart (the 2) which shows an example of the specific process sequence of a pane area | region specific process. It is explanatory drawing which shows the specific example of a pane area | region list. It is a flowchart which shows an example of the specific process sequence of a pane area | region list creation process.

  Exemplary embodiments of an information processing apparatus, an information processing program, and an information processing method according to the present invention will be explained below in detail with reference to the accompanying drawings.

(One example of this area identification method)
First, an example of the region specifying method according to the present embodiment will be described. FIG. 1 is an explanatory diagram showing an example of the region specifying method according to the present embodiment.

  In this region specifying method, another display region whose display content changes depending on one display region among a plurality of display regions in the screen is specified. In FIG. 1, the screen P has a plurality of display areas A to G. The display areas A to G are display frames divided by dividing the screen P, and are, for example, panes, toolbars, scroll bars, and the like.

  (1) In this area specifying method, based on the coordinate positions of the display areas A to G on the screen P, the one including the coordinate position of the point x arbitrarily specified from the plurality of display areas A to G Detect the display area. Here, as a result of designating the point x1, the display area D is detected from the plurality of display areas A to G.

  (2) In this area specifying method, the image data Da of the screen P at the time before the time when the point x is designated is compared with the image data Db of the screen P at the time after the time when the point x is designated. Here, as the comparison result, difference data SD representing a screen difference between the image data Da and the image data Db is displayed.

  (3) In this area specifying method, before and after the time point x is designated from among the plurality of display areas A to G based on the coordinate positions of the display areas A to G on the screen P and the difference data SD To detect other display areas whose display contents have changed. Here, display areas A, E, F, and G are detected from the plurality of display areas A to G.

  As a result, a combination of one display area including the arbitrarily designated point x and another display area whose display contents have changed before and after the time at which the point x is designated is detected. Here, a combination of the display area D and the display areas A, E, F, and G is detected.

  (4) In this area specifying method, the above combinations (1) to (3) are repeated a plurality of times to detect a plurality of combinations of one display area and another display area.

  (5) In this area specifying method, the display content changes depending on the one display area for other display areas included in the combination having a higher appearance frequency than the other combinations based on the appearance frequency of each combination. Specify as a display area. Here, the display area F is specified as a display area whose display contents change depending on the display area D.

  Thus, according to this area specifying method, it is possible to specify another display area whose display contents change depending on one display area from among the plurality of display areas A to G in the screen P. . The two display areas having such a dependency relationship include, for example, a list area and a text area in the screen of the mail software. The list area is a display area for displaying a list group, and the text area is a display area for displaying a text corresponding to a list selected from the list group displayed in the list area.

  In other words, the text area is a display area in which display content changes depending on the list area. Therefore, by replacing one display area specified by this area specifying method and another display area with a list area and a text area, the list area and the text area can be specified from a plurality of display areas on the screen. it can.

(Hardware configuration of information processing apparatus 200)
Next, a hardware configuration of the information processing apparatus 200 according to the present embodiment will be described. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the information processing apparatus. In FIG. 2, an information processing apparatus 200 includes a CPU (Central Processing Unit) 201, a ROM (Read-Only Memory) 202, a RAM (Random Access Memory) 203, a magnetic disk drive 204, a magnetic disk 205, and an optical disk. A drive 206, an optical disk 207, a display 208, an I / F (Interface) 209, a keyboard 210, a mouse 211, a scanner 212, and a printer 213 are provided. Each component is connected by a bus 220.

  Here, the CPU 201 governs overall control of the information processing apparatus 200. The ROM 202 stores a program such as a boot program. The RAM 203 is used as a work area for the CPU 201. The magnetic disk drive 204 controls reading / writing of data with respect to the magnetic disk 205 according to the control of the CPU 201. The magnetic disk 205 stores data written under the control of the magnetic disk drive 204.

  The optical disk drive 206 controls reading / writing of data with respect to the optical disk 207 according to the control of the CPU 201. The optical disk 207 stores data written under the control of the optical disk drive 206, or causes the computer to read data stored on the optical disk 207.

  The display 208 displays data such as a document, an image, and function information as well as a cursor, an icon, or a tool box. As the display 208, for example, a CRT, a TFT liquid crystal display, a plasma display, or the like can be adopted.

  The I / F 209 is connected to a network 214 such as a LAN (Local Area Network), a WAN (Wide Area Network), and the Internet through a communication line, and is connected to other devices via the network 214. The I / F 209 controls an internal interface with the network 214 and controls data input / output from an external device. For example, a modem or a LAN adapter may be employed as the I / F 209.

  The keyboard 210 includes keys for inputting characters, numbers, various instructions, and the like, and inputs data. Moreover, a touch panel type input pad or a numeric keypad may be used. The mouse 211 performs cursor movement, range selection, window movement, size change, and the like. A trackball or a joystick may be used as long as they have the same function as a pointing device.

  The scanner 212 optically reads an image and takes in the image data into the information processing apparatus 200. The scanner 212 may have an OCR (Optical Character Reader) function. The printer 213 prints image data and document data. As the printer 213, for example, a laser printer or an ink jet printer can be employed.

(Functional configuration of information processing apparatus 200)
Next, a functional configuration of the information processing apparatus 200 will be described. FIG. 3 is a block diagram illustrating a functional configuration of the information processing apparatus. In FIG. 3, the information processing apparatus 200 includes an acquisition unit 301, a first selection unit 302, a comparison unit 303, a first detection unit 304, a second detection unit 305, a specification unit 306, 2 selection unit 307, frequency calculation unit 308, integration unit 309, and output unit 310. Specifically, each functional unit (acquisition unit 301 to output unit 310) causes the CPU 201 to execute a program stored in a storage device such as the ROM 202, the RAM 203, the magnetic disk 205, and the optical disk 207 illustrated in FIG. Or the I / F 209 realizes the function.

  Note that the processing results of the functional units (acquisition unit 301 to output unit 310) are stored in a storage device such as the RAM 203, the magnetic disk 205, and the optical disk 207, for example. Further, various DBs (databases) 400, 600, 700, 900, 1000, 1100, 1200, and 1300, which will be described later, are stored in a storage device such as the RAM 203, the magnetic disk 205, and the optical disk 207, for example.

  First, the acquisition unit 301 has a function of acquiring a log of events that occurred during the execution of the target application. Here, the target application is, for example, mail software, an RSS (RDF Site Summary) reader, an electronic book, or the like. However, an application that does not have a log issuing function is assumed here.

  The log is, for example, an event log (click event) issued by an OS (Operating System) in response to a user operation input (click operation) using the keyboard 210 or the mouse 211 shown in FIG. This log includes, for example, the coordinate position of the point designated (clicked) on the screen P assigned to the target application and the time at which the point was designated.

  Specifically, for example, the acquisition unit 301 may acquire an event log from the OS or an external computer that is executing the target application. The acquired log is stored, for example, in the event DB 400 shown in FIG. Here, the contents stored in the event DB 400 will be described.

  FIG. 4 is an explanatory diagram of an example of the contents stored in the event DB. In FIG. 4, the event DB 400 has fields for log ID, click time, and click coordinates, and stores event logs L1 to Ln as records.

  The log ID is an identifier that identifies an event log. The click time is the time when the click event occurs. The click coordinate is the coordinate position of the point clicked on the screen P. The event logs L1 to Ln are stored in the event DB 400 in order from the oldest click time. The event DB 400 is stored in a storage device such as the RAM 203, the magnetic disk 205, and the optical disk 207, for example.

  Returning to the description of FIG. 3, the acquisition unit 301 has a function of acquiring the image data and the window structure of the screen P at a predetermined time interval T. Here, the image data is binary data of a screen image displayed on the screen P of the target application. The window structure is structural data that represents a plurality of elements constituting the screen P in a hierarchical manner.

  The elements include panes, toolbars, scroll bars, and the like that are actually displayed on the screen P as views, and elements that are not actually displayed on the screen P as views. Specifically, for example, the window structure is represented by a tree structure on a tree that represents a parent-child relationship between elements. However, there is one element that is the parent of each element, and the elements having the same parent do not have overlapping ranges.

  Note that the image data may include the window size of the image. The window size is the size of the screen P assigned to the application to be monitored, and is represented by the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the screen P, for example. Here, a specific example of the window structure will be described.

  FIG. 5 is an explanatory diagram showing a specific example of a window structure. In FIG. 5, a tree structure 500 is shown in which a plurality of elements E1 to E14 constituting the screen P are hierarchically structured to express the window structure of the screen P. The element E2 corresponds to the display area A (see FIG. 1) on the screen P that is actually displayed as a view.

  The element E7 corresponds to the display area B on the screen P. The element E10 corresponds to the display area C on the screen P. The element E11 corresponds to the display area D on the screen P. The element E12 corresponds to the display area E on the screen P. The element E14 corresponds to the display area F on the screen P. The element E4 corresponds to the display area G on the screen P.

  In the tree structure 500, the element E1 is an element of the highest hierarchy, and the element E14 is an element of the lowest hierarchy (deepest). Moreover, the coordinate position attached | subjected to each element E1-E14 represents the range on the screen P of each element E1-E14. For example, the range of the element E1 is a rectangular area having a point (0, 0) and a point (1000, 700) as diagonal vertices.

  Returning to the description of FIG. 3, specifically, for example, the acquisition unit 301 may acquire the image data of the screen P and the window structure from the OS at a predetermined time interval T, and the target application is being executed. You may acquire from an external computer. The time interval T can be arbitrarily set (for example, an interval of 100 milliseconds).

  The acquired image data is stored in, for example, the screen DB 600 shown in FIG. The acquired window structure is stored in, for example, the window structure DB 700 shown in FIG. Here, the contents stored in the screen DB 600 and the window structure DB 700 will be described.

  FIG. 6 is an explanatory diagram showing an example of the contents stored in the screen DB. In FIG. 6, the screen DB 600 has fields of image ID, image acquisition time, and data, and stores screen data D1 to Dm as records.

  The image ID is an identifier for identifying image data. The image acquisition time is the acquisition time of image data. The data is binary data of a screen image. The screen data D1 to Dm are stored in the screen DB 600 in order from the oldest image acquisition time.

  FIG. 7 is an explanatory diagram showing an example of the contents stored in the window structure DB. In FIG. 7, the window structure DB 700 has fields of structure acquisition time, window handle, element range, parent element, and child element. By setting information in each field, structure data (for example, structure data 700-1, 700-2) is stored as a record.

  The structure acquisition time is the acquisition time of the window structure of the screen P. The window handle is an identifier for identifying an element constituting the screen P. The element range is a range of each element on the screen P. Note that the range of each element is changed following the change in the position and size of the element on the screen P by the user.

  The parent element is a window handle of an element that is a parent of each element. A child element is a window handle of an element that is a child of each element. The child element field has a variable length, and the subfield changes according to the number of child elements (1, 2,...). The structure data is stored in the window structure DB 700 in order from the oldest structure acquisition time.

  The acquisition process for acquiring the log, the image data, and the window structure described above is continuously executed, for example, from the start of the process of the target application to the end thereof. As a result, various types of information are accumulated as new records in the various DBs 400, 600, and 700, respectively.

  Returning to the description of FIG. 3, the first selection unit 302 has a function of selecting an arbitrary log from the acquired log group. Specifically, for example, the first selection unit 302 may select any event log Li from the event DB 400 shown in FIG. 4 (i = 1, 2,..., N). .

  The comparison unit 303 includes the image data of the screen P at a time before the time included in the selected log and the time after the time included in the log among the series of image data of the screen P obtained at the time interval T. It has a function of comparing the image data of the screen P. The time included in the log is the time when a point on the screen P is designated (clicked).

  Specifically, for example, first, the comparison unit 303 extracts the image data Da of the screen P at the time immediately before the click time of the selected event log Li from the screen DB 600 shown in FIG. Further, the comparison unit 303 extracts the image data Db of the screen P at the time immediately after the click time from the screen DB 600.

  Then, the comparison unit 303 extracts difference data SD representing the screen difference between the image data Da and the image data Db from the image data Db. The difference data SD is, for example, a set of pixels representing a screen difference between the image data Da and the image data Db. Here, the difference data SD is a set of pixels p1 to pK.

  Note that the image data at the time immediately before the click time may be the latest image data whose image acquisition time is before the click time, and any image within a certain time (for example, 300 milliseconds) before the click time. Data may be used. Similarly, the image data at the time immediately after the click time may be the oldest image data whose image acquisition time is after the click time, or may be image data within a certain time after the click time.

  The first detection unit 304 has a function of detecting one element including the coordinate position of the point included in the log from a plurality of elements based on the coordinate position of each element on the screen P. Specifically, for example, the first detection unit 304 refers to the window structure DB 700 illustrated in FIG. 7 and selects an element including the click coordinates of the event log Li from the plurality of elements E1 to E14 as one element. Detect as.

  However, when there are a plurality of elements including the click coordinates of the event log Li, the element at the lowest hierarchy (the deepest) of the window structure is detected as one element. Here, an example of detecting an element including the click coordinate of the event log Li will be described.

  FIG. 8 is an explanatory diagram showing an outline of an example of element detection. Here, a case where an element including a click coordinate (300, 200) is detected from the tree structure 500 will be described as an example. In FIG. 8, each element E <b> 1 to E <b> 14 constituting the tree structure 500 describes the depth h of the hierarchy from the highest hierarchy.

  Specifically, the depth h of the element E1 is “h = 0”, the depth h of the elements E2 to E4 is “h = 1”, the depth h of the elements E5 and E6 is “h = 2”, and the element E7. The depth h of .about.E9 is "h = 3", the depth h of the elements E10 to E13 is "h = 4", and the depth h of the element E14 is "h = 5". Hereinafter, an example of the detection procedure (i) to (vi) of the element including the click coordinate will be described.

  (I) The detection unit detects an element having a depth “h = 0” including the click coordinates (300, 200). Note that including the click coordinates means that the click coordinates are included in the range of the element. Here, the element E1 is detected. (Ii) The detection unit detects an element having a depth “h = 1” including the click coordinate and having the element E1 as a parent. Here, the element E3 is detected.

  (Iii) The detecting unit detects an element having a depth “h = 2” including the click coordinate and having the element E3 as a parent. Here, the element E6 is detected. (Iv) The detection unit detects an element of depth “h = 3” including the click coordinate and having the element E6 as a parent. Here, the element E8 is detected.

  (V) The detection unit detects an element including a click coordinate and having a depth “h = 4” having the element E8 as a parent. Here, the element E11 is detected. (Vi) The detection unit detects an element of depth “h = 5” including the click coordinate and having the element E11 as a parent. Here, it is not detected. As a result, the detection unit detects the element E11 as one element.

  As described above, the detection unit recursively detects the element including the click coordinate, thereby detecting the deepest element E11 including the click coordinate as one element from the plurality of elements E1 to E14. Can do. The detected detection result is stored in the click count DB 900 shown in FIG.

  FIG. 9 is an explanatory diagram showing an example of the contents stored in the click count DB. In FIG. 9, the click count DB 900 has a window handle and a click count field, and stores count information 900-1 to 900-14 for each element E1 to E14 as a record.

  A window handle is an identifier that identifies an element. The number of clicks is the number of times each element E1 to E14 has been detected as one element, that is, the number of clicks. Here, when each element E1 to E14 is detected as one element, the number of clicks of that element is incremented each time. For example, when the element E2 is detected as one element, the number of clicks of the element E2 is incremented to “26”.

  Returning to the description of FIG. 3, the second detection unit 305 determines the time before and after the time included in the log from among a plurality of elements based on the coordinate position of each element on the screen P and the comparison result of the comparison unit 303. It has a function of detecting other elements whose display contents have changed.

  Specifically, for example, first, the second detection unit 305 selects an arbitrary pixel pk from the difference data SD representing the screen difference between the image data Da and the image data Db (k = 1, 2, ..., K). Thereafter, the second detection unit 305 refers to the window structure DB 700 and detects the deepest element including the pixel pk as another element from the plurality of elements E1 to E14.

  Note that the second detection unit 305 may repeat the above processing until there is no unselected pixel pk that is not selected from the difference data SD, for example. Thereby, with respect to one element detected by the first detection unit 304, an element whose display content has changed before and after the click time can be detected as another element.

  The method for detecting the deepest element including the pixel pk is the same as the method described with reference to FIG. The detection result detected by the second detection unit 305 is stored in, for example, the simultaneous screen change frequency DB 1000 shown in FIG.

  FIG. 10 is an explanatory diagram showing an example of the contents stored in the simultaneous screen change frequency DB. In FIG. 10, the simultaneous screen change frequency DB 1000 has fields of click elements and screen change elements, and stores frequency information 1000-1 to 1000-14 for each of the elements E1 to E14 as records.

  Here, the click element is one element detected as an element including the click coordinates. The screen change element is another element detected as an element whose display content has changed before and after the click time. Specifically, the simultaneous screen change frequency DB 1000 stores the number of detections of the elements E1 to E14 detected as other elements when each element E1 to E14 is detected as one element.

  For example, when the element E2 is detected as one element and the elements E2 and E4 are detected as other elements, the values of the columns of the elements E2 and E4 of the frequency information 1000-2 are incremented to “26” and “ 21 ".

  Returning to the description of FIG. 3, as a result of detecting a plurality of combinations of one element and another element, the identifying unit 306 subordinates the other element to the one element based on the appearance frequency of each combination. It has a function of specifying as an element whose display content changes. Here, the appearance frequency may be, for example, the number of appearances of each combination or the appearance rate of each combination with respect to all combinations.

  An element whose display content changes depending on one display area is an element whose display content changes in response to a click operation on one element, for example. Specifically, for example, the list area in the screen of the mail software is an element whose display content changes in response to a click operation on the folder area. Also, the text area in the mail software screen is an element whose display contents change in response to a click operation on the list area.

  The folder area is a display area for displaying a folder group indicating a storage destination of data (for example, mail) to be stored. The list area is a display area for displaying a list group corresponding to a folder selected from the folder group displayed in the folder area. The text area is a display area that displays text corresponding to a list selected from the list group displayed in the list area.

  Specifically, for example, the specifying unit 306 specifies other elements of the combination having the highest appearance frequency among the detected combinations as elements whose display contents change depending on one element. It may be. That is, the appearance frequency of each combination is absolutely evaluated, and other elements of the combination having the highest appearance frequency are specified as elements whose display contents change depending on one element.

  In addition, the specifying unit 306 specifies another element of the combination having an appearance frequency equal to or higher than a predetermined threshold value α as an element whose display content changes depending on one element among the plurality of detected combinations. May be. That is, the appearance frequency of each combination is relatively evaluated, and another element of the combination having an appearance frequency of a certain level or more is specified as an element whose display contents change depending on one element.

  Note that the threshold value α is arbitrarily set in advance and stored in a storage device such as the ROM 202, the RAM 203, the magnetic disk 205, and the optical disk 207, for example.

  Further, the specifying unit 306 may exclude combinations including one element in which the number of detections detected by the first detection unit 304 is equal to or less than the specified number N among the plurality of detected combinations. . In this case, the specifying unit 306 specifies other elements as elements whose display contents change depending on one element based on the appearance frequency of the remaining combinations.

  That is, if the number of times of detection of an element detected as one element is small, the number of combinations including that element is also small and the appearance frequency is low. Therefore, by excluding such combinations in advance, the subsequent processing can be simplified.

  The prescribed number N is, for example, the total number of detections of an element detected as one element by the first detection unit 304, or the number of detections of the element detected the most as one element (maximum detection number). Calculate based on Specifically, for example, the specifying unit 306 may calculate the total number of detections or a [%] of the maximum number of detections as the specified number N.

  The total number of detections and the maximum number of detections can be specified with reference to the click count DB 900, for example. That is, the total number of detections can be obtained by adding the number of clicks of each element E1 to E14. The maximum number of detections can be obtained by specifying the maximum number of clicks among the number of clicks of each element E1 to E14. However, the prescribed number N may be arbitrarily set in advance and stored in a storage device such as the ROM 202, the RAM 203, the magnetic disk 205, and the optical disk 207, for example.

  The specifying unit 306 may exclude combinations including one element whose size on the screen P is smaller than the specified size from among the detected combinations. In general, pane areas such as a folder area, a list area, and a body area have a certain size on the screen P. Therefore, combinations including one element that is obviously small in size as the pane area are excluded in advance.

  The specified size can be specified using, for example, the area, height, width, etc. of the elements on the screen P. The area, height, and width can be expressed by the number of pixels, for example. Specifically, for example, when the specified size is “height of 30 pixels or more”, the specifying unit 306 selects one element having a height on the screen P of less than 30 pixels among a plurality of combinations. Exclude combinations that contain

  The output unit 310 has a function of outputting the specified specific result. Specifically, for example, the output unit 310 uses each element as a body area on the screen P and another element identified as elements whose display area changes depending on one element as a list area. The window handle may be output. As will be described in detail later, the output unit 310 may output the pane area list 2100 shown in FIG.

  Examples of the output format include display on the display 208, print output to the printer 213, and transmission to an external device via the I / F 209. Alternatively, the data may be stored in a storage area such as the RAM 203, the magnetic disk 205, and the optical disk 207.

(Specific processing contents of the specifying unit 306)
Here, an example of specific processing contents of the specifying unit 306 will be described. Here, it is assumed that the folder area, the list area, and the body area in the screen P have a top-down simultaneous change relationship, and an optimal combination that matches this relationship is obtained. Identify the list area and body area.

  First, the second selection unit 307 selects the first, second, and third elements from one element group detected by the first detection unit 304. Here, the first, second, and third elements are elements that are candidates for the folder area, the list area, and the text area. Specifically, for example, the second selection unit 307 refers to the click count DB 900, and from among the remaining one element group excluding the elements with the click count “0” among the elements E1 to E14, Select the first, second and third elements.

The selected selection result is stored in, for example, the combination table 1100 shown in FIG. The selection process for selecting the first, second, and third elements is repeated until there is no unselected combination from one element group, for example. For example, when one element group is seven element groups, the total number of combinations of the first, second, and third elements is 35 (= ₇ C ₃ ).

  The frequency calculation unit 308 calculates the appearance frequency f1 of the combination C1 of the first element detected as one element and the second element detected as another element. Specifically, for example, the number of times the second element is detected when the frequency calculation unit 308 uses the first element as the click element and the second element as the screen change element from the screen simultaneous change frequency DB 1000. Is extracted as the appearance frequency f1. The calculated appearance frequency f1 of the combination C1 is stored in the combination table 1100 shown in FIG.

  Further, the frequency calculation unit 308 calculates the appearance frequency f2 of the combination C2 of the first element detected as one element and the third element detected as another element. Specifically, for example, the number of detections of the third element when the frequency calculation unit 308 uses the first element as the click element and the third element as the screen change element from the screen simultaneous change frequency DB 1000. Is extracted as the appearance frequency f2. The calculated appearance frequency f2 of the combination C2 is stored in the combination table 1100 shown in FIG.

  In addition, the frequency calculation unit 308 calculates the appearance frequency f3 of the combination C3 of the second element detected as one element and the third element detected as another element. Specifically, for example, the number of times of detection of the third element when the frequency calculation unit 308 selects the second element as the click element and the third element as the screen change element from the screen simultaneous change frequency DB 1000. Is extracted as the appearance frequency f3. The calculated appearance frequency f3 of the combination C3 is stored in the combination table 1100 shown in FIG.

  Here, the contents stored in the combination table 1100 will be described. Here, a combination table is described by taking as an example the case where the second selection unit 307 selects the elements E2, E4, and E7 as the first, second, and third elements from one element group. The stored contents 1100 will be described.

  FIG. 11 is an explanatory diagram showing an example of the contents stored in the combination table. In FIG. 11, the combination table 1100 has fields of a click element and a screen change element, and stores appearance frequencies f1 to f3 for each combination C1 to C3 as records by setting information in each field.

  In FIG. 11, (1) the element E2, E4, and E7 are selected as the first, second, and third elements by the second selection unit 307. As a result, the element E2 is displayed in the field of the click element and the screen change element. , E4, E7 window handles are set.

  In FIG. 11, (2) the frequency calculation unit 308 sets the number of detections “20” of the element E4 from the screen simultaneous change frequency DB 1000 as the appearance frequency f1 of the combination C1 (element E2, element E4).

  In FIG. 11, (3) the frequency calculation unit 308 sets the detection frequency “15” of the element E7 from the screen simultaneous change frequency DB 1000 as the appearance frequency f2 of the combination C2 (element E2, element E7).

  In FIG. 11, (4) the frequency calculation unit 308 sets the detection frequency “13” of the element E7 from the screen simultaneous change frequency DB 1000 as the appearance frequency f3 of the combination C3 (element E4, element E7).

  The combination table 1100 is created for the combination of the first, second, and third elements selected by the second selection unit 307. Therefore, for example, when the total number of combinations of the first, second, and third elements is 35, 35 combinations of tables 1100 are created.

  Returning to the description of FIG. 3, the integrating unit 309 calculates a cumulative frequency F obtained by accumulating the appearance frequencies f1 to C3 of the combinations C1 to C3 based on the calculated appearance frequencies f1 to C3 of the combinations C1 to C3. Specifically, for example, the integration unit 309 refers to the combination table 1100 illustrated in FIG. 11 and calculates the cumulative frequency F by adding the appearance frequencies f1 to f3 of the combinations C1 to C3.

  That is, in the example of the combination table 1100 shown in FIG. 11, the cumulative total of the appearance frequency f1 “20” of the combination C1, the appearance frequency f2 “15” of the combination C2, and the appearance frequency f3 “13” of the combination C3. The frequency F “48” is calculated. The calculated cumulative frequency F is stored in, for example, the cumulative frequency table 1200 shown in FIG.

  FIG. 12 is an explanatory diagram of an example of the contents stored in the cumulative frequency table. In FIG. 12, the cumulative frequency table 1200 has a cumulative frequency F for each combination of the first, second, and third elements. For example, the cumulative frequency F of the combination of the elements E2, E4, E7 is “48”.

  The identification unit 306 identifies the second element as an element whose display content changes depending on the first element, based on the cumulative frequency F calculated by the accumulation unit 309. Furthermore, the specifying unit 306 specifies the third element as an element whose display content changes depending on the second element, based on the cumulative frequency F.

  Specifically, for example, the specifying unit 306 changes the display content depending on the first element for the second element of the combination of the first, second, and third elements having the maximum cumulative frequency F. Identifies as an element. Furthermore, the specifying unit 306 specifies the third element of the combination of the first, second, and third elements that has the maximum cumulative frequency F as an element whose display content changes depending on the second element.

  Here, the first, second, and third elements described above can be replaced with a folder area, a list area, and a body area in the screen P. Specifically, the specifying unit 306 specifies the first element as the folder area, the second element as the list area, and the third element as the body area from the combination having the maximum cumulative frequency F.

  That is, assuming the top-down relationship in which the display contents of the list area and the text area change depending on the folder area and the display contents of the text area change depending on the list area, the first and second relationships are assumed. The third element is replaced with a folder area, a list area, and a body area. Thus, even when a plurality of other elements are detected when a single element is clicked (for example, there are a plurality of candidates for the body area), the folder area in the screen P, A list area and a text area can be specified.

  The specifying unit 306 changes the display content of the second element of the combination of the first, second, and third elements whose cumulative frequency F is equal to or greater than the predetermined threshold value β according to the operation input to the first element. You may decide to specify as an element to do. Similarly, the specifying unit 306 displays the third element of the combination of the first, second, and third elements whose cumulative frequency F is equal to or greater than the predetermined threshold value β according to the operation input to the second element. You may decide to specify as a changing element. The threshold β is arbitrarily set in advance and stored in a storage device such as the ROM 202, RAM 203, magnetic disk 205, and optical disk 207, for example.

  The identified result is stored in, for example, the pane area DB 1300 shown in FIG. FIG. 13 is an explanatory diagram of an example of the contents stored in the pane area DB. In FIG. 13, the pane area DB 1300 stores the window handles of the list area and the text area in the screen P.

  Specifically, the window handle of the list area is the window handle of the second element specified as the list area by the specifying unit 306. Further, the window handle of the text area is the window handle of the third element specified as the text area by the specifying unit 306. Here, the window handle of the element E7 as the list area and the element E14 as the body area are stored.

  Since the list area and the text area in the screen P can be accurately specified, it is possible to accurately capture the screen change only in the list area and the screen change only in the text area. As a result, it is possible to acquire a log for determining work contents such as whether the user is referring to the mail list or the mail text by any method.

  For example, first, the information processing apparatus 200 compares screen data at an arbitrary time t (image acquisition time) in the screen DB 600 with image data at a time before the time t. Then, the information processing apparatus 200 determines whether the display contents of the list area and the text area are both changed on the screen at time t based on the compared result of comparison.

  More specifically, for example, the information processing apparatus 200 detects the highlighted display of the line in the list area, and further determines whether the display content of the text area has changed. Here, when the display contents of the list area and the text area both change, the information processing apparatus 200 outputs a log related to the selection operation for switching from one list to another list in association with the time t. To do.

  As a result, it is possible to determine that the list selection operation has been performed efficiently and with high accuracy without requiring a log issued from the application. Further, it is possible to calculate the mail viewing time from the time when one list in the list area is switched to another list, which can be used for business analysis.

(Pain area identification processing procedure of the information processing apparatus 200)
Next, a pane area specifying process procedure of the information processing apparatus 200 will be described. Here, a case where the target application is mail software will be described as an example. Here, one element is referred to as a “click element” and the other element is referred to as a “screen change element”.

  FIG. 14 is a flowchart illustrating an example of the pane area specifying process procedure of the information processing apparatus. In the flowchart of FIG. 14, first, the acquisition unit 301 determines whether or not a mail software process has been started (step S1401).

  Here, after waiting for the process to start (step S1401: No) and when started (step 1401: Yes), the storage contents of the various DBs 400, 600, 700, 900, 1000, 1300 are initialized (step S1402). ).

  Next, the acquisition unit 301 starts execution of a log acquisition process for acquiring an event log (step S1403), and starts execution of an image acquisition process for acquiring image data of the screen P (step S1404). Thereafter, the first selection unit 302 determines whether or not the execution of the log acquisition process and the image acquisition process is completed (step S1405).

  Here, after waiting for the execution to end (step S1405: No), if the execution ends (step S1405: Yes), the first and second detection units 304 and 305 cause the plurality of elements E1 to E14 to be selected. Element detection processing for detecting the click element and the screen change element is executed (step S1406). Then, the specifying unit 306 executes a pane area specifying process for specifying a list area and a body area from the plurality of elements E1 to E14 (step S1407).

  Next, a list creation process for creating a pane area list is executed by the specifying unit 306 (step S1408). Finally, the created pane area list is output by the output unit 310 (step S1409). A series of processes according to the flowchart ends. Thereby, it is possible to specify the list area and the body area from the plurality of elements E1 to E14 in the screen P.

<Log acquisition processing procedure>
Next, a specific processing procedure of the log acquisition processing in step S1403 shown in FIG. 14 will be described. FIG. 15 is a flowchart illustrating an example of a specific processing procedure of the log acquisition processing. In the flowchart of FIG. 15, first, the acquisition unit 301 acquires an event log of an event that occurred during execution of the mail software (step S1501), and determines whether the acquired event log is a click event. (Step S1502).

  Here, when it is not a click event (step S1502: No), it transfers to step S1504. On the other hand, in the case of a click event (step S1502: Yes), the acquisition unit 301 registers the click time and click coordinates included in the acquired event log in the event DB 400 (step S1503).

  Thereafter, the acquisition unit 301 determines whether or not the mail software process has ended (step S1504). If the process has not ended (step S1504: No), the process returns to step S1501. On the other hand, when the process ends (step S1504: Yes), a series of processes according to this flowchart ends. Thereby, the event log regarding the click operation performed with respect to any of the elements E1 to E14 in the screen P can be acquired.

<Image acquisition processing procedure>
Next, a specific processing procedure of the image acquisition processing in step S1404 shown in FIG. 14 will be described. FIG. 16 is a flowchart illustrating an example of a specific processing procedure of the image acquisition process.

  In the flowchart of FIG. 16, first, the acquisition unit 301 acquires the image data of the screen P assigned to the mail software (step S1601). Then, the acquisition unit 301 registers the acquired image data of the screen P in the screen DB 600 in association with the image acquisition time (step S1602).

  Further, the acquisition unit 301 acquires the window structure of the screen P (step S1603). Then, the acquisition unit 301 registers the acquired window structure of the screen P in the window structure DB 700 in association with the structure acquisition time (step S1604).

  Thereafter, the acquisition unit 301 waits for a predetermined time T (step S1605), and determines whether or not the mail software process has ended (step S1606). If the process has not ended (step S1606: NO), the process returns to step S1601.

  On the other hand, when the process is finished (step S1606: Yes), a series of processes according to this flowchart is finished. Thereby, the image data of the screen P can be acquired at a constant time interval T.

<Element detection processing procedure>
Next, a specific processing procedure of the element detection processing in step S1406 shown in FIG. 14 will be described. FIG. 17 is a flowchart illustrating an example of a specific processing procedure of element detection processing. In the flowchart of FIG. 17, first, the first selection unit 302 sets “i = 1” (step S1701), and selects an event log Li from the event DB 400 (step S1702).

  Thereafter, the comparison unit 303 identifies the click time from the selected event log Li (step S1703). Then, the comparison unit 303 determines whether there is image data before and after the click time in the screen DB 600 (step S1704). If there is no image data before and after the click time (step S1704: NO), the process proceeds to step S1714.

  On the other hand, when there is image data before and after the click time (step S1704: Yes), the comparison unit 303 extracts the image data Da of the screen P at the time immediately before the click time of the event log Li from the screen DB 600 (step S1705). . Further, the comparison unit 303 extracts the image data Db of the screen P at the time immediately after the click time of the event log Li from the screen DB 600 (step S1706).

  Thereafter, the comparison unit 303 extracts difference data SD representing the screen difference between the extracted image data Da of the screen P and the image data Db of the screen P (step S1707). Next, the first detection unit 304 specifies click coordinates from the selected event log Li (step S1708).

  Then, the first detection unit 304 refers to the window structure DB 700 to detect the deepest element including the click coordinates of the event log Li as the click element Ek from the plurality of elements E1 to E14 (step S1709). Thereafter, the second detection unit 305 executes a screen change element detection process for detecting a screen change element set whose display contents have changed before and after the click time from among the plurality of elements E1 to E14 (step S1710). .

  Next, the first detection unit 304 determines whether or not the detected click element Ek is included in the screen change element set (step S1711). Here, when the click element Ek is not included (step S1711: No), the process proceeds to step S1714.

  On the other hand, when the click element Ek is included (step S1711: Yes), the first detection unit 304 increments the click count of the click element Ek in the click count DB 900 (step S1712). The click element Ek is an element detected as a click element from the elements E1 to E14 in step S1709.

  Then, the second detection unit 305 increments the number of detections of the screen change elements Ek (1) to Ek (m) included in the screen change element set in the screen simultaneous change frequency DB 1000 (step S1713). Screen change elements Ek (1) to Ek (m) are elements detected as screen change elements from elements E1 to E14 in step S1710.

  Thereafter, the first selection unit 302 increments i (step S1714), and determines whether or not “i> n” (step S1715). Here, if “i ≦ n” (step S1715: No), the process returns to step S1702. On the other hand, if “i> n” (step S1715: YES), the process proceeds to step S1407 shown in FIG.

  Thereby, the number of clicks of each click element Ek can be counted, and the number of appearances of combinations of the click element Ek and the screen change elements Ek (1) to Ek (m) can be counted for each click element Ek. Can do. Further, in step S1711, by determining whether or not the click element Ek is included in the screen change element set, it is possible to count only the number of clicks of the click operation accompanied by the display screen change of other elements.

<Screen change element detection processing procedure>
Next, a specific processing procedure of the screen change element detection processing in step S1710 shown in FIG. 17 will be described. FIG. 18 is a flowchart illustrating an example of a specific processing procedure of the screen change element detection process.

  In the flowchart of FIG. 18, first, the second detection unit 305 sets an empty set screen change element set (step S1801). Then, the second detection unit 305 sets “k = 1” (step S1802), and selects an arbitrary pixel pk from the difference data SD extracted in step S1707 illustrated in FIG. 17 (step S1803).

  Thereafter, the second detection unit 305 refers to the window structure DB 700 and detects the deepest element including the pixel pk as a screen change element from the plurality of elements E1 to E14 (step S1804). Then, the second detection unit 305 determines whether or not the detected screen change element is included in the screen change element set (step S1805).

  If a screen change element is included (step S1805: YES), the process proceeds to step S1807. On the other hand, when the screen change element is not included (step S1805: No), the second detection unit 305 adds the screen change element to the screen change element set (step S1806).

  Then, the second detection unit 305 increments k (step S1807), and determines whether or not “k> K” (step S1808). If “k ≦ K” is satisfied (step S1808: NO), the process returns to step S1803. On the other hand, if “k> K” (step S1808: YES), the process proceeds to step S1711 shown in FIG.

  Thereby, the screen change elements Ek (1) to Ek (m) whose display contents are changed before and after the click time when the click element Ek is clicked can be detected.

<Pain area identification processing procedure>
Next, a specific processing procedure of the pane area specifying process in step S1407 shown in FIG. 14 will be described. 19 and 20 are flowcharts showing an example of a specific processing procedure of the pane area specifying process.

In the flowchart of FIG. 19, first, the specifying unit 306 refers to the click count DB 900 to specify the maximum click count N _max (step S1901). Then, the specifying unit 306 calculates the specified number of times N from the specified number of clicks _Nmax (step S1902).

  Thereafter, the specifying unit 306 specifies a click element set whose number of clicks is equal to or greater than the specified number N from the click number DB 900 (step S1903). Then, the specifying unit 306 excludes click elements smaller than the specified size from the click element set (step S1904).

  Next, the second selection unit 307 selects the first, second, and third elements from the click element set (step S1905). Then, the frequency calculation unit 308 extracts the appearance frequency f1 of the combination C1 from the screen simultaneous change frequency DB 1000 (step S1906). The extraction result is stored in, for example, the combination table 1100. Thereafter, the frequency calculation unit 308 determines whether or not the appearance frequency f1 of the extracted combination C1 is “0” (step S1907).

  If “f1 = 0” (step S1907: YES), the process proceeds to step S1914 shown in FIG. On the other hand, when “f1 ≠ 0” (step S1907: No), the frequency calculation unit 308 extracts the appearance frequency f2 of the combination C2 from the screen simultaneous change frequency DB 1000 (step S1908).

  Thereafter, the frequency calculation unit 308 determines whether or not the appearance frequency f2 of the extracted combination C2 is “0” (step S1909). If “f2 = 0” (step S1909: YES), the process proceeds to step S1914 shown in FIG. On the other hand, when “f2 ≠ 0” (step S1909: No), the frequency calculation unit 308 extracts the appearance frequency f3 of the combination C3 from the screen simultaneous change frequency DB 1000 (step S1910).

  Thereafter, the frequency calculation unit 308 determines whether or not the appearance frequency f3 of the extracted combination C3 is “0” (step S1911). Here, when “f3 = 0” (step S1911: YES), the process proceeds to step S1914 shown in FIG. On the other hand, when “f3 ≠ 0” (step S1911: No), the process proceeds to step S1912 shown in FIG.

  Next, in the flowchart of FIG. 20, the accumulating unit 309 adds the appearance frequencies f1 to f3 of the combinations C1 to C3 to calculate the cumulative frequency F (step S1912). Then, the accumulated unit 309 registers the calculated accumulated frequency F in the accumulated frequency table 1200 together with the window handles of the first, second, and third elements (step S1913).

  Thereafter, the second selection unit 307 determines whether or not there is an unselected combination of the first, second, and third elements not selected from the click element set (step S1914). If there is an unselected combination (step S1914: YES), the process proceeds to step S1905 shown in FIG.

  On the other hand, if there is no unselected combination (step S1914: No), the identifying unit 306 refers to the accumulated frequency table 1200 and identifies the combination of the first, second, and third elements having the maximum accumulated frequency F. (Step S1915). Next, the specifying unit 306 specifies the second element of the specified combination as a list area, and specifies the third element as a body area (step S1916).

  The specifying unit 306 registers the window handles of the elements specified as the list area and the text area in the pane area DB 1300 (step S1917), and the process proceeds to step S1408 shown in FIG. Thereby, the pane area representing the list area and the body area can be specified from the plurality of elements E1 to E14 in the screen P.

<Pain area list creation process>
Next, a specific processing procedure of the pane area list creation processing in step S1408 shown in FIG. 14 will be described. Here, a specific example of the pane area list 2100 to be created will be described first.

  FIG. 21 is an explanatory diagram of a specific example of the pane area list. In FIG. 21, a pane area list 2100 has fields of range change time, list area range, and body area range. By setting information in each field, pane area information (for example, pane area information 2100-1, 2100-2) is stored as a record.

  Here, the range change time is the time when the ranges of the list area and the text area change. The list area range is a list area range on the screen P. The text area range is a text area range on the screen P. The pane area information is registered in the pane area list 2100 in order from the oldest range change time.

  FIG. 22 is a flowchart illustrating an example of a specific processing procedure of the pane area list creation processing. In the flowchart of FIG. 22, first, the specifying unit 306 extracts arbitrary structure data from the window structure DB 700 (step S2201).

  Next, the specifying unit 306 refers to the pane area DB 1300 to specify the window handle of the list area (step S2202). The specifying unit 306 specifies the range of the list area by specifying the range of the specified element of the window handle with reference to the extracted structure data (step S2203).

  Further, the identifying unit 306 refers to the pane area DB 1300 to identify the window handle of the body area (step S2204). The identifying unit 306 identifies the range of the body area by referring to the extracted structure data and identifying the range of the identified window handle element (step S2205).

  Thereafter, the identifying unit 306 refers to the pane area list 2100 shown in FIG. 21 and identifies the previous list area and body area range (step S2206). If the previous list area and text area range have not been specified (step S2207: NO), the process proceeds to step S2209.

  On the other hand, if the previous list area and text area range are specified (step S2207: Yes), whether the list area and text area ranges specified in steps S2203 and S2205 by the specifying unit 306 match the previous range. It is determined whether or not (step S2208). Here, when it corresponds with the last time (step S2208: Yes), it transfers to step S2210.

  On the other hand, if there is a mismatch with the previous time (step S2208: No), the specifying unit 306 registers the list area and body area specified in steps S2203 and S2205 in the pane area list 2100 in association with the structure acquisition time ( Step S2209). The structure acquisition time is the structure acquisition time of the structure data extracted in step S2201.

  Thereafter, the specifying unit 306 determines whether there is unextracted structure data that has not been extracted from the window structure DB 700 (step S2210). Here, when there is unextracted structure data (step S2210: Yes). The process returns to step S2201. On the other hand, when there is no unextracted structure data (step S2210: No), the process proceeds to 1409 shown in FIG.

  Thereby, the range of the pane area representing the list area and the body area in the screen P can be listed and output in association with the range change time when the positions and sizes of the elements E1 to E14 in the screen P are changed. .

  As described above, according to the present embodiment, a combination of one element including a point designated on the screen P and another element whose display content has changed before and after the time at which the point is designated. Multiple ways can be detected. Then, from the appearance frequency of each combination, other elements included in the combination having a higher appearance frequency than the other combinations can be specified as elements whose display contents change depending on one element.

  As a result, it is possible to specify an element representing “folder area and list area” or “list area and body area” having a dependency relationship between one element and another element from among the elements E1 to E14. In addition, when specifying the pane area such as the list area and the body area, the element class name and the element arrangement position on the screen P are not used as conditions. be able to.

  Further, according to the present embodiment, based on the window structure representing the plurality of elements E1 to E14 in the screen P in a hierarchical structure, the element having the deepest depth from the highest hierarchy is designated as one element or another element. By detecting as an element, one element or another element can be uniquely identified.

  In addition, according to the present embodiment, a combination having a low appearance frequency is excluded in advance by excluding a combination in which the number of detections of one element is equal to or less than the specified number N from a plurality of detected combinations. Thus, the subsequent processing can be simplified.

  In addition, according to the present embodiment, by excluding combinations in which the size of one element is smaller than the specified size from among a plurality of combinations, the combination that includes one element that is clearly smaller in size as a pane area Can be eliminated in advance.

  In addition, according to the present embodiment, it is possible to specify other elements included in the combination having the highest appearance frequency among a plurality of combinations as elements whose display contents change depending on one element. Thereby, the appearance frequency of each combination is absolutely evaluated, and the other element of the combination having the highest appearance frequency can be specified as an element whose display content changes depending on one element.

  In addition, according to the present embodiment, other elements included in a combination whose appearance frequency is equal to or higher than the predetermined threshold value α among a plurality of combinations are specified as elements whose display contents change depending on one element. be able to. Thereby, the appearance frequency of each combination can be relatively evaluated, and other elements of the combination having a certain appearance frequency or more can be specified as elements whose display contents change depending on one element.

  According to the present embodiment, the second element is specified as the element whose display content changes depending on the first element among the first, second, and third elements, and the third element Can be specified as an element whose display content changes depending on the second element. Thus, by considering the dependency between the three elements, it is possible to specify a pane area that causes a top-down screen change such as a folder area, a list area, and a text area.

  The information processing method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. The information processing program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, and a DVD, and is executed by being read from the recording medium by the computer. The information processing program may be distributed through a network such as the Internet.

  The following additional notes are disclosed with respect to the embodiment described above.

(Appendix 1) An acquisition means for acquiring a coordinate position of an arbitrary point designated on a screen composed of a plurality of elements and a time at which the arbitrary point is designated;
Among a series of image data of the screen obtained at a predetermined time interval, the image data of the screen at a time before the time acquired by the acquisition means, and the image data of the screen at a time after the time A comparison means for comparing;
First detection means for detecting one element including the coordinate position of the arbitrary point from the plurality of elements based on the coordinate position of each element on the screen stored in a database;
Based on the coordinate position of each element on the screen and the comparison result compared by the comparison means, the display content has changed before and after the time when the arbitrary point is designated from among the plurality of elements. Second detection means for detecting the elements of
As a result of detecting a plurality of combinations of the one element and the other element by the first and second detection means, the other element is changed to the one element based on the appearance frequency of each combination. A specifying means for specifying as an element whose display content changes dependently;
An information processing apparatus comprising:

(Additional remark 2) The said specific means is the appearance frequency of the remaining combination except the combination containing the one element in which the detection frequency detected by the said 1st detection means becomes below a regulation frequency among the said several combinations. The information processing apparatus according to claim 1, wherein the other element is specified as an element whose display content changes depending on the one element.

(Additional remark 3) The said specific means is said other element based on the appearance frequency of the remaining combination except the combination containing the one element in which the size on the said screen is smaller than regulation size among the said several combinations. The information processing apparatus according to appendix 1 or 2, wherein the information is specified as an element whose display content changes depending on the one element.

(Additional remark 4) The said specific means specifies the said other element of the combination with the largest appearance frequency among the said detected several combinations as an element which a display content changes depending on the said one element. The information processing apparatus according to any one of supplementary notes 1 to 3, wherein:

(Additional remark 5) The said specific means makes the said other element of the combination whose appearance frequency is more than a predetermined threshold among the said detected several combinations as an element from which display content changes depending on the said one element. The information processing apparatus according to any one of appendices 1 to 3, wherein the information processing apparatus is specified.

(Appendix 6) Based on the first and second detection results detected by the first and second detection means, the first element detected as the one element and the other element detected as the other element First calculating means for calculating the appearance frequency of the combination with the second element;
Second calculating the frequency of appearance of a combination of the first element detected as the one element and the third element detected as the other element based on the first and second detection results; Means for calculating
Calculating a frequency of appearance of a combination of the second element detected as the one element and the third element detected as the other element based on the first and second detection results; 3 calculating means, and
The specifying means specifies the second element as an element whose display content changes depending on the first element based on the calculation results calculated by the first, second and third calculating means. Then, the information processing apparatus according to any one of appendices 1 to 5, wherein the third element is specified as an element whose display content changes depending on the second element.

(Appendix 7)
An acquisition means for acquiring a coordinate position of an arbitrary point specified on a screen composed of a plurality of elements and a time at which the arbitrary point is specified;
Among a series of image data of the screen obtained at a predetermined time interval, the image data of the screen at a time before the time acquired by the acquisition means, and the image data of the screen at a time after the time Comparison means to compare,
First detection means for detecting one element including the coordinate position of the arbitrary point from the plurality of elements based on the coordinate position of each element on the screen stored in a database;
Based on the coordinate position of each element on the screen and the comparison result compared by the comparison means, the display content has changed before and after the time when the arbitrary point is designated from among the plurality of elements. Second detection means for detecting the elements of
As a result of detecting a plurality of combinations of the one element and the other element by the first and second detection means, the other element is changed to the one element based on the appearance frequency of each combination. A specifying means for specifying as an element whose display content changes depending on
An information processing program that functions as a computer program.

(Additional remark 8) The said specific means is the appearance frequency of the remaining combination except the combination containing the one element in which the detection frequency detected by the said 1st detection means becomes below a regulation frequency among the said several combinations. 8. The information processing program according to appendix 7, wherein the other element is specified as an element whose display content changes depending on the one element.

(Additional remark 9) The said specific means is said other element based on the appearance frequency of the remaining combination except the combination containing the one element in which the size on the said screen is smaller than regulation size among the said several combinations. The information processing program according to appendix 7 or 8, wherein the information is specified as an element whose display content changes depending on the one element.

(Additional remark 10) The said identification means specifies the said other element of the combination with the largest appearance frequency among the detected said several combinations as an element from which display content changes depending on said one element. The information processing program according to any one of appendices 7 to 9, characterized by:

(Additional remark 11) The said specific means makes the said other element of the combination whose appearance frequency is more than a predetermined threshold value among the said detected several combinations as an element from which display content changes depending on the said one element. The information processing program according to any one of appendices 7 to 9, characterized by being specified.

(Supplementary note 12)
Based on the first and second detection results detected by the first and second detection means, the first element detected as the one element and the second element detected as the other element First calculating means for calculating the appearance frequency of the combination with
Second calculating the frequency of appearance of a combination of the first element detected as the one element and the third element detected as the other element based on the first and second detection results; Means for calculating,
Calculating a frequency of appearance of a combination of the second element detected as the one element and the third element detected as the other element based on the first and second detection results; 3 function as a calculation means,
The specifying means specifies the second element as an element whose display content changes depending on the first element based on the calculation results calculated by the first, second and third calculating means. The information processing program according to any one of appendices 7 to 11, wherein the third element is specified as an element whose display content changes depending on the second element.

(Supplementary note 13)
An acquisition step of acquiring a coordinate position of an arbitrary point specified on a screen composed of a plurality of elements and a time at which the arbitrary point is specified;
Of the series of image data of the screen obtained at a predetermined time interval, the image data of the screen at a time before the time acquired by the acquisition step, and the image data of the screen at a time after the time A comparison process to compare;
A first detection step of detecting one element including the coordinate position of the arbitrary point from the plurality of elements based on the coordinate position of each element on the screen stored in a database;
Based on the coordinate position of each element on the screen and the comparison result compared in the comparison step, the display content has changed before and after the time when the arbitrary point is designated from among the plurality of elements. A second detection step for detecting an element of
Appearance of each combination as a result of detecting a plurality of combinations of the one element and the other element by repeating a series of processes of the acquisition step, the comparison step, and the first and second detection steps. A specific step of identifying the other element as an element whose display content changes depending on the one element based on the frequency;
An information processing method characterized in that

(Additional remark 14) The said specific process is made into the appearance frequency of the remaining combination except the combination containing the one element from which the detection frequency detected by the said 1st detection process becomes below a regulation frequency among the said several combinations. 14. The information processing method according to appendix 13, wherein the other element is specified as an element whose display content changes depending on the one element.

(Additional remark 15) The said specific process is based on the appearance frequency of the remaining combination except the combination containing the one element in which the size on the said screen is smaller than regulation size among the said several combinations, said other element The information processing method according to appendix 13 or 14, wherein the display content is specified as an element whose display content changes depending on the one element.

(Additional remark 16) The said specific process specifies the said other element of the combination with the largest appearance frequency among the detected said several combinations as an element from which display content changes depending on said one element. The information processing method according to any one of supplementary notes 13 to 15, characterized in that:

(Supplementary Note 17) In the specifying step, the other element of the combination whose appearance frequency is a predetermined threshold or more among the detected combinations is selected as an element whose display content changes depending on the one element. The information processing method according to any one of supplementary notes 13 to 15, characterized in that the information processing method is specified.

(Supplementary note 18)
Based on the first and second detection results detected by the first and second detection steps, the first element detected as the one element and the second element detected as the other element A first calculation step of calculating the appearance frequency of the combination with
Second calculating the frequency of appearance of a combination of the first element detected as the one element and the third element detected as the other element based on the first and second detection results; The calculation process of
Calculating a frequency of appearance of a combination of the second element detected as the one element and the third element detected as the other element based on the first and second detection results; 3 is further executed, and the specific step is dependent on the first element based on the calculation results calculated by the first, second and third calculation steps. Any one of appendices 13 to 17, wherein the display element is specified as an element whose display contents change, and the third element is specified as an element whose display contents change depending on the second element. Information processing method described in 1.

200 Information Processing Device 301 Acquisition Unit 302 First Selection Unit 303 Comparison Unit 304 First Detection Unit 305 Second Detection Unit 306 Identification Unit 307 Second Selection Unit 308 Frequency Calculation Unit 309 Integration Unit 310 Output Unit 400 Event DB
600 screen DB
700 Window structure DB
900 clicks DB
1000 screen simultaneous change frequency DB
1100 Combination table 1200 Cumulative frequency table 1300 Pain area DB
2100 Pane area list

Claims (6)

An acquisition means for acquiring a coordinate position of an arbitrary point designated on a screen composed of a plurality of elements and a time at which the arbitrary point is designated;
Among a series of image data of the screen obtained at a predetermined time interval, the image data of the screen at a time before the time acquired by the acquisition means, and the image data of the screen at a time after the time A comparison means for comparing;
First detection means for detecting one element including the coordinate position of the arbitrary point from the plurality of elements based on the coordinate position of each element on the screen stored in a database;
Based on the coordinate position of each element on the screen and the comparison result compared by the comparison means, the display content has changed before and after the time when the arbitrary point is designated from among the plurality of elements. Second detection means for detecting the elements of
As a result of detecting a plurality of combinations of the one element and the other element by the first and second detection means, the other element is changed to the one element based on the appearance frequency of each combination. A specifying means for specifying as an element whose display content changes dependently;
An information processing apparatus comprising: The specifying means is:
Based on the appearance frequency of the remaining combination excluding the combination including one element in which the number of detections detected by the first detection unit is equal to or less than the specified number of the combinations, the other elements are The information processing apparatus according to claim 1, wherein the information processing apparatus is specified as an element whose display content changes depending on the one element. The specifying means is:
Of the plurality of combinations, the other elements are subordinated to the one element based on the appearance frequency of the remaining combinations excluding the combination including one element whose size on the screen is smaller than a predetermined size. The information processing apparatus according to claim 1, wherein the information content is specified as an element whose display content changes. Based on the first and second detection results detected by the first and second detection means, the first element detected as the one element and the second element detected as the other element First calculating means for calculating the appearance frequency of a combination with
Second calculating the frequency of appearance of a combination of the first element detected as the one element and the third element detected as the other element based on the first and second detection results; Means for calculating
Calculating a frequency of appearance of a combination of the second element detected as the one element and the third element detected as the other element based on the first and second detection results; 3 calculating means, and
The specifying means is:
Based on the calculation results calculated by the first, second, and third calculation means, the second element is identified as an element whose display content changes depending on the first element, and the third element The information processing apparatus according to claim 1, wherein the element is specified as an element whose display content changes depending on the second element. Computer
An acquisition means for acquiring a coordinate position of an arbitrary point specified on a screen composed of a plurality of elements and a time at which the arbitrary point is specified;
Among a series of image data of the screen obtained at a predetermined time interval, the image data of the screen at a time before the time acquired by the acquisition means, and the image data of the screen at a time after the time Comparison means to compare,
First detection means for detecting one element including the coordinate position of the arbitrary point from the plurality of elements based on the coordinate position of each element on the screen stored in a database;
Based on the coordinate position of each element on the screen and the comparison result compared by the comparison means, the display content has changed before and after the time when the arbitrary point is designated from among the plurality of elements. Second detection means for detecting the elements of
As a result of detecting a plurality of combinations of the one element and the other element by the first and second detection means, the other element is changed to the one element based on the appearance frequency of each combination. A specifying means for specifying as an element whose display content changes depending on
An information processing program that functions as a computer program. Computer
An acquisition step of acquiring a coordinate position of an arbitrary point specified on a screen composed of a plurality of elements and a time at which the arbitrary point is specified;
Of the series of image data of the screen obtained at a predetermined time interval, the image data of the screen at a time before the time acquired by the acquisition step, and the image data of the screen at a time after the time A comparison process to compare;
A first detection step of detecting one element including the coordinate position of the arbitrary point from the plurality of elements based on the coordinate position of each element on the screen stored in a database;
Based on the coordinate position of each element on the screen and the comparison result compared in the comparison step, the display content has changed before and after the time when the arbitrary point is designated from among the plurality of elements. A second detection step for detecting an element of
Appearance of each combination as a result of detecting a plurality of combinations of the one element and the other element by repeating a series of processes of the acquisition step, the comparison step, and the first and second detection steps. A specific step of identifying the other element as an element whose display content changes depending on the one element based on the frequency;
An information processing method characterized in that

Images