JP6662748B2 - Display control device, display control method, and display control program - Google Patents

Description

  The present invention relates to a display control device, a display control method, and a display control program.

  2. Description of the Related Art An annotation technology that provides a user with visual interpretation by overlay display when an application is used has been known. For example, by applying annotation technology to real-time video, the interpretation result at each moment is displayed as an overlay, and the display is updated in synchronization with the change or movement of the interpretation target, so that the user can interpret the original video and its interpretation. Can be presented at the same time. This enables information support for the user.

  As a specific application example of this field, for example, an annotation that continuously captures a screen screen of a PC, searches an area similar to a designated template image from the screen, and overlays an icon at an obtained position. There is a system (see Patent Document 1).

International Publication No. WO 2014/002812 International Publication No. WO 2016/039273 Japanese Patent No. 5916136

  However, any of the above-mentioned conventional techniques has a problem in that when applying an annotation to a plurality of windows, the processing load of the annotation increases in proportion to the number of windows, and the processing time of the annotation also increases accordingly. is there. Therefore, an object of the present invention is to solve the above-described problem and suppress an increase in the processing load of the annotation when displaying the annotation on a plurality of windows.

  In order to solve the above-described problem, the present invention provides a change detection unit that detects a change in the screen of the window at a cycle set for each of a plurality of windows on a desktop, and a change detection unit that detects a change in the screen. For each window in which a change has been detected, a change area of the screen is set as a search range, an area similar to the template image set in the annotation rule for the window is searched, and the similar area is set in the annotation rule. A display update unit that performs a display update process of overlaying the icons.

  According to the present invention, it is possible to suppress an increase in the processing load of the annotation when displaying the annotation for a plurality of windows.

FIG. 1 is a diagram illustrating a configuration example of a display control device according to each embodiment. FIG. 2 is a diagram illustrating an example of the annotation rule. FIG. 3 is a diagram illustrating an outline of a control unit of the display control device according to the first embodiment. FIG. 4 is a diagram illustrating in detail a control unit of the display control device according to the first embodiment. FIG. 5 is a flowchart illustrating an example of a processing procedure of the display control device according to the first embodiment. FIG. 6 is a diagram illustrating search ranges of regular update and provisional update in the display control device according to the second embodiment. FIG. 7 is a diagram illustrating the update processing time of each of the regular update and the provisional update in the display control device according to the second embodiment. FIG. 8 is a diagram illustrating detection separately set by the regular update and the provisional update polling periods in the display control device according to the second embodiment. FIG. 9 is a diagram illustrating the setting of the polling period for each of the regular update and the provisional update for each pattern of the window change tendency in the display control device according to the second embodiment. FIG. 10 is a diagram illustrating an overview of the display control device according to the third embodiment. FIG. 11 is a diagram illustrating a computer that executes a display control program.

  Hereinafter, embodiments (embodiments) for implementing the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below. In the following description, display control using annotation as an example is described, but is not limited to annotation.

[First Embodiment]
First, the display control device 10 of each embodiment will be described with reference to FIG. The display control device 10 monitors a plurality of windows displayed on the desktop of the computer and performs annotation. That is, when detecting a change in the screen of each of the plurality of windows, the display control device 10 applies the annotation rule to a change area in the changed window.

  The display control device 10 performs polling to detect a change in the screen of each window. The polling cycle is set for each window. For example, the display control apparatus 10 sets a short polling cycle for a window that is frequently operated and changes frequently among a plurality of windows on the desktop, and sets a short polling cycle for other windows. Sets the polling cycle longer. That is, when performing the annotation, the display control device 10 allocates a limited number of CPU (Central Processing Unit) resources to windows with a large change, and allocates small resources to windows with a small change.

  By doing so, the display control device 10 can effectively utilize limited CPU resources when performing annotation on a plurality of windows. As a result, the display control device 10 can suppress an increase in processing load and processing time when performing annotation on a plurality of windows.

(Constitution)
The configuration of the display control device 10 will be described. As shown in FIG. 1, the display control device 10 includes an input unit 11, an output unit 12, a control unit 13, and a storage unit 14.

  The input unit 11 is for inputting a screen scroll instruction, an active window selection instruction, and the like, and includes a keyboard, a mouse, and the like. The output unit 12 displays a screen, and includes a display, a speaker, and the like.

  The storage unit 14 stores data and programs required for various kinds of processing by the control unit 13, and particularly has an annotation DB (Data Base) that is closely related to the present invention. For example, the storage unit 14 is a semiconductor memory device such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The annotation DB stores, for example, a plurality of annotation rules for each window for displaying an annotation text.

  This annotation rule has, for example, a structure in which a window rule and an annotation rule are in a parent-child relationship. The control unit 13 first applies a window rule to each window, and applies an annotation rule under the window rule when a condition described in the window rule is satisfied.

  For example, the annotation rule has a tree structure as shown in FIG. 2, and a window tag describing a window rule and an annotate tag describing an annotation rule are arranged in order from immediately below the root. The window tag has a title attribute, and the subordinate annotation rules are applied to windows that match the window title character string specified here. The annotate tag holds information on each annotation icon and message tooltip, and corresponds to direct rule information for generating an annotation. Note that the img tag is an element necessary for rule editing, template image generation, and the like, but description thereof is omitted here.

  In the annotation rule illustrated in FIG. 2, a plurality of annotation rules (annotate tags) are associated under a window rule (window tag). Among them, the window rule holds the window title character string (“calculator”) specified by the user at the time of rule creation, and the annotate rule holds the template image specified by the user at the time of rule creation. Based on such an annotation rule, the control unit 13 searches for an image similar to the specified template image from the window of the window title “calculator” (matches the template image). When finding an image similar to the template image, the control unit 13 displays a predetermined icon in the vicinity of the image, for example, in an overlay manner.

  Next, the control unit 13 will be described. First, the outline of the control unit 13 will be described with reference to FIG. First, when acquiring the desktop window set, the control unit 13 enumerates all windows on the desktop, and updates the display position of the annotation in accordance with the change in the position of any one of the windows.

  Further, the control unit 13 lists all windows on the desktop, and if there is a change in the presence or absence of any of the windows, refers to the window title of the changed window to determine the window to which the annotation rule is applied. Refine (filter by window title).

  Then, the control unit 13 lists, among the narrowed windows, windows having a screen change. Next, when the control unit 13 specifies the enumerated windows and the rectangular coordinates of the change area in the windows, the control unit 13 performs matching of the specified rectangular coordinates (difference area) with the template image set in the annotation rule. When finding an image that matches the template image, the control unit 13 displays (updates) the annotation on the window based on the annotation rule.

  As described above, the control unit 13 detects a change in the screen of the window, and performs matching with the template image for displaying the annotation in the area of the difference from the previous display. In other words, the control unit 13 excludes the window in which the screen change does not occur and the area in which the window does not change in the screen from the target of the matching with the template image. This allows the display control device 10 to efficiently match the window image and the template image when displaying the annotation.

  Note that, as described above, the control unit 13 detects a change in the screen of the window at the polling period set for each window to which the annotation rule is applied. Here, the polling period for each window is set relatively long for windows to which the annotation rule is applied, where the image (screen) change in the window is small, and the image (screen) change in the window is large. Set windows relatively short. Thus, the control unit 13 can detect a change in the screen of each window by effectively using the limited CPU resources.

  The setting of the polling period for each window may be manually set by the user of the display control device 10 or may be automatically set as follows. The automatic setting of the polling cycle will be described with reference to FIG.

  For example, as shown in FIG. 4, the control unit 13 monitors the presence / absence and position of the annotation in the window. Then, the control unit 13 totalizes the presence / absence / position change amount / change frequency of the annotation in the window for each window. Thereafter, the control unit 13 changes the polling cycle of the window based on the result of the aggregation.

  For example, the control unit 13 changes the polling cycle to be shorter than the current time for a window in which at least one of the frequency and the amount of change per time of the icon of the annotation in the window tends to increase. On the other hand, the control unit 13 changes the polling cycle to be longer than the current polling period for a window in which at least one of the frequency and the amount of change per hour of the icon of the annotation in the window tends to decrease.

  In addition, the control unit 13 considers the monitoring result of the image change in the window (for example, the change amount / change frequency of the image change in the window) in addition to the presence / absence / position change amount / change frequency of the annotation in the window. The polling period of the window may be changed. Furthermore, the control unit 13 may change the polling cycle based on a preset window priority order. The details of the change of the polling cycle will be described later.

  Returning to FIG. 1, the control unit 13 will be described in detail. The control unit 13 includes a change detection unit 131, a change tendency measurement unit 132, and a display update unit 133. The display order evaluation unit 134 indicated by a broken line may or may not be equipped, and the case where it is equipped will be described later.

  The change detection unit 131 performs polling on a plurality of windows on the desktop at a polling cycle set for each window, and detects a change in the window screen. The windows on the desktop change in various ways. For this reason, the change detection unit 131 needs to detect the change so that the time delay that occurs asynchronously with the display update of the window screen by the display update unit 133 is minimized. Therefore, the change detection unit 131 executes the polling process in a thread independent of the display update unit 133.

  Specifically, when any change occurs in the window during polling, the change detection unit 131 sets a flag (change detection flag) corresponding to the type to True. The type of flag is, for example, a change in the set of all windows on the desktop (for example, a change in the presence or absence of a window), and a change in the position of the window. When the change detection unit 131 receives an inquiry about the presence / absence of a window change and the type of change from the display update unit 133, the change detection unit 131 refers to the flag, and in the case of True, the corresponding detailed information (for example, Window ID list, etc.) and return the flag to False at the same time.

  Note that the following may be performed in order to avoid repeating the inquiry from the display update unit 133 to the change detection unit 131 for a long time even when there is no change in the window. That is, when the change detection unit 131 receives the next inquiry from the display update unit 133 and the flag remains False, the display update unit 133 shifts to a state of waiting for a notification of a change event from the change detection unit 131. I do. Then, the change detection unit 131 operates to return the thread of the display update unit 133 to the return state in response to the notification of the change event. For example, in the case of the Java (registered trademark) language, the wait procedure can be realized by using wait () and notifyAll () of the Object class.

  If at least one flag held in the change detecting unit 131 is True when the display updating unit 133 calls the standby procedure, the process is continued without waiting. On the other hand, if all the flag states are False, the thread of the display update unit 133 is set to a wait state (wait ()). Then, if the display update unit 133 is in a standby state when the change detection unit 131 sets the flag to True, the thread of the display update unit 133 is restarted (notifyAll ()). Thereby, the control unit 13 can suppress useless inquiries from the display update unit 133 to the change detection unit 131, so that the CPU load accompanying the detection of the change in the window can be reduced.

  In addition, the change detection unit 131 may set the polling cycle of the window as follows based on the frequency and amount of change per hour of the annotation icon on the window measured by the change tendency measurement unit 132. Good (setting of polling cycle based on criterion 1).

  For example, the change detection unit 131 sets a short polling cycle for a window in which at least one of the frequency and the amount of change of the annotation icon on the window measured by the change tendency measurement unit 132 tends to increase. . On the other hand, the change detection unit 131 sets a longer polling cycle for a window in which at least one of the frequency of change and the amount of change of the annotation icon on the window measured by the change trend measurement unit 132 tends to decrease. .

  For example, when the content of the output annotation is almost the same as the immediately preceding frame even after the window display is updated, the change detection unit 131 may set the polling cycle too short. Therefore, the change detection unit 131 changes the polling cycle to be longer. On the other hand, when the content of the annotation output after performing the display update of the window is different from that of the immediately preceding frame, the change detection unit 131 may set the polling cycle too long. Therefore, the change detection unit 131 changes the polling cycle to a short period.

  Note that the change detection unit 131 sets the polling cycle short if the variation in the change frequency / change amount per hour of the annotation icon on the window measured by the change tendency measurement unit 132 is increasing, and conversely decreases it. In the case of the tendency, the polling cycle may be set longer.

  Further, after setting the polling cycle based on the above-described criterion 1, the change detecting unit 131 changes the window screen change and the change frequency / change amount of the icon of the annotation per unit time, which are measured by the change tendency measuring unit 132, significantly. When the polling cycle is exhausted, the polling cycle may be changed (adjusted) according to the following criterion 2.

  That is, the change detection unit 131 sets the polling cycle based on the above-described criterion 1 and then changes the frequency and amount of change in the annotation with respect to the change frequency and change amount of the window screen measured by the change tendency measurement unit 132. Obtain the calculation result of the ratio of Then, the change detecting unit 131 changes the polling cycle longer based on the polling cycle set in the criterion 1 when the value indicated by the calculation result (or the variation in the value) is decreasing, and conversely, when the value indicates an increasing trend. Then, the polling cycle is shortened with reference to the polling cycle set in the criterion 1 (adjustment based on the criterion 2).

  As described above, when the change detection unit 131 sets the polling period of the window, the change is made based on the reference 2 in addition to the reference 1, so that when the screen of the window changes, the change actually affects the content of the annotation. In many cases, the polling cycle can be shortened as much as possible within the range allowed by the CPU resources.

  In addition, the change detection unit 131 performs the adjustment based on the criterion 2 in addition to the criterion 1 so that when there is a change in the screen of the window, if there are few cases where the change actually affects the content of the annotation, the polling cycle is changed. Can be longer. Thereby, the change detection unit 131 can adjust the polling cycle to a suitable state within a range allowed by the CPU resources of the display control device 10.

  The change detection unit 131 sets the polling cycle of each window according to the above-described criteria 1 and 2, and then obtains the measurement result of the change per window of each window by the change tendency measurement unit 132 again. The above-mentioned criteria 1 and 2 may be applied to change the polling cycle of each window. That is, the change detection unit 131 may dynamically change the polling cycle of each window based on the measurement result of the change per time of each window.

  The change tendency measuring unit 132 measures a change per time for each of the plurality of windows. The measurement target here measures, for example, the frequency and amount of change per hour of the icon of the annotation displayed on the window by the display update unit 133, the frequency of change and the amount of change of the screen per unit time of the window itself, and the like. .

  The display update unit 133 sets, as a search range, a change area of the screen for each of the windows in which the change of the screen has been detected by the change detection unit 131, and an image similar (matching) to the template image set in the annotation rule for the window. Search the area of. Then, the display updating unit 133 overlays the icon set in the annotation rule on an area of the image similar to the template image.

  According to such a display control device 10, when performing annotation on a plurality of windows, CPU resources necessary for the annotation can be effectively used. As a result, the display control device 10 can suppress an increase in processing load and processing time when performing annotation on a plurality of windows.

  When the display control device 10 detects that the CPU resources are depleted, for example, when another application consumes a large amount of the CPU resources, the display control device 10 temporarily increases the polling cycle or temporarily stops the display update operation. It may have a function of stopping itself.

  Further, the display control device 10 may further include the display order evaluation unit 134 in FIG. The display order evaluation unit 134 determines the order of priority between windows displayed on the desktop. The display order evaluation unit 134 determines whether the window is on the near side or on the rear side (Z-axis order), the order of the rectangular size of the window frame, or the center of the window on the desktop screen. The priority of the window is determined in the order close to the center of the window.

  When a plurality of windows are displayed on the desktop in an overlapping manner, the display order evaluation unit 134 considers the overlapping of the windows and the size of the area actually exposed as an area visible to the user in consideration of the context. May be determined in this order.

  Then, the display update unit 133 preferentially displays the annotation icon from the window with the highest priority determined by the display order evaluation unit 134. By doing so, the display update unit 133 can promptly display the annotation icon for a window that is likely to be more important to the user.

  In addition, the change detection unit 131 may adjust the polling cycle to be shorter for the window with the higher priority determined by the display order evaluator 134, and may be longer for the window with the lower priority. By doing so, the display update unit 133 can allocate CPU resources to each window according to the degree of importance to the user when displaying annotations on a plurality of windows.

(Processing procedure)
Next, a processing procedure for displaying an annotation by the display control device 10 will be described with reference to FIG.

  First, the change detection unit 131 of the display control device 10 acquires IDs and positions of all windows on the desktop (S101: enumerate (position) all windows on the desktop). For example, when the OS used for window display is a Microsoft Windows (registered trademark) OS, the change detection unit 131 obtains window handles (corresponding to IDs) of all windows on the desktop by calling the EnumWindows () function. get. Then, the change detection unit 131 acquires the window ID, the window position coordinates, the window title character string, and the screen capture image from the OS as necessary.

  After S101, when the change detection unit 131 determines that the position of any of the windows on the desktop has changed (Yes in S102), the control unit 13 proceeds to the following S103 and S104 for all target windows. Start the indicated loop. That is, the display update unit 133 changes the annotation display position for the window whose position has been changed (Yes in S103) (S104), and has changed the annotation display position for the window whose position has not been changed (No in S103). , S104 are skipped. When the control unit 13 completes the above loop for all target windows, the process returns to S101.

  After S101, if the change detection unit 131 determines that there is no change in the position of any window on the desktop (No in S102), it enumerates all windows on the desktop and obtains information on the presence or absence of all windows. (S105: list all windows on the desktop (presence or absence of windows)).

  After S105, when the change detection unit 131 determines that there is a change in the presence or absence of any window on the desktop (Yes in S106), the following process is performed on all target windows. That is, the change detection unit 131 compares the title of the target window with the titles of all windows set in the window rule, and if there is a matching window title (S107: Compare titles and match? It is registered in the screen change detection target list (S108).

  On the other hand, if there is no matching window title (S107: Compare titles and match ?, No), change detection unit 131 skips the process of S108. Then, when the above-described loop is completed for all the target windows, the control unit 13 returns to S101.

  After S105, when the change detection unit 131 determines that there is no change in the presence or absence of any window on the desktop (No in S106), the window change is not detected among the windows registered in the screen change detection target list. List windows In other words, the change detection unit 131 lists the window set in which the presence or absence of the window has changed, and the rectangular coordinates of the change area of the window in which the screen has changed in the window set (S109: The window in which the screen has changed is displayed. Enumeration (presence of window, rectangular coordinates).

  After S109, when the change detection unit 131 determines that there is a change in the window registered in the screen change detection target list (Yes in S110), that is, when the change detection unit 131 detects a screen change, the display update is performed. The unit 133 starts a loop of annotation rules under all window rules for all screen change detected windows. That is, the display update unit 133 searches for the template image set in the annotation rule under all the applicable window rules for all the screen change detected windows, and if there is a match (Yes in S111), the icon display is performed. Perform (S112).

  On the other hand, the display update unit 133 searches for a template image set in the annotation rule under all the applicable window rules, and if there is no match (No in S111), skips the processing of S112. When the above-described loop ends for all the screen change detected windows, the display update unit 133 returns to S101.

  If the change detection unit 131 determines that there is no change in any of the windows registered in the screen change detection target list (No in S110), the process returns to S101.

  By performing the above-described processing, the display control device 10 narrows down to a change area of a window where a screen has changed, performs matching with a template image set in an annotation rule, and can display an icon of an annotation.

[Second embodiment]
Next, a display control device 10 according to a second embodiment will be described. When the display update unit 133 updates the display of the annotation on the window, the display control device 10 of the second embodiment performs a regular update process and a tentative update process (see Patent Document 2). Then, when detecting a change in the window screen, the change detection unit 131 sets an individual polling cycle for each of the regular update process and the provisional update process. Hereinafter, the regular update process is abbreviated as “regular update”, and the provisional update process is appropriately abbreviated as “temporary update”.

  First, an outline of the regular update process and the provisional update process will be described with reference to FIG. As described above, the display update unit 133 performs matching with the template image set in the annotation rule with the change area of the window where the screen has changed as the search range. Here, it may take a long time to perform matching with the above-described template image using the entire changing region of the window as a search range. Therefore, as a search range when performing matching with the template image, an area narrowed down to an area that is particularly likely to match the template image is used among the change areas of the window.

  For example, consider a case where the window area 601 shown in FIG. 6 is a change area. In this case, for example, assuming that scrolling of the window in the vertical direction has been performed, an area that is vertically swept from the match position of the previous frame in the area 601, that is, a narrowed area is narrowed down to a change area The region 602 is extracted. Then, the display update unit 133 performs matching with the template image having the area 601 as a search range, and updates the display of the annotation (regular update processing). Further, the display update unit 133 performs matching with the template image having the area 602 as a search range, and updates the display of the annotation (provisional update processing). Although the accuracy of this provisional update process is not guaranteed as compared to the regular update, the provisional update process has a merit that the response is faster than the regular update because the search range is narrower than the regular update.

  For example, as shown in FIG. 7, when a change in the screen of the window (for example, scroll movement of the screen) is detected, the display update unit 133 performs matching with the template image in each thread of the regular update and the provisional update. Here, since the normal update takes time, the detection of the screen change cannot catch up, and the flag of the change detection unit 131 is True when the normal update loop returns, so that the next loop is immediately executed. I do. On the other hand, since the provisional update is completed in a relatively short time, when the provisional update loop returns, the process shifts to the standby state, and the event update from the thread of the change detection unit 131 restarts the provisional update loop. The display of the annotation icon is also updated following the scroll direction of the window screen as shown in FIG.

  As described above, in the case where the display update unit 133 performs both the regular update and the provisional update, by using the change detection flag, the update processing time can be longer or shorter than the unit period. An operation to obtain a correct update result is possible.

  That is, if the unit period is lengthened, the display update unit 133 is not notified immediately when the screen of the window changes, but is notified collectively of changes before and after the predetermined period. As a result, even if the CPU of the display control device 10 has sufficient processing capacity, the CPU does not consume it and the update processing is in a standby state. On the other hand, when the unit period is shortened, a plurality of change detections are notified to the display update unit 133 during the execution of one update process. Thereby, if the processing capability of the CPU of the display control device 10 is high, the update is performed for each notification at a ratio close to one-to-one, but if the processing capability of the CPU is low, the content of a plurality of changes is correspondingly updated by one update. Process together. These movements cause a loss of waiting time and wasteful detection too fine, but do not break the basic mechanism of the operation. Here, the meaning of the unit period is a time interval of a detection event notified from the change detection unit 131 when a change in the screen occurs continuously, and is the same as the polling cycle of the change detection. is there.

  By applying the above operation, the change detecting unit 131 can develop a configuration in which a polling cycle is set for each of the provisional update and the regular update, and notification of a screen change is individually notified.

  That is, as illustrated in FIG. 8, the change detection unit 131 performs the detection of the ongoing change for the regular update and the detection of the continuous change for the provisional update at polling periods that are separately set. Then, when the change of the window is detected by the polling for the regular update of the change detecting unit 131, the display updating unit 133 calculates the regular update (for example, matching with the template image or the like) for the window change area. To obtain the result area. In addition, the provisional update is calculated for the speculative range (an area speculatively narrowed down from the above-mentioned change area to an area that is highly likely to match the template image) by polling for the provisional update by the change detection unit 131, Get the result area. Then, the display update unit 133 updates the display of the annotation icon in the result area.

  After that, the change tendency measuring unit 132 measures the change tendency of the window, and the change detecting unit 131 adjusts the balance between the polling periods of the regular update and the provisional update using the measurement result. For example, the change detection unit 131 performs setting (balance adjustment) of the polling cycle of each of the regular update and the provisional update based on the following tendency based on the change tendency of the window. Here, the case where the change tendency of the window is one of the pattern 1 shown in FIG. 9A and the pattern 2 shown in FIG. 9B will be described as an example.

  Pattern 1 is a pattern in which the ratio of change in the presence or absence of a match target is greater than the ratio of change in the position of the match target with the template image in the window. For example, a case where a GUI element (match target) frequently appears / disappears. Pattern 2 is a pattern in which the ratio of the change in the location of the match target is greater than the ratio of the change in the presence or absence of the match target in the window. For example, this corresponds to a case where scrolling in the vertical direction of the screen always occurs.

  As in Pattern 1, when the rate of change in the “existence” of the match target is large, it is not very important to display a smooth animation in accordance with the change in the position of the match target, etc. Is more preferred. Therefore, as illustrated in FIG. 9C, the change detecting unit 131 sets the regular update polling cycle short and sets the provisional update polling cycle long.

  On the other hand, when the rate of change in the “existing position” of the match target is large, such as when the screen scrolls constantly, as in pattern 2, it is important to display the animation in real time in accordance with the position change. It is preferable to carry out the procedure carefully. Therefore, as illustrated in FIG. 9C, the change detection unit 131 sets a longer regular update polling cycle and a shorter temporary update polling cycle.

  For example, the change tendency measuring unit 132 measures the frequency of the change (first change frequency) due to the appearance and disappearance of the icon displayed by the annotation per time for each of the plurality of windows on the desktop. Further, the change tendency measuring unit 132 measures the frequency of change (second change frequency) due to the movement of the position of the icon displayed by the annotation for each of the plurality of windows on the desktop.

  Then, for a window in which the ratio of the frequency of the second change to the frequency of the first change is larger than a predetermined value, the change detection unit 131 performs polling of the provisional update with respect to the length of the regular update polling cycle in the window. The period length ratio is set so as to be larger than a predetermined value. On the other hand, for a window in which the ratio of the frequency of the second change to the frequency of the first change is equal to or less than a predetermined value, the change detection unit 131 determines whether the temporary update polling is performed for the length of the regular update polling cycle in the window. The ratio of the length of the cycle is set to be equal to or less than a predetermined value.

  That is, when the change frequency of the position of the match target is higher than the change frequency of the presence or absence of the match target, the change detection unit 131 reduces the ratio of the polling cycle of the provisional update to the polling cycle of the regular update. On the other hand, when the change frequency of the position is smaller than the change frequency of the presence or absence of the match target, the change detection unit 131 increases the ratio of the provisional update polling cycle to the regular update polling cycle.

  As described above, for each window, the change detection unit 131 performs the regular update and the tentative update of the window in accordance with which change tendency of the change of the presence or absence of the match target and the change of the position of the match target is stronger. Set the polling cycle of

  The change detecting unit 131 changes the polling cycle of the provisional update by changing the ratio based on the polling cycle of the regular update, for example. In this case, if the regular update polling period is made longer, the provisional update polling period becomes longer accordingly, and if the regular update polling period is made shorter, the provisional update polling period becomes shorter correspondingly.

  Here, if the polling cycle of the provisional update is longer than the polling cycle of the regular update, the regular update processing is completed before the provisional update processing, and the provisional update processing is kicked based on the result (FIG. 8). "+" Link shown below). This eliminates the necessity of the provisional update process, and the provisional update polling process is substantially meaningless. That is, the process of the provisional update alone in the polling cycle is temporarily canceled, and the polling standby from the latest provisional update output is started again. Therefore, the ratio of the provisional update polling cycle to the regular update polling cycle is set to 1.0 at the maximum. The regular update polling cycle may be adjusted using the criteria 1 and the criteria 2 described in the first embodiment. In this case, the position of the icon displayed by the annotation is ignored and the polling cycle of the icon is ignored. It is only necessary to monitor for changes in the presence or absence.

[Third Embodiment]
Next, a display control device 10 according to a third embodiment will be described. The display control device 10 according to the third embodiment, when detecting the end of the screen change, triggers the update of the window display to prevent phenomena such as omission of window update and incorrect display contents remaining. It is characterized by doing.

  As in the first embodiment and the second embodiment, when the change detection unit 131 of the display control device 10 dynamically changes the polling cycle, a state in which the polling cycle is set to be extremely long may occur. There is.

  If the screen change is not continuous for a long time like a scroll operation, but is, for example, a change of only one frame, the user recognizes that the still screen image has been replaced. It is expected that the display of the annotation icon will be updated. At this time, if the polling cycle is set to be extremely long, the time lag after the change will be long, and the user experience will be greatly reduced as compared with the case where the display update cycle during continuous change is thinned out. For this reason, it is desirable that the display update when the screen is stopped be started promptly regardless of the polling cycle.

  Therefore, the change detection unit 131 of the display control device 10 according to the third embodiment detects the stop (end) of the change of the screen of each of the plurality of windows. Then, when the change detecting unit 131 detects the stop of the screen change, the display updating unit 133 performs the updating process without waiting for the polling cycle. When the display control device 10 performs the regular update and the provisional update as in the second embodiment, as illustrated in FIG. 10, the display update unit 133 performs the regular update when the end of the screen change is detected. Regular update without waiting for the polling cycle of.

  In this way, when the display control device 10 detects the end of the screen change, it is guaranteed that the display update by the regular update is always performed once. As a result, even when the polling cycle is set to be extremely long or when the polling cycle changes significantly during the display update process, the display control device 10 can accurately update the display of the annotation icon. Further, even when the polling cycle is set to be extremely long, when the screen change converges, the display control device 10 can promptly update the display.

  The change detection unit 131 detects the stop (end) of the screen change, for example, in addition to a normal bitmap comparison method (see Patent Document 1), a scan line method (see Patent Document 3), and the like. It is also possible to combine methods that are implemented quickly with low load.

[Other Embodiments]
In the display control device 10 of the second embodiment, one thread is assigned to each of the regular update and the provisional update, but the number of threads to be processed is increased to increase the degree of parallelism. You can also. For example, it is conceivable that the change detection unit 131 holds a set of change detection flags for each target window, and allocates and executes a thread for each target window. At this time, the same number of threads as the target window may be dynamically generated, or a predetermined number of threads may be generated in advance, and the target window set may be divided and assigned to the same number of groups. At the time of grouping, it is conceivable that the priority of the target window element is determined by using the output of the display order evaluation unit 134 and the change tendency measuring unit 132, and the target window element is divided by the priority division. The configuration in which the degree of parallelism is increased is effective when the number of windows to be processed simultaneously per polling loop frequently increases in each thread of the regular update and the provisional update.

  Further, the display control device 10 described in the above embodiment can be implemented by installing a display control program for realizing the function of the display control device 10 in a desired information processing device (computer). For example, by causing the information processing device to execute the display control program provided as package software or online software, the information processing device can function as the display control device 10. The information processing device referred to here includes a desktop or notebook personal computer. In addition, the information processing apparatus includes a mobile communication terminal such as a smartphone, a mobile phone, or a PHS (Personal Handyphone System), and a PDA (Personal Digital Assistants). Further, the display control device 10 may be mounted on a cloud server.

  Hereinafter, an example of a computer that executes the display control program will be described. FIG. 11 is a diagram illustrating a computer that executes a display control program. As shown in FIG. 11, the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. In the disk drive 1100, for example, a removable storage medium such as a magnetic disk or an optical disk is inserted. For example, a mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050. For example, a display 1130 is connected to the video adapter 1060.

  Here, as shown in FIG. 11, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various data and information described in the above embodiments are stored in, for example, the hard disk drive 1090 or the memory 1010.

  Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as necessary, and executes the above-described procedures.

  The program module 1093 and the program data 1094 relating to the program are not limited to being stored in the hard disk drive 1090. For example, the program module 1093 and the program data 1094 are stored in a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. You may. Alternatively, the program module 1093 and the program data 1094 relating to the program are stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and are stored in the CPU 1020 via a network interface 1070. May be read.

Reference Signs List 10 display control device 131 change detection unit 132 change tendency measurement unit 133 display update unit 134 display order evaluation unit

Claims (5)

A change detection unit that detects a change in the screen of the window at a cycle set for each of the windows on the desktop,
The change area of the screen as the search range for each window change screen is detected by the change detection unit, the window to search an area similar to the set template image in the annotation rules for the similar is searched A display update unit that performs a display update process of overlay-displaying an icon set in the annotation rule in an area ;
A first change frequency, which is a frequency of a change due to the appearance and disappearance of the icon over time displayed by the display update unit with respect to each of the plurality of windows, and a change frequency due to a movement of the position of the icon. A measurement unit for measuring the frequency of the second change ,
The display update unit,
In accordance with a change in the screen of each of the windows detected by the change detection unit, a predetermined area generated based on the previously searched similar area is used as the search range to search for an area similar to the template image. A provisional update process of overlaying the icon on the searched similar area, and searching for an area similar to the template image using the area wider than the predetermined area as the search range. Performing a regular update process of overlaying the icon in the similar area,
The change detection unit,
For a window in which the ratio of the frequency of the second change to the frequency of the first change measured by the measurement unit is larger than a first predetermined value, a change in a screen required for the normal update process in the window is detected. Setting a ratio of a period length for detecting a change in the screen required for the provisional update process to a period length to be larger than a second predetermined value;
For a window in which the ratio of the frequency of the second change to the frequency of the first change measured by the measurement unit is equal to or less than the first predetermined value, a change in the screen required for the normal update process in the window is determined. A display control device , wherein a ratio of a length of a cycle for detecting a change in a screen required for the provisional update processing to a length of a cycle for detection is set to be equal to or less than the second predetermined value . The change detection unit,
Detecting the stop of the screen change of each of the plurality of windows,
The display update unit,
The display control device according to claim 1, wherein the display update process is performed when the change detection unit detects that the change of the screen has stopped. The change detection unit,
The display control device according to claim 1, wherein a change in an image on the screen is detected for a window having a designated predetermined window title among the plurality of windows on the desktop. A display control method executed by a display control device,
A change detection step of detecting a change in the screen of the window at a cycle set for each of the windows on the desktop,
For each of the windows in which the screen change has been detected by the change detection step, a screen change area is set as a search range, an area similar to the template image set in the annotation rule for the window is searched, and the searched similar area is searched. A display update step of overlaying an icon set in the annotation rule ,
A first change frequency, which is a frequency of a change due to the appearance and disappearance of the icon over time displayed by the display update unit with respect to each of the plurality of windows, and a change frequency due to a movement of the position of the icon. the step of measuring the frequency of the second variation seen including,
In the display update step,
In accordance with a change in the screen of each of the windows detected in the change detection step, a predetermined area generated based on the previously searched similar area is used as the search range to search for an area similar to the template image. A provisional update process of overlaying the icon on the searched similar area, and searching for an area similar to the template image using the area wider than the predetermined area as the search range. Performing a regular update process of overlaying the icon in the similar area,
In the change detecting step,
For a window in which the ratio of the frequency of the second change to the frequency of the first change measured in the measurement step is larger than a first predetermined value, a change in a screen required for the normal update process in the window is detected. Setting a ratio of a period length for detecting a change in the screen required for the provisional update process to a period length to be larger than a second predetermined value;
For a window in which the ratio of the frequency of the second change to the frequency of the first change measured in the measuring step is equal to or less than the first predetermined value, a change in a screen required for the normal update process in the window is determined. A display control method , wherein a ratio of a length of a cycle for detecting a change in a screen required for the provisional update processing to a length of a cycle for detection is set to be equal to or less than the second predetermined value . Display control program for executing each step included in the display control method according to the computer to claim 4.

Images