JP4160997B1 - Operation image reproduction apparatus and operation image reproduction program - Google Patents

Abstract

A technique for facilitating confirmation of a specific operation when paying attention to the specific operation while reproducing an operation image on a terminal.
An operation image reproduction apparatus for acquiring an operation image and an operation event log describing an operation event in a terminal and reproducing the operation image. An operation event is classified by operation category, and an operation including at least an attention event as one or more operation events indicating a notable operation in the operation category and a preceding event as an operation event indicating an operation preceding the attention event is included. When the preceding event of the event group by operation category is temporally associated with the operation image that is grouped and reproduced as an event group by category, the contents of the event of interest of the event group by this specific operation category are played back Output with the operation image.
[Selection] Figure 1

Description

  The present invention relates to an operation image reproduction technique for acquiring an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal and reproducing the operation image.

  In a system in which the remote control server and client terminal are connected via a communication network, the monitor screen displayed on the monitor is transmitted to the remote control server and received by the remote control server when the operator remotely operates the terminal. The captured monitor screen is displayed on the monitor, and the captured image recording means of the remote control server records the monitor screen of the client terminal as a captured image (operation image) in the operation event log at a predetermined interval. There is one in which a log reproduction processing unit of a remote control server reproduces a captured image recorded in an operation event log (see, for example, Patent Document 1). In this prior art, the operation image stored in the data storage unit can be reproduced on the monitor monitor to monitor the operation content of the terminal by the user. However, when the user wants to monitor by paying attention to a specific dangerous operation. Even if there are, other operations appear frequently, and the monitoring work becomes distracting. For this reason, it is difficult to reliably find an operation that is prohibited in some cases, such as copying a file and copying it to a portable memory.

  In addition to receiving information such as the water level and gate opening from each district river facility, the river management facilities that should be operated (monitored) based on the importance and safety based on these information are listed in the monitoring list display area in order. A wide area monitoring system that displays and displays an image of a CCTV selected in a facility monitoring area in a monitoring operation area is known (see, for example, Patent Document 2). In this wide-area monitoring system, information such as the water level and gate opening that describes the state of the river as a monitoring target and CCTV images that display the state of the river are sent from the monitoring target site to the monitoring center. While monitoring the state of the river. However, since this monitoring system is intended to monitor the state of rivers that change from moment to moment, it is naturally built for real-time river monitoring, and sent measurement information and monitoring images (TV images). It is not considered to monitor the river while recording and reproducing it, and there is no suggestion about facilitating monitoring when the surveillance image is reproduced.

Furthermore, in order to visually confirm the user operation on the image, the time position and spatial coordinate data in the video when the user clicks on a specific subject in the video is saved as a click operation event log, and the video is being played back In addition, a technique for superimposing and displaying the click operation event log is known (for example, see Patent Document 3). With this technique, it is possible to visually confirm a click operation such as what scene the user clicked at the time of reproduction. However, this technique only displays the user's click operation on an image that was not visible in real time as an image at the playback time when the playback was clicked. It is not possible to contribute to a request that facilitates confirmation of specific operations.
JP 2006-108947 A (paragraph numbers 0094 to 0099, FIGS. 1 and 17) Japanese Patent Laying-Open No. 2006-41770 (paragraph numbers 0035 to 0051, FIGS. 1 and 2) Japanese Patent Laying-Open No. 2004-110096 (paragraph numbers [0022] to 0030, FIG. 3)

  In view of the above situation, an object of the present invention is to provide a technique that facilitates confirmation of a specific operation when paying attention to a specific operation while reproducing an operation image on a terminal.

In order to solve the above-mentioned problem, an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal are acquired, and the operation image reproduction apparatus performs reproduction of the operation image. In the apparatus according to the present invention, a data storage unit that stores the operation image and an operation event included in the operation event log, and the operation event is classified according to an operation category, and a notable operation in the operation category is determined. A category-specific event group determination unit for grouping as an event group by operation category including at least an attention event as one or more operation events to be displayed and a preceding event as an operation event indicating an operation preceding the attention event; the operation image is a temporally corresponding steering playback operation image intoxicated to A reproduction information processing means for outputting the implied image to imply contents of event of interest of the operation category event group including an event as predecessor event together with the reproduction operation image, it is provided.

  User operations to be confirmed in reproduction of operation screens for various purposes such as monitoring purposes and simple browsing purposes can be divided into various categories, and these user operations are expressed as operation events as user operation units handled in the present invention. Generally, it consists of a series of events that start from an initial event, reach an intermediate event, and reach a final event. At least one attention event as a notable operation event exists among the operation events following the initial event. become. In the present application, the initial event and the intermediate event (including a plurality of events) are referred to as preceding events. For example, one of the notable actions that falls into the operation category of copying an important file to a USB memory and taking it out is an operation event for selecting a file, an operation event for duplicating a file, an operation event for changing a duplicate file name, or duplication It is composed of a group of events including operation events for cutting and pasting files into a USB memory. Then, if the event of interest in this event group by operation category is an operation event of “recording a file to a USB memory”, in the terminal monitoring apparatus according to the present invention described above, for example, a preceding event of “selecting a file” The event contents of “Record file to USB memory” as a noticed event brought about by this preceding event at the time of playback of the operation image corresponding in time to the occurrence of the preceding event “change file name” are also output and monitored Can inform those who are watching. As a form of output of the contents of the attention event, it is preferable to display the contents of the final event together with the operation image on the monitor, but instead of or in combination with this, the contents of the attention event are given by voice. Also good. In addition, if there are two events of interest in the event group by operation category, that is, the operation event “change the duplicate file name” and the operation event “record the file to the USB memory”, “select file”. The event content “change the duplicate file name” and the event “record the file to the USB memory” as the attention event brought about by the preceding event when the operation image corresponding in time to the occurrence of the preceding event occurs The contents are output. In any case, it is displayed on the monitor. Since operation events such as “Select File” always occur, it is difficult to always pay attention, but “Select File” included in the event group by operation category to which the attention event that should be particularly problematic belongs When an operation event occurs, the content of the event of interest is displayed on the operation screen, so that the operation event can be checked with special attention. As a result, the occurrence of a specific important operation event (attention event) can be suggested at the stage of occurrence of the preceding operation event that leads to it, and the confirmation of the attention event is facilitated.

  A specific case where the reproduction information processing means displays on the monitor together with the reproduction operation image the content of the event of interest of a specific operation category event group, that is, a series of operation event groups generated by a user operation that is highly necessary to be monitored. As a configuration, it is proposed that the reproduction information processing means includes an implied image creation unit that creates an implied image implying the content of the event of interest, and an image composition unit that synthesizes the implied image with the reproduction operation image. In this configuration, an implied image that implys the content of the event of interest is combined with the playback operation image, so it is possible to confirm in advance the content of an important user operation guided by a specific operation without feeling uncomfortable just by looking at the monitor. be able to. In particular, when the implied image is synthesized in association with the occurrence location of the preceding event in the playback operation image linked to the preceding event, the person viewing the operation image as the terminal operation screen through the monitor It is possible to obtain an advantage that a user operation to be noticed after that can be confirmed only by gazing at the operation. Alternatively, if there is a purpose of informing beforehand the occurrence location of the event of interest (the operation point of the event of interest), the implied image is synthesized in association with the occurrence location of the event of interest in the playback operation image linked to the preceding event. May be. Further, if the implied image to be synthesized adopts a display form that is easy to understand for anyone, such as a balloon frame starting from the occurrence location and the content of the attention event drawn in characters in this balloon frame, When a preceding event occurs, it is possible to confirm the contents of a serious operation event led to this preceding event. In addition, if a time lapse image representing a period until the final event occurs is employed together with such an implied image or as a single implied image, an operation event to be noticed after that occurs when a preceding event occurs. It is convenient because it can grasp the period until. For example, this time lapse image may be expressed as an hourglass or a slide bar, but if it is a countdown time, it is possible to easily know the time until an operation event occurs.

In an operation image reproduction program for acquiring an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal, storing the operation image in a data storage unit, and reproducing the operation image In order to solve the above problem, an operation image reproduction program according to the present invention includes a function for storing the operation image and an operation event included in the operation event log in the data storage unit, and operating the operation event. An event classified by operation category including at least an attention event as one or more operation events indicating notable operations in the operation category and a preceding event as an operation event indicating an operation preceding the attention event. A function of grouping as a group and the operation to be reproduced. A function of outputting along with the reproducing operation image implied image to imply contents of event of interest of the operation category event group including a predecessor event operation event temporally corresponding to the reproduction operation image is an image, to the computer make it happen. The operation image reproduction program configured in this way also has the operational effects described in the previous operation image reproduction device, and can have various additional characteristic configurations described above.

  An event group by operation category including an attention event as an operation event indicating an operation stage to be noticed in an operation category having a particularly high degree of interest, and an initial event or an advance event such as an intermediate event preceding the attention event. , And the contents of the attention event of the event group by operation category having the operation event linked to the reproduction operation image as a preceding event when the operation image is reproduced (when the reproduction operation image is displayed). The basic principle of the operation image reproduction technique according to the present invention, which can confirm in advance the content of a user operation to be noticed that is guided by a specific operation by outputting it in some form together with the display on the monitor, is shown in FIG. Explained.

  FIG. 1 shows a terminal 1 composed of a general-purpose notebook computer or desktop personal computer, and a terminal monitoring device 3 that monitors user operations on the terminal 1 by reproducing its operation screen. A computer system connected through a data transmission line 2 such as a network in a simple organization or an open wide area network such as the Internet) is shown.

In the terminal 1, an operation event log that sequentially describes operation events representing each user operation is created, and an image that creates a monitor screen (operation screen) at a predetermined interval (for example, every 1 second) is captured as an operation image. (# 01). The operation event log and the operation image are sent to the terminal monitoring device 3 through the data transmission line 2 (# 02). The operation event described in the operation event log sent to the terminal monitoring device 3 and the operation image sent to the terminal monitoring device 3 are stored in a data storage unit such as a large-capacity hard disk (# 03). Before or after storing, referring to a time code indicating an acquisition time such as a time stamp, for example, the matching time or the approximate time is linked as a link condition (# 04). With this link, an operation event that occurs at an arbitrary time is temporally associated with an operation image that represents an operation screen that substantially corresponds to the occurrence time, and identifies a user operation performed on a certain operation screen. It becomes possible. If the operation event acquisition timing and the operation image capture timing are significantly different from each other, and there is no operation image temporally corresponding to the operation event to be processed, interpolation representing the operation event is performed. An operation image may be generated.
User operations in the terminal 1 can be classified into various operation categories such as copying important files to a portable memory (USB memory or the like), attaching and sending important files, and downloading illegal files from the outside. The user operation of each operation category has a notable operation event such as copying or attaching to an important file, and is a start event that is a starting point prior to this notable operation event (notable event). An operation event (for example, selection of important file) is performed. The operation events belonging to the preset operation category are grouped as an operation category-specific event group including a start event, an intermediate event, and a final event, but a target event is designated from the intermediate event and the final event (# 05). Furthermore, an implied image representing the content of the operation is created from an attention event regarded as an attention operation event from a specific operation category event group (# 06). For example, if the operation content of the event of interest is “file copy to USB memory”, an illustration image in which characters indicating the content are drawn in a balloon frame can be adopted as an implied image. In addition, the time to the preceding event and the event of interest belonging to this event group by operation category is calculated from the time stamps assigned to each, and the time period until the event of interest occurs is visualized (time elapsed Image: count time, time lapse bar graph, etc.) may be added as an implied image.

  Next, in order to check the operation screen of the specific terminal, the operation image is sequentially read from the data storage unit along the time axis, and the operation image is reproduced on the monitor (# 07). If the operation event linked to the operation image read out for reproduction is a preceding event belonging to the specific operation category event group set in advance in step # 05, the event of interest of the operation category event group The implied image created based on the above is called, synthesized with the read operation image, and displayed on the monitor (# 08). At that time, if the combined position in the operation image of the implied image is the operation point of the linked preceding event, the content of the final operation guided from that operation to the position indicating some operation in the reproduced operation image Combined with the display, can attract attention. Further, when the combined position of the implicit image in the operation image is set as the operation point of the event of interest, it is possible to know in advance at which position on the monitor the operation to be noted occurs. In particular, when a time-lapse bar graph or a countdown value indicating a period until a noticeable event occurs is displayed as one of the implicit images, the concentration is increased, and it is easy to confirm a noticeable operation. In this description, the output of the content of the attention event belonging to the event group by operation category is performed in the form of an implicit image combined with the operation image corresponding to the preceding event. May be carried out in a voice form, or these may be used in combination.

  FIG. 2 shows a configuration of a terminal monitoring system as an example of a terminal operation confirmation system that employs the basic principle of the operation image reproduction technique according to the present invention described above. In this terminal monitoring system, a large number of terminals 1 are connected to a terminal monitoring device 3 via a network 2. The terminal 1 is a general personal computer, and operations related to various data processing and data browsing, such as creation, browsing, copying, correction, deletion, printing, Internet browsing, mail transmission, and the like of files, etc. are performed by the user. A user name or a user code input through a login operation performed before the user uses the terminal 1 is sent to the terminal monitoring apparatus 3 as user attribute information together with the terminal ID, so that the terminal monitoring apparatus 3 The user who is using can be specified. Furthermore, from the terminal 1 to the terminal monitoring device 3, an operation event log describing an operation event performed between the time when the user logs in and the time when the user logs off, and an operation image obtained by capturing the operation screen each time are provided in a timely manner. Sent. The terminal monitoring device 3 can monitor the operation performed on the terminal by reproducing the transmitted operation image for each terminal 1 in real time or after recording.

  The terminal 1 includes a monitor 51 such as a liquid crystal display, a keyboard and a mouse as the input operation device 52, a stationary hard disk 53 for storing various data, and a removable recording medium (USB memory or the like) 54. Various operation commands input by the user through the operation screen displayed on the monitor 51 are executed by the GUI unit 10 created by the OS installed in the terminal 1 as an intermediary. Data processing based on user instructions including various file processing is performed by the data processing unit 11. At this time, in order to display various operation screens on the monitor 51, these display data are temporarily stored in the video memory 12, but the display data developed in the video memory 12, that is, operation images representing the operation screens. Is captured by the operation screen capture unit 13. In addition, all operations performed by the user through the input operation device 52 are described in the operation event log generation unit 14 by listing operation execution program names, operation target data names (file names), operation command names, and the like as operation events. These operation events are scripted as an operation event log. The operation event log generated by the operation event log generation unit 14 and the captured image captured by the operation screen capture unit 13 are sent to the terminal monitoring device 3 connected by the network 2 through the network IF. In order to facilitate management, a common time stamp and user attribute information are assigned to the operation event log and the captured image.

  Normally, in order to use the terminal 1, the user must first log in by entering the user name and password. The user name obtained through the login process is used as it is or in the login / logoff control unit 15. Accordingly, it is converted to a user ID or the like, and becomes user attribute information to be sent to the terminal monitoring device 3 together with a terminal ID for identifying the terminal 1.

  The terminal monitoring apparatus 3 is also generally constituted by a general-purpose computer, and is attached with a monitoring monitor 55 such as a liquid crystal display, a data storage unit 57 including a keyboard, a mouse, a large-capacity hard disk, etc. as an input operation device 56. . A terminal that manages the terminal specified based on the user attribute information related to the terminal operating user sent from the GUI unit 30 or the terminal 1 by the OS installed in the terminal monitoring device 3 and the user operating this terminal Various basic functions such as the management unit 31 are created. Further, most of the functions related to the present invention in the terminal monitoring device 3 are created along with the execution of the terminal monitoring program installed in the terminal monitoring device 3, and have been sent from the terminal 1 as typical ones. An operation event log acquisition unit 32, an operation image acquisition unit 33, and an operation event log acquisition that receive an operation event log describing an operation event performed by the user and an operation image indicating an operation screen at that time via the network 2. A type code is assigned to each operation type such as “mouse click”, “mouse drag”, and “keyboard character input” with respect to the operation event for each terminal extracted by the unit 32 and formed in the data storage unit 57. The operation event classification unit 35 to be transferred to the operation event storage unit 57a, and the operation event and time by a time stamp or the like The operation image management unit 36 which stores provided in series in readable operation image storage unit 57b, which is formed in the data storage unit 57 as when and operation images that are associated in specific at another terminal. In addition, when a link code such as an index code is added in order to enable high-speed search and extraction of the operation event extracted by the operation event log acquisition unit 32 and the operation image at the time of the operation event occurrence, A link unit 34 for linking the operation image and the operation event is provided. Furthermore, the operation events stored in the operation event storage unit 57a are classified by operation category as described with reference to FIG. 1, and the attention event as the operation event indicating the operation to be noticed in the operation category and the above-described event A category-specific event group determination unit 37 that groups as an operation-category event group that includes at least a preceding event that represents an operation preceding the event of interest, and an operation category determined by the category-specific event group determination unit 37 An event group registration table 37a for storing the contents of different event groups in a table and a function for extracting an operation event linked to a reproduction operation image that is an operation image to be reproduced from the operation event storage unit 57a or a reproduction operation Operation events that correspond to images temporally An operation event management unit 38 having a function of checking whether the event belongs to an event group classified by operation category registered in the event group registration table 37a, and a reproduction operation image by link determination by the operation event management unit 38 When it is determined that the preceding event of the specific operation category event group corresponds to the time, the reproduction information processing means for outputting the contents of the attention event of the specific operation category event group together with the reproduction operation image Monitoring information processing means 40 functioning as: Furthermore, as described above, when there is no operation image that temporally corresponds to the operation event to be processed, the interpolation operation image generation unit 45 that generates an interpolation operation image representing the operation event is provided. ing.

  The monitoring information processing means 40 includes an implied image creating unit 41 that creates an implied image that implys the content of the event of interest included in the event group for each operation category, and an event for each operation category to which the final event belongs. An image composition unit 42 that composes a reproduction operation image linked to a preceding event of the group, a reproduction operation image synthesized with the implied image, and a reproduction operation image without such composition are processed so as to be reproduced on the monitor monitor 55. An operation image reproduction unit 43 is included. Furthermore, although not shown in the figure, when the content of the final event is notified by voice, the monitoring information processing means 40 is provided with a voice generation unit.

  Further, in order to be able to arbitrarily designate a specific operation category, a noticed operation category designation unit 39a that gives the designated operation category input from the input operation device 56 to the category-specific event group determination unit 37; An attention event designation unit 39b for designating an attention event in the event group is provided. There may be a plurality of attention events specified by the attention event specifying section 39b. If the attention event in the operation category designated by the attention operation category designation unit 39a can be defined as the operation event at the final stage of the operation category, the attention event is automatically specified by the designation of the attention operation category. Since it is determined, the attention event designation | designated part 39b can be abbreviate | omitted.

  The implied image representing the content of the event of interest belonging to the operation category designated by the operation category designation unit 39a is synthesized and displayed when the operation image temporally corresponding to the preceding event is reproduced and displayed in FIG. Although schematically shown, in this embodiment, the final event of the operation category is set as the event of interest. In other words, user operations performed on the terminal 1 called X1, for example, “Exit ABC application”, “Select xxx.xls file”, “Duplicate xxx.xls file”,. "Selection", "Write to USB memory", etc. are performed, and FIG. 3 shows an operation image captured in the operation process and an operation event log created in the operation process. When these operation images and operation event log are sent to the terminal monitoring device 3, each operation image and each operation event (cut out from the operation event log) are linked by a link code based on the time stamp assigned to each. Linked. The category-specific event group determination unit 37 evaluates the contents of all operation events linked in this way, and extracts operation events belonging to the operation category specified by the attention operation category specifying unit 39a. Here, an example is shown in which an operation event belonging to the operation category “USB memory writing” is extracted. Event ID: E022 “select xxx.xls file”, event ID: E023 “xxx.xls file” ”, Event ID: E026“ select xxx.xls file ”, and event ID: E027“ write to USB memory ”are extracted and grouped as a group of events by operation category. In addition, the operation events included in the event group by operation category are divided into processing events, intermediate events, and final events, and a link code that defines an operation image corresponding to each operation event in time is additionally added. It is registered in the group registration table 37a.

In this embodiment, the implied image as shown in FIG. 4 in which the word “write file xxx.xls to USB memory”, which is the operation content of the final event designated as the event of interest, is drawn in a balloon frame. Is created by the implied image creation unit 41, and the storage address of the implied image is also registered in the event group registration table 37a. In addition, the countdown variable inserted in parentheses in the wording is a variable for displaying the calculated value of the time until the final event as the event of interest occurs, depending on the occurrence time of the operation event as the starting point The contents of the variable, that is, the calculated value (countdown value) will be different. In addition, when the playback speed in the operation image playback is the actual speed, the countdown value may be the difference in time when each operation event occurs, but when the playback speed is not the actual speed, that is, in the case of slow playback or fast forward playback, The countdown value may be changed in accordance with the playback speed so that the actual time is increased so that it becomes longer during slow playback and shorter during fast forward playback. As a result, it is possible to grasp the time until the final event as the event of interest occurs in actual time without being aware of whether the playback screen is slow playback or fast-forward playback slow .

  As shown in FIG. 3, when an operation category-specific event group is registered in the event group registration table 37a, when the operation image is reproduced, a screen as shown in FIG. When the operation image linked to the preceding event (event ID: E022) is to be reproduced, an implied image in which the time from the preceding event to the final event that is the attention event is given to the countdown variable is the operation image. The image is displayed on the monitoring monitor 55 in a form combined with the image (see FIG. 5A). From the screen of the monitor monitor 55, the monitor can know that the file xxx.xls will be written to the USB memory after 20 seconds, and the concentration of the monitor can be enhanced. Further, the operation image linked to the intermediate event (event ID: E026) is reproduced (see FIG. 5B). Here again, the same implied image is combined with the operation image and displayed, but only the countdown value is recalculated and rewritten to the correct value. FIG. 5C shows a state in which the operation image linked to the final event (event ID: E027), which is the event of interest, is being reproduced. Here, as an implicit image, there is no countdown value. , A balloon indicating only the content “file xxx.xls is written to USB memory” is displayed.

  In the example described with reference to FIG. 3 and FIG. 5, the event of interest is one of the final events, but as the event of interest, one of the intermediate events and the final event are designated, or the intermediate event is A plurality of operation events may be specified as the event of interest, such as specifying a plurality of events. In that case, a plurality of implicit images representing the contents of a plurality of events of interest are combined and displayed when the operation image corresponding to the preceding event preceding the event of interest is reproduced.

  Also, the importance level may be set in advance for the event of interest, and the form of the implied image representing the content of the event of interest displayed during the reproduction of the operation image may be changed according to the importance level. For example, it is convenient that the implied image for the attention event with high importance is enlarged or flashed and displayed in a more emphasized form, so that the confirmation is easy.

  The terminal monitoring device 3 stores the operation image and the operation event log for each terminal 1 acquired by the operation image acquisition unit 33 and the operation event log acquisition unit 32 in the data storage unit 57 as described above. In addition to the mode for selectively reproducing, various display modes for terminal monitoring such as a mode for directly displaying the operation image and the operation event log on the monitoring monitor 51 before storing them in the data storage unit 57 are provided. FIG. 6 shows a terminal monitoring basic screen 100 displayed on the monitoring monitor 55 in order to select a desired mode from such various display modes. On the terminal monitoring basic screen 100, an operation image playback button 101, a log browsing button 102, a PC information button 103, a remote control button 104, an alert setting button 105, and the like are arranged in the upper position in the form of icons. At the bottom of the basic screen 100, an operation image display area 110 for each terminal, a message display unit 111, and a frame 120 for displaying a real-time operation image of each terminal 1 are displayed in a matrix. A terminal identification display unit 121 indicating a terminal name or a terminal ID is arranged at the bottom.

  When the operation image playback button 101 is operated on the terminal monitoring basic screen 100, the operation image of the terminal 1 selected at that time is read from the operation image storage unit 57b as shown in FIG. A replay screen 140 is displayed. Before operating the operation image playback button 101, the terminal 1 displayed in the operation image display area 110 or the terminal list display unit 112 of the terminal monitoring basic screen 100 is selected. In this case, a dialog box (not shown) may be displayed, and information specifying the terminal 1 may be input from the input operation device 56. The playback screen 140 includes a noticed operation category designation field 141, a playback start date / time input unit 142, a playback end date / time input unit 143, an OK button 144, a playback area 145, a plurality of playback control buttons 146, a playback A slider 147 is arranged.

  A typical method for displaying the operation image on the playback screen 140 is to input the playback start date and time from the keyboard or the like in the playback start date and time input unit 142 and input the playback end time to the playback end date and time input unit 143 as necessary. After that, the user clicks the OK button 144. As a result, the specified operation image of the terminal 1 is read from the operation image storage unit 57 b by the operation image management unit 36 and displayed in the reproduction area 145 in time series. At that time, as described above, the timing at which an operation event belonging to the attention operation category input through the attention operation category designation field 141 and set by the attention operation category designation unit 39a, that is, an operation event included in the event group for each operation category occurs. Thus, an implied image indicating the content of the final event of the event group for each operation category is displayed together with the operation image.

  When the log browsing button 102 is operated on the basic screen 100 shown in FIG. 6, the operation event log list is displayed in a separate window. When the PC information button 103 is operated on the basic screen 100, a process of displaying PC information such as physical data of the selected terminal 1 and an installed application name is performed. When the remote control button 104 is operated on the basic screen 100, it is possible to remotely control individual terminals by operating icons or executing commands on the remote control screen.

  In the above-described embodiment, operation images and operation event logs in a large number of terminals 1 are sent to the terminal monitoring device 3 via the network 2, and the operation images are reproduced and displayed as necessary on the monitoring monitor 55 of the terminal monitoring device 3. Is done. However, the operation image reproduction function according to the present invention is modularized to be an application that can be executed on the terminal 1 and installed in each terminal 1, so that the terminal 1 itself becomes a stand-alone terminal with an operation image reproduction function (that is, a terminal). A terminal incorporating a monitoring device). Compared to the server / client type terminal monitoring system described above, there is a difference that the operation event log and the operation image are transmitted via the network 2 via the personal computer data bus. The content is the same.

  The term operation image used in this specification refers to a still image file obtained by capturing an operation screen, a collection of still image files, and a moving image obtained by converting continuous still images into animations. A collection of files and video files. Therefore, the operation image transferred from the terminal 1 to the terminal monitoring device 3 may be a still image or a moving image. Further, when the operation image is displayed on the monitoring monitor 55, a continuous display of a still image (style like a slide show) or a moving image display may be used, and an appropriate still image-moving program or moving image- A still image program is implemented. Furthermore, various types of capture images can be selected when recorded in the operation image storage unit 57b. For example, JPEG or GIF is used for still images, and MPEG or AVI is used for moving images.

  In the above-described embodiment, the terminal monitoring device 3 has been described as being configured by a single computer. However, the functions of the terminal monitoring device 3 may be divided and distributed to a plurality of computers. For example, the data storage unit 57 in the terminal monitoring device 3 is allocated to a dedicated data storage device such as a file server, and an operation image or an operation event is read from the dedicated data storage device, and the above-described processing is performed and displayed on the monitoring monitor. A monitoring terminal may be arranged. By connecting this monitoring terminal to a network incorporating a dedicated data storage device, the terminal can be monitored anytime and anywhere.

Explanatory drawing explaining the basic principle of operation screen reproduction by this invention The system block diagram which shows an example of the terminal monitoring system by this invention Explanatory drawing explaining the flow which synthesizes an implied image related to a specific operation event on the operation screen Schematic diagram showing a balloon as a form of implied image Screen diagram showing the operation image with the implied image combined Screen diagram showing the terminal monitoring basic screen Screen diagram showing the playback screen

Explanation of symbols

1: terminal 3: terminal monitoring device 13: operation screen capture unit 14: operation event log generation unit 32: operation event log acquisition unit 33: operation image acquisition unit 34: link unit 37: event group determination unit 37a by operation category: event Group registration table 38: Operation event management unit 39a: Attention operation category designation unit 39b: Attention event designation unit 40: Monitor information processing means 41: Implicit image creation unit 42: Image composition unit 43: Operation image reproduction unit 45: Interpolation operation image Generation unit 57: Data storage unit 57a: Operation event storage unit 57b: Operation image management unit 55: Monitoring monitor

Claims (2)

In the operation image reproduction device that acquires the operation image obtained by capturing the operation screen in the terminal and the operation event log describing the operation event in the terminal, and reproduces the operation image,
A data storage unit for storing the operation image and an operation event included in the operation event log;
The operation event is classified by operation category, and at least an attention event as one or more operation events indicating a notable operation in the operation category and a preceding event as an operation event indicating an operation preceding the attention event are at least A category-specific event group determination unit for grouping as an operation category-specific event group including,
An implied image implying the content of the event of interest of the event group for each operation category including an operation event corresponding in time to the reproduction operation image that is the operation image to be reproduced as a preceding operation event is output together with the reproduction operation image Replay information processing means,
An operation image reproduction apparatus provided with In an operation image reproduction program for acquiring an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal, storing the operation image in a data storage unit, and reproducing the operation image ,
A function of storing the operation image and the operation event included in the operation event log in the data storage unit;
The operation event is classified by operation category, and at least an attention event as one or more operation events indicating a notable operation in the operation category and a preceding event as an operation event indicating an operation preceding the attention event are at least A grouping function as a group of events by operation category,
An implied image implying the content of the event of interest of the event group for each operation category including an operation event corresponding in time to the reproduction operation image that is the operation image to be reproduced as a preceding operation event is output together with the reproduction operation image Function to
An operation image reproduction program for causing a computer to realize the above.

Images