JP2013174964A - Information processing device and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the time of an operation in an information processing device on an operation side to be recognized in an information processing device on an observation side.SOLUTION: An information processing device comprises: a reception unit that receives screen information indicating a screen displayed on another information processing device that performs screen display in accordance with input operation and an operation signal indicating the status of input operation which are transmitted from the other information processing device; a generation unit that generates an operation status notification image indicating operation status on the basis of the type of the received operation signal; and a drawing unit that draws the generated operation status notification image on the screen indicated by the received screen information.

Description

  Embodiments described herein relate generally to an information processing apparatus and an information processing method for receiving and displaying screen information displayed on another information processing apparatus.

  2. Description of the Related Art Conventionally, a screen transfer system that displays a screen of a remote terminal connected via a network on a monitor and display of a terminal at hand is known. It is possible to monitor changes in the computer screen remotely, and the locus of the mouse cursor on the screen is also transferred as part of the screen, so it can be applied as a remote monitoring system that remotely monitors other people's operations. The

  For example, it is also possible to construct a training system in which a trainer's screen operation is monitored by a leader in an adjacent room or other building using a screen transfer system. Such a screen transfer system is also used as a presentation application added to a video conference system and the like, and the audience can know the position indicated by the mouse pointer on the screen by the presenter.

  In the screen transfer system, screen information is transmitted from a screen transmission side terminal to a screen reception side terminal by a screen transfer protocol. For example, an RFB (Remote Frame Buffer) protocol is used in a VNC (Virtual Network Computing) system which is one of screen transfer systems.

The RFB Protocol Version 3.8 Last updated 26 February 2009

  A screen transfer system that can confirm a remote screen and operation status is called a remote screen monitoring system. As shown in FIG. 26, the observation side terminal (screen receiving terminal) (PC2) can recognize the movement of the mouse cursor executed on the operation side terminal (screen transmission terminal) (PC1). The state of mouse clicks and key operations at the screen transmission terminal, that is, when these operations are performed cannot be recognized by the screen reception terminal on the observation side.

  The problem to be solved by the present invention is to provide an information processing apparatus and an information processing method capable of recognizing when an operation on the operation-side screen transmission terminal is performed by the observation-side screen reception terminal. There is.

  According to the embodiment, the information processing apparatus transmits screen information indicating a screen displayed on the information processing apparatus transmitted from another information processing apparatus that performs screen display according to the input operation, and input in the information processing apparatus A receiving unit that receives an operation signal indicating an operation status, a generation unit that generates an operation status notification image indicating an operation status in the information processing device based on the type of the received operation signal, and the generated A drawing unit that draws an operation state notification image on the screen indicated by the received screen information.

  According to the present invention, it is possible to recognize at the screen receiving terminal on the observation side when the operation on the screen transmitting terminal on the operating side has been performed.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a simplified diagram showing a remote screen monitoring system (screen transfer system) in a first embodiment. The block diagram which shows the function structural example of the screen transmission terminal of the remote screen monitoring system in 1st Embodiment. The block diagram which shows the function structural example of the screen receiving terminal of the remote screen monitoring system in 1st Embodiment. The block diagram which shows the function structural example of the screen holding | maintenance part 23 of the screen receiving terminal 2 of the remote screen monitoring system in 1st Embodiment. The figure which shows an example of the event generation | occurrence | production position notification image and mouse image in the remote screen monitoring system in 1st Embodiment. The flowchart which shows an example of the process sequence by the screen transmission terminal of the remote screen monitoring system in 1st Embodiment. The flowchart which shows an example of the process sequence by the screen receiving terminal of the remote screen monitoring system in 1st Embodiment. The figure explaining an example of operation | movement of the remote screen monitoring system in 1st Embodiment. The block diagram which shows the function structural example of the screen transmission terminal of the remote screen monitoring system in 2nd Embodiment. The block diagram which shows the function structural example of the screen receiving terminal of the remote screen monitoring system in 2nd Embodiment. The block diagram which shows the function structural example of the screen holding | maintenance part 23 of the screen receiving terminal 2 of the remote screen monitoring system in 2nd Embodiment. The figure which shows an example of the event generation | occurrence | production position notification image and mouse image in the remote screen monitoring system in 2nd Embodiment. The flowchart which shows an example of the process sequence by the screen transmission terminal of the remote screen monitoring system in 2nd Embodiment. The flowchart which shows an example of the process sequence by the screen receiving terminal of the remote screen monitoring system in 2nd Embodiment. The figure explaining an example of operation | movement of the remote screen monitoring system in 3rd Embodiment. The block diagram which shows the function structural example of the screen holding | maintenance part 23 of the screen receiving terminal of the remote screen monitoring system in 3rd Embodiment. The figure which shows an example of the key image in the remote screen monitoring system in 4th Embodiment. The flowchart which shows an example of the process sequence by the screen receiving terminal of the remote screen monitoring system in 4th Embodiment. The block diagram which shows the function structural example of 1 A of screen transmission terminals of the remote screen monitoring system in 5th Embodiment. The block diagram which shows the function structural example of 2 A of screen receiving terminals of the remote screen monitoring system in 5th Embodiment. The figure which shows an example of the audio | voice image used with the remote screen monitoring system in 5th Embodiment. The flowchart which shows an example of the process sequence by the screen transmission terminal of the remote screen monitoring system in 5th Embodiment. The flowchart which shows an example of the process sequence by the screen receiving terminal of the remote screen monitoring system in 5th Embodiment. The block diagram which shows the function structural example of 1 A of screen transmission terminals of the remote screen monitoring system in 6th Embodiment. The block diagram which shows the function structural example of 2 A of screen receiving terminals of the remote screen monitoring system in 6th Embodiment. The figure which shows an example of each usage form of the screen transmission terminal and screen receiving terminal of a remote monitoring system.

Hereinafter, embodiments will be described with reference to the drawings.
In the present embodiment, the screen information indicating the screen displayed on the screen transmission terminal and the operation signal indicating the status of the input operation on the screen transmission terminal transmitted from the screen transmission terminal that performs screen display according to the input operation are displayed on the screen. Received by the receiving terminal, the screen receiving terminal generates an operation status notification image indicating the operating status at the screen transmitting terminal based on the type of the received operation signal, and received the generated operation status notification image It is characterized by drawing on a screen indicated by screen information.

  In the present embodiment, it is possible to recognize the position and timing when the mouse is clicked on the operation-side screen transmission terminal on the screen of the observation-side screen reception terminal. For example, it is possible to recognize that a mouse click has been made at an incorrect location or an unnecessary mouse click, and as a usage scene, an appropriate instruction is given from the observer of the screen receiving terminal to the operator of the screen transmitting terminal. Is also possible.

(First embodiment)
First, the first embodiment will be described.
FIG. 1 is a simplified diagram showing a remote screen monitoring system (hereinafter referred to as a screen transfer system) in the first embodiment.
In this screen transfer system, a screen transmission terminal 1 and a screen reception terminal 2 are connected via a network 3. The screen transmission terminal 1 is a device (screen transfer server device) that transmits screen information in the screen transfer system, and the screen reception terminal 2 is a device (screen transfer device) that receives screen information in the screen transfer system. Client device). The network 3 here is a LAN or the Internet, and may be wired or wireless.

  In this embodiment, since the screen transfer system is described, the two devices shown in FIG. 1 are called the screen transmission terminal 1 and the screen reception terminal 2, but for example, between two notebook PCs When the screen is transferred, the notebook PC that transmits the screen is the screen transmitting terminal 1 and the notebook PC that receives the screen is the screen receiving terminal 2.

  In the screen transmission terminal 1, application software for generating screen information is operating. In the present embodiment, a screen information transmission function is incorporated in the screen transmission terminal 1, and the screen information generated by the application software is transmitted to the screen reception terminal 2 via the network 3. As the screen transmission function of the screen transmission terminal 1, screen transfer using a push-type screen transfer protocol is used in the present embodiment.

  In the push type, the screen transmission side (screen transmission terminal 1 in the case of this embodiment) does not require a screen acquisition request from the screen reception side (screen reception terminal 2 in the case of this embodiment). Sends screen information to the screen receiver. On the other hand, the screen receiving terminal 2 receives the screen information from the screen transmitting terminal 1, and generates and draws a screen to be displayed on the display device of the screen receiving terminal 2 from the received screen information. The screen receiving function on the screen receiving terminal 2 side corresponds to the screen transmitting function on the screen transmitting terminal 1 side.

  As described above, in the screen transfer system shown in FIG. 1, the screen information generated by the application software in the screen transmission terminal 1 is transmitted by the screen transfer protocol and displayed on the display device of the screen reception terminal 2 that receives the screen information. The

  The screen transfer protocol is not limited to the push type, and may be a pull type as will be described later. For example, in the screen transfer method known as VNC described above, the screen receiving side (screen receiving terminal 2 in this embodiment) sends a screen acquisition request called FramebufferUpdateRequest to the screen transmitting side (screen sending terminal 1 in this embodiment). Is sent to the screen receiving side as a response called FramebufferUpdate.

  When VNC is used as a pull-type screen transfer protocol, “VNC server” software is activated on the screen transmission terminal 1 side, and “VNC client” software is activated on the screen reception terminal 2 side.

FIG. 2 is a block diagram illustrating a functional configuration example of the screen transmission terminal 1 of the remote screen monitoring system according to the first embodiment. However, FIG. 2 is a schematic diagram showing only the functions related to the features of the present embodiment compared to the prior art.
The screen transmission terminal 1 includes a screen change detection unit 11-1, a transmission image data generation unit 12-1, a transmission unit 13-1, and an operation event detection unit 14-1. In addition, a time detection that is a feature of the present embodiment is provided. A storage device 31 which is a storage medium such as a unit 15-1, a time information addition unit 16-1, an operation signal generation unit 17-1, and a nonvolatile memory.

  The screen change detection unit 11-1 detects a changed part in the screen update that has occurred on the screen of the screen transmission terminal 1. For example, the screen change detection unit 11-1 stores the previous screen information, which is the screen information before the update, in the frame buffer in the storage device 31, and the stored previous screen information and the current state after the update. The difference between the previous screen information and the current screen information is detected by comparing the pixels with the screen information, and the detected difference is stored in the storage device 31 as update screen data to update the screen. The transmission image data generation unit 12-1 is notified of data generation.

  Upon receiving notification of screen update data generation, the transmission image data generation unit 12-1 acquires the update screen data stored in the storage device 31, performs data compression, and uses the compressed screen data obtained by this data compression. The data is output to the transmission unit 13-1.

  The transmission unit 13-1 has a function of transmitting and receiving data using a network protocol such as TCP / IP. The transmission unit 13-1 packetizes the compressed screen data output from the transmission image data generation unit 12-1, and uses the packetized compressed screen data using a connection set with the screen receiving terminal 2. To the screen receiving terminal 2.

  The operation event detection unit 14-1 detects an operation event from an input interface such as a mouse of the screen transmission terminal 1. The input interface is not limited to a mouse, and may be a keyboard. Examples of the operation event include a mouse operation event indicating a mouse operation such as mouse cursor movement, single click, or double click, a key operation event indicating a pressed keyboard key, and the like. Hereinafter, a single click is simply referred to as a click.

  These operation events are notified to the OS (operating system) and application software running on the screen transmission terminal 1, and as a result, the contents drawn on the screen are updated according to the specifications of the OS and application software. For example, when a mouse cursor movement is notified to the screen receiving terminal 2 by a mouse operation event, a movement of the mouse cursor moving in the screen of the screen receiving terminal 2 is drawn.

  Further, when an application software window is displayed on the screen of the screen transmission terminal 1, for example, a mouse is used on a button for minimizing the window arranged at the upper right of the window or a button for closing the window. When a click event occurs, the window is minimized or closed in the screen. When a playback button is displayed on the screen of the screen transmission terminal 1 in the video playback application, video playback starts when a mouse click event occurs on the playback button.

  That is, the contents of the screen are updated as a result of some operation being performed with the mouse, keyboard, or other external input interface on the screen transmission terminal 1 side. When the contents of the screen are updated, the screen information is transmitted to the screen receiving terminal 2 by the screen change detection unit 11-1, the transmission image data generation unit 12-1, and the transmission unit 13-1.

  Next, the time detection unit 15-1, the time information addition unit 16-1, and the operation signal generation unit 17-1 which are features of the present embodiment will be described. In the present embodiment, the following description will be made assuming that the external input interface is a mouse.

  First, in the screen transmission terminal 1, the operation event detecting unit 14-1 has a function of detecting the occurrence of an operation event and notifying the operation signal generating unit 17-1 of operation event information indicating the generated operation event.

  The time detection unit 15-1 detects the time when the screen change detection unit 11-1 notifies the transmission image data generation unit 12-1 that the screen update data has been generated, and outputs the detected time to the time information addition unit 16-1. Further, the time detection unit 15-1 detects the time at which the operation event detection unit 14-1 notifies the operation signal information to the operation signal generation unit 17-1, and outputs the detected time to the time information addition unit 16-1. In the present embodiment, description will be made assuming that the time detection unit 15-1 outputs the time information of the click to the time information adding unit 16-1 only when the operation event is a mouse click operation event.

  When the time information regarding the generation of the screen update data is input from the time detection unit 15-1, the time information addition unit 16-1 observes the generation process of the compressed screen data in the transmission image data generation unit 12-1. When the time information addition unit 16-1 observes that the compressed image data corresponding to the time information from the time detection unit 15-1 is generated by the transmission image data generation unit 12-1, the transmission image data generation unit In 12-1, the input time information is output to the transmission image data generation unit 12-1 so that the compressed screen data and the time information are associated with each other.

  The transmission image data generation unit 12-1 outputs the compressed screen data and time information corresponding to the compressed screen data to the transmission unit 13-1. Further, the transmission unit 13-1 transmits the compressed screen data output from the transmission image data generation unit 12-1 to the screen reception terminal 2 together with time information as screen information.

  When the operation event information is notified from the operation event detection unit 14-1, the operation signal generation unit 17-1 confirms the type of the operation event and the coordinate information on the screen on which the operation event has occurred, and information about them. Is generated, and the generated operation signal is output to the transmitter 13-1. In the present embodiment, as a result of confirming the type of operation event, an operation signal is generated and output to the transmission unit 13-1 only when the mouse is clicked among mouse operation events.

  When the time information addition unit 16-1 described above inputs time information related to a mouse click among mouse operation events from the operation event detection unit 14-1, the operation signal generation unit 17-1 performs an operation signal generation process. Observe. When the time information adding unit 16-1 observes that the operation signal corresponding to the time information is generated by the operation event detecting unit 14-1, the operation signal generating unit 17-1 corresponds the operation signal to the time information. As described, the input time information is output to the operation signal generator 17-1.

  Note that the operation signal generation unit 17-1 described above outputs the operation signal and time information corresponding to the operation signal to the transmission unit 13-1. The transmission unit 13-1 transmits the output operation signal and time information together to the screen receiving terminal 2.

FIG. 3 is a block diagram illustrating a functional configuration example of the screen receiving terminal of the remote screen monitoring system according to the first embodiment. However, FIG. 3 is a schematic diagram showing only functions related to features of the prior art.
The screen receiving terminal 2 includes a receiving unit 21-1, a screen constructing unit 22-1, a screen holding unit 23-1, a drawing unit 24-1, a display time adjusting unit 25-1, which is a feature of the present embodiment, A display control unit 26-1, an operation signal analysis unit 27-1, and an operation state notification image generation unit 28-1 are provided.

  The reception unit 21-1 has a function of transmitting and receiving data using a network protocol (NTP) such as TCP / IP, and constructs a screen for screen information received using a connection set with the screen transmission terminal 1. To the unit 22-1.

  The screen construction unit 22-1 decompresses the data of the screen information input from the reception unit 21-1, and writes this screen information in the frame buffer in the screen holding unit 23-1. This frame buffer is a buffer for holding screen information. The drawing unit 24-1 has a function of displaying screen information held in the frame buffer of the screen holding unit 23-1 on an external display device.

Next, the display time adjustment unit 25-1, the display control unit 26-1, the operation signal analysis unit 27-1, and the operation status notification image generation unit 28-1 which are features of the present embodiment will be described.
When the receiving unit 21-1 receives the time information together with the screen information, the display time adjusting unit 25-1 determines the delay time T1 with respect to this time, and sends the information on the determined delay time T1 to the display control unit 26-1. Output. The display time adjustment unit 25-1 has a function of synchronizing the time measured by the clock of the screen transmission terminal 1 that is the communication partner and the time measured by the clock of the screen reception terminal 2 that is the own device. For example, the display time adjustment unit 25-1 may estimate the time information measured by the screen transmission terminal 1 from the time information received from the screen transmission terminal 1 together with the screen information.

  As a simple example, the display time adjustment unit 25-1 may use the time information initially received together with the screen information as the current time measured by the own device, or estimate the transmission delay in the network from the time information. A value in consideration of the value may be the current time measured by the own device. Note that the synchronization mechanism is not limited to these, and the screen transmitting terminal 1 and the screen receiving terminal 2 may use in advance a technique such as adjusting the time measured by NTP in advance.

As a method of determining the delay time T1, for example, a fixed value such as 1 second or 2 seconds may be determined. For example, when the screen information is frequently updated, a buffer that temporarily holds the screen information, which will be described later. The value of the delay time T1 may be shortened in consideration of the size.
In the present embodiment, the display control unit 26-1 has only a function of notifying the screen holding unit 23-1 of the delay time T1 when the delay time T1 is input from the display time adjusting unit 25-1.

FIG. 4 is a block diagram illustrating a functional configuration example of the screen holding unit 23 of the screen receiving terminal 2 of the remote screen monitoring system according to the first embodiment.
On the other hand, in the present embodiment, the screen holding unit 23-1 includes a buffer 23-1A in the previous stage of the frame buffer 23-1B, as shown in FIG. This buffer 23-1A temporarily holds the screen information from the screen construction unit 22-1. It is assumed that the temporarily held time corresponds to the delay time T1, and the screen information is written in the frame buffer 23-1B after the delay time T1 has elapsed since the hold. The drawing unit 24-1 acquires the screen information written in the frame buffer 23-1B of the screen holding unit 23 and displays it on the display device.

  When the receiving unit 21-1 receives the operation signal, the operation signal analyzing unit 27-1 analyzes the received operation signal. Specifically, the type of operation event, the coordinate information on the screen where the event occurred, and the time information when the operation event occurred are detected, and these pieces of information are sent to the operation status notification image generation unit 28-1. Output.

  The operation signal analysis unit 27-1 takes into account the size and display area difference between the screen of the screen transmission terminal 1 and the screen of the display device of the screen reception terminal 2 for the coordinate information. It may be output to the operation status notification image generation unit 28-1 after processing such as conversion to appropriate coordinate information on the screen of the display device.

  When the operation status notification image generation unit 28-1 inputs the type of the operation event, the coordinate information on the screen where the event occurred, and the time information when the operation event occurred, from the operation signal analysis unit 27-1, Check the type of operation event.

  In the present embodiment, since only the mouse click when the operation event is a mouse operation event is targeted, the operation situation notification image generation unit 28-1 generates an operation situation notification image when the operation event is a mouse click. The screen holding unit 23-is an image stored in advance in the internal memory of the unit 28-1, that is, a mouse image A indicating a mouse click state and an event occurrence notification image for notifying the event occurrence position. 1 is written in the frame buffer 23-1B.

  However, with respect to the event occurrence position notification image, the operation status notification image generation unit 28-1 is displayed at the coordinates on the screen where the event has occurred on the screen of the display device on the screen receiving terminal 2 side, and The mouse image A is written in the frame buffer 23-1B of the screen holding unit 23-1 so as to be displayed in the vicinity near the display location of the event occurrence position notification image.

FIG. 5 is a diagram illustrating an example of an event occurrence position notification image and a mouse image in the remote screen monitoring system according to the first embodiment.
FIG. 5A shows an example of the event occurrence position notification image and the mouse image A and an example of their relative display positions. In FIG. 5A, the event occurrence position notification image is “x”, but the center position of this “x” is the coordinates on the screen where the event occurred.

  Here, the operation status notification image generation unit 28-1 notifies the drawing unit 24-1 that the event occurrence position notification image and the mouse image A have been written to the frame buffer 23-1B of the screen holding unit 23-1. . The drawing unit 24-1 acquires the written screen information and displays it on the display device. It should be noted that the event occurrence position notification image and the mouse image A may be regularly displayed on the display device so as to be periodically written to the frame buffer 23-1B until the delay time T1 elapses.

  Further, the operation status notification image generation unit 28-1 obtains the information on the delay time T1 determined by the display time adjustment unit 25-1 from the display time adjustment unit 25-1, and then receives the time information received together with the operation signal. After the delay time T1 elapses, the mouse image B is displayed in the frame buffer 23-1B of the screen holding unit 23-1 so that the mouse image B shown in FIG. 5B is displayed on the display device by the drawing unit 24-1. Write.

  In this case, the operation state notification image generation unit 28-1 writes the mouse image B into the frame buffer 23-1B of the screen holding unit 23-1 after the delay time T1 has elapsed, or frames the mouse image B after the delay time T1 has elapsed. There is a method in which the buffer 23-1A in the previous stage of the buffer 23-1B is temporarily held, and the screen holding unit 23-1 writes it in the frame buffer 23-1B after the delay time T1 has elapsed.

FIG. 6 is a flowchart illustrating an example of a processing procedure by the screen transmission terminal of the remote screen monitoring system according to the first embodiment.
First, when the screen change detection unit 11-1 of the screen transmission terminal 1 detects a screen change (step S1), the screen change data generation unit 12-1 is notified of the screen update data generation (step S2). The time detection unit 15-1 detects the notification time of the screen change and outputs it to the time information addition unit 16-1 (step S3). When the transmission image data generation unit 12-1 acquires the update screen data from the screen change detection unit 11-1 (step S4), the transmission image data generation unit 12-1 performs data compression and compressed screen data generation (step S5). The time information adding unit 16-1 outputs the time information input in step S3 to the transmission image data generating unit 12-1 (step S6). Then, the transmission image data generation unit 12-1 outputs the compressed screen data and time information to the transmission unit 13-1 (Step S7). The transmission unit 13-1 transmits the compressed screen data and the time information from the transmission image data generation unit 12-1 to the screen reception terminal 2 (step S8).

  When the operation event detection unit 14-1 of the screen transmission terminal 1 detects an operation event (step S11), the operation event information is notified to the operation signal generation unit 17-1 (step S12). The time detection unit 15-1 detects the time when the operation event has occurred and outputs it to the time information addition unit 16-1 (step S13). When obtaining the operation event information notification, the operation signal generation unit 17-1 confirms the operation event type and coordinate information (step S14), and generates an operation signal indicating these pieces of information (step S15). The time information adding unit 16-1 outputs the time information input in Step S13 to the operation signal generating unit 17-1 (Step S16). Then, the operation signal generation part 17-1 outputs an operation signal and time information to the transmission part 13-1 (step S17). The transmission unit 13-1 transmits the operation signal and time information from the operation signal generation unit 17-1 to the screen receiving terminal 2 (step S18).

FIG. 7 is a flowchart illustrating an example of a processing procedure performed by the screen receiving terminal of the remote screen monitoring system according to the first embodiment.
First, when the receiving unit 21-1 of the screen receiving terminal 2 receives the compressed screen data and time information (step S21), the display time adjusting unit 25-1 determines a delay time T1 for the time information and displays the display control unit. It outputs to 26-1. The display control unit 26-1 notifies the delay time T1 to the screen holding unit 23-1 (step S22).

  The screen construction unit 22-1 decompresses the screen information data for the received compressed screen data and outputs the decompressed screen information data to the screen holding unit 23-1 (step S23). The screen holding unit 23-1 writes and holds the screen information from the screen construction unit 22-1 in the buffer 23-1A (step S24).

When the delay time T1 notified in step S22 has elapsed (step S25), the screen holding unit 23-1 writes the screen information held in the buffer 23-1A into the frame buffer 23-1B (step S26).
The drawing unit 24-1 displays the screen information written in the frame buffer 23-1B as a difference image on the display device (step S27).

  Further, when the receiving unit 21-1 of the screen receiving terminal 2 receives the operation signal (step S31). The operation signal analysis unit 27-1 analyzes the operation signal (step S32), and outputs the operation event type, coordinate information, and time information obtained by this analysis to the operation state notification image generation unit 28-1 (step S33). .

  Then, the operation status notification image generation unit 28-1 recognizes that the operation event is a mouse click (step S34), writes the event occurrence position notification image and the mouse image A to the frame buffer 23-1B (step S35), This writing is notified to the drawing unit 24-1 (step S36). As a result, a notification image is added to the screen information. The drawing unit 24-1 displays the event occurrence position notification image and the mouse image written in the frame buffer 23-1B on the display device (step S37).

  Then, when the time T1 has elapsed from the reception time of the operation signal in step S31 (step S38), the screen holding unit 23-1 writes the mouse image B into the frame buffer 23-1B (step S39), and the drawing unit 24- 1 displays the image written in the frame buffer 23-1B of the screen holding unit 23-1 on the display device (step S40).

FIG. 8 is a diagram for explaining an example of the operation of the remote screen monitoring system in the first embodiment. FIG. 8 shows a state in which screen information is transmitted from the screen transmission terminal 1 of this embodiment to the screen reception terminal 2.
Although FIG. 8 shows that the difference image is transmitted and received as the screen information, the processing such as compression and expansion of the screen information is omitted, but the above processing is actually performed. To do. In the screen transmission terminal 1, a mouse operation is performed by the operator, and the position of the mouse cursor changes on the screen. On the other hand, on the screen receiving terminal 2 side, the observer can observe how the position of the mouse cursor changes on the screen of the display device.

  FIG. 8 shows that a difference image D1 is generated at time t11 counted by the screen transmission terminal 1, a difference image D2 is generated at time t12, and a difference image D3 is transmitted at time t14, and these are transmitted as screen information. . Further, although a mouse click is performed at time t13, there is no screen change at this timing, no difference image is generated, and no screen information is transmitted.

  In FIG. 8, the screen receiving terminal 2 receives the difference image D1 transmitted from the screen transmitting terminal 1 as screen information at time t21 timed by the screen receiving terminal 2, and the difference image transmitted from the screen transmitting terminal 1. D2 is received as screen information at time t22 timed by the screen receiving terminal 2, and the difference image D3 transmitted from the screen transmitting terminal 1 is received as screen information at time t23 timed by the screen receiving terminal 2. It is. The screen receiving terminal 2 shows a state in which the received screen information is displayed on the display device after the delay time T1 has elapsed from the reception time of the screen information by the control processing of the present embodiment.

In FIG. 8, a mouse click occurs at time t <b> 13 timed by the screen transmission terminal 1, an operation signal is transmitted to the screen reception terminal 2, and this signal is received at time t <b> 23 timed by the screen reception terminal 2. Is shown. When the screen receiving terminal 2 recognizes that the operation signal is an operation signal related to a mouse click by the control processing of the present embodiment, the event occurrence position notification image and the mouse are recognized at the time of recognition without waiting for the delay time T1 to elapse. An image A is displayed on the display device.
Further, FIG. 8 shows that the screen receiving terminal 2 displays the event occurrence position notification image and the mouse image B on the display device after the delay time T1 has elapsed from the reception time of the operation signal by the control processing of the present embodiment. ing.

  Here, a case is described in which the time t13 when the mouse click occurs at the screen transmission terminal 1 is later than the time t21 when the difference image D1 is received at the screen reception terminal 2 plus the delay time T1. Not limited to this, the time t13 may be earlier than the time obtained by adding the delay time T1 to the time t21. In this case, the display device on the screen receiving terminal 2 side adds the delay time T1 to the time t21. At the time, the mouse image before the mouse click is displayed together with the difference image D1.

  The observer at the screen receiving terminal 2 observes that the position of the mouse cursor changes on the screen after a delay time T1 from the reception at the screen receiving terminal 2. In such a situation, since the event occurrence position notification image and the mouse image A are displayed when the operation signal is received by the screen receiving terminal 2, the observer is delayed from the timing at which the screen on which the mouse click is performed is displayed. The mouse image A before the mouse click can be recognized at a timing earlier by T1. Then, the observer can know the relationship between the mouse clicked screen and its position by displaying the event occurrence position notification image and the mouse image B on the display device after T1 has elapsed from the display of the mouse image A. it can.

  In general, the mouse click is instantaneous, but according to the present embodiment, the observer on the screen receiving terminal 2 side can know in advance that the mouse is clicked on the screen. It is also possible to continue displaying the information on the screen for the delay time T1, and the observer can more easily recognize the mouse click operation of the operator.

(Second Embodiment)
Next, a second embodiment will be described. In addition, description of the same part as what was shown in 1st Embodiment among the structures of the screen transfer system in each following embodiment is abbreviate | omitted.
In the second embodiment, the screen transmitting terminal 1 and the screen receiving terminal 2 shown in the first embodiment have different internal configurations.

FIG. 9 is a block diagram illustrating a functional configuration example of the screen transmission terminal 1 of the remote screen monitoring system according to the second embodiment. However, FIG. 9 is a schematic diagram showing only functions related to the features of the present embodiment with respect to the prior art.
The screen transmission terminal 1 includes a screen change detection unit 11-2, a transmission image data generation unit 12-2, a transmission unit 13-2, an operation event detection unit 14-2, a time detection unit 15-2, and an operation signal generation unit 17-. 2. A storage device 31 is provided.

  The screen change detection unit 11-2 performs the same processing as the screen change detection unit 11-1 described in the first embodiment, and stores the detected difference image in the storage device 31 as update screen data. When notifying the transmission image data generation unit 12-2 of the occurrence, current time information is acquired from the time detection unit 15-2, and this time information is also notified to the transmission image data generation unit 12-2.

  When the screen change detection unit 11-2 receives the refresh notification from the operation signal generation unit 17-2, the screen change detection unit 11-2 treats the entire screen as update image data, assuming that a change has occurred in the entire screen. The generation and current time information are notified to the transmission image data generation unit 12-2.

  Upon receiving notification of screen update data generation and time information, the transmission image data generation unit 12-2 acquires the update screen data stored in the storage device 31, performs data compression, and creates compressed screen data. The generated compressed screen data and the notified time information are output to the transmission unit 13-2 together.

  The transmission unit 13-2 has a function of transmitting and receiving data using a network protocol such as TCP / IP, packetizes the compressed screen data and time information input from the transmission image data generation unit 12-2, and receives the screen. The screen is transmitted to the screen receiving terminal 2 using a connection established with the terminal 2.

  The operation event detection unit 14-2 performs the same processing as the operation event detection unit 14-1 described in the first embodiment, and when detecting an operation event from an input interface such as a mouse, the time detection unit 15- The current time information is acquired from No. 2, and the operation event information and the time information are notified to the operation signal generation unit 17-2 together.

  When the operation signal generation unit 17-2 receives the notification of the operation event information and the time information from the operation event detection unit 14-2, the operation signal generation unit 17-2 confirms the type of the operation event and the coordinate information on the screen where the event has occurred. Then, an operation signal including the information is generated, and the generated operation signal and the notified time information are output together to the transmission unit 13-2.

  The operation signal generation unit 17-2 has a filtering function for generating an operation signal only for a specific operation event. In the present embodiment, the operation signal generation unit 17-2 generates an operation signal indicating the operation only when the mouse operation event is a mouse click or double click, and outputs the operation signal to the transmission unit 13-2.

  Further, the operation signal generation unit 17-2 notifies the screen change detection unit 11-2 of a refresh notification immediately after generating the operation signal. In addition, the transmission unit 13-2 packetizes the operation signal and the time information, and transmits the packet to the screen reception terminal 2 using a connection set with the screen reception terminal 2.

  The time detection unit 15-2 has a function of acquiring current time information, and provides the current time information to the screen change detection unit 11-2 and the operation event detection unit 14-2. In addition, the time detection unit 15-2 has a function of using a time synchronization protocol such as NTP to match the time measured by the reference clock with the time measured by the own device clock.

FIG. 10 is a block diagram illustrating a functional configuration example of the screen receiving terminal 2 of the remote screen monitoring system according to the second embodiment. However, FIG. 10 is a schematic diagram showing only functions related to the features of the present embodiment with respect to the prior art.
The screen receiving terminal 2 includes a receiving unit 21-2, a screen construction unit 22-2, a screen holding unit 23-2, a drawing unit 24-2, a display time adjusting unit 25-2, a display control unit 26-2, and an operation signal analysis. Unit 27-2 and operation status notification image generation unit 28-2.

  First, the display time adjustment unit 25-2 has a function of using a time synchronization protocol such as NTP to match the time measured by the reference clock with the time measured by the own device clock. The display time adjustment unit 25-2 determines the delay time T1, and outputs information on the delay time T1 to the screen construction unit 22-2 and the operation status notification image generation unit 28-2.

  The receiving unit 21-2 has a function of transmitting and receiving data using a network protocol such as TCP / IP, and displays screen information and time information received using a connection set with the screen transmission terminal 1. Output to the construction unit 22-2.

  The screen construction unit 22-2 performs data expansion of the screen information input from the reception unit 21-2. On the other hand, the screen construction unit 22-2 adds the delay time T1 notified from the display control unit 26-2 to the time information input from the reception unit 21-2. The screen construction unit 22-2 outputs the screen information obtained by decompressing the data and the time information added with the delay time T1 to the screen holding unit 23-2.

FIG. 11 is a block diagram illustrating a functional configuration example of the screen holding unit 23 of the screen receiving terminal 2 of the remote screen monitoring system according to the second embodiment.
As shown in FIG. 11, the screen holding unit 23-2 of the screen receiving terminal 2 according to the present embodiment is provided with a memory 23-2A in the preceding stage of the frame buffer 23-2B, and the time information and screen are stored in the memory 23-2A. Information is stored in association with each other.

The display control unit 26-2 confirms the information stored in the memory 23-2A of the screen holding unit 23-2, confirms the time information held in the memory 23-2A, and is timed with the clock of its own device. When the time becomes the time, the screen information associated with the time is written in the frame buffer 23-2B of the screen holding unit 23-2.
The drawing unit 24-2 has a function of causing the display device to display screen information held in the frame buffer 23-2B.

The operation signal analysis unit 27-2 performs the same processing as the operation signal analysis unit 27-1 described in the first embodiment.
When the operation status notification image generation unit 28-2 receives from the operation signal analysis unit 27-2 the type of operation event, coordinate information on the screen where the event occurred, and time information when the operation event occurred, First, the type of operation event is confirmed.

  In this embodiment, when the operation event is a mouse operation, only mouse clicks and double-clicks are targeted. Therefore, when the operation event is a mouse click, a mouse image indicating the state of a mouse click prepared in advance is displayed. A and an event occurrence notification image for notifying the event occurrence position are written into the frame buffer 23-2B of the screen holding unit 23-2. The event occurrence position notification image and the mouse image A are written in the frame buffer as described in the first embodiment.

FIG. 12 is a diagram illustrating an example of an event occurrence position notification image and a mouse image in the remote screen monitoring system according to the second embodiment.
FIG. 12 shows a mouse image C and a mouse image D which are event occurrence notification images when the operation event is a double click.

  When the operation event is a double click, the operation status notification image generation unit 28-2 stores the mouse image C indicating the double click state and the event occurrence notification image for notifying the event occurrence position in the storage device 31. Are written in the frame buffer 23-2B of the screen holding unit 23-2.

  In addition, the operation status notification image generation unit 28-2 confirms the delay time T1 determined by the display time adjustment unit 25-2. If the operation event is a mouse click, T1 has elapsed from the time information received together with the operation signal. The mouse image B is written in the screen holding unit 23-2 so that the mouse image B is displayed on the display device later by the drawing unit 24-2. That is, the operation status notification image generation unit 28-2 writes the mouse image B into the frame buffer 23-2B of the screen holding unit 23-2 after the delay time T1 has elapsed from the reception time.

  On the other hand, when the operation event is a double click, the operation state notification image generation unit 28-2 causes the drawing unit 24-2 to display the mouse image D on the display device after T1 has elapsed from the time information received together with the operation signal. The mouse image D is written into the frame buffer 23-2B of the screen holding unit 23-2.

FIG. 13 is a flowchart illustrating an example of a processing procedure performed by the screen transmission terminal of the remote screen monitoring system according to the second embodiment.
First, when a screen change is detected (step S1), the screen change detection unit 11-2 of the screen transmission terminal 1 acquires current time information from the time detection unit 15-2 (step S51), and generates transmission image data. The screen update data generation notification and the time information are notified to the section 12-2 (step S2). When the transmission image data generation unit 12-2 obtains the updated screen data together with the time information from the screen change detection unit 11-2 (step S4), the transmission image data generation unit 12-2 performs data compression and compressed screen data generation (step S5). Then, the transmission image data generation unit 12-2 outputs the compressed screen data and the acquired time information to the transmission unit 13-2 (Step S7). The transmission unit 13-2 transmits the compressed screen data and the time information from the transmission image data generation unit 12-2 to the screen reception terminal 2 (Step S8).

  When the operation event detection unit 14-2 of the screen transmission terminal 1 detects an operation event (step S11), the operation event detection unit 14-2 acquires current time information from the time detection unit 15-2 (step S52), and an operation signal generation unit In 17-2, notification of operation event information and notification of time information are performed (step S12). When the operation signal generation unit 17-2 obtains the operation event information notification together with the time information, the operation signal generation unit 17-2 confirms the operation event type and the coordinate information (step S14), and generates an operation signal indicating the information (step S15). Then, the operation signal generation unit 17-2 outputs the operation signal and the acquired time information to the transmission unit 13-2 (Step S17). The transmission unit 13-2 transmits the operation signal and time information from the operation signal generation unit 17-2 to the screen receiving terminal 2 (step S18).

FIG. 14 is a flowchart illustrating an example of a processing procedure performed by the screen receiving terminal of the remote screen monitoring system according to the second embodiment.
First, the receiving unit 21-2 of the screen receiving terminal 2 receives the compressed screen data and time information (step S21). The display time adjustment unit 25-2 determines the delay time T1 for the time information from the screen transmission terminal 1, and outputs the delay time T1 to the screen construction unit 22-2 and the operation status notification image generation unit 28-2 (step S53).

  The screen construction unit 22-2 adds the delay time T1 to the time when the reception unit 21-2 has received the screen information obtained by performing the screen information data expansion on the compressed screen data received by the reception unit 21-2. The time information is output to the screen holding unit 23-2 (step S23). The screen holding unit 23-2 writes and holds the screen information from the screen construction unit 22-2 and the time information after adding the delay time T1 in the memory 23-2A (step S54).

  When the time measured by a clock circuit (not shown) is the time after adding the delay time T1 written in the memory 23-2A (step S55), the screen holding unit 23-2 stores the screen information held in the memory 23-2A. Is written into the frame buffer 23-2B (step S26). The drawing unit 24-2 displays the screen information written in the frame buffer 23-2B on the display device as a difference image (step S27).

  Further, when the receiving unit 21-2 of the screen receiving terminal 2 receives the operation signal (step S31). The operation signal analysis unit 27-2 analyzes the operation signal (step S32), and outputs the operation event type, coordinate information, and time information obtained by this analysis to the operation state notification image generation unit 28-2 (step S33). .

  Then, the operation status notification image generation unit 28-2 determines whether or not the operation event is a mouse click (step S56), and when the operation event is recognized as a mouse click (YES in step S56), the event The occurrence position notification image and the mouse image A are written in the frame buffer 23-2B (step S35), and this writing is notified to the drawing unit 24-2 (step S36). As a result, a notification image is added to the screen information. The drawing unit 24-2 displays the event occurrence position notification image and the mouse image A written in the frame buffer 23-2B on the display device (step S37).

  When the time measured by the clock circuit (not shown) is the time after adding the delay time T1 written in the memory 23-2A (step S38), the screen holding unit 23-2 displays the mouse image B as the frame buffer 23-2B. (Step S39), the drawing unit 24-2 displays the image written in the frame buffer 23-2B of the screen holding unit 23-2 on the display device (step S40).

  If the operation status notification image generation unit 28-2 recognizes that the operation event is a double click (YES in step S61), the event occurrence position notification image and the mouse image C are written in the frame buffer 23-2B (step S61). (S62) This writing is notified to the drawing unit 24-2 (step S63). As a result, a notification image is added to the screen information. The drawing unit 24-2 displays the event occurrence position notification image and the mouse image C written in the frame buffer 23-2B on the display device (step S64).

Then, when the time measured by a clock circuit (not shown) is the time after adding the delay time T1 written in the memory 23-2A (step S65), the screen holding unit 23-2 displays the mouse image D in the frame buffer 23-2B. (Step S66), the drawing unit 24-2 displays the image written in the frame buffer 23-2B of the screen holding unit 23-2 on the display device (step S67).
By performing the processing as described above, as in the first embodiment, the observer of the screen receiving terminal 2 can recognize the mouse image before the mouse click and the mouse image on which the mouse click has been made.

(Third embodiment)
In the first embodiment, a method has been described in which the screen information is displayed on the display device after a delay time T1 from the time when the screen receiving terminal 2 receives the screen information. However, when the screen information is transmitted / received via the network, the arrival time of the screen information fluctuates due to the effects of delay and loss that occur in the network. Therefore, in the third embodiment, a method considering the arrival fluctuation of screen information will be described.

  FIG. 15 is a diagram for explaining an example of the operation of the remote screen monitoring system according to the third embodiment. FIG. 15 shows a state in which screen information is transmitted from the screen transmission terminal 1 to the screen reception terminal 2 in the third embodiment.

  In the present embodiment, first, the time measured by the screen transmission terminal 1 and the time measured by the screen reception terminal 2 are matched in advance using the time synchronization protocol such as NTP. I shall let you. In FIG. 15, the screen receiving terminal 2 displays not the reception time at the screen receiving terminal 2 but the screen information received after the delay time T <b> 1 elapses from the generation time of the differential image at the screen transmitting terminal 1. Is shown. For example, the difference image D1 shown in FIG. 15 is generated at time t11 counted by the screen transmission terminal 1 and transmitted to the screen reception terminal 2. On the other hand, since the time measured by the clock matches the time measured by the clock of the screen transmission terminal 1, the screen receiving terminal 2 adds the delay time T1 to the above-described time t11. Display on the display device at the specified time.

FIG. 16 is a block diagram illustrating a functional configuration example of the screen holding unit 23 of the screen receiving terminal of the remote screen monitoring system according to the third embodiment.
As shown in FIG. 16, the screen holding unit 23-3 of the screen receiving terminal 2 of the present embodiment includes a memory 23-3A in the previous stage of the frame buffer 23-3B, and time information and screen information are stored in the memory 23-3A. Are held in association with each other. Furthermore, the screen holding unit 23-3 includes an acquisition writing unit 23-3C. The acquisition writing unit 23-3C confirms the time information held in the memory 23-3A, and when the time counted by the clock of the own device becomes the time, the screen associated with the time Information is written into the frame buffer 23-3B.

  Note that the internal configuration of the screen receiving terminal 2 of the third embodiment is almost the same as that of FIG. 3 described in the first embodiment, except that the internal configuration of the screen holding unit 23-3 is different as described above. The operations of the unit 21-1 and the screen construction unit 22-1 are also different correspondingly. In the present embodiment, when receiving the time information together with the screen information, the receiving unit 21-1 outputs the received screen information and time information to the screen building unit 22-1, and the screen building unit 22-1 is expanded. Along with the screen information, the time information is written in the memory 23-3A of the screen holding unit 23-1.

  In FIG. 15, a mouse click occurs at time t <b> 13 timed by the screen transmission terminal 1, an operation signal indicating this mouse click is transmitted to the screen reception terminal 2, and this operation signal is timed by the screen reception terminal 2. Also shown is the reception at time t23. Also in this embodiment, if the screen receiving terminal 2 recognizes that the operation signal is an operation signal related to mouse click, the event occurrence position notification image and the mouse click are recognized at the time of recognition without waiting for the delay time T1 to elapse. A mouse image A indicating the front is displayed on the display. In the present embodiment, the screen receiving terminal 2 does not display the event occurrence position notification image and the mouse after the mouse click after the delay time T1 has elapsed from the occurrence of the mouse click at the screen transmission terminal 1, not after the delay time T1 has elapsed since reception. The image B is displayed on the display device.

  Even if the arrival fluctuation occurs when the screen information is transmitted through the network by the processing described above, the third embodiment absorbs the fluctuation and is the same as the first embodiment. In addition, the observer can know in advance that the mouse is clicked.

(Fourth embodiment)
In the first and second embodiments, the case where the operation event is a mouse operation has been described. However, the present invention is not limited to this. Therefore, in the present embodiment, a case where the operation event is a key operation event will be described.
FIG. 17 is a diagram illustrating an example of a key image in the remote screen monitoring system according to the fourth embodiment.
When only the pressing of a specific key, for example, the Enter key in the screen transmission terminal 1 is set as a detection target as an operation event, the screen receiving terminal 2 recognizes that the operation signal is an operation signal related to the pressing of the Enter key. For example, without waiting for the elapse of the delay time T1 from the reception of the operation signal, a key image A for indicating that the Enter key is immediately before the operation shown in FIG. And a key image B for indicating that the Enter key has been operated as shown in FIG. 17B after the delay time T1 has elapsed from the reception of the operation signal. Display in the center.

  In addition, if the purpose of the screen receiving terminal 2 is to know that any key on the screen transmitting terminal 1 has been pressed, the screen receiving terminal 2 can operate the operation signal regardless of the type. If it is recognized that the operation signal is related to the pressing of the key on the 1 side, any key shown in FIG. 17A is immediately before the operation at the time of recognition without waiting for the delay time T1 to elapse after receiving the operation signal. A key image C for indicating the presence of the key is displayed, for example, on the upper right corner on the screen of the display device, and any key shown in FIG. 17B is operated after a delay time T1 has elapsed since the operation signal was received. May be displayed at the upper right corner on the screen of the display device.

The functional configuration of the screen transmitting terminal and the functional configuration of the screen receiving terminal of the remote screen monitoring system in the fourth embodiment are the same as the configurations shown in FIGS. 2 and 3 described in the first embodiment. The processing procedure by the screen transmission terminal of the remote screen monitoring system in the fourth embodiment is the same as that in the first embodiment. FIG. 18 is a flowchart illustrating an example of a processing procedure performed by the screen receiving terminal of the remote screen monitoring system according to the fourth embodiment.
Here, only pressing of the Enter key is set as a detection target as an operation event, and processing from reception to display of the difference image in steps S21 to S27 described in the first embodiment is performed.
Further, when the receiving unit 21-1 of the screen receiving terminal 2 receives the operation signal (step S31). The operation signal analysis unit 27-1 analyzes the operation signal (step S32), and outputs the operation event type, coordinate information, and time information obtained by this analysis to the operation state notification image generation unit 28-1 (step S33). .

  Then, the operation status notification image generation unit 28-1 recognizes that the operation event is an operation of the Enter key (step S71), and writes the event occurrence position notification image and the key image A to the frame buffer 23-1B (step S72). This writing is notified to the drawing unit 24-1 (step S36). As a result, a notification image is added to the screen information. The screen holding unit 23-1 displays the event occurrence position notification image and the mouse image written in the frame buffer 23-1B on the display device (step S37).

  Then, when the time T1 has elapsed from the reception time of the operation signal in step S31 (step S38), the screen holding unit 23-1 writes the key image B into the frame buffer 23-1B (step S73), and the drawing unit 24- 1 displays the image written in the frame buffer 23-1B on the display device (step S40).

(Fifth embodiment)
In each of the above-described embodiments, processing for an operation signal by an external input interface such as a mouse or a keyboard has been described. However, in the present embodiment, a case where audio generated in the screen transmission terminal 1 is notified to the screen reception terminal 2 will be described. .
FIG. 19 is a block diagram illustrating a functional configuration example of the screen transmission terminal 1A of the remote screen monitoring system according to the fifth embodiment.
The screen transmission terminal 1A in the present embodiment includes a voice detection unit 18-1. The other components of the screen transmission terminal 1A are the same as those of the screen transmission terminal 1 shown in FIG.
The voice detection unit 18-1 detects the voice generated in the screen transmission terminal 1A. In addition, the voice detection unit 18-1 obtains time information when the voice is generated by a timing circuit (not shown), and then outputs the type of the voice signal and the time information to the operation signal generation unit 17-1. For example, it is assumed that the voice detection unit 18-1 monitors a voice file and has a mechanism for detecting that a specific voice file has been reproduced.

  When a specific sound file, for example, a warning sound file for transmitting a warning to the user of the screen transmission terminal 1A is started to be played on the screen transmission terminal 1A, the sound detection unit 18-1 includes the file name of the warning sound, The time information indicating the time when the reproduction of this file is started is output to the operation signal generator 17-1.

  When the operation signal generation unit 17-1 receives the warning sound file name and time information from the sound detection unit 18-1, the “start” information and the warning sound file name as one of the “operation signals”. Is generated, and the audio information signal and the input time information are output to the transmitter 13-1. Then, the transmission unit 13-1 transmits the audio information signal and the time information together to the screen receiving terminal 2.

  The voice detector 18-1 can also detect the end of playback of the voice file. In this way, when the reproduction of the audio file is completed, the audio detection unit 18-1 gives the operation signal generation unit 17-1 the alarm sound file name and the time information when the reproduction ends as the alarm sound end information. Notice. When the warning sound end and time information are input, the operation signal generation unit 17-1 generates a voice information signal including information “end” as one of the “operation signals” and the file name of the warning sound. The corresponding time information is output to the transmitter 13-1 together with the audio information signal. Moreover, the transmission part 13-1 transmits the audio | voice information signal output from the operation signal generation part 17-1, and time information to the screen receiving terminal 2 together.

  FIG. 20 is a block diagram illustrating a functional configuration example of the screen receiving terminal 2A of the remote screen monitoring system according to the fifth embodiment. The screen receiving terminal 2 </ b> A of this embodiment includes an audio playback unit 29. The other components of the screen receiving terminal 2A are the same as those of the screen receiving terminal 2 shown in FIG.

  As described above, when the audio detection unit 18-1 of the screen transmission terminal 1A detects the start of reproduction of the audio file, the operation signal analysis unit 27-1 of the screen reception terminal 2A indicates that the reception unit 21-1 When the voice information signal is received, the received operation signal is analyzed. The operation signal analysis unit 27-1 recognizes that the operation signal is a voice information signal as a result of the analysis.

  The operation signal analysis unit 27-1 detects the information “start” included in the audio information signal, the file name of the warning sound, and the time information indicating the reproduction start time of the file. As a result of detecting the file name of the warning sound, the operation signal analysis unit 27-1 outputs the warning sound start event, the file name of the warning sound, and the time information of the file to the operation status notification image generation unit 28-1. .

  Further, as described above, when the audio detection unit 18-1 of the screen transmission terminal 1A detects the end of reproduction of the audio file, the operation signal analysis unit 27-1 of the screen reception terminal 2A receives the operation signal from the reception unit 21-1. When a voice information signal is received as one of the above, the received operation signal is analyzed. The operation signal analysis unit 27-1 recognizes that the operation signal is a voice information signal as a result of the analysis.

  The operation signal analysis unit 27-1 detects information “end” included in the audio information signal, a file name of the warning sound, and time information indicating the reproduction end time of the file. As a result of detecting the warning sound file name, the operation signal analysis unit 27-1 outputs the warning sound end event, the warning sound file name, and the time information of the file to the operation status notification image generation unit 28-1. .

FIG. 21 is a diagram illustrating an example of an audio image used in the remote screen monitoring system according to the fifth embodiment.
When the operation state notification image generation unit 28-1 receives a warning sound start event from the operation signal analysis unit 27-1, the operation state notification image generation unit 28-1 is illustrated in FIG. Audio image A to be written is written into the frame buffer 23-1B of the screen holding unit 23-1.

  Note that the audio image A is written in the frame buffer 23-1B so as to be displayed at the upper left corner on the screen of the display device, for example. The operation state notification image generation unit 28-1 periodically transmits the audio image A to the frame buffer 23-1B of the screen holding unit 23-1 until the warning sound end event is input from the operation signal analysis unit 27-1. And the audio image A is always displayed on the screen of the display device.

  Further, the operation status notification image generation unit 28-1 can know the information of the delay time T1 determined by the display time adjustment unit 25-1, and the file name of the warning sound and the reproduction start time information of the file are included. If it is input, at the time when the delay time T1 is added to the reproduction start time, the audio reproduction unit 29 starts reproduction of the warning sound file stored in the internal memory.

  On the other hand, when the operation status notification image generation unit 28-1 inputs the file name of the warning sound and the reproduction end time information of the file from the operation signal analysis unit 27-1, the delay time T1 is added to the reproduction end time. When the time comes, the audio playback unit 29 ends the file playback of the warning sound.

  Not only the warning sound but also information on another audio file reproduced on the screen transmitting terminal 1A side can be transmitted to the screen receiving terminal 2A by the same mechanism. In the case of another audio file, for example, an audio image B shown in FIG. 21 may be displayed.

FIG. 22 is a flowchart illustrating an example of a processing procedure performed by the screen transmission terminal of the remote screen monitoring system according to the fifth embodiment.
Here, the difference image transmission process in steps S1 to S8 described in the first embodiment is performed, and the operation signal transmission process in steps S11 to S18 described in the first embodiment is performed.

  Then, when the voice detection unit 18-1 of the screen transmission terminal 1A detects the start of voice playback or the end of voice playback in the screen transmission terminal 1A, the voice detection unit 18-1 acquires time information when the playback of the voice starts or ends. Then, the type of the audio signal and the time information are output to the operation signal generator 17-1. The sound detection unit 18-1 outputs a file name of the warning sound and time information indicating the time when the reproduction is started or ended to the operation signal generation unit 17-1 (Step S81).

  When the warning signal file name and time information are input from the voice detection unit 18-1, the operation signal generation unit 17-1 includes information indicating the start or end of voice playback and a voice including the warning sound file name. An information signal is generated (step S82), and the audio information signal and the input time information are output to the transmission unit 13-1. Then, the transmission unit 13-1 transmits the input time information together with the audio information signal as the operation signal to the screen receiving terminal 2A (step S83).

FIG. 23 is a flowchart illustrating an example of a processing procedure performed by the screen receiving terminal of the remote screen monitoring system according to the fifth embodiment.
Here, the processing from reception to display of the difference image in steps S21 to S27 described in the first embodiment is performed.
Further, when the receiving unit 21-1 of the screen receiving terminal 2A receives the operation signal transmitted from the screen transmitting terminal 1A as a voice information signal (step S31). The operation signal analysis unit 27-1 analyzes the operation signal (step S32). As a result of the analysis, the operation signal analysis unit 27-1 recognizes that the operation signal is a voice information signal (step S91), and the voice information signal indicates the start of voice. If it is a signal (YES in step S92), the information “start” included in the audio information signal, the file name of the warning sound, and the time information indicating the reproduction start time of the file are detected. As a result of detecting the file name of the warning sound, the operation signal analysis unit 27-1 outputs the warning sound start event, the file name of the warning sound, and the time information of the file to the operation status notification image generation unit 28-1. (Step S93).

  Then, the operation status notification image generation unit 28-1 writes the audio image A into the frame buffer 23-1B of the screen holding unit 23-1 (Step S35), and notifies the drawing unit 24-1 of this writing (Step S36). ). As a result, a notification image is added to the screen information. The screen holding unit 23-1 displays the audio image written in the frame buffer 23-1B on the display device (step S37).

  Then, when time T1 has elapsed from the reception time of the operation signal in step S31 (step S38), the screen holding unit 23-1 outputs a warning sound file reproduction start instruction signal to the audio reproduction unit 29. Then, the audio reproduction unit 29 starts reproducing the warning sound file stored in the internal memory (step S95).

  On the other hand, if the operation signal analysis unit 27-1 recognizes that the operation signal is an audio information signal (step S91) and the audio information signal is a signal indicating the end of the audio (YES in step S92), the audio information signal The information “end”, the file name of the warning sound, and the time information indicating the reproduction end time of the file are detected. As a result of detecting the warning sound file name, the operation signal analysis unit 27-1 outputs the warning sound end event, the warning sound file name, and the time information of the file to the operation status notification image generation unit 28-1. (Step S96).

  Then, the operation status notification image generation unit 28-1 outputs the time information of the warning sound file to the screen holding unit 23-1. When the time T1 has elapsed from the reception time of the operation signal in step S31 (step S97), the screen holding unit 23-1 outputs a warning sound file reproduction end instruction signal to the audio reproduction unit 29. Then, the audio reproduction unit 29 ends the reproduction of the warning sound file stored in the internal memory (step S98).

  By performing the processing as described above, it is possible to transmit information of the audio file reproduced on the screen transmission terminal 1A side to the screen reception terminal 2A. Further, by displaying the audio image before starting the audio reproduction of the screen receiving terminal 2A, the observer of the screen receiving terminal 2A can grasp in advance that the audio is reproduced.

(Sixth embodiment)
In the above embodiments, the case where the push type is used as the screen transfer protocol has been described. However, in this embodiment, the case where the pull type is used will be described. In this embodiment, those not particularly specified are the same as those described in the second embodiment.
FIG. 24 is a block diagram illustrating a functional configuration example of the screen transmission terminal 1A of the remote screen monitoring system according to the sixth embodiment. FIG. 25 is a block diagram illustrating a functional configuration example of the screen receiving terminal 2A of the remote screen monitoring system according to the sixth embodiment.
In the present embodiment, since the screen transfer protocol corresponds to the pull type, the screen receiving terminal 2 includes the screen acquisition request transmission unit 210-2 as a component, and other configurations are the same as those in FIG. 9 described in the second embodiment. The configuration is the same as shown. The screen transmission terminal 1 includes a screen acquisition request receiving unit 19-2 as a constituent element, and the other configuration is the same as the configuration illustrated in FIG. 10 described in the second embodiment.

  The screen acquisition request transmission unit 210-2 of the screen reception terminal 2 periodically transmits a screen acquisition request to the screen transmission terminal 1. In addition, when a screen acquisition notification is input from the outside, the screen acquisition request transmission unit 210-2 transmits a forced screen acquisition request to the screen transmission terminal 1 as a trigger.

  In addition to the processing described in the second embodiment, the operation signal analysis unit 27-2 of the screen reception terminal 2 analyzes the operation signal from the screen transmission terminal 1, and as a result, the operation signal indicates a mouse, key operation, or voice. That is, an operation signal serving as a trigger for writing an image such as a mouse image, a key image, or a sound image to the frame buffer 28-2B of the operation status notification image generation unit 28-2 in the latter stage, for example, in the screen holding unit 23-2. If it is determined, a screen acquisition notification is sent to the screen acquisition request transmitter 210-2. Then, the screen acquisition request transmission unit 210-2 transmits a screen acquisition notification to the screen transmission terminal 1.

  The operation signal analysis unit 27-2 may request to update the entire screen in the screen acquisition notification, or the operation state notification image generation unit 28-2 may request the frame of the screen holding unit 23-2. If an area in which an image is written in the buffer 23-2B can be predicted, it may be requested to update only that area.

  When receiving the screen acquisition request from the screen acquisition request transmission unit 210-2 of the screen reception terminal 2, the screen acquisition request reception unit 19-2 of the screen transmission terminal 1 notifies the screen change detection unit 11-2 of the screen update notification. To do. Further, when receiving the forced screen acquisition request from the screen receiving terminal 2, the screen acquisition request receiving unit 19-2 notifies the screen change detecting unit 11-2 of a forced screen update notification.

  The screen acquisition request receiving unit 19-2 updates the full screen by the screen update notification or the forced screen update notification when the update of the full screen is instructed by the screen acquisition request or the forced screen acquisition request. Notify the screen change detection unit 11-2. On the other hand, when the screen acquisition request receiving unit 19-2 is instructed to update a part of the screen in the screen acquisition request or the forced screen acquisition request, the screen acquisition request receiving unit 19-2 receives the screen update request or the forced screen update notification. The screen update is notified to the screen change detection unit 11-2.

  When the screen change detection unit 11-2 receives the screen update notification, the screen change detection unit 11-2 targets the entire screen or a part of the screen instructed by this notification, and detects the changed portion on the screen. For example, the previous screen information is stored in the storage device 31, and the screen change detection unit 11-2 detects the difference by comparing the current screen information with the pixel, and uses the detected difference as the update screen data. The data is stored in the storage device 31 and the generation of screen update data is notified to the transmission image data generation unit 12-2.

  On the other hand, when the screen change detection unit 11-2 receives the forced screen update notification, the screen change detection unit 11-2 determines that a screen change has occurred on the entire screen or a part of the screen instructed by this notification, and treats all the screen areas as differences. The update screen data is stored in the storage device 31 and the generation of the screen update data is notified to the transmission image data generation unit 12-2.

  In the second embodiment, the process in which the operation signal generation unit 17-2 notifies the screen change detection unit 11-2 of the refresh notification has been described. However, in this embodiment, it is compulsory to correspond to the refresh notification. This screen update notification is notified to the screen change detection unit 11-2 when the screen acquisition request receiving unit 19-2 receives a forced screen acquisition request from the screen receiving terminal 2 side. Further, the forced screen acquisition request is triggered by a screen acquisition notification sent to the screen acquisition request transmission unit 210-2 by the determination of the operation signal analysis unit 27-2 on the screen receiving terminal 2 side.

  In the present embodiment, the case where the screen transfer protocol is replaced with the pull type based on the configuration of the second embodiment has been described. The main features of this embodiment are as described in the second or first embodiment.

According to each of these embodiments, there is provided an information processing apparatus and an information processing method capable of recognizing at the screen receiving terminal on the observation side when the operation on the screen transmitting terminal on the operation side has been performed. Can do.
Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1, 1A ... Screen transmission terminal 2, 2A ... Screen reception terminal, 3 ... Network, 11-1, 11-2 ... Screen change detection part, 12-1, 12-2 ... Transmission image data generation part, 13-1 , 13-2 ... transmission unit, 14-1, 14-2 ... operation event detection unit, 15-1, 15-2 ... time detection unit, 16-1 ... time information addition unit, 17-1, 17-2 ... Operation signal generation unit, 18 ... voice detection unit, 19 ... screen acquisition request reception unit, 21-1, 21-2 ... reception unit, 22-1, 22-2 ... screen construction unit, 23-1, 23-2 23-3 ... Screen holding unit, 23-1A ... Buffer, 23-1B, 23-2B, 23-3B ... Frame buffer, 23-2A, 23-3A ... Memory, 23-3C ... Acquisition writing unit, 24-1 , 24-2 ... Rendering unit, 25-1, 25-2 ... Display time adjustment unit, 26-1, 26 2 ... display control unit, 27-1, 27-2 ... operation signal analysis unit, 28-1, 28-2 ... operation status notification image generation unit, 29 ... audio reproduction unit, 31 ... storage device, 210-2 ... screen Acquisition request transmission unit.

Claims (10)

Reception for receiving screen information indicating a screen displayed on the information processing apparatus and an operation signal indicating the status of the input operation in the information processing apparatus, transmitted from another information processing apparatus that performs screen display according to the input operation And
Based on the type of operation signal received by the reception unit, a generation unit that generates an operation status notification image indicating an operation status in the information processing device;
An information processing apparatus comprising: a drawing unit that draws the generated operation state notification image on a screen indicated by screen information received by the receiving unit. The drawing unit
The operation state notification image generated by the generation unit is rendered on the screen indicated by the screen information received by the reception unit when a predetermined time has elapsed since the reception time of the operation signal received by the reception unit. The information processing apparatus according to claim 1. The operation signal includes coordinate information of an input operation position on the screen of the other information processing apparatus,
The drawing unit
3. The information processing according to claim 2, wherein the operation status notification image generated by the generation unit is drawn at a coordinate indicated by the coordinate information on a screen indicated by the screen information received by the reception unit. apparatus. The information processing apparatus according to claim 2, wherein the input operation includes at least one of a mouse click operation of the other information processing apparatus and a keyboard operation of the other information processing apparatus. The operation signal includes a first operation signal that is an operation other than an operation for determining a process on the screen and a second operation signal that is an operation for determining a process on the screen,
The drawing unit
A screen indicated by the screen information received by the receiving unit when an operation status notification image is generated according to the first operation signal when a predetermined time has elapsed from the reception time of the first operation signal received by the receiving unit. Draw on top
When the second operation signal is received by the receiving unit, before the processing determination, which is generated according to the second operation signal before a predetermined time elapses from the reception time of the second operation signal. A first operation status notification image as an image is drawn on the screen indicated by the screen information received by the receiving unit;
The processing decision generated according to the second operation signal, which is different from the first operation state notification image when a certain time has elapsed from the reception time of the second operation signal received by the receiving unit, is shown. The information processing apparatus according to claim 2, wherein a second operation state notification image that is an image is drawn on a screen indicated by the screen information received by the receiving unit. The operation signal includes coordinate information of a mouse cursor position on the screen of the other information processing apparatus or a mouse click position on the screen,
The drawing unit
On the screen indicated by the screen information received by the receiving unit, the operation status notification image generated by the generating unit in response to the operation signal indicating the cursor movement of the mouse of the other information processing apparatus on the screen In the coordinates indicated by the coordinate information,
The operation status notification generated by the generation unit in response to the operation signal indicating a mouse click operation of the other information processing apparatus on the screen is generated in response to the cursor movement. The information processing apparatus according to claim 2, wherein the image is drawn in the vicinity of coordinates drawn on the screen of the image. Perform time synchronization in synchronization with the time measured by the other information processing device,
The receiver is
Receiving information on the occurrence time of the operation signal transmitted from the other information processing apparatus in the other information processing apparatus;
The drawing unit
When it is timed that a predetermined time has elapsed from the time when the operation signal received by the reception unit is generated in the other information processing apparatus, the operation state notification image generated by the generation unit according to the operation signal is received. The information processing apparatus according to claim 2, wherein the information processing apparatus draws on a screen indicated by the screen information. The receiving unit further receives a signal that is transmitted from the other information processing apparatus and notifies generation of information that does not involve a change on the screen of the information processing apparatus,
The generator is
The information processing apparatus according to claim 2, wherein a notification image is generated according to a signal that is received by the receiving unit and notifies the occurrence of information that is not accompanied by a change on the screen. The signal for notifying the occurrence of information without a change on the screen in the other information processing apparatus is a signal for notifying the occurrence of sound in the other information processing apparatus,
The receiving unit receives the screen information and a signal indicating the generation of the sound,
The generator is
In response to the signal indicating the generation of the sound received by the receiving unit, a notification image and a notification sound indicating the generation of the sound are generated,
The drawing unit
When a predetermined time has elapsed from the time of generation of the signal indicating the generation of the sound received by the reception unit, a notification image corresponding to the signal indicating the generation of the sound is drawn on the screen indicated by the received screen information,
The apparatus according to claim 2, further comprising: a reproduction unit that reproduces the notification sound generated by the generation unit when the predetermined time has elapsed from a generation time of the signal indicating the generation of the sound received by the reception unit. The information processing apparatus described. Received from the other information processing device that performs screen display according to the input operation, screen information indicating the screen displayed on the information processing device, and an operation signal indicating the status of the input operation in the information processing device,
Based on the type of the received operation signal, generate an operation status notification image indicating an operation status in the information processing device,
The information processing method characterized by drawing the produced | generated operation condition notification image on the screen shown by the received screen information.

Images