JP5683859B2 - Screen receiving device, screen receiving method and screen receiving program - Google Patents

Description

  Embodiments described herein relate generally to a screen transfer technique for transferring screen information to an information terminal and displaying the received screen information on the information terminal side.

  A plurality of devices such as information terminals can transmit and receive various types of information by configuring a system connected by a communication network even if they are remotely installed. In particular, the screen transfer system can also display a screen displayed on the monitor of a remote device on the monitor of the device at hand.

  Various protocols are used as a protocol for realizing the screen transfer system. For example, VNC (Virtual Network Computing), which is known as one of screen transfer systems, uses an RFB (Remote Frame Buffer) protocol.

  An example of the application of the screen transfer system is a remote monitoring system. In the remote monitoring system, when a device or communication network abnormality occurs, a mechanism for detecting the abnormality is required. This is because it is necessary to continue monitoring by the monitoring person immediately discovering that it is not operating normally due to such an abnormality and promptly dealing with it.

  As an example of a method for detecting such an abnormality, there is a method for transmitting and receiving a keep alive signal. The keep-alive signal is a signal for notifying other devices that each device is operating normally at a predetermined time. It is also possible to transmit / receive failure information between devices. For example, a device in which an abnormality has occurred transmits failure information, and the device that has received the failure information can detect the abnormality.

  However, there is a screen transfer protocol that does not include a keep-alive or failure information transmission / reception mechanism. When such a screen transfer protocol is used, an abnormality in screen transfer processing cannot be detected. In order to cope with this, for example, it is conceivable to separately prepare an application for transmitting and receiving a keep-alive signal. Thereby, the occurrence of an abnormality in the communication network can be detected.

The RFB Protocol Version 3.8 Last updated 26 February 2009

  However, even if the application as described above is prepared, it is not known whether the screen transfer protocol is operating normally or whether the screen is correctly transmitted. For example, when an internal abnormality that cannot be detected from the outside occurs, even if a black screen continues to be sent, this cannot be detected as an abnormality.

Embodiments of the present invention generate a screen holding unit that holds screen information, a screen information transmission unit that transmits screen information held by the screen holding unit via a communication network, and a change image that changes at a predetermined cycle. Communication from a screen transfer device including: an image generation unit that includes: an image addition unit that includes a change image generated by the image generation unit included in screen information held by the screen holding unit; A screen receiving apparatus that receives screen information via a network has the following technical features.

That is, a screen acquisition request transmission unit that transmits a screen information acquisition request to the screen transfer device, a screen information reception unit that receives screen information transmitted from the screen transfer device in response to the acquisition request, and the received screen A detection unit that detects a change image that changes in a predetermined cycle is included in the information, and the screen acquisition request transmission unit transmits a screen information acquisition request at a timing for acquiring an image different from the previously received change image. It is characterized by doing.

It is a simplified diagram showing an example of a screen transfer system. It is a block diagram which shows an example of an application server. It is explanatory drawing which shows an example of the screen content before (a) drawing of the image A, and after drawing (b). It is a block diagram which shows an example of a client terminal. It is explanatory drawing which shows an example of the reception interval of a screen acquisition request, and the period change of the image A. FIG. It is a flowchart which shows an example of the production | generation and drawing process of the image A in a period change image generation part. It is a flowchart which shows an example of the detection process of the change of the image A in a period change image confirmation part. It is a figure which shows an example of the period change image containing numerical information. It is a block diagram which shows an example of the application server which has a self-device fault detection part. It is a block diagram which shows an example of the client terminal which has an other apparatus failure detection part. It is explanatory drawing which shows an example of a period change image.

(1) 1st Embodiment 1st Embodiment is described with reference to FIGS.
[Basic equipment configuration]
FIG. 1 is a simplified diagram showing a screen transfer system S (hereinafter referred to as the present system S).

  The system S has an application server 1 and a client terminal 2. The application server 1 and the client terminal 2 are configured to be able to send and receive various types of information via a communication network (not shown). The application server 1 functions as a device (screen transfer device or screen transfer server device) that transmits screen information. The client terminal 2 functions as a device (screen receiving device or screen transfer client device) that receives screen information.

  The application server 1 and the client terminal 2 can be configured by a computer, for example. In this case, the function of each unit described below can be realized by physically utilizing computer hardware by a predetermined program. As described above, a program that realizes the functions of the respective units or a recording medium that records the program is also realized as an embodiment.

  The computer constituting the application server 1 and the client terminal 2 has display units 31 and 32 and input units 41 and 42, respectively, as shown in FIGS.

  The display units 31 and 32 are components that output various data including a screen to be processed in the system S so that a user including an administrator and an operator can view the data. The display units 31 and 32 include all display devices that can be used now or in the future. In addition, as an output unit other than the display units 31 and 32, for example, a printer, an audio output device, and the like may be provided.

  The input units 41 and 42 are components that input information necessary for processing of the system S, input of processing selections, and instructions. As the input units 41 and 42, for example, a keyboard, a mouse, a touch panel (including one configured in a display device) and the like can be considered. However, all input devices available now or in the future are included. Therefore, a voice input device for inputting by voice is also included. Also, a simple switch or the like may be used.

  Information necessary for the processing of the system S is stored in a storage unit set in the computer. Necessary information includes, for example, frames and data for communication and data processing (including screen acquisition requests and screen information), and various settings (including a predetermined cycle). As will be described later, screen information, a cycle, and the like can be collected in a setting file and stored in a storage unit. As the storage unit, any storage medium such as a memory or a disk that can be used at present or in the future is applicable. A buffer (including a frame buffer) that temporarily stores information is also included in the storage unit.

  As a communication network, any wired or wireless transmission path or transmission medium can be applied. As long as information can be transmitted and received, it does not matter what LAN or WAN is routed or not routed along the route. Any communication protocol that can be used at present or in the future can be applied.

  In this system S, the two devices shown in FIG. 1 are called an application server 1 and a client terminal 2. However, for example, when screen transfer is performed by two notebook PCs, the notebook PC that transmits the screen is the application server 1, and the notebook PC that receives the screen is the client terminal 2. The computer can also be configured to perform the functions of both the application server 1 and the client terminal 2 by a program.

  In the application server 1, application software is activated. In the system S, a screen transmission function is incorporated in the application server 1 and the screen information generated by the application software is transmitted to the client terminal 2 via the network.

  In order to realize such a screen transmission function of the application server 1 and a screen reception function of the client terminal 2 corresponding thereto, the present system S uses a screen transfer protocol. One example is a pull-type screen transfer protocol. The pull type is a protocol for transmitting information in response to a request from the receiving side.

  For example, the screen transfer protocol known as “VNC” is a pull type as shown below. In FIG. 1, the exchange of signals of the pull type screen transfer protocol is shown in a simplified manner.

  First, the client terminal 2 transmits a screen acquisition request “FramebufferUpdateRequest” to the application server 1. Then, the application server 1 transmits screen information to the client terminal 2 as a response “FramebufferUpdate”.

  The client terminal 2 receives the screen information transmitted from the application server 1 as described above. The client terminal 2 displays a screen on the display unit 32 based on the received screen information.

  As described above, when using VNC, the application server 1 activates “VNC server” software installed in advance. Further, the client terminal 2 activates “VNC client” software installed in advance. Existing “VNC server” software and “VNC client” software can perform basic functions related to screen transmission and reception without changing the software itself.

  For example, the screen capture unit 130 and the transmission / reception unit 140 described later can be realized by “VNC server” software. Further, the transmission / reception unit 210 to be described later can be realized by “VNC client” software. As shown in this example, the present system S can be used without changing the existing screen transfer protocol.

  As described above, the screen information generated by the application software in the application server 1 is transmitted to the system S. And based on the transmitted screen information, the display part of the client terminal 2 which received it displays a screen.

Configuration details
Next, details of the configuration of the system S will be described.
[Application server 1]
FIG. 2 is a block diagram illustrating an example of the configuration of the application server 1. FIG. 2 is a schematic diagram showing only functions related to the system S. The application server 1 includes application software 110, a frame buffer 120, a screen capture unit 130, a transmission / reception unit 140, a start / end control unit 150, a periodic change image generation unit 160, a setting storage unit 170, and the like.

  The application software 110 includes a screen generation unit 111, a drawing execution unit 112, and the like. The screen generation unit 111 is a processing unit that generates a screen in which the entire screen, a part of the screen, or a plurality of parts of the screen changes. The contents of the screen to be generated and the timing of change are not limited to specific ones. For example, when the application software 110 is moving image reproduction software for reproducing a moving image, the screen generation unit 111 may generate a moving image. Further, when the application software 110 is text editor software for editing a text file, the screen generation unit 111 may generate a text display screen.

  The drawing execution unit 112 is a processing unit that executes a drawing process in accordance with a drawing command provided by the OS or middleware. By executing this drawing process, the pixel information of the screen generated by the screen generation unit 111 is written in the frame buffer 120. In FIG. 2, the drawing process is indicated by an arrow from the application software 110 to the frame buffer 120.

  The frame buffer 120 is a storage unit in which pixel information to be a screen displayed on the display unit 31 is written. The frame buffer 120 is a screen holding unit that holds pixel information. The display unit 31 displays a screen based on the pixel information written in the frame buffer 120.

  The screen capture unit 130 includes a drawing detection unit 131, an image holding unit 132, a difference detection unit 133, a difference area output unit 134, and the like. The drawing detection unit 131 is a processing unit that detects drawing processing in the frame buffer 120. The image holding unit 132 is a storage unit that holds the drawn pixel information as a backup image.

  The difference detection unit 133 is a processing unit that detects the difference area by acquiring the pixel information of the portion where the change has occurred from the frame buffer 120 and comparing it with the backup image. The difference area output unit 134 is a processing unit that outputs information about the detected difference area to the transmission / reception unit 140. In the backup image of the image holding unit 132, the difference area is overwritten and updated. In FIG. 2, the difference area detection process is indicated by an arrow from the frame buffer 120 to the screen capture unit 130.

  The transmission / reception unit 140 includes a screen acquisition request reception unit 141 and a screen information transmission unit 142. The screen acquisition request reception unit 141 is a processing unit that receives a screen acquisition request from the client terminal 1. The screen information transmission unit 142 is a processing unit that transmits screen information as a response to the received screen acquisition request.

  The screen information is pixel information acquired from the frame buffer 120, information on a difference area input from the screen capture unit 130, and the like. When transmitting screen information, the amount of transmission data may be reduced by executing compression processing. In order to transmit the screen information in this way, the transmission / reception unit 140 sets, for example, a TCP connection with the client terminal 2.

  In this system S, it is assumed that the screen acquisition request from the client terminal 2 is repeatedly transmitted. In other words, the client terminal 2 periodically transmits a screen acquisition request or receives screen information from the application server 1 to transmit a screen acquisition request as a trigger.

  The start / end control unit 150 includes a detection unit 151, a start processing unit 152, and an end processing unit 153. The detection unit 151 is a processing unit that detects the start and end of the screen transmission process. Detection of the start and end of the screen transmission process is performed, for example, in the application server 1 when the input unit 41 receives start and end instructions from the user. However, for example, the detection unit 151 may detect the disconnection detection as the end of the screen transmission process.

  The start processing unit 152 is a processing unit that controls the processes of the screen capture unit 130 and the transmission / reception unit 140 to start in accordance with the detection of the start of the screen transmission process. The termination processing unit 153 is a processing unit that controls the processing of the screen capture unit 130 and the transmission / reception unit 140 to be terminated in accordance with detection of termination of the screen transmission processing. Note that the start / end control unit 150 similarly controls the start and end of the process of the period change image generation unit 160 described below.

  The cycle change image generation unit 160 includes a cycle processing unit 161, an image generation unit 162, an image addition unit 163, and the like. The cycle processing unit 161 is a processing unit that controls each unit such as the image generation unit 162 and the image addition unit 163 according to a predetermined cycle. Note that, for example, an internal timer / counter is used in the process of starting, determining, and ending the periodic process. The image generation unit 162 is a processing unit that generates a “change image” (hereinafter referred to as an image A) that changes at a predetermined cycle. The image adding unit 163 is a processing unit that includes the image A on the screen. The image adding unit 163 is a drawing execution unit that draws the image A in the frame buffer 120, for example.

  The setting storage unit 170 is a storage unit that stores information necessary for processing of the system S. For example, screen information (including information such as color information on which the image A is drawn, position information, and size information), predetermined cycle information, and the like are collectively stored in a setting file.

  Here, FIGS. 3A and 3B show an example in which a part of the screen content 1011 drawn in the frame buffer 120 changes. FIG. 3A shows the screen content 1011 at a certain point in time before the processing by the periodic change image generation unit 160 is started. On the other hand, FIG. 3B shows the screen content 1011 immediately after the process of the periodic change image generation unit 160 is started. Here, the image A is displayed on the screen content 1011.

  As described later, there are various methods for changing the image A at a predetermined cycle, and the method is not limited to a specific method. The period for changing the image A is not limited to a specific one. For example, as the period determined by the period processing unit 161, a value described in advance in the setting file may be used as described above.

  Further, for example, the cycle processing unit 161 may calculate the cycle. An example of a method for calculating the period in this way will be described. That is, the screen acquisition request receiving unit 141 receives a screen acquisition request from the client terminal 2. The period processing unit 161 detects the reception timing, and calculates the reception period of the screen acquisition request.

[Client terminal 2]
FIG. 4 is a block diagram illustrating an example of the client terminal 2. FIG. 4 is a schematic diagram showing only functions related to the system S. The client terminal 2 includes a transmission / reception unit 210, a drawing execution unit 230, a frame buffer 240, a start / end control unit 260, a periodic change image confirmation unit 270, a setting storage unit 280, and the like.

  The transmission / reception unit 210 includes a screen acquisition request transmission unit 211, a screen information reception unit 212, and the like. The screen acquisition request transmission unit 211 is a processing unit that transmits a screen acquisition request to the application server 1. The screen acquisition request is periodically transmitted based on a preset value. Alternatively, when a reception completion notification is input from the screen information receiving unit 212, it is transmitted in response to the input.

  The screen information receiving unit 212 is a processing unit that receives screen information from the application server 1 and outputs the screen information to the drawing execution unit 230. When the screen information is compressed, the screen information receiving unit 212 has a function of expanding the screen information. The screen information receiving unit 212 also has a function of outputting a reception completion notification to the screen acquisition request transmitting unit 211. In order to receive the screen information in this way, the transmission / reception unit 210 sets, for example, a TCP connection with the application server 1.

  The drawing execution unit 230 is a processing unit that writes input screen information as pixel information in the frame buffer 240. The pixel information written in the frame buffer 240 is displayed on the display unit 32 as a screen.

  The start / end control unit 260 includes a detection unit 261, a start processing unit 262, an end processing unit 263, and the like. The detection unit 261 is a processing unit that detects the start and end of the screen reception process. Detection of the start and end of the screen reception process is performed, for example, when the input unit 42 receives a start and end instruction from the user in the client terminal 2. However, for example, the detection unit 261 may detect the disconnection detection as the end of the screen reception process.

  The start processing unit 262 is a processing unit that controls the processes of the screen acquisition request transmission unit 211 and the screen information reception unit 212 to be started in accordance with the detection of the start of the screen reception process. The termination processing unit 263 is a processing unit that controls the processing of the screen acquisition request transmission unit 211 and the screen information reception unit 212 to be terminated in accordance with the detection of the termination of the screen reception processing.

  Further, the termination processing unit 263 controls the processing of the screen acquisition request transmission unit 211 and the screen information reception unit 212 to be terminated even when the abnormality in the screen transfer processing is notified from the periodic change image confirmation unit 270. . Note that the start / end control unit 260 similarly controls the start and end of the processing of the period change image confirmation unit 270 described below.

  The period change image confirmation unit 270 includes a period processing unit 271, a detection unit 272, a determination unit 272, and the like. The cycle processing unit 271 is a processing unit that controls each unit such as the detection unit 272 and the determination unit 273 according to a predetermined cycle. Note that, for example, an internal timer / counter is used in the process of starting, determining, and ending the periodic process. The detection unit 272 is a processing unit that detects a change in the image A generated by the periodic change image generation unit 160 on the application server 1 side. The determination unit 273 is a processing unit that determines an abnormality in the screen transfer process based on the change detected by the detection unit 272.

  The setting storage unit 280 is a storage unit that stores information necessary for processing of the system S. For example, screen information (including color information on which the image A is drawn, position information, size information, etc.), predetermined cycle information, abnormality detection time information, and the like are collectively stored in the setting file.

  Thereby, the detection unit 272 in the periodic change image confirmation unit 270 can detect the change by comparing the screen information stored in the setting storage unit 280 with the screen information acquired or referred to from the frame buffer 240. In the detection by the detection unit 272, for example, pixel information (position information, size information, etc.) is acquired or referenced from the frame buffer 240.

  The abnormality determination by the periodic image change image confirmation unit 270 can be performed as follows, for example. Assume that the time information of abnormality detection is set to 5 seconds, for example. In this case, it is assumed that the pixel information corresponding to the position information and size information of the image A is not changed for 5 seconds (a change is not detected by the detection unit 272). Then, the determination unit 273 determines that an abnormality has occurred in the screen transfer process between the application server 1 and the client terminal 2. Then, the periodic change image confirmation unit 270 notifies the start / end control unit 260 of abnormality in the screen transfer process. It should be noted that the abnormality detection time information can be described in advance in the setting file as described above. The above 5 seconds is an example, and is not limited to a specific time.

[Action]
The operation of the present embodiment as described above will be described. Note that the basic screen transfer process is the same as the process of each unit described in the above configuration, and the details are omitted. [Processing overview], [Image change example], [Image A generation and drawing], and [Image A change detection] will be described below.

[Process overview]
First, the outline of the processing of this embodiment is as follows.
On the application server 1 side (screen transmission side), in the screen transfer state, an image A that changes at a constant cycle is displayed on the screen.
On the application server 1 side, an update screen in which the image A changes at a constant cycle is transmitted to the client terminal 2 side (screen reception side).
The client terminal 2 displays the received update screen with a fixed period.
On the client terminal 2 side, it is confirmed whether an abnormality has occurred in the screen transfer protocol or whether it is operating normally by observing screen updates at a fixed period.

[Image change example]
Next, a change example of the image A transmitted from the application server 1 side and received and displayed on the client terminal 2 side will be described. First, for example, there is a method in which drawing and non-drawing of the image A are repeated (flashing). For example, there is a method in which two images A are prepared and drawn alternately. Furthermore, for example, there is a method of preparing three or more images A and drawing them in order. As described above, when a plurality of images A are prepared, each image A is different in at least one of color (part or all), shape, and size, for example. Furthermore, the image A may include information representing characters (including images, text, etc.).

  As a more specific example, as shown in FIG. 5, a case is assumed where image A repeats, for example, black and white. It is assumed that the reception interval time of the screen acquisition request is regular, for example, 1 second. In this case, control is performed so that black and white change every second as in the first change example of the image A. Or you may control to change every 0.25 second like the example 3 of a change of the image A.

  On the other hand, for example, as shown in the change example 2 or the change example 4 of the image A, a period in which the same color (black in the example of FIG. 5) is always synchronized with the reception timing of the screen acquisition request is avoided. That is, when the reception interval time is regularly T1 seconds, it is efficient to control so that black and white change every T1 seconds or every T1 / 4 seconds. Further, even when the average time is T1 seconds, black and white may be set every T1 seconds or T1 / 2N (N> 1). Note that the number of pixels of the image A that changes at a constant period is not limited to a specific number. However, it may not be noticeable by using one pixel. That is, in general, an image is considered to be composed of a plurality of pixels. However, the image A is not limited to a plurality of pixels, and includes a case of one pixel.

  Further, the display position of the image A is not limited to a specific position. The image A may be a mode in which the position is changed as well as a mode in which the image A changes (including changes in color, shape, etc.) without moving at the same position, such as blinking. For example, the image A may reciprocate in the vertical, horizontal, or diagonal directions. Further, for example, the place where the image A that repeats drawing / non-drawing is drawn (the place where the image A blinks) may be changed every time. At this time, it may be devised to make the change of the image A inconspicuous, for example, by displaying it at a place that is not close to each other.

  Further, the number of images A generated by the periodic change image generation unit 160 may be plural instead of one. The display position on the screen when there are a plurality of images A is not limited to a specific position. For example, a plurality of images A may be scattered on the screen. This makes it possible to determine whether screen transfer is operating correctly using a wide range of information, not locally on the screen.

Further, the position where the image A is displayed may be made inconspicuous, such as at the corner or edge of the screen. Furthermore, the image A may be displayed in a screen area that is not displayed on the client terminal 2 side under the following conditions.
-The screen capture unit 130 can capture (hold)-The screen information transmission unit 142 can transmit as screen information-The client terminal 2 can detect it

  Further, when drawing / non-drawing of the image A is repeated, for example, the following processing may be performed. In a portion where the image A is overlapped, the color before being overlapped is detected, and the image A is made a color similar to that color. For example, in the case of FIG. 3, the color before overlapping is the color of the portion where the image A of the screen content 1011 is overlapped. As a more specific example, it is assumed that the image A can be selected from two types of blinking of different colors (for example, blinking of blue or blinking of orange). Then, for example, when the color of the overlapping portion is red, the image A is set to the same color system, and blinking of orange having a close color is selected.

  In such a case, for example, a color determination unit that determines the color before superimposition is set in the period change image generation unit 160. Then, the image generation unit 162 is set to generate an image A having a similar color with respect to the color determined by the color determination unit.

  Furthermore, it is conceivable that the image A that changes at a constant cycle is sized according to the characteristics of the encoding method at the time of screen transfer. For example, when JPEG is used as an encoding method, a block of 8 × 8 pixels is set as an image A. Thereby, the image A can be encoded efficiently. For example, when the image A is changed in binary, the following processing is also possible. That is, the binary difference is a color that does not deteriorate due to the encoding process on the application server 1 side and the decoding process on the client terminal 2 side. Thereby, the detection error of the change of the image A can be suppressed.

  The client terminal 2 side can determine whether or not normal screen transfer is operating correctly by detecting such an image A that changes at a constant cycle. That is, the detection unit 272 in the periodic change image confirmation unit 270 can detect a change in the image A generated by the periodic change image generation unit 160 according to a rule. It should be noted that the rules regarding the mode of change of the fixed period of the image A are, for example, stored in advance in the setting storage unit 170, the setting storage unit 280, and the like. As a change rule, for example, various modes as described above are conceivable, such as when the color, shape, and position of the image A change, or when a change with a constant period occurs at a plurality of locations.

[Generation and drawing of image A]
Next, an example of processing for generating and drawing the image A that periodically blinks in the cycle change image generation unit 160 will be described with reference to the flowchart of FIG. That is, as described above, the start processing unit 152 causes the periodic change image generation unit 160 to start image generation processing based on the detection result of the detection unit 151 in the start / end control unit 150.

  Then, the period change image generation unit 160 acquires color information, position information, size information, period information, and the like as information related to the image A from the setting file (step 01). Then, the cycle processing unit 161 sets the counter value held therein to 0 (i = 0) (step 02).

  The image generation unit 162 generates an image A using information acquired from the setting file. Then, the image adding unit 163 draws the generated image A in the frame buffer 120 (step 03). At the same time, the period processing unit 161 starts a timer for the time acquired as the period information (step 04). When receiving the event (step 05), the cycle processing unit 161 confirms the content of the event.

  If the event is an end notification from the start / end control unit 150 (YES in step 06), the cycle change image generation unit 160 ends all internal processes. If the event is a time-out notification of the time acquired as the cycle information (YES in step 07), the cycle processing unit 161 increments the counter value by one (i = i + 1) (step 08).

  If the counter value is an even number (YES in step 09), the image adding unit 163 performs the image A drawing process (step 10). On the other hand, if the counter value is an odd number, the image adding unit 163 performs a non-rendering process for the image A (for example, including a process for not rendering or rendering an image before superimposing). And the period process part 161 starts the timer of the time acquired as period information again (step S11).

[Change detection of image A]
Furthermore, an example of processing for detecting a change in the image A that periodically blinks in the periodic change image confirmation unit 270 will be described with reference to a flowchart of FIG. That is, as described above, the start processing unit 262 starts the confirmation process by the periodic change image confirmation unit 270 based on the detection result of the detection unit 261 in the start / end control unit 260.

  Then, the period change image confirmation unit 270 acquires color information, position information, size information, period information, abnormality detection time information, and the like as information about the image A from the setting file (step 21). And the period process part 271 starts the timer for the period determination based on period information (step 22).

  When receiving the event (step S23), the cycle processing unit 271 confirms the content of the event. If the event is an end notification from the start / end control unit 260 (YES in step S24), the periodic change image confirmation unit 270 ends all internal processes. If the event is a time-out notification of the time acquired as the period information (YES in step 25), the period change image confirmation unit 270 uses the position information, size information, and the like of the image A acquired from the setting file, and the frame buffer 240. Corresponding pixel information is acquired from (step S26).

  The detecting unit 272 confirms the acquired pixel information, and confirms whether or not it is periodically blinking (changes normally) based on the color information of the image A acquired from the setting file. If the detection unit 272 determines that the cycle blinks normally (NO in step S27), the cycle processing unit 271 starts the timer for the time acquired as the cycle information again (step S29).

  On the other hand, the detection unit 272 detects that there is no change in the pixel information for the time corresponding to the “abnormality detection time information” acquired from the setting file (NO in step S27). In this case, the determination unit 273 determines that the periodic blinking is not performed normally, and notifies the start / end control unit 260 of an abnormality in the screen transfer process (step S28). Thereafter, the cycle processing unit 271 starts a timer for the time acquired as the cycle information again (step S29).

[effect]
As described above, according to the present embodiment, it is possible to reliably detect an abnormality from a screen including a changing image and deal with it immediately. Thereby, it is possible to prevent the user or the like from performing an erroneous operation in a state where the screen transfer is not operating normally. For example, when an abnormality is detected, the screen display can be immediately terminated.

(2) Second Embodiment The screen transfer system S according to the second embodiment is characterized in that the above-described periodic change image generation unit 160 includes numerical information in an image that changes at a constant time interval. The configuration of this embodiment is basically the same as described above. However, the internal processes of the periodic change image generation unit 160 on the application server 1 side and the periodic change image confirmation unit 270 on the client terminal 2 side are different.

  That is, in the periodic change image generation unit 160, the image generated by the image generation unit 162 represents numerical information. Examples of the numerical information include a sequence number. Further, in the periodic change image confirmation unit 270, the detection unit 272 detects numerical information from the image. It is assumed that the rule that the image represents numerical information is stored in advance in the setting storage unit 170, the setting storage unit 280, and the like, for example.

  For example, it is assumed that the change image generated by the image generation unit 162 on the application server 1 side is 3 pixels. Each pixel is, for example, white or black. For example, if black is treated as 1 and white is treated as 0, the image generation unit 162 can express 3-bit information. FIG. 8 shows an example in which a 3-bit sequence number (101 → 110 → 111) is expressed by an image that changes at a constant time interval.

  On the other hand, the detection unit 272 on the client terminal 2 side recognizes the change of the three pixels as a sequence number, and detects whether the sequence number is missing. When there is a missing sequence number, the determination unit 273 notifies the start / end control unit 260 of an abnormality in the screen transfer process.

  Here, changing the image to three pixels is only an example. The number of pixels can be freely determined (one pixel may be used as described above). For example, it is possible to reduce the numerical value that can be expressed as two pixels. For example, if the number of pixels is 4 or more, more numerical values can be expressed. The color, shape, size, display position, and display number of the change image are not limited to specific ones. The plurality of images generated by the image generation unit 162 may have different numbers of pixels. Although FIG. 8 shows a case where pixels are adjacent to each other, they may be separated without being adjacent. That is, the display position of the change image is not limited to a specific position.

  For example, let us consider a case where an abnormality occurs in 3 seconds, a state in which screen transfer processing cannot be performed continues, and then screen transfer starts to operate normally. Assume that the image A switches between white and black every second as in the above embodiment. Assume that the period-change image confirmation unit 270 of the client terminal 2 confirms the frame buffer 240 and detects black three times while an abnormality occurs.

  In this case, it is considered that black has not been updated due to the occurrence of an abnormality. However, although it has switched between white and black, it may be possible to confirm by the timing of black.

  In this embodiment, the sequence number is detected. For this reason, when it is not updated, it becomes the same (not changing) number, and when it is updated, it becomes a different (changed) number. Therefore, the above difference can be determined. For this reason, it is possible to grasp the abnormality of the communication network in more detail.

(3) Third Embodiment A screen transfer system S according to a third embodiment will be described with reference to FIGS. The present embodiment is characterized in that the period change image generation unit 160 includes operation information (for example, failure information) in an image that changes at a constant time interval.

  The configuration of this embodiment is basically the same as described above. However, the internal processes of the periodic change image generation unit 160 on the application server 1 side and the periodic change image confirmation unit 270 on the client terminal 2 side are different. As shown in FIG. 9, the application server 1 includes a self-device failure detection unit 180. Furthermore, as illustrated in FIG. 10, the client terminal 2 includes an other device failure detection unit 290.

  The image generation unit 162 on the application server 1 side includes the operation information from the own device failure detection unit 180 in the changing image. Then, the detection unit 272 on the client terminal 1 side detects operation information from the received image. Note that a rule in which an image represents device operation information is stored in advance in the setting storage unit 170, the setting storage unit 280, and the like, for example.

  An example of a method for including failure information in a periodic change image will be described below. It is assumed that the image generated by the image generation unit 162 is, for example, 4 pixels. Each pixel is white or black. In the application server 1, three different programs (P1, P2, and P3) are operating. Each program state is assigned to three of the four pixels.

FIG. 11 shows an example of a periodic change image including failure information.
[FIG. 11 (a)]
It is an example of the change image at the time of normal operation. Here, all black and all white are transmitted alternately.
[FIG. 11 (b)]
It is an example of a change image when abnormality occurs in the program P1. Here, all black, only black corresponding to the program P1, and black for the other three are transmitted alternately.

[FIG. 11 (c)]
It is an example of a change image when abnormality occurs in programs P1 and P2. In this case, black images are all transmitted, black images corresponding to the programs P1 and P2, and white images are transmitted alternately.
[FIG. 11 (d)]
It is an example of a change image when abnormality occurs in the programs P1, P2, and P3. The images corresponding to all black, the programs P1, P2, and P3 corresponding to black and the other one alternately transmit white images.

  In the present embodiment, the own device failure detection unit 180 of the application server 1 monitors the operating states of the three programs (P1, P2, and P3). When an abnormality occurs in the program, the own device failure detection unit 180 outputs the abnormality information to the periodic change image generation unit 160. In the periodic change image generation unit 160, when abnormality information is input, the image generation unit 162 generates a periodic change image (for example, the change image shown in FIG. 11) using the information.

  On the other hand, in the client terminal 2, the detection unit 272 in the periodic change image confirmation unit 270 detects the periodic image change and outputs the detection result to the other device failure detection unit 290. The other device failure detection unit 290 can know the operating states (whether or not it is operating normally) of the three programs (P1, P2, and P3) on the application server 1 side using the input information.

  In the above, an example in which a program abnormality is reported is shown. However, the abnormality detection target is not limited to this. For example, an abnormality related to hardware may be transmitted. As an example, it is assumed that a camera module (imaging unit) is connected to the application server 1 (for example, USB connection). Then, it is assumed that the own device failure detection unit 180 can detect the following three.

(A) Hardware error of camera module (B) Software error of driver that drives camera module (C) Abnormality of display application that displays video acquired from camera module on screen

  In this case, similarly to the example of the three programs P1, P2, and P3 described above, different change images are set according to the detected abnormality. Thereby, the other device failure detection unit 290 of the client terminal 2 can know the operation states (whether or not it is operating normally) of (A), (B), and (C).

  Note that the number of pixels of the image for expressing the abnormality is not limited to the above. For example, fewer target abnormalities may be expressed with 3 pixels or less (1 pixel or 2 pixels). If there are more than four pixels, more abnormalities of the object can be expressed. The color, shape, size, display position, display number, etc. of the image are also free.

  As described above, in the present embodiment, the operating state of the application server 1 on the screen transmission side can be known. For example, it is possible to grasp on the client terminal 2 side what kind of software abnormality or hardware abnormality has occurred on the application server 1 side as the operating state.

(4) Other embodiments

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and equivalents thereof. Further, various embodiments can be realized by appropriately combining a plurality of constituent elements disclosed in each of the above embodiments. Some constituent elements may be omitted from all the constituent elements shown in the above embodiments.

  For example, each of the above-described embodiments can use a screen transfer method using a push-type screen transfer protocol that does not require a request on the screen receiving side. For example, necessary components such as the periodic change image generation unit 160 are set in the device on the screen transmission side. Necessary components such as the period change image confirmation unit 270 are also set in the device on the screen receiving side. And even if there is no screen acquisition request from the screen receiving side, the screen transmitting side transmits the change screen to the screen receiving side at a predetermined cycle. The generation of the change image and the detection of the change are as described above. That is, in each of the above-described embodiments, the screen acquisition request receiving unit 141, the screen acquisition request transmitting unit 211, and the processing corresponding thereto are not necessarily essential.

  Moreover, you may combine said each embodiment suitably. For example, one or both of the numerical information of the second embodiment and the device operation information of the third embodiment can be included in the change image of the first embodiment. In addition, specific contents, values, and the like of each information used in the embodiment are free and are not limited to specific contents and values.

  In addition, a part or all of the above embodiments can be configured as a dedicated circuit. Such a circuit may be variously conceived, for example, an IC chip such as an ASIC or CPU that realizes each function, other peripheral circuits, a system LSI that integrates a plurality of functions, and is not limited to a specific one. The scope of hardware processing and software processing is also free.

DESCRIPTION OF SYMBOLS 1 ... Application server 2 ... Client terminal 31, 32 ... Display part 41, 42 ... Input part 110 ... Application software 111 ... Screen generation part 112 ... Drawing execution part 120 ... Frame buffer 130 ... Screen capture part 131 ... Drawing detection part 132 ... Image holding unit 133 ... difference detection unit 134 ... difference area output unit 140 ... transmission / reception unit 141 ... screen acquisition request reception unit 142 ... screen information transmission unit 150 ... start / end control unit 151 ... detection unit 152 ... start processing unit 153 ... end processing Unit 160 ... periodic change image generation unit 161 ... period processing unit 162 ... image generation unit 163 ... image addition unit 170 ... setting storage unit 180 ... own device failure detection unit 210 ... transmission / reception unit 211 ... screen acquisition request transmission unit 212 ... screen information Reception unit 230 ... Drawing execution unit 240 ... Frame buffer 260 ... Start / end control unit 2 1 ... detector unit 262 ... start processing unit 263 ... end processing unit 270 ... period changes the image checking unit 271 ... detecting unit 271 ... periodic processing unit 273 ... determination unit 280 ... setting storage unit 290 ... other equipment failure detector

Claims (10)

A screen holding unit for holding screen information;
A screen information transmission unit that transmits screen information held by the screen holding unit via a communication network;
An image generation unit that generates a change image that changes in a predetermined cycle, and an image addition unit that includes the change image generated by the image generation unit in the screen information held by the screen holding unit. When,
In a screen receiving device that receives screen information via a communication network from a screen transfer device having:
A screen acquisition request transmitter for transmitting a screen information acquisition request to the screen transfer device;
A screen information receiving unit for receiving screen information transmitted from the screen transfer device in response to the acquisition request;
From the received screen information, having a detection unit that detects a change image that changes in a predetermined cycle,
The screen acquisition device, wherein the screen acquisition request transmitter transmits a screen information acquisition request at a timing of acquiring an image different from the previously received change image.   The screen receiving device according to claim 1, further comprising: a determination unit that determines an abnormality of the screen transfer device based on pixel information of the change image when the detection unit detects a change image. The change image is at least two pixels or more,
The screen reception device according to claim 2, wherein the determination unit determines an operation state of the screen transfer device based on a change in a combination of pixel information.   The screen receiving device according to claim 1, wherein the change image is one pixel.   5. The screen receiving apparatus according to claim 1, wherein the change image is an image that is displayed in a plurality of dots on the screen. 6.   The screen change apparatus according to claim 1, wherein the change image represents numerical information.   The screen receiving apparatus according to claim 6, wherein the numerical information is a sequence number. Drawing and non-drawing of the change image are repeated,
The screen receiving apparatus according to claim 1, wherein a place where the change image is drawn is displayed at a place away from a place where the change image was drawn last time. A screen holding unit that holds screen information, a screen information transmission unit that transmits screen information held by the screen holding unit via a communication network, an image generation unit that generates a change image that changes in a predetermined cycle, and From a screen transfer device having a periodic change image generation unit including an image addition unit including a change image generated by the image generation unit to screen information held by the screen holding unit, a computer or an electronic circuit transmits a communication network In the screen receiving method for receiving screen information via
The computer or electronic circuit includes a screen acquisition request transmission unit, a screen information reception unit, and a detection unit,
The screen acquisition request transmission unit transmits an acquisition request for screen information to the screen transfer device,
The screen information receiving unit receives screen information transmitted from the screen transfer device in response to the acquisition request,
From the screen information received by the detection unit, a change image that changes in a predetermined cycle is detected,
The screen acquisition request transmitting unit transmits a screen information acquisition request at a timing of acquiring an image different from the previously received change image. A screen holding unit that holds screen information, a screen information transmission unit that transmits screen information held by the screen holding unit via a communication network, an image generation unit that generates a change image that changes in a predetermined cycle, and From a screen transfer device having a periodic change image generation unit including an image addition unit including a change image generated by the image generation unit to screen information held by the screen holding unit, to a computer via a communication network In the screen reception program that receives screen information,
In the computer,
Send the screen information acquisition request to the screen transfer device,
Receiving the screen information transmitted from the screen transfer device in response to the acquisition request;
From the received screen information, the change image that changes at a predetermined cycle is detected,
The screen acquisition request transmission unit causes a screen information acquisition request to be transmitted at a timing of acquiring an image different from the previously received change image.

Images