JPWO2014174802A1 - Screen transmission system, image information transmitting device, and image information receiving device - Google Patents

Abstract

Provided is a screen transmission system capable of preventing a shift in display timing of a plurality of difference areas to be displayed simultaneously. The difference area set extraction unit 54 extracts a difference area set, which is a set of difference areas to be displayed simultaneously, from the difference area. The communication unit 56 transmits the encoded data of the difference area to the screen information receiving device 61. When the encoded data of the difference area is decoded, the determination unit 63 determines whether the encoded data of all the difference areas belonging to the same difference area set as the difference area has been decoded. When the encoded data of all the difference areas belonging to the difference area set is decoded, the update unit 64 updates the image information of the area corresponding to the difference area with the decoded image information.

Description

  The present invention relates to a screen transmission system that efficiently transmits image information, and an image information transmission device, an image information reception device, a screen transmission method, an image information transmission program, and an image information reception program applied to the screen transmission system.

  There is known a screen transmission system that transmits image information of a computer screen from a server to a client. In such a screen transmission system, in order to reduce the amount of communication data between the server and the client, a technique in which the server transmits image information of a difference area, which is an area whose display content has changed from the previous screen, is applied. Japanese Patent Application Laid-Open No. 2004-228561 describes a technique for determining the priority of each difference area and controlling the transmission order of the difference areas according to the priority. With the technique described in Patent Document 1, it is possible to shorten the delay time until a high-priority difference area is displayed on the client side, and to increase the display content update frequency.

  Patent Document 1 describes, as an example of a method for determining the priority, determining the priority based on the pixel value of the pixel included in the difference region and the area of the difference region. Furthermore, based on the pixel value of the pixel included in the difference area and the area of the difference area, a change information density indicating the degree of change in the pixel value with respect to the previous screen is calculated, and the priority is determined based on the change information density. It is described. As a calculation example of the change information density, it is described that the total change amount of the pixel value is divided by the square of the area. A technique for weighting the change information density is also described in Patent Document 1. For example, it describes that the change information density is weighted based on the difference between the time when the image information in the difference area was last transmitted and the current time, or the transmission frequency of the blocks included in the screen. Yes. It also describes that the change information density is weighted based on operation information sent from the client.

Table 2009/004996

  The system described in Patent Literature 1 determines the priority for each difference area extracted from a screen when the latest screen is generated. At this time, different priorities may be set for a plurality of difference areas to be displayed simultaneously on the client side. Then, the timing at which the difference area is updated on the client side is shifted, and the display becomes difficult for the user to see. Specifically, the difference area (hereinafter, sometimes referred to as the same meaning image display area) displaying the semantically identical contents should be displayed simultaneously on the client side. However, when different priorities are set for the difference area, the display timing of the difference area is shifted. For example, one moving image area may be divided and extracted as a plurality of difference areas. Such a plurality of difference areas correspond to the same meaning image display area. In such a case, if different priorities are assigned to the respective difference areas, the image update timing of each difference area divided from the moving image area is shifted on the client side. As a result, it becomes difficult for the user to view the moving image. Further, as in the technique described in Patent Document 1, when the difference between the time when the image information of the difference area was last transmitted and the current time is reflected in the priority, the timing difference once occurred is the later time. This may affect the transmission order of the difference areas detected in the above, and there is a problem that the visibility of the moving image is lowered.

  Accordingly, the present invention provides a screen transmission system capable of preventing a shift in display timing of a plurality of difference areas to be displayed simultaneously, and an image information transmitting device, an image information receiving device, and screen transmission applied to the screen transmission system. It is an object to provide a method, an image information transmission program, and an image information reception program.

  An image transmission system according to the present invention includes a screen information transmitting device that transmits image information on a screen, and a screen information receiving device that receives image information and displays a screen. The screen information transmitting device includes a current screen, a current screen, and a current screen. Compared with the previous screen before changing to the screen, the difference area extraction means for extracting the area on the current screen that changed with respect to the previous screen as the difference area, and the set of difference areas to be displayed simultaneously from the difference area Difference area set extraction means for extracting a certain difference area set, encoding means for encoding image information of a difference area belonging to the difference area set for each difference area set, and encoded data of the difference area as screen information receiving device The screen information receiving apparatus includes a decoding means for decoding the encoded data in the difference area received from the screen information transmitting apparatus, and the encoded information in the differential area is decoded when the encoded data in the differential area is decoded. A determination means for determining whether or not encoded data of all the difference areas belonging to the difference area set to which the difference area belongs is decoded, and it is determined that the encoded data of all the difference areas belonging to the difference area set has been decoded. Update means for updating the image information of the areas corresponding to all the difference areas belonging to the difference area set among the image information of the screen stored in the screen information receiving apparatus with the decoded image information; It is characterized by including.

  Further, the screen information transmitting apparatus according to the present invention compares the current screen with the previous screen before changing to the current screen, and extracts the difference area extraction that extracts the area on the current screen that has changed with respect to the previous screen as the difference area. Means, a difference area set extraction means for extracting a difference area set, which is a set of difference areas to be displayed at the same time, from the difference area, and for each difference area set, encodes image information of the difference area belonging to the difference area set. It is characterized by comprising encoding means and communication means for transmitting encoded data of the difference area.

  Further, the screen information receiving apparatus according to the present invention includes a decoding unit that decodes encoded data of a difference area that is an area changed from a stored screen, and the difference between the encoded data of the difference area when the encoded data of the difference area is decoded. A determination means for determining whether or not encoded data of all the difference areas belonging to the difference area set, which is a set of difference areas to be displayed at the same time, has been decoded, and all the difference areas belonging to the difference area set When it is determined that the encoded data has been decoded, the image information of the areas corresponding to all the difference areas belonging to the difference area set among the image information of the stored screen is the decoded image information. Update means for updating.

  Also, in the screen transmission method according to the present invention, the screen information transmitting apparatus compares the current screen with the previous screen before changing to the current screen, and the area on the current screen that has changed with respect to the previous screen is used as a difference area. Extract and extract a difference area set, which is a set of difference areas to be displayed simultaneously, from the difference area, encode the image information of the difference area belonging to the difference area set for each difference area set, and encode the difference area encoded data Is transmitted to the screen information receiving device, and when the screen information receiving device decodes the encoded data of the difference region received from the screen information transmitting device and decodes the encoded data of the difference region, the difference to which the differential region belongs When it is determined whether the encoded data of all the difference areas belonging to the area set has been decoded, and when it is determined that the encoded data of all the difference areas belonging to the difference area set has been decoded, the screen information receiving device Record Of this image as information on the screen, the image information of a region corresponding to all the differential area belonging to the differential area set, and updates the decoding image information.

  The image information transmission program according to the present invention compares the current screen with the previous screen before changing to the current screen, and extracts the area on the current screen that has changed with respect to the previous screen as a difference area. Difference region extraction processing, difference region set extraction processing for extracting a difference region set, which is a set of difference regions to be displayed simultaneously from the difference region, and encoding of image information of difference regions belonging to the difference region set for each difference region set And a communication process for transmitting the encoded data of the difference area.

  In addition, the image information receiving program according to the present invention is a decoding process for decoding the encoded data of the difference area, which is an area changed from the stored screen, when the encoded data of the difference area is decoded. A determination process for determining whether or not encoded data of all difference areas belonging to the difference area set, which is a set of difference areas to be displayed at the same time, has been decoded, and all belonging to the difference area set When it is determined that the encoded data of the difference area is decoded, the image information of the areas corresponding to all the difference areas belonging to the difference area set is decoded from the stored screen image information. An update process for updating with image information is executed.

  ADVANTAGE OF THE INVENTION According to this invention, the shift | offset | difference of the display timing of the several difference area which should be displayed simultaneously can be prevented.

It is a block diagram which shows the example of the screen transmission system of this invention. It is a schematic diagram which shows the example of the block in a screen. It is explanatory drawing which shows the example of the log | history for every block which a change log | history memory | storage part memorize | stores. It is a flowchart which shows the example of the process progress of the screen transmission system of this invention. It is explanatory drawing which shows the example of the area | region which can move within a screen. It is a block diagram which shows the principal part of the screen transmission system of this invention. It is a block diagram which shows the principal part of the image information transmitter of this invention. It is a block diagram which shows the principal part of the image information receiver of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an example of a screen transmission system of the present invention. The screen transmission system of the present invention includes a screen transmission server (image information transmission device) 1 and a screen reception client (image information reception device) 2. The screen transmission server 1 transmits image information of a computer screen to a screen reception client 2 via a communication network (not shown), and the screen reception client 2 displays a screen based on the image information. For example, the screen reception client 2 is a thin client, and the screen transmission server 1 is a server that transmits image information to the thin client.

  The screen transmission server 1 includes a screen generation unit 8, a screen acquisition unit 9, a current screen information storage unit 11, a transmission screen information storage unit 12, a difference area extraction unit 13, and a change history storage unit 14. A degree calculation unit 15, a difference area set extraction unit 16, a priority determination unit 17, an encoding unit 18, a server communication unit 19, and a transmission screen information update unit 10.

  The current screen information storage unit 11 stores image information of the latest screen (hereinafter sometimes referred to as the current screen).

  The screen generation unit 8 generates image information of the current screen. The screen acquisition unit 9 acquires image information of the current screen generated by the screen generation unit 8 and stores it in the current screen information storage unit 11.

  The transmission screen information storage unit 12 stores image information of the most recent screen (previous screen) that the screen transmission server 1 has already transmitted to the screen reception client 2. After the image information of the difference area extracted from the current screen is encoded and the encoded image information of the difference area is transmitted to the screen reception client 2, the transmission screen information update unit 10 transmits the transmission screen information storage unit 12 The image information of the area corresponding to the difference area is updated in the image information of the screen stored in. With this operation, the transmission screen information storage unit 12 stores the image information of the previous screen.

  The difference area extraction unit 13 compares the image information of the current screen stored in the current screen information storage unit 11 with the image information of the previous screen stored in the transmission screen information storage unit 12, and starts from the previous screen. A difference area that is a changing area is extracted. It can also be said that the difference area is a continuous set of pixels changed from the current screen. The difference area extraction unit 13 extracts each difference area when there are a plurality of difference areas.

  The change history storage unit 14 stores a history of whether or not there is a change for each block in the screen. A block is an individual area when a screen is divided into areas of a fixed size. One pixel may be defined as one block. That is, the change history storage unit 14 may store a history of whether or not there is a change for each pixel in the screen.

  FIG. 2 is a schematic diagram illustrating an example of a block. Although FIG. 2 shows an example in which one screen is divided into 16 blocks, the number of divisions for one screen is not particularly limited. As described above, one pixel may be one block. FIG. 3 is an explanatory diagram illustrating an example of a history for each block stored in the change history storage unit 14. The change history storage unit 14 stores the presence / absence of a change in each block in the past N images. Here, the time on the screen is represented by t. The time immediately before the screen at time t is t-1. When a certain block i on the screen at a certain time t changes from the same block i on the previous screen at the time t−1, the change history storage unit 14 “changes” with respect to the block i at the time t. Remember. When a certain block i on the screen at a certain time t has not changed from the same block i on the previous screen at the time t−1, the change history storage unit 14 relates to the block i at the time t. Memorize “no change”. Hereinafter, “with change” is represented by “1”, and “no change” is represented by “0”.

  The change history storage unit 14 stores the presence / absence of a change for each block regarding the past N screens. In FIG. 2, the change history in the blocks 1 and 2 is illustrated, but the change history storage unit 14 similarly stores the change history for other blocks. In the example shown in FIG. 3, the change history of block 1 is represented by an N-dimensional vector of (1, 0,..., 0, 1, 0). The change history of other blocks can also be represented by N-dimensional vectors.

  Hereinafter, the presence or absence of a change at time t of block i is represented as f (i, t). At this time, if a vector representing a history of changes in the past N screens in the block i is x (i), the vector x (i) is expressed by the following equation (1).

x (i) = (f (i, t-1), f (i, t-2), ..., f (i, t-N)) Equation (1)

The nth element of the N-dimensional vector x (i) is denoted as x _n (i). Here, the block i has been described as an example, but the same applies to other blocks.

  The operation for storing the change history for each block in the change history storage unit 14 may be performed by, for example, the difference area extraction unit 13.

  The dissimilarity calculation unit 15 calculates the dissimilarity between the difference areas extracted from the current screen based on the history of the presence / absence of changes for each block. The “dissimilarity” is a value representing the degree of dissimilarity between the difference areas. The dissimilarity becomes larger as the difference areas are not similar. Further, the dissimilarity becomes smaller as the difference areas are more similar.

  Hereinafter, an example of calculating the dissimilarity between the difference areas will be shown. The shape of the difference area extracted by the difference area extraction unit 13 is an arbitrary shape, but here, in order to simplify the description, the case where each difference area corresponds to one block will be described as an example. . In each example shown below, the dissimilarity calculation unit 15 calculates the dissimilarity between the difference regions using an N-dimensional vector indicating a change history in the block corresponding to the difference region as a feature vector. In each example shown below, an example is given in which one difference area is block i and the other difference area is block j, and the dissimilarity between the two difference areas is obtained.

[Example 1]
The dissimilarity calculation unit 15 uses the simple Hamming distance (in other words, the number of mismatches) between the feature vector x (i) of the block i and the feature vector x (j) of the block j as the dissimilarity between them. It may be calculated. Let d (i, j) be the distance between feature vectors x (i) and x (j). In Example 1, the dissimilarity calculation unit 15 calculates d (i, j) by the following expression (2), and sets d (i, j) as the dissimilarity between blocks.

  Note that g (a, b) in equation (2) is the calculation shown in equation (3).

That is, g (a, b) = 0 if a and b are equal, and g (a, b) = 1 if a and b are different. In equation (2), x _n (i) and x _n (j) being equal means that the block i, j has changed at the same timing, or both the blocks i, j have the same timing. It means no change has occurred. In Expression (2), x _n (i) and x _n (j) being different means that a change has occurred on the one hand, but no change has occurred on the other hand.

  When calculating dissimilarity as described above, when a change occurs in one block, the change also occurs in the other block, and when no change occurs in one block, there is a tendency that no change occurs in the other block. The stronger, the lower the dissimilarity. That is, the dissimilarity becomes lower as the tendency of the timing of the change is more similar.

  Example 1 is a method of calculating dissimilarity by similarly treating the fact that changes occur at the same time and no change at the same time.

[Example 2]
In Example 2, the difference between the evaluation that the blocks i and j are simultaneously changed and the evaluation that the blocks i and j are not changed at the same time is different. Assess dissimilarity low (evaluate that the degree of similarity is high). In other words, Example 2 is a method of focusing on the fact that changes occur at the same time, and evaluating the dissimilarity as low if there is a strong tendency for the number of changes to occur simultaneously. In Example 2, the dissimilarity calculation unit 15 calculates d (i, j) by, for example, the following expression (4), and sets d (i, j) as the dissimilarity between blocks.

  However, g (a, b) in Expression (4) is, for example, an operation shown in Expression (5).

That is, if both a and b are 1, g (a, b) = 0, and if both a and b are 0, g (a, b) = 1/4, and a and b are different. In this case, g (a, b) = 1. In Expression (4), x _n (i) and x _n (j) are both 1 means that the blocks i and j have changed at the same timing. In Expression (4), x _n (i) and x _n (j) are both 0, which means that neither block i nor j has changed at the same timing. In Expression (4), x _n (i) and x _n (j) being different mean that a change has occurred on the one hand, but no change has occurred on the other hand.

[Example 3]
Example 3 evaluates the dissimilarity lower when the tendency that the blocks i and j change at the same time or both do not change at the same timing becomes stronger toward the present than the past (similar Method). For example, the dissimilarity calculation unit 15 calculates d (i, j) by the calculation of Expression (6) below, and sets d (i, j) as the dissimilarity between blocks.

However, w ₁ > w ₂ >...> W _N and w ₁ , w ₂ ,..., W _N are all 0 or more. For example, C may be a value less than 1 and w _n = C ⁿ⁻¹ .

  Note that g (a, b) in Example 3 may be the calculation shown in Expression (3) or the calculation shown in Expression (5).

  The dissimilarity calculation methods shown in Examples 1 to 3 are methods for calculating the dissimilarity based on the change in history and the tendency of no change.

[Example 4]
Example 4 is a method of calculating the dissimilarity by using the distance between the image features of the blocks i and j together with the change in history and the tendency of no change. In Example 4, the dissimilarity calculation unit 15 calculates d (i, j) by the following equation (7), and sets d (i, j) as the dissimilarity between blocks.

[Example 5]
Example 5 is a method of calculating the dissimilarity by using the spatial distance of the blocks i and j in the screen together with the change in the history and the tendency of no change. In Example 5, the dissimilarity calculation unit 15 calculates d (i, j) by the following equation (8), and sets d (i, j) as the dissimilarity between blocks.

  What is necessary is just to calculate the 1st term of the right side in Formula (7) and Formula (8) similarly to Example 1 or Example 2. Further, the right side of Expression (6) shown in Example 3 may be used as the first term of the right side in Expression (7) and Expression (8).

  In the above example, the case where the dissimilarity d (i, j) between two blocks is calculated based on the history of the presence or absence of past changes has been shown. When calculating the dissimilarity, the dissimilarity calculating unit 15 may add a term whose value decreases as the in-screen positions of the two blocks are closer to the value calculated based on the history. In addition, when information indicating the possibility that two blocks belong to areas representing semantically identical contents is obtained from an OS (Operating System) or application software running on a computer, The similarity calculation unit 15 may calculate the dissimilarity between blocks using the information.

  In each of the above examples, in order to simplify the description, the case where each difference area corresponds to one block has been described as an example. Since the shape of the difference area is an arbitrary shape, one difference area does not necessarily correspond to one block, and the difference area is configured by a set of one or a plurality of blocks. In this case, for example, the difference area extraction unit 13 divides the difference area so that the number of blocks included in the difference area is 1, and the dissimilarity calculation unit 15 determines the dissimilarity between the difference areas between the blocks. It is one method for calculating the dissimilarity between the difference areas to calculate by replacing with the similarity.

In addition, the dissimilarity calculation unit 15 may use an average value of dissimilarities between blocks constituting two difference areas as a dissimilarity between difference areas. Specifically, block i ₁ of the differential area 1 are K, i _2, ..., is composed of i _K, the difference region 2 of the L block j _1, j _2, ..., from j _L When configured, the dissimilarity calculation unit 15 calculates an average value of the K × L sets of interblock dissimilarities d (i _k , j _l ), and calculates the average value of the difference region 1 and the difference region 2. Similarity. Here, the block constituting the difference area means a block in which all or part of the pixels in the block belong to the difference area. Another method is to add a low weight to blocks that are partly included in the difference area, and apply a high weight to blocks that are all included in the difference area to calculate a weighted average of dissimilarities between blocks. Also good. Alternatively, instead of the average value of the dissimilarity between blocks, an intermediate value or the like may be used as the dissimilarity between the difference areas.

  In addition, the dissimilarity calculation unit 15 calculates the N-dimensional feature vector of the difference area from the mode value or the intermediate value of the N-dimensional feature vector of the blocks constituting the difference area, and uses the N-dimensional feature vector. Thus, the dissimilarity between the difference areas may be calculated by performing the same calculation as the dissimilarity calculation between the blocks. Alternatively, the dissimilarity calculation unit 15 calculates the dissimilarity between the difference regions according to the equation (4) by using the average value of the N-dimensional feature vectors of the blocks constituting the difference region as the N-dimensional feature vector of the difference region. Also good. However, g (a, b) is defined as a real-valued function defined for 0 ≦ a ≦ 1, 0 ≦ b ≦ 1 that satisfies any of the following conditions.

(Condition 1)
The smaller the difference between a and b, the smaller g (a, b).
(Condition 2)
When condition 1 is satisfied and the difference between a and b is approximately the same, g (a, b) takes a smaller value as a and b are closer to 0 or 1 (the average value of a and b).
(Condition 3)
G (a, b) takes a smaller value when both condition a and b are close to 0 than when both condition a and b are close to 0.

  Examples of g (a, b) satisfying Condition 1 include Expression (9) and Expression (10). In this case, the dissimilarity is evaluated to be small when the ratio of the number of blocks that have changed in the two difference areas is approximately the same as the ratio of the number of blocks that have not changed.

  As an example of g (a, b) satisfying the condition 2, there is a formula (11). In this case, the dissimilarity is evaluated to be small when most of the blocks change in the two difference areas, or when most of the blocks do not change in the two difference areas, respectively. In addition, when there are mixed blocks that have changed and blocks that have not changed in the two difference areas, the dissimilarity is a large value.

  As an example of g (a, b) satisfying the condition 3, there is a formula (12). In this case, the dissimilarity when most of the blocks do not change in the two difference areas (when both a and b are close to 0) is the case where most of the blocks change in the two difference areas (a , b are both close to 1), the evaluation is greater than the dissimilarity.

g (a, b) = | ab | Formula ² (9)

  g (a, b) = | a-b | Formula (10)

  g (a, b) = a (1-b) + b (1-a) Formula (11)

  Note that the method for determining an N-dimensional feature vector corresponding to a difference area having an arbitrary shape is not limited to the above example.

  Based on the dissimilarity between the difference areas, the difference area set extraction unit 16 extracts a set of difference areas to be displayed at the same time (in other words, a set of difference areas that display semantically identical contents). . Hereinafter, each set of difference areas extracted by the difference area set extraction unit 16 is referred to as a semantic set. The difference area set extraction unit 16 extracts a semantic set by performing clustering based on the dissimilarity between the difference areas for each difference area extracted from the current screen by the difference area extraction unit 13. The clustering method may be a general clustering method such as a k-nearest neighbor method, a k-average method, or a self-organizing map. Alternatively, using an image region segmentation method based on clustering that is corrected so that the connectivity of the difference regions included in the set of difference regions displaying not only the small dissimilarity but also the semantically identical contents is increased. Also good.

  Further, the difference area set extraction unit 16 assigns a semantic set identifier to each extracted semantic set. Further, the difference area set extraction unit 16 counts the number of difference areas belonging to the semantic set for each extracted semantic set.

  The priority determination unit 17 determines the priority of the transmission order of the encoded data for each semantic set extracted by the difference region set extraction unit 16. The encoded data is data obtained by encoding the image information in the difference area. For example, for each semantic set, the priority determination unit 17 performs the following calculation based on the pixel values of the pixels included in each differential region belonging to the semantic set and the area of each differential region.

  (Sum of changes in pixel values of pixels included in each difference region belonging to the semantic set) / (Area of the entire difference region belonging to the semantic set) squared

  This calculation result is referred to as change information density. The change information density is a value representing the degree of change of the pixel value in each difference area belonging to the semantic set, that is, the magnitude of change as a picture in each difference area belonging to the semantic set. The priority determination unit 17 determines the priority of each semantic set based on the change information density. Here, a case where the change information density itself is used as a priority is taken as an example.

  Note that the priority determination method is not limited to the above example, and the priority determination unit 17 may determine the priority of the semantic set by another method.

  The encoding unit 18 encodes the image information of the difference areas belonging to the semantic set in order from the semantic set with the highest priority, and writes it in a buffer (not shown) held by the server communication unit 19. As described above, data obtained by encoding the image information in the difference area is referred to as encoded data.

  The server communication unit 19 transmits the encoded data of the difference area written in the buffer to the screen reception client 2 in the order written in the buffer. Accordingly, the server communication unit 19 sequentially transmits the encoded data in the difference area belonging to the semantic set having a high priority.

  Further, when the server communication unit 19 transmits the encoded data of the difference area to the screen receiving client 2, the server communication unit 19 belongs to the encoded identifier of the difference area, the identifier of the semantic set to which the differential area belongs, and the semantic set. Information on the number of difference areas is added and transmitted to the screen receiving client 2. The information on the number of difference areas belonging to the semantic set identifier and the semantic set is information determined by the differential area set extraction unit 16.

  Screen generation unit 8, screen acquisition unit 9, transmission screen information update unit 10, difference region extraction unit 13, dissimilarity calculation unit 15, difference region set extraction unit 16, priority determination unit 17, encoding unit 18 and server communication The unit 19 is realized by a CPU of a computer that operates according to an image information transmission program, for example. For example, the CPU reads the image information transmission program, and according to the program, the screen generation unit 8, the screen acquisition unit 9, the transmission screen information update unit 10, the difference area extraction unit 13, the dissimilarity calculation unit 15, and the difference area set extraction unit 16, the priority determination unit 17, the encoding unit 18, and the server communication unit 19 may be operated. The image information transmission program may be stored in a computer-readable recording medium. Each of these elements may be realized by separate hardware.

  The screen reception client 2 includes a client communication unit 21, a decryption unit 22, a difference area update unit 23, a screen information storage unit 24, and a screen output unit 25.

  The client communication unit 21 receives the combination of the encoded data of the difference area transmitted by the server communication unit 19, the identifier of the semantic set to which the differential area belongs, and the information on the number of differential areas that belong to the semantic set. To do.

  The decoding unit 22 decodes the encoded data of the difference area received by the client communication unit 21.

  When the decoding of the encoded data of the difference area corresponding to the identifier is completed by the number of difference areas belonging to the semantic set specified by one identifier, the difference area update unit 23 stores the difference information in the screen information storage unit 24. The image information of the area corresponding to each difference area is specified from the image information of the displayed screen. The difference area update unit 23 replaces the image information of the specified area with the decoded image information of each difference area. In other words, the difference area update unit 23 receives the image information of the screen stored in the screen information storage unit 24 after the encoding information of all the difference regions belonging to one semantic set is received and decoded by the decoding unit 22. The image information corresponding to each of the difference areas is updated.

  The screen information storage unit 24 stores image information to be output to the screen output unit 25.

  The screen output unit 25 is a display device that displays an image based on the image information stored in the screen information storage unit 24. As described above, after all the encoded information of all the difference areas belonging to one semantic set is decoded, the image information of the areas corresponding to the difference areas is replaced with the decoded information. Accordingly, the screen output unit 25 simultaneously updates the images of all the difference areas belonging to one semantic set.

  The client communication unit 21, the decoding unit 22, and the difference area update unit 23 are realized by a CPU of a computer that operates according to an image information reception program, for example. For example, the CPU may read the image information reception program and operate as the client communication unit 21, the decoding unit 22, and the difference area update unit 23 according to the program. The image information receiving program may be stored in a computer-readable recording medium. Each of these elements may be realized by separate hardware.

  FIG. 4 is a flowchart showing an example of processing progress of the screen transmission system of the present invention.

  When generating the image information of the latest screen (current screen), the screen generation unit 8 passes the image information to the screen acquisition unit 9 (step S1). Then, the screen acquisition unit 9 stores the current screen image information acquired from the screen generation unit 8 in the current screen information storage unit 11 (step S2).

  The difference area extraction unit 13 displays the current difference area based on the current screen image information stored in the current screen information storage unit 11 and the previous screen image information stored in the transmission screen information storage unit 12. Extract from the screen (step S3).

  The dissimilarity calculation unit 15 refers to the history of presence / absence of change for each block stored in the change history storage unit 14, and based on the history of presence / absence of change for each block, the difference region extracted in step S3 The dissimilarity between each other is calculated (step S4).

  For example, after step S4, the difference area extraction unit 13 determines for each block whether or not a change has occurred in the current image from the previous image, and stores the determination result for each block in the change history storage unit 14. . At this time, the difference area extraction unit 13 deletes the oldest history of each block stored in the change history storage unit 14. As a result, the history of changes in each block stored in the change history storage unit 14 (specifically, an N-dimensional vector for each block) is updated. Note that the update method and update timing of the history of each block stored in the change history storage unit 14 are not limited to the above example. In another aspect, the history of each block stored in the change history storage unit 14 may be updated.

  After step S4, the difference area set extraction unit 16 extracts a semantic set by performing clustering on each difference area extracted in step S3 based on the dissimilarity between the difference areas (step S4). S5). Each difference area that displays semantically identical contents belongs to one semantic set. In step S5, the difference area set extraction unit 16 assigns a semantic set identifier to each extracted semantic set. Further, in step S5, the difference area set extraction unit 16 counts the number of difference areas belonging to the semantic set for each semantic set.

  Next, the priority determination part 17 determines a priority with respect to each meaning set extracted at step S5 (step S6).

  Then, the encoding unit 18 encodes the image information of the difference areas belonging to the semantic set in order from the semantic set with the highest priority, and writes the encoded image information in a buffer (not shown) held by the server communication unit 19 (step S7).

  The server communication unit 19 adds the identifier of the semantic set to which the differential area belongs and the number of differential areas that belong to the semantic set to the encoded data of the differential area in the order written in the buffer. The combination of the encoded data, the identifier of the semantic set, and the information of the number of difference areas belonging to the semantic set is transmitted to the screen receiving client 2 (step S8).

  Then, the transmission screen information update unit 10 among the image information of the screen stored in the transmission screen information storage unit 12, the image information of the region corresponding to the difference region to which the encoded data has already been transmitted is stored in the difference region. Update with image information (step S9).

  When the screen generation unit 8 newly generates screen image information, the screen transmission server 1 repeats the processing from step S1 onward.

  In addition, the client communication unit 21 of the screen reception client 2 transmits the encoded data of the difference region transmitted by the server communication unit 19 in step S8, the identifier of the semantic set to which the differential region belongs, and the difference belonging to the semantic set. Each combination of information on the number of areas is received (step S21).

  The decoding unit 22 decodes the encoded data of the difference area received by the client communication unit 21 in the order of reception (step S22).

  When the encoded difference area data is decoded, the differential area update unit 23 decodes encoded data of all the differential areas belonging to the semantic set indicated by the identifier transmitted together with the encoded data of the differential area. It is determined whether or not it is completed (step S23). The difference area updating unit 23 transmits the number of difference areas transmitted together with the identifier of the semantic information (that is, the number of difference areas belonging to the semantic set indicated by the identifier) and the number of decoded difference areas corresponding to the identifier. And compare. Then, when the two match, the difference area update unit 23 determines that all the difference areas of the semantic set to which the decoded difference area belongs have been decoded (Yes in step S23). Further, if the two do not match, the difference area update unit 23 determines that decoding of all the difference areas of the semantic set to which the decoded difference area belongs has not been completed yet (No in step S23).

  If decoding of all the difference areas of the semantic set to which the decoded difference area belongs has not been completed yet (No in step S23), the processes in steps S22 and S23 are repeated.

  When the decoding of all the difference areas of the semantic set to which the decoded difference area belongs is completed (Yes in step S23), the difference area update unit 23 stores the image information on the screen stored in the screen information storage unit 24. From this, the image information of the area corresponding to each difference area belonging to the semantic set is specified. Then, the difference area update unit 23 replaces the image information of the specified area with the image information of each difference area belonging to the semantic set (step S24). Accordingly, in step S24, the difference area update unit 23 updates the image information stored in the screen information storage unit 24 in a semantic set unit.

  When the screen image information stored in the screen information storage unit 24 is updated, the screen output unit 25 displays an image based on the image information (step S25).

  After step S25, when the image information stored in the screen information storage unit 24 is updated again in the semantic set unit (step S24), a new image is displayed based on the image information (step S25). . Thus, steps S24 and S25 are repeated.

  According to this embodiment, the difference area set extraction unit 16 classifies the difference areas extracted from the current screen into semantic sets. Then, the priority determination unit 17 determines the priority of the semantic set, and the image information of the difference area is encoded and transmitted to the screen reception client 2 in the order of the priority of the semantic set. Then, in the screen receiving client 2, when the decoding of the encoded data of all the difference areas belonging to one semantic set is completed, the difference area update unit 23 stores the screen information stored in the screen information storage unit 24. Among them, the image information of the area corresponding to each difference area is updated, and the screen output unit 25 displays an image based on the updated image information.

  Thus, in this embodiment, the screen transmission server 1 determines a semantic set, and the screen reception client 2 updates the image information for each semantic set. A plurality of difference areas to be displayed at the same time are classified into the same semantic set. Accordingly, it is possible to prevent a shift in display timing of a plurality of difference areas to be displayed at the same time.

  In addition, it is considered that the difference areas displaying the semantically identical contents change at the same timing or remain unchanged at the same timing. In the present invention, the dissimilarity between the difference areas is calculated based on the history of whether or not there is a change, and the difference areas are clustered. Therefore, a semantic set can be extracted appropriately.

  Next, various modifications of the embodiment of the present invention will be described.

  In some cases, a partial area is defined in the screen, an image is displayed in the area, and the area is moved by a user operation. FIG. 5 is an explanatory diagram showing an example of a movable area in the screen. An area 32 shown in FIG. 5 is an area defined in the screen 31 and can be moved by a user operation. An example of such an area 32 is a window displayed on the screen.

  FIG. 5A shows an example of a screen before the area 32 is moved. FIG. 5B is an example of the screen after the area 32 is moved. For example, when the operation of moving the area 32 is performed in the screen reception client 2 that is a thin client, the screen transmission server 1 generates screen information of the screen illustrated in FIG.

  A range occupied by the region 32 before the movement is referred to as a range A. A range occupied by the area 32 after movement is referred to as a range B. Assume that the region 32 has moved from the range A to the range B as shown in FIG. In this case, for example, the difference area extracting unit 13 updates the history of whether or not each block overlapping the range B stored in the change history storage unit 14 with the history of whether or not each block overlapping the range A is changed. Also good. That is, the difference area extraction unit 13 may cause each block that overlaps the range B to inherit the history of each block that overlaps the range A. Here, although it demonstrates as operation | movement of the difference area extraction part 13, the log | history of each block which overlaps with the range B which the area | region 32 after movement occupies is updated with the log | history of each block which overlaps with the range A which the area | region 32 before movement moves. The processing may be performed by a processing unit other than the difference area extraction unit 13. Alternatively, a dedicated processing unit that performs this process may be provided in the screen transmission server 1.

  The tendency of the image change in the region 32 is considered to be the same as the tendency of the image change before the movement even after the movement. Therefore, as described above, by updating the history of the presence / absence of each block that overlaps the range B with the history of the presence / absence of each block that overlaps the range A, the dissimilarity between the difference regions is calculated with higher accuracy. As a result, the extraction accuracy of the semantic set can be improved.

  In addition, after the area 32 has moved, the difference area extraction unit 13 updates the history of each block that overlaps the range A occupied by the area 32 before the movement with a value indicating that the presence or absence of the change is indefinite. Also good. For example, 1/2 may be used as a value indicating that the presence or absence of the change is indefinite. For example, after the area 32 has moved, the difference area extraction unit 13 calculates (1/2, 1/2) an N-dimensional vector representing the history of each block overlapping the range A occupied by the area 32 before the movement. ,..., 1/2).

  Further, when a value (1/2 in this example) indicating that there is no change is used for the block history, the dissimilarity calculation unit 15 performs the above-described calculation of g (a, b). Change it.

  For example, when calculating the dissimilarity by the method of Example 1 described above, the dissimilarity calculation unit 15 may perform the following expression (3a) instead of the expression (3).

Next, another modification of the above-described embodiment will be described.
The difference area extraction unit 13 may divide the difference area into separate difference areas when a predetermined condition is satisfied for each extracted difference area. For example, the difference area extraction unit 13 sequentially selects the extracted difference areas one by one. Then, the difference area extraction unit 13 determines a plurality of pixel sets within the selected difference area (that is, determines a plurality of partial areas within the difference area). This determination method is not particularly limited. The difference area extraction unit 13 calculates the dissimilarity between the pixel sets. This dissimilarity calculation method may be similar to the method in which the dissimilarity calculation unit 15 calculates the dissimilarity between the difference regions. When there is a pixel set that has a significantly high degree of dissimilarity with other pixel sets, the difference area extraction unit 13 divides the pixel set from the difference area and sets the pixel set as a separate difference area. Also good. Note that whether or not the dissimilarity with other pixel sets is extremely high may be determined based on whether or not the dissimilarity with other pixel sets is equal to or more than a predetermined threshold.

  Further, in the above-described embodiment, the case where the priority determination unit 17 uses the change information density calculated for each semantic set as a priority has been described as an example. The priority determination unit 17 may determine the priority of the semantic set by other methods.

  For example, the priority determination unit 17 may determine the result of weighting the change information density of the semantic set calculated as described above as the priority of the semantic set. Hereinafter, various methods for weighting the change information density will be described.

  For example, the priority determination unit 17 identifies a block that is at least partially encoded and transmitted to the screen reception client 2 from among the blocks that overlap with the difference region belonging to the semantic set, and determines the transmission time of the encoded data of the block. Calculate time to current time. The priority determination unit 17 may increase the weight as this time is longer, weight the change information density, and determine the result as the priority.

  Further, for example, the priority determination unit 17 quantitatively evaluates the transmission frequency at which each block overlapping with the difference region belonging to the semantic set is encoded and transmitted to the screen reception client 2. Then, the priority determination unit 17 may increase the weight as the transmission frequency decreases, weight the change information density, and determine the result as the priority.

  The priority determination unit 17 may use a reciprocal of the distance between the position of the cursor in the screen of a pointing device such as a mouse and the center of gravity of the entire difference area belonging to the semantic set as a weight. The priority determination part 17 may weight change information density using the weight, and may determine the result as a priority.

  The priority determination unit 17 sets the weight to 1 when the cursor moving speed is less than the threshold value, and increases the smaller the area of the entire difference region belonging to the semantic set when the cursor moving speed is equal to or higher than the threshold value. May be used. The priority determination part 17 may weight change information density using the weight, and may determine the result as a priority.

  The priority determination unit 17 sets the weight to 1 when the key operation frequency in the screen receiving client 2 is less than the threshold, and when the key operation frequency is equal to or greater than the threshold, the area of the entire difference area belonging to the semantic set. You may use the weight which becomes a large value, so that becomes small. The priority determination part 17 may weight change information density using the weight, and may determine the result as a priority.

  Information such as the cursor position, cursor movement speed, or key operation frequency may be transmitted from the screen reception client 2 to the screen transmission server 1 as user operation information.

  Hereinafter, the main part of the present invention will be described. FIG. 6 is a block diagram showing the main part of the screen transmission system of the present invention. FIG. 7 is a block diagram showing the main part of the image information transmitting apparatus of the present invention. FIG. 8 is a block diagram showing the main part of the image information receiving apparatus of the present invention.

  The screen transmission system of the present invention includes an image information transmission device 51 (for example, the screen transmission server 1) and an image information reception device 61 (for example, the screen reception client 2).

  The image information transmission device 51 includes a difference area extraction unit 52, a difference area set extraction unit 54, an encoding unit 55, and a communication unit 56 (see FIGS. 6 and 7).

  The difference area extraction unit 52 (for example, the difference area extraction unit 13) compares the current screen with the previous screen before changing to the current screen, and sets the area on the current screen changed with respect to the previous screen as the difference area. Extract.

  The difference area set extraction unit 54 (for example, the difference area set extraction unit 16) extracts a difference area set that is a set of difference areas to be displayed simultaneously from the difference area.

  The encoding unit 55 (for example, the encoding unit 18) encodes the image information of the difference area belonging to the difference area set for each difference area set.

  The communication unit 56 (for example, the server communication unit 19) transmits the encoded data of the difference area to the screen information receiving device 61.

  Further, the image information receiving apparatus 61 includes a decoding unit 62, a determination unit 63, and an update unit 64 (see FIGS. 6 and 8).

  The decoding unit 62 (for example, the decoding unit 22) decodes the encoded data of the difference area received from the screen information transmitting device 51.

  Whether or not the determination unit 63 (for example, the difference region update unit 23) has decoded the encoded data of all the difference regions belonging to the difference region set to which the difference region belongs when the encoded data of the difference region is decoded. Determine whether.

  The update unit 64 (for example, the difference region update unit 23), when it is determined that the encoded data of all the difference regions belonging to the difference region set has been decoded, the screen information stored in the screen information receiving device 61 is stored. Of the image information, the image information of the areas corresponding to all the difference areas belonging to the difference area set is updated with the decoded image information.

  With such a configuration, it is possible to prevent a shift in display timing of a plurality of difference areas to be displayed simultaneously.

  A part or all of the above embodiments can be described as in the following supplementary notes, but is not limited to the following.

(Supplementary Note 1) A screen information transmitting apparatus that transmits image information of a screen and a screen information receiving apparatus that receives the image information and displays a screen, the screen information transmitting apparatus including a current screen and the current screen A difference area extracting means for comparing the previous screen before the change to the previous screen and extracting the area on the current screen that has changed with respect to the previous screen as a difference area; A difference area set extracting means for extracting a difference area set as a set; an encoding means for encoding image information of a difference area belonging to the difference area set for each difference area set; A communication means for transmitting to the information receiving device, wherein the screen information receiving device decodes the encoded data of the difference region received from the screen information transmitting device, and the encoded data of the difference region is decoded. Determining means for determining whether or not encoded data of all the difference areas belonging to the difference area set to which the difference area belongs has been decoded, and decoding of the encoded data of all the difference areas belonging to the difference area set When it is determined that the image information on the screen stored in the screen information receiving device is stored, the image information of the areas corresponding to all the difference areas belonging to the difference area set is decoded image information. An image transmission system comprising an update means for updating.

(Supplementary note 2) The screen information transmitting apparatus includes dissimilarity calculating means for calculating the dissimilarity between the difference areas, and the difference area set extracting means extracts the difference area set based on the dissimilarity between the difference areas. The image transmission system according to appendix 1.

(Supplementary Note 3) The screen information transmitting apparatus includes a change history storage unit that stores a history of whether or not there is a change for each block in the screen, and the dissimilarity calculation unit includes the presence or absence of a change stored in the change history storage unit The image transmission system according to supplementary note 2, wherein the dissimilarity between the difference areas is calculated based on the history.

(Supplementary note 4) The screen information transmitting apparatus includes priority determining means for determining the priority of the transmission order of the encoded data for each difference area set, and the communication means sets the difference area set in the order of priority of the difference area set. 4. The image transmission system according to appendix 2 or appendix 3, wherein the encoded data of the image information of the difference area belonging to is transmitted.

(Appendix 5) The dissimilarity calculation means sets the dissimilarity between the difference regions to a smaller value as the frequency of both changes at the same timing and the frequency of no change at the same timing increases. The image transmission system according to claim 1.

(Appendix 6) The dissimilarity calculation means places more importance on the frequency at which both changes occur at the same timing and the frequency at which both changes occur at the same timing. The image transmission system according to appendix 5, wherein the dissimilarity between the difference areas is set to a smaller value as the frequency of occurrence of both changes at the same timing increases.

(Supplementary note 7) The dissimilarity between the difference regions is set to a smaller value as the frequency at which the change occurs at the same timing and the frequency at which the change does not occur at the same timing increase at a time closer to the current time. The image transmission system according to appendix 5 or appendix 6.

(Additional remark 8) The difference area extraction means calculates the dissimilarity between the partial areas in the difference area, and in the difference area, the partial area whose dissimilarity with other partial areas is greater than or equal to the threshold The image transmission system according to any one of Supplementary Note 1 to Supplementary Note 7, wherein the differential region is divided from the differential region into another differential region.

(Supplementary note 9) When a movable area moves on the screen, the screen information transmitting apparatus occupies a history of whether or not there is a change in a block overlapping with a range occupied by the area after the movement, before the movement 9. The image transmission system according to any one of appendix 3 to appendix 8, comprising history rewriting means for updating with a history of presence / absence of a change in a block overlapping the range.

(Supplementary Note 10) The communication means transmits the encoded data of the difference area, the identifier of the difference area set to which the difference area belongs, and the number of difference areas that belong to the difference area set to the screen information receiving device. When the encoded data of the region is decoded, based on the identifier of the differential region set transmitted together with the encoded data of the differential region and the number of differential regions belonging to the differential region set, the differential region set indicated by the identifier The image transmission system according to any one of Supplementary Note 1 to Supplementary Note 8, wherein it is determined whether or not the encoded data of all the difference areas belonging to is decoded.

(Supplementary Note 11) A difference area extracting unit that compares a current screen with a previous screen before changing to the current screen and extracts an area on the current screen that has changed with respect to the previous screen as a difference area; Difference area set extraction means for extracting a difference area set that is a set of difference areas to be displayed simultaneously from the difference area, and encoding means for encoding image information of the difference area belonging to the difference area set for each difference area set And a screen information transmitting apparatus comprising: communication means for transmitting encoded data of the difference area.

(Additional remark 12) It is provided with the dissimilarity calculation means which calculates the dissimilarity of difference areas, A difference area set extraction means is described in Additional remark 11 which extracts a difference area set based on the dissimilarity of difference areas Screen information transmitter.

(Additional remark 13) It is provided with the change history storage means which memorizes the history of the existence of change for every block in a screen, and a dissimilarity calculation means is based on the history of the existence of change memorized by the change history storage means, Item 13. The screen information transmitting apparatus according to appendix 12, which calculates dissimilarity between difference areas.

(Supplementary note 14) Priority determining means for determining the priority of the transmission order of the encoded data is provided for each difference area set, and the communication means includes images of difference areas belonging to the difference area set in order of priority of the difference area set. 14. The screen information transmitting apparatus according to appendix 12 or appendix 13, which transmits encoded data of information.

(Supplementary Note 15) The dissimilarity calculation means sets the dissimilarity between the difference regions to a smaller value as the frequency at which the change occurs at the same timing and the frequency at which the change does not occur at the same timing increases. To 15. The screen information transmitting device according to any one of appendix 14.

(Supplementary Note 16) The dissimilarity calculating means places more importance on the frequency at which both changes occur at the same timing and the frequency at which both changes occur at the same timing. The screen information transmitting apparatus according to supplementary note 15, wherein as the frequency of changes occurring at the same timing increases, the dissimilarity between the difference areas is set to a smaller value.

(Supplementary Note 17) The dissimilarity between the difference areas is set to a smaller value as the frequency at which the change occurs at the same timing and the frequency at which the change does not occur at the same timing increase at a time closer to the current time. The screen information transmitting apparatus according to Supplementary Note 15 or Supplementary Note 16.

(Additional remark 18) The difference area extraction means calculates the dissimilarity between the partial areas in the difference area, and in the difference area, the partial area whose dissimilarity with other partial areas is greater than or equal to the threshold 18. The screen information transmitting apparatus according to any one of supplementary note 12 to supplementary note 17, which is divided from the differential area and is set as another differential area.

(Supplementary note 19) When a movable area moves on the screen, the history of whether or not there is a change in the block that overlaps the area occupied by the area after the movement is recorded in the block that overlaps the area occupied by the area before the movement. 19. The screen information transmitting device according to any one of supplementary note 13 to supplementary note 18, comprising history rewriting means for updating with a history of presence / absence of a change.

(Supplementary note 20) The communication means transmits the encoded data of the differential region, the identifier of the differential region set to which the differential region belongs, and the number of differential regions belonging to the differential region set to any one of Supplementary note 11 to Supplementary note 19. The screen information transmitting device described.

(Supplementary note 21) Decoding means for decoding the encoded data of the difference area, which is an area changed from the stored screen, and when the encoded data of the differential area is decoded, the differential area belongs and is displayed at the same time Determining means for determining whether or not encoded data of all difference areas belonging to the difference area set, which is a set of power difference areas, and encoded data of all difference areas belonging to the difference area set are decoded; Update means for updating the image information of the areas corresponding to all the difference areas belonging to the difference area set, with the decoded image information, among the stored image information of the screen. A screen information receiving apparatus comprising:

(Supplementary Note 22) The determination means is based on the difference region set identifier and the number of difference regions belonging to the difference region set transmitted together with the difference region encoded data when the difference region encoded data is decoded. The screen information receiving apparatus according to attachment 21, wherein it is determined whether or not the encoded data of all the difference areas belonging to the difference area set indicated by the identifier has been decoded.

  This application claims the priority on the basis of the JP Patent application 2013-093153 for which it applied on April 26, 2013, and takes in those the indications of all here.

  Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

Industrial applicability

  The present invention is suitably applied to a screen transmission system that transmits image information between an image information transmitting device and an image information receiving device.

DESCRIPTION OF SYMBOLS 1 Screen transmission server 2 Screen reception client 11 Current screen information storage part 12 Transmission screen information storage part 13 Difference area extraction part 14 Change history storage part 15 Dissimilarity calculation part 16 Difference area set extraction part 17 Priority determination part 18 Encoding Unit 19 Server communication unit 21 Client communication unit 22 Decoding unit 23 Difference area update unit 24 Screen information storage unit 25 Screen output unit

Claims (23)

A screen information transmitting device for transmitting screen image information;
A screen information receiving device for receiving the image information and displaying a screen;
The screen information transmitting device
A difference area extracting means for comparing a current screen with a previous screen before changing to the current screen, and extracting an area on the current screen changed with respect to the previous screen as a difference area;
A difference area set extraction means for extracting a difference area set, which is a set of difference areas to be displayed simultaneously, from the difference area;
For each difference area set, encoding means for encoding the image information of the difference area belonging to the difference area set;
Communication means for transmitting the encoded data of the difference area to the screen information receiving device,
The screen information receiving device,
Decoding means for decoding the encoded data of the difference area received from the screen information transmitting device;
A determination unit that determines whether encoded data of all the difference areas belonging to the set of difference areas to which the difference area belongs is decoded when the encoded data of the difference area is decoded;
When it is determined that the encoded data of all the difference areas belonging to the difference area set has been decoded, all the differences belonging to the difference area set among the screen image information stored in the screen information receiving device An image transmission system comprising: update means for updating image information of an area corresponding to the area with the decoded image information. Screen information transmitter
Dissimilarity calculation means for calculating the dissimilarity between the difference areas,
The image transmission system according to claim 1, wherein the difference area set extraction unit extracts the difference area set based on a dissimilarity between the difference areas. The screen information transmitting device includes a change history storage unit that stores a history of whether or not there is a change for each block in the screen,
The image transmission system according to claim 2, wherein the dissimilarity calculating unit calculates a dissimilarity between the difference areas based on a history of presence / absence of a change stored in the change history storing unit. The screen information transmission device includes a priority determination unit that determines the priority of the transmission order of the encoded data for each difference region set,
The image transmission system according to claim 2 or 3, wherein the communication unit transmits the encoded data of the image information of the difference areas belonging to the difference area set in order of priority of the difference area set.   The dissimilarity calculation means sets the dissimilarity between the difference areas to a smaller value as the frequency of the change occurring at the same timing and the frequency of the change not changing at the same timing increase. 5. The image transmission system according to claim 1. The dissimilarity calculation means gives more importance to the frequency at which changes occur at the same timing and the frequency at which changes do not occur at the same timing. The image transmission system according to claim 5, wherein as the frequency of changes in both increases, the dissimilarity between the difference areas is set to a smaller value. The dissimilarity between the difference areas is set to a smaller value as the frequency at which the change occurs at the same timing and the frequency at which the change does not occur at the same timing increase at a time closer to the current time. Or the image transmission system of Claim 6. The difference area extraction means calculates the dissimilarity between the partial areas in the difference area, and divides the partial area in the difference area whose dissimilarity with other partial areas is equal to or greater than a threshold from the difference area. The image transmission system according to any one of claims 1 to 7, wherein the difference transmission region is set as another difference region. The communication means transmits the encoded data of the difference area, the identifier of the difference area set to which the difference area belongs, and the number of difference areas belonging to the difference area set to the screen information receiving device,
The determination means, based on the identifier of the difference area set transmitted together with the encoded data of the difference area and the number of difference areas belonging to the difference area set, when the encoded data of the difference area is decoded. The image transmission system according to any one of claims 1 to 8, wherein it is determined whether or not encoded data of all the difference areas belonging to the difference area set indicated by is decoded. A difference area extracting means for comparing a current screen with a previous screen before changing to the current screen, and extracting an area on the current screen changed with respect to the previous screen as a difference area;
A difference area set extraction means for extracting a difference area set, which is a set of difference areas to be displayed simultaneously, from the difference area;
For each difference area set, encoding means for encoding the image information of the difference area belonging to the difference area set;
A screen information transmitting apparatus comprising: communication means for transmitting encoded data of a difference area. Dissimilarity calculation means for calculating the dissimilarity between the difference areas,
The screen information transmitting apparatus according to claim 10, wherein the difference area set extraction unit extracts the difference area set based on a dissimilarity between the difference areas. A change history storage means for storing a history of whether or not there is a change for each block in the screen;
The screen information transmitting apparatus according to claim 11, wherein the dissimilarity calculating unit calculates a dissimilarity between the difference areas based on a history of presence / absence of a change stored in the change history storing unit. Priority determining means for determining the priority of the transmission order of encoded data for each differential region set,
The screen information transmission device according to claim 11 or 12, wherein the communication unit transmits encoded data of image information of difference areas belonging to the difference area set in order of priority of the difference area set. The dissimilarity calculation means sets the dissimilarity between the difference regions to a smaller value as the frequency of both changes at the same timing and the frequency of no change at the same timing increases. 14. The screen information transmission device according to claim 1. The dissimilarity calculation means gives more importance to the frequency at which changes occur at the same timing and the frequency at which changes do not occur at the same timing. The screen information transmitting apparatus according to claim 14, wherein the dissimilarity between the difference areas is set to a smaller value as the frequency of changes in both increases. 15. The dissimilarity between difference regions is set to a smaller value as the frequency at which changes occur at the same timing and the frequency at which changes do not occur at the same timing increase at a time closer to the current time. Or the screen information transmitter of Claim 15. The difference area extraction means calculates the dissimilarity between the partial areas in the difference area, and divides the partial area in the difference area whose dissimilarity with other partial areas is equal to or greater than a threshold from the difference area. The screen information transmitting apparatus according to any one of claims 10 to 16, wherein the screen information transmitting apparatus is a different difference area. The communication means transmits the encoded data of the difference area, the identifier of the difference area set to which the difference area belongs, and the number of difference areas belonging to the difference area set. The screen information transmitting device described. Decoding means for decoding the encoded data of the difference area, which is an area changed from the stored screen;
When encoded data of the difference area is decoded, it is determined whether the encoded data of all the difference areas belonging to the difference area set, which is the set of difference areas to be displayed at the same time, has been decoded. Determination means to perform,
Areas corresponding to all the difference areas belonging to the difference area set among the image information of the stored screen when it is determined that the encoded data of all the difference areas belonging to the difference area set has been decoded. A screen information receiving apparatus comprising: update means for updating the image information with the decoded image information. The determination means, based on the identifier of the difference area set transmitted together with the encoded data of the difference area and the number of difference areas belonging to the difference area set, when the encoded data of the difference area is decoded. The screen information receiving apparatus according to claim 19, wherein it is determined whether or not encoded data of all difference areas belonging to the difference area set indicated by is decoded. Screen information transmitter
Compare the current screen with the previous screen before changing to the current screen, extract the area on the current screen that has changed with respect to the previous screen as a difference area,
From the difference area, extract a difference area set that is a set of difference areas to be displayed simultaneously,
For each difference area set, encode the image information of the difference area belonging to the difference area set,
Send the encoded data of the difference area to the screen information receiving device,
The screen information receiving device is
Decode the encoded data of the difference area received from the screen information transmitting device,
When decoding the encoded data of the difference area, determine whether or not the encoded data of all the difference areas belonging to the difference area set to which the difference area belongs,
When it is determined that the encoded data of all the difference areas belonging to the difference area set has been decoded, all the difference areas belonging to the difference area set among the image information of the screen stored in the screen information receiving device A screen transmission method characterized in that the image information of the area corresponding to is updated with the decoded image information. On the computer,
A difference area extraction process for comparing the current screen and the previous screen before changing to the current screen, and extracting an area on the current screen that has changed with respect to the previous screen as a difference area;
A difference area set extraction process for extracting a difference area set, which is a set of difference areas to be displayed simultaneously, from the difference area;
For each difference area set, an encoding process for encoding image information of difference areas belonging to the difference area set, and
An image information transmission program for executing communication processing for transmitting encoded data of a difference area. On the computer,
A decoding process for decoding the encoded data of the difference area, which is an area changed from the stored screen;
When encoded data of the difference area is decoded, it is determined whether the encoded data of all the difference areas belonging to the difference area set, which is the set of difference areas to be displayed at the same time, has been decoded. Determination process to perform, and
Areas corresponding to all the difference areas belonging to the difference area set among the image information of the stored screen when it is determined that the encoded data of all the difference areas belonging to the difference area set has been decoded. An image information receiving program for executing update processing for updating the image information with the decoded image information.

Images