JP6229315B2 - Information communication apparatus and data linkage program - Google Patents

Description

  The present invention relates to an image data processing apparatus and an image data processing program.

  Conventionally, when an image such as a screen is transmitted to another device via a network at regular time intervals, the image data processing device divides the image into blocks of a predetermined size, and the previous image for each block And the compressed image data of the block in which the difference is detected is transmitted. Alternatively, the image data processing apparatus transmits the compressed image data of the rectangular area where the difference is detected in the block where the difference is detected.

  In addition, there is a conventional technique in which transparent color data is written and transmitted to image pixels having no difference in a block having a difference (see, for example, Patent Document 1). In addition, in order to reduce the size of image data to be transmitted, there is a conventional technique in which a transmitting apparatus transmits an original image by thinning out the original image vertically and horizontally for each line, and the receiving apparatus enlarges and restores the image (see, for example, Patent Document 2). In addition, there is a conventional technique in which as the number of colors used in each block is smaller, the number of indexes for displaying each pixel is reduced, and the image is compressed and transmitted by converting the index into a smaller number of bits (for example, , See Patent Document 3).

JP 2007-212229 A JP 7-296158 A JP-A-10-74173

  However, when the performance of the receiving device is low compared to the transmitting device, such as when a screen image is transmitted from a personal computer (hereinafter referred to as “PC”) to a smartphone or tablet terminal, There is a problem that the load of restoration processing is heavy. That is, the receiving apparatus needs to perform processing for receiving and restoring a compressed image for each block or rectangular area, and a situation occurs in which processing cannot be performed in time for the compressed image to be transmitted.

  An object of one aspect of the present invention is to reduce the load of image restoration processing in a receiving apparatus.

In one aspect, an information communication device disclosed in the present application, in one aspect, a division unit that divides a target image, which is an image captured on a screen, into a plurality of blocks having a predetermined size, and a block obtained by division by the division unit Each has a comparison unit for comparing with the previous image. The information communication apparatus further includes an extraction unit that extracts an update rectangle based on the update location from each block when there is an update location from the previous image as a result of the comparison by the comparison unit. Furthermore, the information communication device specifies a rectangle based on the update rectangle extracted from each block by the extraction unit as an entire rectangle, an image of the entire rectangle specified by the specifying unit, and information on the entire rectangle and an update Rectangle information is transmitted to another information communication apparatus that has performed mutual authentication as a data linkage target, and the other information communication apparatus is based on the transmitted whole rectangle image, whole rectangle information, and updated rectangle information. And a transmission unit that receives information on an operation event generated from the touch operation on the screen displayed in step S <b> 1 and performs an operation corresponding to the operation event on the own apparatus .

  According to one embodiment, it is possible to reduce the load of image restoration processing in the receiving device.

FIG. 1 is a functional block diagram illustrating the configuration of the PC-smartphone cooperation system according to the embodiment. FIG. 2 is a functional block diagram illustrating the configuration of the cooperation unit. FIG. 3 is a diagram illustrating an example of a data structure of profile information. FIG. 4 is a functional block diagram showing the configuration of the remote operation control unit. FIG. 5 is a functional block diagram showing the configuration of the image processing unit on the PC side. FIG. 6A is a diagram illustrating an update rectangle extracted for each block. FIG. 6B is a diagram illustrating black painting of the entire rectangle and unchanged portions. FIG. 7 is a diagram illustrating an example of a data structure of screen information transmitted from the PC to the smartphone. FIG. 8 is a diagram illustrating an example of screen information transmitted from the PC to the smartphone. FIG. 9 is a diagram for explaining an image restoration method by the image processing unit. FIG. 10 is a flowchart showing a flow of screen data transmission processing. FIG. 11 is a diagram illustrating an example of the PC cooperation screen. FIG. 12 is a diagram illustrating an example of a smartphone cooperation screen. FIG. 13 is a flowchart showing the flow of image restoration processing. FIG. 14 is a diagram illustrating a software configuration. FIG. 15 is a diagram illustrating a hardware configuration of the smartphone. FIG. 16 is a diagram illustrating a hardware configuration of the PC.

  Embodiments of an image data processing apparatus and an image data processing program disclosed in the present application will be described below in detail with reference to the drawings. Note that this embodiment does not limit the disclosed technology.

  First, the PC-smartphone cooperation system according to the embodiment will be described. FIG. 1 is a functional block diagram illustrating the configuration of the PC-smartphone cooperation system according to the embodiment. As illustrated in FIG. 1, the PC-smartphone cooperation system 1 includes a PC 2 and a smartphone 3, and performs cooperation between the PC 2 and the smartphone 3. Here, “cooperation” refers to exchanging data such as photos and videos.

  The PC 2 includes a WLAN unit 21, a cooperation unit 22, an input control unit 23, and an output control unit 24.

  The WLAN unit 21 performs communication using a wireless LAN, and here, communicates with the WLAN unit 31 of the smartphone 3. The WLAN unit 21 includes an AP unit 211 that causes the WLAN unit 21 to function as an access point in wireless LAN communication, and an STA unit 212 that causes the WLAN unit 21 to function as a station (client device) in wireless LAN communication.

  The cooperation unit 22 cooperates with other information communication devices using the WLAN unit 21, and here cooperates with the cooperation unit 32 of the smartphone 3. Details of the cooperation unit 22 will be described later.

  The input control unit 23 receives instructions and data input by the user using an input device such as a mouse and a keyboard, and passes the received instructions and data to the linkage unit 22. The output control unit 24 displays the screen information output by the PC 2 on the display device, and outputs the sound output by the PC 2 to the speaker.

  The smartphone 3 includes a WLAN unit 31, a cooperation unit 32, a wireless control unit 33, an input control unit 34, an output control unit 35, a call control unit 36, a microphone control unit 37, a call recording unit 38, A telephone directory recording unit 39 and a location detection control unit 3a are provided.

  The WLAN unit 31 communicates using a wireless LAN, and here, communicates with the WLAN unit 21 of the PC 2. The WLAN unit 31 includes an AP unit 311 that causes the WLAN unit 31 to function as an access point in wireless LAN communication, and an STA unit 312 that causes the WLAN unit 31 to function as a station (client device) in wireless LAN communication.

  The cooperation unit 32 cooperates with other information communication apparatuses using the WLAN unit 31, and here cooperates with the cooperation unit 22 of the PC 2. Details of the cooperation unit 32 will be described later.

  The wireless control unit 33 performs voice communication and data communication wirelessly with other smartphones, landline phones, computers installed by mobile phone service providers, and the like. The input control unit 34 receives instructions and data input by the user using the touch screen, and passes the received instructions and data to the linkage unit 32. The output control unit 35 displays information on the screen output by the smartphone 3 on the display device, and outputs the sound output by the smartphone 3 to the speaker.

  The call control unit 36 performs control such as connection, release, and monitoring of the telephone. The microphone control unit 37 controls a microphone used for a call, and the call recording unit 38 records the call. The phone book recording unit 39 manages the phone book, and the whereabouts detection control unit 3 a performs control for detecting the position of the smartphone 3.

  Next, the structure of the cooperation parts 22 and 32 is demonstrated. Note that the cooperation unit 32 basically has the same configuration as the cooperation unit 22, and therefore, here, the cooperation unit 22 will be described as an example. That is, the case where the cooperation unit 22 of the PC 2 cooperates with the smartphone 3 as another information communication device will be described as an example. Moreover, about a different part from the cooperation part 22 among the cooperation parts 32, a different point is specified clearly. FIG. 2 is a functional block diagram illustrating the configuration of the cooperation unit 22.

  As illustrated in FIG. 2, the linkage unit 22 includes a setup unit 221, a device linkage unit 222, a remote operation control unit 223, a linkage data storage unit 224, a GUI unit 225, and a control unit 226.

  The setup unit 221 performs mutual authentication with other information communication devices that have not been authenticated and registered, and mutually registers SSID (Service Set Identifier) and device information after mutual authentication. Here, the setup unit 221 performs mutual authentication with the smartphone 3 and registers SSID and device information with each other. As a result of the authentication registration, the setup unit 221 stores information necessary for cooperation such as SSID and device information in the cooperation data storage unit 224.

  The device cooperation unit 222 establishes cooperation with an information communication device that has already been authenticated and registered, that is, the smartphone 3. Here, establishment of cooperation refers to establishing a wireless LAN connection with the smartphone 3 and acquiring the latest profile information of the smartphone 3. Then, the device cooperation unit 222 performs a process of capturing a photo, a moving image, the number of steps / activity, or starts the remote operation control unit 223 based on information at the time of connection by the wireless LAN. On the other hand, the device cooperation unit 322 of the smartphone 3 performs transmission processing of a photo, a moving image, the number of steps / activity, or starts the remote operation control unit 323 of the smartphone 3.

  The remote operation control unit 223 controls the remote operation of the PC 2 by the smartphone 3. Specifically, the remote operation control unit 223 transmits the screen data and audio data of the PC 2 to the smartphone 3. The remote operation control unit 323 of the smartphone 3 that has received the screen data and audio data of the PC 2 performs screen display and audio output, and accepts user operations. Then, the remote operation control unit 323 of the smartphone 3 transmits the received operation information to the remote operation control unit 223 of the PC 2, and the remote operation control unit 223 of the PC 2 operates the PC 2 based on the received operation information. Details of the remote operation control unit 223 and the remote operation control unit 323 will be described later.

  The cooperation data storage unit 224 stores data necessary for cooperation with other information communication apparatuses, and the SSID when the own apparatus operates as an AP, the SSID of other information communication apparatuses that have been mutually registered for authentication, and encryption. Store key, profile information, etc. Here, the cooperation data storage unit 224 stores an SSID when the PC 2 operates as an AP, an SSID of the smartphone 3, an encryption key, profile information, and the like. When there are a plurality of information communication devices that have been mutually registered for authentication, the cooperative data storage unit 224 stores the SSID, profile information, encryption key, and the like for each information communication device.

  FIG. 3 is a diagram illustrating an example of a data structure of profile information. As shown in FIG. 3, the profile information stores the version number, model, MAC address, hardware information, individual function, and linkage function.

  In the version number, version information of the link unit 22 or 32 in the own apparatus is stored. The model stores the category information of the own device. For example, the model stores a distinction between a notebook computer, a desktop computer, and a mobile phone. The content of this model may be determined as appropriate according to the category of the device. The MAC address stores the MAC address information of the device itself. For example, information on the number of pixels is stored in the hardware information for a liquid crystal display (LCD) and a camera. In the individual function, whether or not the WOL (Wake On LAN) is supported is stored.

  The cooperation function stores information indicating whether or not there is support for each cooperation function. Linking functions include capturing photos, videos, steps / activity, and remote operations. For example, when the remote operation information is “1”, the remote operation function is supported. Note that the profile information is stored in the cooperation data storage unit 224 or 324 provided in the own apparatus for each apparatus, and is stored at the time of shipment, for example.

  The GUI unit 225 performs a dialog with a user using a GUI (Graphical User Interface). Here, the GUI unit 225 receives data input by the user from the input control unit 23 and passes it to the control unit 226, receives display data and audio data from the control unit 226, and passes them to the output control unit 24.

  The control unit 226 controls the entire linkage unit 22. Specifically, the control unit 226 moves the control between the function units and transfers data between the function unit and the storage unit, thereby making the whole as the linkage unit 22. Make it work.

  Next, details of the remote operation control units 223 and 323 will be described. FIG. 4 is a functional block diagram showing the configuration of the remote operation control units 223 and 323. FIG. 4 shows the remote operation control unit 223 on the PC 2 side and the remote operation control unit 323 on the smartphone 3 side.

  The remote operation control unit 223 on the PC 2 side includes an image processing unit 41, an audio processing unit 42, an operation processing unit 43, and a communication control unit 44.

  The image processing unit 41 captures a screen of the PC 2, extracts a difference from the previous screen, compresses an image of a portion where there is a difference, and creates screen data to be transmitted to the smartphone 3. For example, the image processing unit 41 creates data of 11 screens / second. Details of the image processing unit 41 will be described later.

  The audio processing unit 42 performs audio capture or format conversion of the PC 2 and creates audio data to be transmitted to the smartphone 3. Further, the sound processing unit 42 silences the speaker of the PC 2. Since the sound processing unit 42 silences the speaker during the remote operation, it is possible to eliminate unnecessary sound output from the PC 2 during the remote operation.

  The operation processing unit 43 reflects remote operations such as key input, mouse operation, and character string input performed by the user on the smartphone 3 on the PC 2.

  The communication control unit 44 transmits screen data and audio data to the smartphone 3, and receives operation event information such as key input, mouse operation, and character string input from the smartphone 3.

  The remote operation control unit 323 on the smartphone 3 side includes an image processing unit 51, an audio processing unit 52, a touch control unit 53, and a communication control unit 54.

  The image processing unit 51 restores the screen from the screen data transmitted from the PC 2 and displays it on the display device of the smartphone 3. The image processing unit 51 includes a reception unit 511, a restoration unit 512, a generation unit 513, and a display unit 514.

  The receiving unit 511 receives screen information transmitted from the PC 2 via the communication control unit 54. The restoration unit 512 restores the updated rectangular image based on the screen information received by the reception unit 511. The generation unit 513 generates a screen by pasting the updated rectangular image restored by the restoration unit 512 to a corresponding position on the previous screen. The display unit 514 displays the screen generated by the generation unit 513 on the display device.

  The audio processing unit 52 performs stream reproduction of the audio data transmitted from the PC 2. The touch control unit 53 generates operation event information such as key input, mouse operation, and character string input corresponding to the touch operation performed by the user on the smartphone 3 on the PC 2. The communication control unit 54 receives screen data and audio data from the communication control unit 44, and transmits operation event information such as key input, mouse operation, and character string input to the communication control unit 44.

  Next, the configuration of the image processing unit 41 on the PC side will be described. FIG. 5 is a functional block diagram showing a configuration of the image processing unit 41 on the PC side. As illustrated in FIG. 5, the image processing unit 41 includes a capture unit 61, a difference extraction unit 62, a black painting unit 63, an image compression unit 64, an image transmission unit 65, and an image data storage unit 66. .

  The capture unit 61 captures the screen currently displayed on the display device of the PC 2 and stores the image data of the captured screen in the image data storage unit 66.

  The difference extraction unit 62 extracts, from the current screen, the smallest rectangular image that includes a different portion between the current screen captured by the capture unit 61 and the previous screen. The difference extracting unit 62 includes a dividing unit 621, a comparing unit 622, an extracting unit 623, a converting unit 624, and a specifying unit 625.

  The dividing unit 621 divides the current screen and the previous screen into blocks of 64 × 64 pixels. Here, although divided into blocks of 64 × 64 pixels, the dividing unit 621 can divide into blocks of other sizes such as 128 × 128 pixels.

  The comparison unit 622 compares the current screen with the previous screen for each block. The comparison unit 622 compares the two blocks in the order of odd-numbered rows → even-numbered rows → odd-numbered rows →. That is, the comparison unit 622 performs the process of comparing the first block with an odd number of rows, comparing the next block with an even number of rows, comparing the third block with an odd number of rows, and so on. The comparison unit 622 can compare two blocks efficiently by changing the comparison of two blocks in the order of odd-numbered rows → even-numbered rows → odd-numbered rows →.

  The extraction unit 623 extracts a minimum rectangular area including a different part in the comparison by the comparison unit 622 as an update rectangle from each block. Then, the extraction unit 623 stores the coordinates of the block and the coordinates of the update rectangle in the image data storage unit 66 for each block. The stored coordinates are the upper left and lower right X and Y coordinates of the block or update rectangle.

  FIG. 6A is a diagram illustrating an update rectangle extracted for each block. In FIG. 6A, squares arranged in two dimensions are blocks 71 of 64 × 64 pixels. However, the block at the right end of the screen is a block 72 of 48 × 64 pixels, the block at the bottom of the screen is a block 73 of 64 × 12 pixels, and the block at the lower right corner of the screen is a block 74 of 48 × 12 pixels. . A rectangle indicated by diagonal lines is an update rectangle 75. In FIG. 6A, 14 update rectangles 75 are extracted.

  When the screen resolution is higher than 1280 × 720, the conversion unit 624 converts the screen resolution to 1280 × 720, and stores the converted image in the image data storage unit 66. The conversion unit 624 converts the coordinates of the block and the coordinates of the update rectangle stored for each block in correspondence with the conversion of the resolution, and stores the converted data in the image data storage unit 66. Here, the screen resolution is converted to 1280 × 720, but the conversion unit 624 can convert the screen resolution to another screen resolution based on the screen resolution of the receiving device. Further, the conversion unit 624 can convert the resolution within 1280 × 720 while maintaining the aspect ratio of the resolution.

  The specifying unit 625 uses the coordinates of each update rectangle stored in the image data storage unit 66 to specify the minimum rectangle including each update rectangle as an overall rectangle, and the image of the current screen of the specified overall rectangle as an overall rectangle. The image is stored in the image data storage unit 66 together with the coordinates.

  The black painting unit 63 creates a black painting image in which a portion other than the update rectangle in the entire rectangular image, that is, a portion that has not been changed from the previous screen, is created and stored in the image data storage unit 66. FIG. 6B is a diagram illustrating black painting of the entire rectangle and unchanged portions. As shown in FIG. 6B, portions of the entire rectangle 76 excluding the 14 update rectangles 75 are painted black.

  When the black painting unit 63 paints the entire rectangle 76 excluding the update rectangle 75, the compression ratio of the image compression unit 64 can be increased. Therefore, the remote operation control unit 223 can efficiently transmit the screen of the PC 2 to the smartphone 3.

  The image compression unit 64 compresses a black-painted image by JPEG (Joint Photographic Experts Group) when the entire rectangular image is less than 30 kilobytes (KB). Note that the image compression unit 64 can also compress the image using a compression method other than JPEG.

  The image transmission unit 65 transmits the entire rectangular image to the smartphone 3 when the entire rectangular image is less than 30 KB. On the other hand, when the size is 30 KB or more, the image transmission unit 65 compares the size of the entire rectangular image and the compressed image by JPEG, and transmits the image having the smaller size to the smartphone 3.

  In this manner, the image transmission unit 65 transmits an image to the smartphone 3 in a format in which a compressed image by JPEG and an entire rectangular image that is an uncompressed image are mixed. Therefore, when the screen update does not occur for a certain time (for example, 1 second), mosquito noise may occur at the boundary between the compressed image and the uncompressed image. Here, mosquito noise is image disturbance that occurs in a portion where the color change is severe in a compressed image.

  Therefore, the image transmission unit 65 counts the number of times that the current image is the same as the previous transmission image as the number of unchanged times, and transmits the current entire screen image to the smartphone 3 when the number of unchanged times reaches 10. To refresh the screen. When the number of times of no change is 10, the image transmission unit 65 can suppress the occurrence of mosquito noise by transmitting an image of the entire current screen to the smartphone 3.

  The image data storage unit 66 stores data required when the image processing unit 41 performs processing. Specifically, the image data storage unit 66 stores the captured screen image data, the image data of each block, the coordinates of each block, and the coordinates of each update rectangle for the current screen and the previous screen. The image data storage unit 66 stores data before and after the resolution change for the image data of the screen, the coordinates of each block, and the coordinates of each update rectangle. Further, the image data storage unit 66 stores the data of the entire rectangular image, the coordinates of the entire rectangle, the data of the black image, the image data after JPEG compression, the aspect ratio, and the like.

  Next, the data structure of the screen data transmitted from the PC 2 to the smartphone 3 will be described. FIG. 7 is a diagram illustrating an example of a data structure of screen information transmitted from the PC 2 to the smartphone 3.

  As shown in FIG. 7, the screen information transmitted from the PC 2 to the smartphone 3 includes the width and height of the screen resolution, the width and height of the image size, and the entire rectangular data. Also, the entire rectangular data includes an image format indicating whether the image is compressed or uncompressed, an entire rectangular image X and Y indicating the start X coordinate and Y coordinate of the entire rectangular data, and an entire indicating the width and height of the entire rectangular data. The rectangular image width and height, the number of updated rectangles stored, and the updated rectangle size are included. Further, the entire rectangular data further includes an image data size and image data. Each update rectangle size includes rectangles X and Y indicating the start X coordinate and Y coordinate of the rectangle, and a rectangle width and height indicating the width and height of the rectangle.

  FIG. 8 is a diagram illustrating an example of screen information transmitted from the PC 2 to the smartphone 3. The screen information in the example illustrated in FIG. 8 includes “1280” and “720” as the width and height of the screen resolution, and “1150” and “520” as the width and height of the image size.

  Also, the entire rectangular data includes data of the entire rectangular image compressed by JPEG, the start X coordinate and start Y coordinate of the entire rectangle are “100” and “40”, and the width and height of the entire rectangle are “1000” and “380”. The whole rectangle data includes the start X coordinate, start Y coordinate, width, and height of the 14 update rectangles 75 shown in FIG. 6A.

  Next, an image restoration method by the image processing unit 51 of the smartphone 3 will be described. FIG. 9 is a diagram for explaining an image restoration method by the image processing unit 51. FIG. 9 shows an example when the screen information shown in FIG. 8 is sent from the PC 2.

  The restoration unit 512 of the image processing unit 51 decompresses a black-painted image compressed by JPEG. Then, the restoration unit 512 extracts 14 update rectangular images (1) to (14) from the expanded black image based on the start X coordinate, start Y coordinate, width, and height of the update rectangle. Then, the generation unit 513 generates 14 update rectangle images based on the start X coordinate and start Y coordinate and the start X coordinate, start Y coordinate, width, and height of the update rectangle sent from the PC 2 for the entire rectangle. Paste it to the image on the previous screen.

  As described above, the image processing unit 51 extracts the update rectangle image from the expanded black image based on the start X coordinate, the start Y coordinate, the width, and the height of the entire rectangle and the update rectangle, and the previous rectangle is updated. By pasting the image on the screen, the processing load for image restoration can be reduced.

  Next, the flow of screen data transmission processing will be described. FIG. 10 is a flowchart showing a flow of screen data transmission processing. Note that the process illustrated in FIG. 10 is executed when “remote operation” is selected by the user as the linkage function. The user selects “remote operation” by selecting “operate PC” on the PC cooperation screen shown in FIG. 11 from the smartphone 3. Further, the user selects “remote operation” by selecting “PC operation on smartphone” on the smartphone cooperation screen shown in FIG.

  As shown in FIG. 10, the capture unit 61 captures the screen of the PC 2 (step S1), and the difference extraction unit 62 determines whether or not the capture screen is the first screen (step S2). If it is the first screen, the difference extraction unit 62 proceeds to step S9.

  On the other hand, if it is not the first screen, the difference extraction unit 62 performs a process of extracting a difference from the previous screen. That is, the dividing unit 621 blocks the captured screen in units of 64 × 64 pixels (step S3), and the comparison unit 622 compares the screen captured last time with the screen captured this time in block units (step S4).

  Then, the difference extraction unit 62 determines whether or not there is a changed portion between the blocks (step S5). If there is no changed portion, the process proceeds to step S7, and if there is a changed portion, the extracting unit 623. However, the minimum rectangle including the changed portion is set as the update rectangle, and the coordinates of the update rectangle are calculated and stored (step S6). Then, the difference extraction unit 62 determines whether or not the comparison has been completed for all the blocks (step S7). If completed, the process proceeds to step S9. If not completed, 1 is set as the number of comparison blocks. In addition (step S8), the process returns to step S5 to process the next block. The number of comparison blocks is the number of compared blocks, and is initialized to “1”. When the number of comparison blocks reaches the number of blocks obtained by dividing the screen into blocks, the comparison of all blocks is completed.

  Then, the difference extraction unit 62 determines whether or not there is a change from the previously transmitted image (step S9), and when there is no change, adds 1 to the number of unchanged images (step S10). Here, the number of unchanged images is the number of times that the captured screen has not changed, and is initialized to “0”. Then, the difference extraction unit 62 determines whether or not the number of unchanged images is “10” (step S11). As a result, if the number of unchanged images is not “10”, the difference extraction unit 62 returns to step S1, and the capture unit 61 captures the screen.

  On the other hand, when the number of unchanged images is “10”, the conversion unit 624 converts the screen resolution of the captured screen to 1280 × 720 (step S12), and transmits an image to suppress generation of mosquito noise. The unit 65 performs full screen transmission (step S13). Then, the difference extraction unit 62 initializes the number of unchanged images to “0” (step S14), the control returns to step S1, and the capture unit 61 captures the screen.

  If there is a change from the previously transmitted image, the conversion unit 624 converts the screen resolution of the captured screen to 1280 × 720 (step S15), and sets the resolution of the captured image and the aspect ratio of 1280 × 720. Calculate (step S16). Here, the aspect ratio is a vertical ratio and a horizontal ratio between the resolution of the captured image and 1280 × 720. Then, the conversion unit 624 corrects the coordinates of the update rectangle with the aspect ratio (step S17).

  Then, the specifying unit 625 sets the minimum rectangle including all the update rectangles as the entire rectangle from the comparison result of all the blocks, and calculates the coordinates of the entire rectangle based on the coordinates of the update rectangle corrected by the conversion unit 624 (step S18). . Then, the specifying unit 625 stores the coordinates of the entire rectangle in the image data storage unit 66 (step S19).

  Then, the black painting unit 63 paints a portion of the entire rectangular image that is not changed compared to the previous image (step S20), and stores the data of the black painted image in the image data storage unit 66. save.

  Then, the image transmission unit 65 determines whether or not the size of the data of the entire rectangular image before blacking is less than 30 KB (step S21). If the data size is not less than 30 KB, the image compression unit 64 converts the blacked image. JPEG conversion is performed (step S22). Then, the image transmission unit 65 compares the data size of the unpainted image and the JPEG compressed image (step S23), and sets the smaller data size as the transmission image (step S24). On the other hand, when the data size of the whole rectangular image before black painting is less than 30 KB, the image transmission unit 65 sets the whole rectangular image before black painting as a transmission image (step S25).

  And the image transmission part 65 transmits the image information of a transmission image to the smart phone 3 (step S26). Specifically, the image transmission unit 65 adds the entire rectangle information and the rectangle information of the update rectangle unit to the header of a packet for transmitting image data, and transmits the image information as image information.

  As described above, the image transmission unit 65 assigns the entire rectangle information and the rectangle information of the update rectangle unit and transmits one image to the smartphone 3, so that the image processing unit 41 loads the image restoration process of the smartphone 3. Can be reduced.

  Although the case where the screen resolution is converted after extracting the update rectangle from each block has been described here, the image processing unit 41 can also perform block division and extraction of the update rectangle after converting the screen resolution. Further, the image processing unit 41 can always send the data of the entire rectangular image without blacking or JPEG compression regardless of the size of the data of the entire rectangular image.

  Next, the flow of image restoration processing by the smartphone 3 will be described. FIG. 13 is a flowchart showing the flow of image restoration processing. As illustrated in FIG. 13, the reception unit 511 of the smartphone 3 receives screen information from the PC 2 (Step S41), and the restoration unit 512 analyzes the screen information (Step S42).

  Then, the restoration unit 512 expands the image data included in the screen information in the buffer area (step S43). If the image data is JPEG compressed image data as a result of the screen information analysis, the restoration unit 512 expands the image data in the buffer area after performing the image expansion process.

  Then, the restoration unit 512 sets the update rectangle storage number included in the screen information in the rectangle counter (step S44). Then, the restoration unit 512 acquires the coordinates, width, and height of the update rectangle from the entire rectangle data (step S45), and cuts the update rectangle from the buffer (step S46).

  Then, the generation unit 513 pastes the updated rectangle on the displayed whole image based on the coordinates of the entire rectangle and the updated rectangle (step S47), and the image processing unit 51 subtracts “1” from the rectangle counter (step S48). ). Then, the image processing unit 51 determines whether or not the value of the rectangle counter is “0”, that is, whether or not all the update rectangles have been processed (step S49). If the rectangle counter is not “0” In step S45, the next update rectangle is processed. On the other hand, when the rectangular counter is “0”, the reception of the next screen information is awaited. When the screen information is received (step S50, Yes), the process returns to step S42.

  As described above, the image processing unit 51 can restore an image by processing data of one image, and can update the screen of the smartphone 3 with a small load.

  As described above, in the embodiment, the dividing unit 621 of the PC 2 divides the current screen and the previous screen into blocks of 64 × 64 pixels, and the comparing unit 622 compares each block. Then, the extraction unit 623 extracts a minimum rectangular area including a different portion from each block as an update rectangle. Then, the specifying unit 625 specifies the minimum rectangle including each update rectangle as the entire rectangle. And the image transmission part 65 transmits the image data of a whole rectangle to the smart phone 3 with the information of a whole rectangle and each update rectangle. Further, the image processing unit 51 of the smartphone 3 performs a process of extracting each update rectangular image from the received image data and pasting it on the screen being displayed based on the information of the entire rectangle and each update rectangle. Therefore, the smartphone 3 can restore the screen with a small processing load.

  In the embodiment, the black painting unit 63 paints a black portion of the entire rectangular image that is not changed compared to the previous image, and the image compression unit 64 JPEG compresses the black painting image. Then, the image transmission unit 65 transmits the JPEG-compressed image to the smartphone 3. Therefore, the PC 2 can efficiently transfer the screen to the smartphone 3.

  In the embodiment, the conversion unit 624 converts the screen resolution to 1280 × 720, and corrects the coordinates of the update rectangle corresponding to the conversion. Therefore, the smartphone 3 can display the screen of the PC 2 with a display device having a resolution of 1280 × 720.

  In the embodiment, the difference extraction unit 62 counts the number of unchanged images, and when the number of unchanged images reaches “10”, the image transmission unit 65 performs full screen transmission. Therefore, the image processing unit 41 can suppress the generation of mosquito noise in the smartphone 3.

  Next, a software configuration for realizing cooperation between the PC 2 and the smartphone 3 will be described. FIG. 14 is a diagram illustrating a software configuration. As illustrated in FIG. 14, software for realizing cooperation between the PC 2 and the smartphone 3 includes an OS 710, a driver 720, and an application 730. The OS 710, the driver 720, and the application 730 cooperate with the hardware 700 to realize cooperation processing.

  The driver 720 includes a WLAN program 721. The WLAN program 721 executed in the PC 2 realizes the function of the WLAN unit 21 shown in FIG. The WLAN program 721 executed in the smartphone 3 realizes the function of the WLAN unit 31 illustrated in FIG. The WLAN program 721 includes an STA program 721a and an AP program 721b. The STA program 721a and the AP program 721b executed on the PC 2 realize the functions of the STA unit 212 and the AP unit 211 shown in FIG. In addition, the STA program 721a and the AP program 721b executed in the smartphone 3 realize the functions of the STA unit 312 and the AP unit 311 illustrated in FIG.

  The application 730 includes a cooperation program 731. The cooperation program 731 executed in the PC 2 realizes the function of the cooperation unit 22 shown in FIG. Moreover, the cooperation program 731 executed in the smartphone 3 realizes the function of the cooperation unit 32 illustrated in FIG. The cooperation program 731 uses the WLAN program 721 to cooperate with other information communication devices. That is, the cooperation program 731 of the PC 2 (or the smartphone 3) executes cooperation processing with the cooperation program 731 of the smartphone 3 (or PC 2).

  Next, the hardware 700 illustrated in FIG. 14 will be described. FIG. 15 is a diagram illustrating a hardware configuration of the smartphone. As illustrated in FIG. 15, the smartphone 800 includes a wireless communication unit 810, a display unit 820, a voice input / output unit 830, an input unit 840, a processor 850, and a storage unit 860. The wireless communication unit 810, the display unit 820, the voice input / output unit 830, the input unit 840, and the storage unit 860 are each connected to the processor 850.

  The storage unit 860 includes a program storage unit 861, a data storage unit 862, and a RAM (Random Access Memory) 863. The program storage unit 861 stores programs such as a cooperation program 731 that realizes the functions of the cooperation unit 32 shown in FIG. The data storage unit 862 stores various data such as the SSID, encryption key, and profile information stored in the cooperative data storage unit 224 illustrated in FIG. The RAM 863 stores data generated by executing the program.

  The processor 850 is, for example, an electronic circuit such as an application specific integrated circuit (ASIC), a central processing unit (CPU), or a micro processing unit (MPU). Then, the processor 850 reads a program such as the cooperation program 731 from the storage unit 860 and loads it into the RAM 863, so that the program such as the cooperation program 731 functions as a task such as a cooperation task. Then, the cooperation task loads information read from the data storage unit 862 to the area allocated to itself on the RAM 863 as appropriate, and executes various data processing based on the loaded data.

  FIG. 16 is a diagram illustrating a hardware configuration of the PC. As shown in FIG. 16, the PC 900 includes a main memory 910, a CPU (Central Processing Unit) 920, a LAN (Local Area Network) interface 930, and an HDD (Hard Disk Drive) 940. The PC 900 includes a super IO (Input Output) 950, a DVI (Digital Visual Interface) 960, and an ODD (Optical Disk Drive) 970. The PC 900 includes a speaker that outputs sound.

  The main memory 910 is a memory for storing a program and a program execution result. The CPU 920 is a central processing unit that reads and executes a program from the main memory 910. The CPU 920 includes a chip set having a memory controller.

  The LAN interface 930 is an interface for connecting the PC 900 to another computer via the LAN. The HDD 940 is a disk device that stores programs and data, and the super IO 950 is an interface for connecting an input device such as a mouse or a keyboard. The DVI 960 is an interface for connecting a liquid crystal display device, and the ODD 970 is a device for reading / writing a DVD.

  The LAN interface 930 is connected to the CPU 920 by PCI Express, and the HDD 940 and the ODD 970 are connected to the CPU 920 by SATA (Serial Advanced Technology Attachment). The super IO 950 is connected to the CPU 920 by LPC (Low Pin Count).

  A program executed in the PC 900 is stored in the DVD, read from the DVD by the ODD 970, and installed in the PC 900. Alternatively, the program is stored in a database or the like of another computer system connected via the LAN interface 930, read from these databases, and installed in the PC 900. The installed program is stored in the HDD 940, read into the main memory 910, and executed by the CPU 920.

  In the embodiment, the case where the PC 2 is remotely operated from the smartphone 3 has been described. However, the present invention is not limited to this, and an image is transmitted from one device to another device and the image is transmitted to another device. The same applies to the display.

  Further, in this embodiment, a case has been described in which the minimum rectangle including the update location is the update rectangle, and the minimum rectangle including the entire update rectangle is the entire rectangle, but the present invention is not limited to this, The present invention can be similarly applied to a case where a rectangle based on an update location is an update rectangle and a rectangle based on an all update rectangle is an overall rectangle.

1 PC-smartphone linkage system 2 PC
3 Smartphone 21, 31 WLAN unit 22, 32 Cooperation unit 23, 34 Input control unit 24, 35 Output control unit 33 Wireless control unit 36 Call control unit 37 Microphone control unit 38 Call recording unit 39 Phone book recording unit 3a Location detection control unit 41 Image processing unit 42 Audio processing unit 43 Operation processing unit 44 Communication control unit 51 Image processing unit 52 Audio processing unit 53 Touch control unit 54 Communication control unit 61 Capture unit 62 Difference extraction unit 63 Blackening unit 64 Image compression unit 65 Image transmission Unit 66 image data storage unit 71, 72, 73, 74 block 75 update rectangle 76 overall rectangle 211, 311 AP unit 212, 312 STA unit 221 setup unit 222 device linkage unit 223, 323 remote operation control unit 224 linkage data storage unit 225 GUI unit 226 Control unit 511 Reception unit 5 2 recovery unit 513 generating unit 514 display unit 621 splitting unit 622 comparison unit 623 extracting unit 624 conversion unit 625 identifying unit 700 hardware 710 OS
720 driver 721 WLAN program 721a STA program 721b AP program 730 application 731 cooperation program 800 smartphone 810 wireless communication unit 820 display unit 830 voice input / output unit 840 input unit 850 processor 860 storage unit 861 program storage unit 862 data storage unit 863 RAM
900 PC
910 main memory 920 CPU
930 LAN interface 940 HDD
950 Super IO
960 DVI
970 ODD

Claims (7)

A dividing unit that divides a target image, which is an image captured on a screen, into a plurality of blocks of a predetermined size;
A comparison unit for comparing with the previous image for each block obtained by the division by the division unit;
As a result of the comparison by the comparison unit, when there is an update location from the previous image, an extraction unit that extracts an update rectangle based on the update location from each block;
A specifying unit that specifies a rectangle based on the update rectangle extracted from each block by the extraction unit as an overall rectangle;
The entire rectangular image identified by the identifying unit, the entire rectangular information, and the updated rectangular information are transmitted to other information communication devices that have performed mutual authentication as data linkage targets , and the transmitted entire rectangular image and the entire Based on the information on the rectangle and the information on the updated rectangle, information on an operation event generated from a touch operation on the screen displayed on the other information communication device is received, and an operation corresponding to the operation event is performed on the own device. An information communication apparatus comprising: a transmission unit. A black painted portion that generates a black painted image by black painting a portion that is not included in any of the update rectangles of the entire rectangular image identified by the identifying unit;
A compression unit that compresses the black image generated by the black portion;
The information communication apparatus according to claim 1, wherein the transmission unit transmits the image compressed by the compression unit as an overall rectangular image.   The transmission unit compares the data size of the entire rectangular image and the black image after compression, and transmits the black image after compression when the data size of the black image after compression is small. 3. The information communication apparatus according to claim 2, wherein when the data size of the black-painted image after compression is not small, an entire rectangular image is transmitted. When the resolution of the target image is larger than a predetermined resolution, the calculation unit further includes a vertical and horizontal ratio between the resolution of the target image and the predetermined resolution,
The information communication apparatus according to claim 1, wherein the update rectangle is a rectangle corrected based on a vertical and horizontal ratio calculated by the calculation unit.   The information communication apparatus according to claim 2, wherein the transmission unit transmits the target image when the extraction unit repeats a predetermined number of times that no update rectangle can be extracted. Rectangle extracted as an update rectangle based on the change location from the block that has changed from the previous image among multiple blocks obtained by dividing the target image, which is the image captured on the screen , into a predetermined size A receiving unit that receives the information of the rectangle, the information of the rectangle specified as the entire rectangle based on the updated rectangle, and the information of the image of the entire rectangle from another information communication device that has performed mutual authentication as a data linkage target; ,
A restoration unit that restores an update rectangle image based on the update rectangle information, the entire rectangle information, and the entire rectangle image information received by the reception unit;
A generating unit that generates an entire image by replacing the corresponding rectangular image of the previous image with the updated rectangular image restored by the restoring unit;
The entire image generated by the generation unit is displayed on a display device, an operation event is generated from a touch operation on the displayed screen, and the operation event is transmitted to the other information communication device. And a display unit that performs an operation corresponding to the information communication device. Divide the target image, which is the image captured on the screen, into a plurality of blocks of a predetermined size,
Compare with the previous image for each block obtained by division,
As a result of the comparison, if there is an update location from the previous image, an update rectangle based on the update location is extracted from each block,
Specify a rectangle based on all the update rectangles extracted from each block as an overall rectangle,
The identified entire rectangular image, the entire rectangular information, and the updated rectangular information are transmitted to the information communication apparatus that has performed mutual authentication as data linkage targets, and the transmitted entire rectangular image, the entire rectangular information, and the updated rectangular information Receiving information on an operation event generated from a touch operation on the screen displayed on the information communication device based on the information, and causing the computer to execute processing corresponding to the operation event. Data linkage program.

Images