JP2010020159A - Screen output converter, display, display system, and screen display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display a screen of a device such as a computer in a remote display efficiently. <P>SOLUTION: The screen output converter 300 is connected with a screen output terminal of the device to supply image data output from the device to be displayed on the screen to the display system. A buffer 320 buffers the image data on a current screen in a block unit. An updating detection part 330 obtains an error detection code value of image data in each block on the current screen and compares it with an error detection code value of the image data in the block corresponding to a preceding screen to detect the presence or absence of the updating of image data in each block. An image coding part 340 compression-codes the image data in the block having been updated. A packet transmission part 350 transmits the image data in the block having been updated to the remote display system together with information for distinguishing the block. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a screen output converter for converting image data for screen display output from a computer, a display device incorporating the screen output converter, and display on the screen upon receipt of image data for screen display. It relates to a display system.

  Computer networks are also widely used in homes, and it is common to connect computers in rooms to each other via a wireless LAN and share printers. Also, in order for family members to view pictures taken with digital cameras and videos downloaded from the Internet, they can easily display still images and videos stored on computers on the screens of game consoles and television (TV) systems in the living room. There is also a growing need for it.

  Therefore, a game machine and a TV system are also connected to a computer network, and a function for displaying a computer screen on a display of a game machine or a TV system connected to the network instead of a PC monitor directly connected to the computer is required. It has become to.

A desktop screen of a remote computer is virtually displayed on a screen on another computer connected to the network, and the operation on the virtual desktop screen can be transmitted to a remote computer via a network using a specific protocol. A service called “remote desktop” that remotely controls a remote computer by transmitting it is also used.
US Patent Application Publication No. 2007 / 020,2956

  In order to display the desktop screen of a remote computer on the screen of another computer and perform remote control on the desktop screen, an operating system (OS) that is the same as or compatible with the remote computer is used. The same desktop environment needs to be realized on the computer on the operation side, support of the OS is indispensable, and the “remote desktop” service is necessarily dependent on the OS. In addition, since it is assumed to be supported by the OS, it is necessary to store the image data of the desktop screen in the main memory, which puts pressure on the main memory.

  There is a need for a technique to be called “remote display” that can efficiently display a computer display screen on the screen of another display system such as a TV system.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for efficiently displaying screen display data of a computer or the like on a screen of a remote display system and remotely operating the screen. is there.

  In order to solve the above problems, a screen output converter according to an aspect of the present invention is connected to a screen output terminal of a device and supplies image data for screen display output from the device to a remote display system. This screen output converter calculates the error detection code value of the image data of each block of the current screen and the buffer that buffers the image data of the current screen in units of blocks, and the image data of the corresponding block of the previous screen. An update detection unit that detects whether or not the image data of each block is updated by comparing with the error detection code value, and transmission that transmits the image data of the updated block together with the identification information of the block to the remote display system Part.

  Another aspect of the present invention is a display device incorporating the screen output converter of the above aspect.

  Yet another aspect of the present invention is a display system that receives image data for screen display from a screen output converter connected to a screen output terminal of the device. This display system receives a current screen image data from the screen output converter together with the block identification information in block units, decodes the received image data, refers to the block identification information, and A decoding unit for updating and buffering image data of a block to be processed, a display control unit for processing the buffered image data in accordance with the specifications of the display system and displaying it on the screen, and a user for the screen of the display system A user interface unit that accepts an operation of the user, and a transmission unit that transmits information related to the user's operation to the image output converter or the device.

  Yet another embodiment of the present invention is a screen display method. This method is a screen display method in which image data for screen display output from a device is supplied to a display system via a network for display, and buffering the image data of the current screen in units of blocks; Detects whether the image data of each block has been updated by obtaining the error detection code value of the image data of each block of the current screen and comparing it with the error detection code value of the image data of the corresponding block of the previous screen A step of compressing and encoding the image data of the updated block, and a step of transmitting the compressed and encoded image data of the updated block to the display system together with the identification information of the block. .

  Yet another embodiment of the present invention is a program for performing conversion for supplying image data for screen display output from a device to a display system. This program has the function to buffer the image data of the current screen in units of blocks, and the error detection code value of the image data of each block of the current screen to detect the error of the image data of the corresponding block of the previous screen The function of detecting whether or not the image data of each block is updated by comparing with the code value, the function of compressing and encoding the image data of the updated block, and the compression encoding of the updated block A function of transmitting image data to the display system together with identification information of the block is realized by a computer.

  This program may be provided as part of basic software or a library incorporated in the operating system. In order to provide such firmware, basic software or library, or to update a part thereof, a computer-readable recording medium storing this program may be provided. It may be transmitted via a communication line.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, apparatus, system, computer program, data structure, recording medium, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to provide a technique for efficiently displaying screen display data of a computer or the like on a screen of a remote display system and remotely operating the screen.

Embodiment 1
FIG. 1 is an overall configuration diagram of a screen display system according to the first embodiment. In this system, the screen output from the personal computer (PC) 200 is displayed on the PC display 100 instead of or on the normal PC display 100 directly connected to the PC 200, and also with Gigabit Ethernet (registered trademark) or the like. Display on a remote display of another display system connected to the network 900, here, an HDTV (High Definition Televison) 600 of the game machine 400.

  An input device such as a keyboard 290 and the PC display 100 are connected to the PC 200. The PC 200 is connected to the network 900 and can communicate with the game machine 400 and the like.

  The PC 200 outputs a digital video signal from the video signal output terminal 202. The video signal output terminal 202 is connected to the video signal splitter 210. The video signal splitter 210 branches the digital video signal output from the PC 200 to the PC display 100 and the screen output converter 300 and inputs them to each.

  The video signal input unit 110 of the PC display 100 receives the video signal output from the video signal splitter 210, demodulates it into pixel data, and supplies it to the display circuit 120. The display circuit 120 displays pixel data on the screen.

  The screen output converter 300 receives the video signal output from the video signal splitter 210, encodes pixel data, converts the pixel data into a packet format that can be transmitted to the network 900, and transmits the packet format to the screen output display system 500. The screen output converter 300 receives graphical user interface (GUI) data from the screen output display system 500, emulates keyboard / mouse commands based on the received data, and converts them into USB (Universal Serial Bus) signals. , Supplied to the USB signal input terminal 204 of the PC 200. Further, the screen output converter 300 receives character-based command data from the screen output display system 500, converts it into a USB signal based on it, and supplies it to the USB signal input terminal 204 of the PC 200.

  The game machine 400 has a built-in screen output display system 500 and can communicate with the screen output converter 300 and the PC 200 via the network 900.

  The screen output display system 500 receives the packetized video data from the screen output converter 300, decodes it to obtain pixel data, and displays it on the HDTV 600.

  FIG. 2 is a configuration diagram of the screen output converter 300. The video signal input unit 310 receives the video signal from the video signal splitter 210, demodulates it, and obtains pixel data of the screen of the current frame. For example, when the DVI (Digital Visual Interface) standard is used as the video output interface, the video signal input unit 310 has a function corresponding to a DVI receiver.

  The video signal input unit 310 stores the acquired pixel data in the buffer 320. The buffer 320 has a memory area divided in units of scan blocks on the screen, and the image data of each block is stored in the memory area of the corresponding block.

  The update detection unit 330 obtains the error detection code value of the image data of each block and compares it with the error detection code value of the image data of the corresponding block on the screen of the previous frame, thereby Detects whether there is an update.

  The image encoding unit 340 receives the notification of the number of the updated block from the update detection unit 330, reads the image data of the updated block from the corresponding memory area of the buffer 320, and compresses the image data of the block Encode.

  The packet transmission unit 350 adds the identification information of the block as a tag to the compression-encoded image data of the updated block, and transmits it to the screen output display system 500 via the network 900.

  In the above description, the update detection unit 330 obtains an error detection code value for the image data of each block before compression encoding and detects the presence or absence of the update, but the image encoding unit 340 performs compression encoding. An error detection code value may be obtained for the image data of each subsequent block, and the presence or absence of update may be detected. However, the former method is efficient in that it saves the trouble of compressing and coding the blocks that have not been updated.

  The GUI data receiving unit 360 receives GUI data transmitted from the screen output display system 500 via the network 900. The GUI data includes operation commands made via the GUI for windows and icons displayed on the remote display screen.

  The keyboard / mouse emulator 370 generates a signal generated when the operation is performed by emulating the operation of the keyboard and the mouse based on the GUI data received by the GUI data receiving unit 360. The keyboard / mouse emulator 370 inputs the generated signal to the USB signal input terminal 204 of the PC 200. As a result, the same effect is obtained in the PC 200 as if the operation was performed using an input device such as a keyboard or a mouse connected to the PC 200. The PC 200 executes the operation, updates the screen data, and outputs it.

  In the above description, the keyboard and mouse operations are emulated and given to the PC 200 based on the GUI data received from the screen output display system 500. When a character-based command is received from the screen output display system 500, The received command may be given to the PC 200 as it is.

  In the screen output converter 300 of the present embodiment, it is preferable that each functional configuration is configured by hardware and is made into one chip.

  FIG. 3 is a configuration diagram of the screen output display system 500. The packet receiving unit 540 receives screen image data packetized from the screen output converter 300 via the network 900. The image decoding unit 530 decodes the received image data, and buffers the decoded image data in the buffer 520.

  The buffer 520 has a memory area divided so that image data can be held in units of screen blocks. The image decoding unit 530 stores the image data of the corresponding block based on the block identification information added as a tag to the image data, whereby the image data of the updated block is rewritten.

  Display control unit 510 displays the image data stored in buffer 520 on the screen of HDTV 600. Thereby, the screen of the PC 200 is reconfigured to the screen of the HDTV 600. Display control unit 510 appropriately processes the image data stored in buffer 520 in accordance with the specifications of HDTV 600. For example, the HDTV 600 executes enlargement / reduction, clipping, filter processing, effect processing, etc. according to the resolution.

  The GUI 550 provides a function of a graphical user interface so that the user can perform various operations on the screen of the HDTV 600 using the input device of the game machine 400.

  Here, since the game machine 400 does not generate the screen of the PC 200 itself, the system does not know what windows and icons are displayed on the screen of the HDTV 600. Therefore, the operation target extraction unit 570 extracts an operation target such as a window or an icon from the screen image reconstructed on the HDTV 600 and notifies the GUI 550 of the position of the operation target window or icon. The GUI 550 provides an interface so that the user can operate on the operation target extracted from the image, and generates GUI data indicating the user's operation content on the operation target.

  Specifically, the operation target extraction unit 570 identifies the type of the extracted operation target based on information such as an icon shape and a character string, and grasps a command that can be executed on the operation target. For example, if the operation target is a file created by a word processor application, a file open command is specified as one of the operation contents. The GUI 550 generates a GUI command for opening the target file when the user performs an operation of clicking at the position of the operation target file.

  If the operation target extracted from the image is a video file, a playback command for the video file is specified as one of the operation contents. When the user performs an operation of clicking at the position of the operation target video file, the GUI 550 generates a GUI command for reproducing the target file.

  The GUI 550 gives the generated GUI data to the GUI data transmission unit 560. The GUI data transmission unit 560 transmits GUI data to the screen output converter 300 via the network 900. Thereby, the screen output converter 300 can emulate the user's operation content performed on the operation target of the remote display screen and supply it to the PC 200, and the PC 200 can update the screen by executing the user's operation content. .

  In the above description, data related to operations performed by the user via the graphical user interface is transmitted to the screen output converter 300. However, operations that the user can perform on the remote display screen are those via the graphical user interface. Not limited to. For example, a command for opening a file can be instructed by operating a specific key on the keyboard. In that case, the screen output display system 500 has a command generation unit, and the command generation unit generates a character-based command and transmits it to the screen output converter 300. The screen output converter 300 gives the command received from the screen output display system 500 to the PC 200.

  For example, when the user simultaneously presses a function key on the keyboard and the letter “O”, the command generation unit generates a command for opening a file and transmits it to the screen output converter 300. The screen output converter 300 supplies the received file open command to the PC 200. The PC 200 executes the file open command to update the screen.

  FIG. 4 is a diagram for explaining a screen image data supply process from the screen output converter 300 to the screen output display system 500.

  First, the operation of the screen output converter 300 on the transmission side will be described. The screen 10 of the PC 200 is divided into n blocks by scan blocks. The buffer 320 is also divided into n buffer memories corresponding to the scan blocks of the screen 10. The pixel data of each block is stored in a corresponding buffer memory of the buffer 320. That is, the image data of block 1 is stored in the buffer memory 1, the image data of block 2 is stored in the buffer memory 2, and the image data of block n is stored in the buffer memory n.

  The update detection unit 330 of the screen output converter 300 includes a CRC calculation unit 332, a CRC holding unit 334, and a comparison unit 336. The CRC calculation unit 332 performs a cyclic redundancy check (CRC), which is a kind of error detection code, on the image data of each block, and calculates a CRC value.

  The CRC holding unit 334 holds the CRC value of each block of the previous frame. The comparison unit 336 compares the CRC value of the image data of each block of the current frame calculated by the CRC calculation unit 332 with the CRC value of each block of the previous frame held in the CRC holding unit 334. . The image of the block whose CRC value does not match is updated. The comparison unit 336 notifies the image encoding unit 340 of the identification numbers of the blocks whose CRC values do not match.

  When the comparison of the CRC value of each block by the comparison unit 336 is completed, the CRC calculation unit 332 writes the CRC value of each block of the current frame in the CRC holding unit 334 and updates the CRC value of each block.

  Based on the block identification number notified from the comparison unit 336, the image encoding unit 340 reads the image data of the block updated in the current frame from the corresponding buffer memory, and performs compression encoding. The image encoding unit 340 adds the identification information of the block as a tag to the compression-encoded image data. The packet transmission unit 350 packetizes the updated block of compression-encoded image data and transmits it to the screen output display system 500. As a result, only the image data of the updated block is compressed and transmitted.

  Next, the operation of the screen output display system 500 on the receiving side will be described. The packet receiving unit 540 receives the compressed and encoded image data of the updated block from the screen output converter 300. The image decoding unit 530 decodes the image data and stores it in the buffer 520.

  The buffer 520 has a buffer memory divided in units of scan blocks of the screen of the HDTV 600, and holds image data of each block. The image decoding unit 530 refers to the block identification information added as a tag to the image data, and stores the decoded image data in the buffer memory corresponding to the block. The display control unit 510 displays the image data stored in the buffer 520 on the screen 1000 of the HDTV 600. Thereby, when the image of any block on the screen 10 is updated on the PC 200, the image of the corresponding block on the screen 1000 of the HDTV 600 is also updated.

  In the above description, the image data stored in the buffer 520 is displayed on the screen 1000 as it is. However, the image data stored in the buffer 520 is used as a texture, and the texture is enlarged, reduced, and effects are applied. You may display on the screen 1000 above. For example, the window screen displayed on the screen 10 of the PC 200 is cut out as a texture and processed so that it looks three-dimensional and displayed on the screen 1000. When the window is opened or closed, the window becomes larger or smaller. You may look like

  Here, when a plurality of windows are overlapped and displayed on the screen 10 of the PC 200, the texture images of the windows cannot be cut out individually. In order to extract the texture of each window by image recognition on the screen output display system 500 side, for example, an operation of switching to a mode in which a plurality of windows are displayed on the screen 10 side by side without being overlapped on the screen 10 on the PC 200 side. There is a need to do. Therefore, the screen output display system 500 generates a command for switching to a mode in which multiple windows are displayed side by side, and transmits the command to the screen output converter 300. The screen output converter 300 gives the command to the PC 200. On the screen 10 of the PC 200, the mode automatically switches to the window parallel display mode.

  As another method, instead of displaying all the plurality of windows on the screen 10 on the PC 200 side, the windows can be displayed one by one in order, and the texture of each window can be acquired on the screen output display system 500 side. You may do it. For this reason, the screen output display system 500 generates a command for opening and closing the windows one by one after closing all the windows once, and transmits the commands to the screen output converter 300. The screen output converter 300 gives the command to the PC 200. On the screen 10 of the PC 200, windows are automatically opened and closed one by one in order.

  The remote operation of the window system for the purpose of obtaining the texture of the window need not be shown to the user. The screen output display system 500 or the screen output converter 300 generates a window operation command in the background and gives it to the PC 200, and only the necessary window image data is acquired and held in the memory on the screen output display system 500 side. . The window is controlled so that the texture of the window can be easily extracted by image recognition, but the operation process is not shown to the user. Although the operation process can be seen on the screen 10 of the PC 200, the screen 1000 of the HDTV 600 does not change during that time. On the screen 1000 of the HDTV 600, the acquired window texture can be freely processed and displayed, and does not need to be linked to the screen 10 of the PC 200.

Embodiment 2
FIG. 5 is an overall configuration diagram of the screen display system according to the second embodiment. The PC display 102 with a transmission function of this system incorporates the video signal splitter 210 and the screen output converter 300 of the first embodiment.

  The video signal output terminal 202 of the PC 200 is directly connected to the video signal input terminal of the PC display 102 with a transmission function. The video signal splitter 210 of the PC display 102 with a transmission function branches the video signal output from the video signal output terminal 202 of the PC 200 to the video signal input unit 110 and the screen output converter 300, and inputs them.

  The operations of the video signal input unit 110 and the display circuit 120 are the same as those in the first embodiment. The functional configuration and operation of screen output converter 300 are the same as those in the first embodiment.

  The high function TV system 602 is a television system incorporating the screen output display system 500 of the first embodiment. For example, the high function TV system 602 includes a multiprocessor system, and the multiprocessor executes the image decoding function and the display control function of the screen output display system 500 by software.

Embodiment 3
FIG. 6 is an overall configuration diagram of the screen display system according to the third embodiment. In this system, the multiprocessor system 220 executes each function of the screen output converter 300 by software.

  For example, the multiprocessor system 220 is a multicore processor in which two or more processor cores are integrated in one package, and includes a main processor 230, a plurality of subprocessors 260, a shared memory 270, and a communication interface 280.

  One sub-processor 260 causes the function of the screen output converter 300 to be executed by software. The video signal output from the video signal output terminal 202 is branched to the video signal input unit 110 of the PC display 100 and the screen output converter 300 by the video signal splitter 210 and is input to each. The configuration and operation of the screen output converter 300 are the same as those in the first embodiment. After the screen image data is compression-encoded, it is packetized and transmitted to the portable terminal 800 via the communication interface 280.

  A wireless LAN (WLAN) 700 is connected to the network 900, and the mobile terminal 800 receives screen image data from the multiprocessor system 220 via the WLAN 700. The mobile terminal 800 decodes the received image data, displays it on the screen of the mobile terminal 800, causes the user to operate the operation target on the screen via the GUI, and sends the GUI data back to the multiprocessor system 220.

  FIG. 7 is a diagram for explaining a screen 1000 of each remote display of HDTV 600 according to the first embodiment, high-function TV system 602 according to the second embodiment, and mobile terminal 800 according to the third embodiment. This screen example is a desktop screen of a multi-window system, and displays a word processor application window 1002, a window 1004 for browsing the contents of a folder, a calendar 1006, various icons 1008, and the like.

  The cursor 1010 is operated by the game controller of the game machine 400 in the first embodiment, the remote controller of the high-function TV system 602 in the second embodiment, and the keys and buttons of the portable terminal 800 in the third embodiment. Operated. In any of the embodiments, a GUI is provided so that the user can perform operations such as movement and click of the cursor 1010 using a pointing device or the like.

  FIG. 8 is a diagram for explaining an operation example on the remote display screen 1000. When the user performs an operation such as pressing a specific key with the input device, the image around the cursor 1010 is clipped and enlarged and displayed on the screen.

  FIG. 9 is a diagram for explaining another example of operation on the remote display screen 1000. The remote display screen 1000 includes windows, icons, buttons, and the like that are user operation targets. However, these operation targets are managed by basic software such as the window system of the PC 200 on the transmission side, and are recognized by the game machine 400 on the reception side, the high-function TV system 602, the portable terminal 800, and the like. It is not. Therefore, the GUI 550 of the screen output display system 500 on the receiving side first extracts operation targets such as windows, icons, and buttons displayed on the remote display screen 1000 by image processing.

  As a result, the window 1040 of the word processor application is specified as an operation target, and menus (reference numerals 1050, 1052, 1054, 1056, 1058) such as files, edits, displays, fonts, tools, etc. are specified as the operation targets in the window 1040. Identified. The GUI 550 provides an interface so that the user can perform a GUI operation for changing the position and size of the window with respect to the window 1040, and generates GUI data indicating the operation content of the window 1040 performed by the user. When the user moves the cursor 1010 to each menu position in the window 1040 and clicks, the GUI 550 generates GUI data indicating the click operation at that position.

  The generated GUI data is transmitted to the screen output converter 300 by the GUI data transmission unit 560. On the PC 200 side, an event for changing the position and size of the window 1040 and an event for selecting a menu occur in the window system, and the image of the window 1040 is updated. The screen output converter 300 transmits the update data of the PC screen, and the remote display screen 1000 on the receiving side is updated.

  Similarly, a window 1042 for browsing a folder is specified as an operation target, and a document file A (reference numeral 1030), a system file B (reference numeral 1032), a folder C (reference numeral 1034), a folder displayed in the window 1042 are displayed. D (reference numeral 1036) and a video file (reference numeral 1038) are specified as the operation targets.

  When the user moves the cursor 1010 to the position of the document file A and performs an operation of double-clicking, GUI data indicating the operation content is generated and transmitted to the screen output converter 300. An event for opening the document file A occurs in the window system of the PC 200, the screen data is updated, and the update result is reflected on the remote display screen 1000 on the receiving side.

  Although the system file B may be operated according to management needs in the PC 200, it is a file that does not need to be operated on the remote display system, and is therefore excluded from the operation target on the remote display system.

  When the operation target extraction unit 570 of the screen output display system 500 extracts the operation target from the image of the remote display screen 1000, the operation target extraction unit 570 determines the file type based on information such as the icon shape and the file name extension character string. judge. The GUI 550 restricts operation targets that are allowed to be operated by the user according to the file type. For example, an application file is set as an operation target, but a system file is excluded from the operation target. In this way, the GUI 550 can limit the operation target permitted by the user among the operation targets extracted by the operation target extraction unit 570 to the range of services used in the remote display system.

  For the calendar 1044, a button 1061 for displaying “next month” and a button 1062 for displaying “previous month” are extracted as operation target buttons.

  Also, icons of various applications (reference numerals 1020, 1022, 1024, 1026, 1028) are extracted as operation target icons on the desktop screen. The operation target extraction unit 570 includes an icon 1020 for displaying a photo album, an icon 1022 for playing music, an icon 1024 for displaying the world clock, and an icon 1026 for displaying the icon. This icon specifies that the map display application corresponds to an application, and the icon 1028 corresponds to an application that customizes system settings. The icons for each application are registered in advance in the remote display system so that the operation target extraction unit 570 can identify the application from the icon design.

  Among these applications, an application for customizing the system setting indicated by the symbol 1028 is executed on the PC 200, but generally should not be executed on the remote display system. Is done.

  The GUI 550 may provide a navigation function for guiding a cursor operated by the user to an operation target icon extracted from the screen. For example, when the user moves the cursor 1010 slightly to the right side of the remote display screen 1000, one of the icons 1020, 1022, 1024, and 1026 of the operation target application is highlighted, and the cursor 1010 is slightly moved up and down. By doing so, the user can easily select an icon of any application by performing navigation such that the highlighted icon is switched up and down.

  In order to highlight the icon designated by the cursor 1010, the operation target extraction unit 570 specifies the position and size of the icon by image recognition. The GUI 550 highlights the icon by changing the color of the icon image designated by the cursor 1010 or combining the icon image with a mark image.

  This navigation function eliminates the need for the user to operate the pointing device to move the point of the cursor to the icon to be operated or finely adjust the position of the icon. Unlike the cursor operation on the PC 200, the cursor operation on the game machine 400 or the high-function TV system 602 is often burdened by the user, so the navigation function is highly convenient.

  The GUI 550 may provide autopilot functionality for specific applications. The PC service used on the TV screen is specified in advance, and the GUI 550 automatically generates a GUI operation for executing a specific process of the specific application. For example, when a specific button of the input device of the remote display system is pressed, an operation of moving the cursor 1010 to the icon 1026 of the map display application and clicking the icon 1026 is automatically executed. In order to enable such automatic control, an agent (robot) function may be installed in the remote display system.

  The GUI 550 generates GUI data indicating these autopilot operations, and the GUI data transmission unit 560 transmits the generated GUI data to the screen output converter 300. As a result, the application is automatically executed on the PC 200 side, and the result is reflected on the remote display screen. This may be said to be macro automatic execution by GUI operation on the remote display.

  In addition, “attraction” is set for an icon of an application or file that may move the cursor 1010, and when the cursor 1010 comes near one of those icons, the speed is increased and the icon is drawn. The cursor may be controlled so that As a result, the user does not need to perform cursor operation more than necessary on the TV screen placed in the living room.

  FIG. 10 is a diagram for explaining still another operation example on the remote display screen 1000. When the aspect ratio of the remote display screen 1000 is different from the aspect ratio of the PC display 100, a blank area is generated on the remote display screen 1000. An image around the position of the cursor 1010 may be enlarged and displayed in this blank area, or information related to the operation target designated by the cursor 1010 may be displayed.

  In the example of FIG. 10, the cursor 1010 points to the video file 1038, and guidance information on how to operate the video file 1038 is displayed in the right margin area 1070 of the desktop screen 1060. For example, operations such as video playback, browsing of thumbnail index images, and scene search using keywords are possible. When the user selects the operation content indicated in the blank area 1070, the GUI 550 generates GUI operation data for executing the operation content designated for the video file 1038. For example, when video playback is selected, GUI data indicating an operation of clicking the video file 1038 is generated. When viewing the thumbnail index image is selected, GUI data indicating a series of operations such as clicking the right mouse button on the video file 1038 and selecting “view thumbnail index” from the submenu displayed at that time. Is generated.

  As described above, according to the embodiment, the function of “remote display” for efficiently displaying the display screen of the PC 200 on the screen of a remote display system such as the game machine 400, the high function TV system 602, and the portable terminal 800 is provided. Can be provided. Further, the screen can be processed and displayed so as to be easy for the user to see according to the specifications of the remote display system. For example, when a high-definition high-definition television system is in a living room, it is easy for the user to view the screen by using the screen size and image quality of the high-definition television system.

  Whereas “Remote Desktop” is a service that enables the remote desktop environment of a remote computer to be used seamlessly and remotely, “Remote Display” in the embodiment is used to transfer image data of a computer screen to a remote display system. Basically, it is transmitted and reconfigured to the screen of the remote display system. “Remote display” does not provide the same environment as the desktop environment of a computer. Since services used in the remote display system are limited, the degree of freedom of GUI operation can be greatly limited, which facilitates the user's GUI operation and improves convenience.

  The “remote display” simply receives image data of a computer screen, extracts an operation target from the received image data, generates data indicating a GUI operation, and sends it back to the computer. Therefore, it does not depend on basic OS software such as a computer window system. Even a screen of a computer operating on an old generation OS or an incompatible OS can be displayed and operated on the screen of the remote display system.

  When displaying the screen of a computer running an old generation window system on the screen of a remote display system, the display format of the window is upgraded and displayed as if it were operating on the new generation window system. You can also For example, if a new generation window system has a function to display a window three-dimensionally or dynamically, if the processing performance of the transmitting computer is low, the new generation window system may be Difficult to implement. If you use a remote display, the remote display system receives the screen of the computer running the old generation window system, processes the image data, and the remote display system screen operates as if it were a new generation window system. The window can be displayed three-dimensionally or dynamically.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In the above description, the image of the computer screen is displayed on the screen of the remote display system. However, the image displayed on the screen of the remote display system is not limited to the image data of the computer screen. Since the “remote display” does not depend on the OS of the computer at all, any video signal output from the screen output terminal of any device is converted by the screen output converter 300 and transmitted to the remote display system. Displayed on the screen and can be operated remotely.

  For example, the image data displayed on the liquid crystal display of the digital camera may be transmitted to the remote display system and displayed on the remote display screen. Since the liquid crystal display of a digital camera has a small screen size, it may be difficult to see or operate, but it is enlarged and displayed on the remote display screen, so that the operation becomes easy. In addition, since the screen of a digital camera's liquid crystal display is narrow, there is a limit to the realization of a multi-window system. However, on a remote display screen, an image browser, an image editor, etc. can be displayed on the multi-window system. Can be increased.

1 is an overall configuration diagram of a screen display system according to Embodiment 1. FIG. It is a block diagram of the screen output converter of FIG. It is a block diagram of the screen output display system of FIG. It is a figure explaining the supply process of the image data of the screen from a screen output converter to a screen output display system. 3 is an overall configuration diagram of a screen display system according to Embodiment 2. FIG. FIG. 10 is an overall configuration diagram of a screen display system according to a third embodiment. It is a figure explaining a remote display screen. It is a figure explaining the example of operation in the remote display screen of FIG. It is a figure explaining another example of operation in the remote display screen of FIG. It is a figure explaining the example of another operation in the remote display screen of FIG.

Explanation of symbols

  100 PC display, 102 PC display with transmission function, 200 PC, 202 video signal output terminal, 204 USB signal input terminal, 210 video signal splitter, 300 screen output converter, 310 video signal input unit, 320 buffer, 330 update detection unit, 332 CRC calculation unit, 334 CRC holding unit, 336 comparison unit, 340 image encoding unit, 350 packet transmission unit, 360 GUI data reception unit, 370 keyboard / mouse emulator, 400 game machine, 500 screen output display system, 510 display control Unit, 520 buffer, 530 image decoding unit, 540 packet receiving unit, 550 GUI, 560 GUI data transmitting unit, 570 operation target extracting unit, 600 HDT V, 602 High-performance TV system, 800 mobile terminal, 900 network.

Claims (15)

A screen output converter connected to a screen output terminal of the device and supplying image data for screen display output from the device to a remote display system,
A buffer that buffers the image data of the current screen in blocks;
By detecting the error detection code value of the image data of each block of the current screen and comparing it with the error detection code value of the image data of the corresponding block of the previous screen, the presence or absence of update of the image data of each block is detected. An update detection unit;
A screen output converter, comprising: a transmission unit that transmits image data of an updated block together with identification information of the block to the remote display system. A receiver that receives information related to an operation performed by the user on the screen displayed in the remote display system;
The screen output converter according to claim 1, wherein after the operation performed by the user is executed in the device, the buffer buffers the screen image data obtained as image data of the next screen.   The operation or the content of the operation that the user is allowed to perform on the screen displayed on the remote display system is limited compared to the operation that the user can perform on the device. The screen output converter according to claim 2.   A display device comprising the screen output converter according to claim 1. A display system for receiving image data for screen display from a screen output converter connected to a screen output terminal of a device,
A receiving unit that receives image data of a current screen from the screen output converter together with block identification information in units of blocks;
A decoding unit that decodes received image data, refers to block identification information, updates and buffers image data of a corresponding block;
A display control unit that processes the buffered image data according to the specifications of the display system and displays it on the screen;
A user interface unit that accepts user operations on the screen of the display system;
A display system comprising: a transmission unit configured to transmit information related to a user operation to the screen output converter or the device. An extraction unit that extracts an operation target operated by the user from an image displayed on the screen of the display system;
The display system according to claim 5, wherein the user interface unit accepts a user operation on an operation target extracted from a screen by the extraction unit.   The user interface unit restricts an operation target that is permitted to be operated by a user among operation targets extracted from a screen by the extraction unit to a range of a service used in the display system. 7. The display system according to 6.   The display system according to claim 6, wherein the user interface unit provides a navigation function for guiding a point destination of a pointing device operated by a user to an operation target extracted from a screen by the extraction unit. .   The display system according to claim 6, wherein the user interface unit provides an autopilot function for executing a specific operation on the operation target extracted from the screen by the extraction unit.   The display system according to claim 5, wherein the display control unit enlarges and displays an image around a point destination of a pointing device operated by a user.   When the aspect ratio of the screen of the display system is different from the aspect ratio of the screen normally assumed for the device, the display control unit points to a blank area generated on the screen of the display system with the point destination of the pointing device operated by the user. The display system according to claim 5, wherein an image around the image is enlarged and displayed.   When the aspect ratio of the screen of the display system is different from the aspect ratio of the screen normally assumed by the device, the display control unit provides guidance on an operation method related to the user's operation target in a blank area generated on the screen of the display system. Information is displayed, The display system in any one of Claim 5 to 9 characterized by the above-mentioned. A program for performing conversion for supplying image data for screen display output from a device to a remote display system,
A function to buffer the current screen image data in blocks,
By detecting the error detection code value of the image data of each block of the current screen and comparing it with the error detection code value of the image data of the corresponding block of the previous screen, the presence or absence of update of the image data of each block is detected. Function and
A function of compressing and encoding the image data of the updated block;
A program for causing a computer to realize a function of transmitting compression-encoded image data of an updated block to the remote display system together with identification information of the block. A screen display method for supplying image data for screen display output from a device to a remote display system via a network for display.
Buffering the current screen image data in blocks;
By detecting the error detection code value of the image data of each block of the current screen and comparing it with the error detection code value of the image data of the corresponding block of the previous screen, the presence or absence of update of the image data of each block is detected. Steps,
Compressing and encoding the image data of the updated block;
Transmitting the compressed and encoded image data of the updated block together with the identification information of the block to the remote display system. Decoding received image data and buffering in block units;
Processing the buffered image data according to the specifications of the remote display system and displaying it on the screen;
Providing the device with information relating to user operations performed on a screen of the remote display system via a user interface;
The screen display method according to claim 14, further comprising: executing the user operation on the device and buffering image data of a screen obtained as a result in units of blocks.

Images