JP2011081591A - Information processing apparatus and system, computer program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing apparatus for enabling a user to use content stored in an information processing apparatus by the other device side by a simple operation regardless of the restriction of the display device of the information processing apparatus even when the user is not familiar with the operation of the other device. <P>SOLUTION: An information processing apparatus includes: a display device; a CPU for executing processing accompanied by the display of an image at the display device; steps 262 and 264 for establishing connection with the other devices in order to share the image displayed at the display device with the other devices; processing 268 and 272 for automatically switching a connection mode with the other devices between a first connection mode and a second connection mode during connection with the other devices; processing for transmitting the image displayed in the first connection mode to the other device; and processing for creating an image separately from the image displayed at the display device in the second connection mode, and for transmitting the image to the other devices. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to cooperation processing between information processing apparatuses having display devices, and particularly relates to cooperation processing for screen display between information processing apparatuses having different screen sizes.

  Technology that allows home appliances to connect to each other over a network and view content held on a mobile phone or the like with a larger device such as a television receiver (hereinafter simply referred to as a “TV receiver”) It has become. As a standard for that purpose, DLNA (Digital Living Network Alliance) is known. In DLNA, electrical appliances are connected to each other via wired and wireless networks, and contents can be mutually used between electrical appliances in the network.

  However, in the case of an electrical appliance compliant with DLNA, content stored in a first device (hereinafter simply referred to as “server”) is to be viewed on a second device (hereinafter referred to as “client”). In this case, there is a problem that the user needs to know the operation of the content by the client. For example, when the owner of the server storing the content visits another person's house and wants to view the content on the TV receiver (client) of the house, for example, the user must remember the operation of the TV receiver. I must. Therefore, it has been quite difficult for server owners to fully utilize the convenience of such a system.

  In order to solve these problems, the following techniques have been put into practical use. In other words, this is a technique in which a screen of a remote device is acquired by communication, displayed on the device at hand, and an operation performed on the screen by the device is sent back to the remote device. This technique is called a remote desktop. For example, a program that causes the general-purpose computer to function as the server is operated on a general-purpose computer, and a client program is operated on another information processing terminal (for example, a PC (Personal Computer)). As a result, a remote server can be operated from the client PC. In the above example, the mobile phone is connected to the mobile phone from the TV receiver with the mobile phone as the server and the TV receiver as the client, and the mobile phone screen is received by the TV receiver and displayed on the screen of the TV receiver. By operating the screen using a remote controller (hereinafter referred to as “remote controller”) of the TV receiver, the mobile phone is operated from the TV receiver side, and the contents of the mobile phone are displayed on the TV receiver. it can. As long as the mobile phone can be operated, it is not always necessary to become familiar with the operation of the TV receiver.

JP 2009-122947 A

  However, on the contrary, in the above technique, the screen to be displayed is only the screen of the mobile phone, and there is a problem that the display capability of the TV receiver cannot be fully utilized. The display resolution, color, and the like are limited by the display capability of the mobile phone, and the display capability of the TV receiver that has a large screen and high definition cannot be fully utilized.

  Therefore, Patent Document 1 discloses the following technique by taking it one step further. That is, a shared area for displaying a part of the server screen is provided on the client screen in the same manner as in the above technique. The input event information generated for the shared area is sent to the server. When an instruction to transfer data from the server to the shared area is generated by an operation on the shared area, the client requests the server to transfer the data, and the transferred data is stored not in the shared area but in the client's memory space. Expand and store. With such a function, the client can display this data by using its own display function instead of the shared area, and can fully utilize the display performance of the client.

  However, the technique disclosed in Patent Document 1 still has the following problems. That is, in the case of the technique disclosed in Patent Document 1, since data transferred to the client is displayed using the display function of the client, if the function for expanding and displaying the data is not provided, the data can be displayed in the first place. Impossible.

  Therefore, even if the object of the present invention is not familiar with the operation of other devices, the contents held in the information processing device can be easily operated regardless of the limitations of the display device of the information processing device. It is providing the information processing apparatus and system which can be utilized by the apparatus side.

  An information processing apparatus according to a first aspect of the present invention includes a display device, a processing execution unit for executing a process involving display of an image on the display device, and an image displayed on the display device by the processing execution unit. In order to share with another device, a communication device that communicates with the other device, and the first connection mode and the second connection mode are connected to the other device via the communication device. Means for automatically switching a connection mode with another device between the two, and operating in the first connection mode and transmitting an image displayed on the display device to the other device via the communication device A first image transmitting means and a second image that operates in the second connection mode and generates an image separately from the image displayed on the display device and transmits the image to another device via the communication device Transmitting means.

  Preferably, in response to being instructed to connect to another device, the communication device makes an inquiry to the other device about display device information related to the display performance of the display device of the other device. In response to receiving the display device information from another device in response to the inquiry by the inquiry unit, the other device that has transmitted the display device information so that the screen data can be communicated at a predetermined resolution. In response to the establishment of the connection by the connection establishment means for establishing a connection with the apparatus, the connection mode with the other apparatus is set to the first connection mode, Means for activating the first image transmitting means, and means for transmitting the image to another apparatus in response to receiving an image from one of the first and second image transmitting means Including.

  The process execution means executes a process involving displaying an image on the display device. The communication device performs communication for sharing an image with another device. While the information processing apparatus is connected to another apparatus by the communication apparatus, the switching means automatically switches the connection mode between the first connection mode and the second connection mode. In the first connection mode, the image displayed on the display device is transmitted to another device by the first image transmission means. In the second connection mode, an image is created separately from the image displayed on the display device by the second image transmission means, and transmitted to another device.

Between the first connection mode in which an image displayed on the display device is displayed on another device and the second connection mode in which an image different from the image displayed on the display device is displayed on the other device The connection mode is automatically switched. By using this information processing device, the user can display a state in which the same screen as the display screen of the information processing device is displayed on another device and a screen different from the display screen of the information processing device. The information processing apparatus automatically switches the connection mode without being conscious of switching to the state to be performed. As a result, even if the user is not familiar with the operation of the other device, the content held in the information processing device can be easily operated on the other device side regardless of the limitations of the display device of the information processing device. An information processing apparatus that can be used can be provided.
Or, in response to an instruction to connect to another device, the communication device makes an inquiry to another device for inquiring about display device information related to the display performance of the display device of the other device. And another device that has transmitted the display device information so that the screen data can be communicated at a predetermined resolution in response to receiving the display device information from another device in response to the inquiry by the inquiry means In response to the establishment of the connection by the connection establishment means, the connection mode between the other devices is set to the second connection mode, Means for activating the second image transmitting means, and means for transmitting the image to another apparatus in response to receiving an image from one of the first and second image transmitting means. May be included.

  More preferably, the display device information includes information on the maximum resolution that can be displayed on the display device, and the predetermined resolution is a resolution having a predetermined relationship with the maximum resolution of the display device included in the display device information.

  More preferably, the predetermined resolution is equal to the maximum resolution of the display device included in the display device information.

  When this information processing apparatus is connected to another apparatus, the connection is made at the maximum resolution of the display apparatus connected to the other apparatus, not the display performance of the display apparatus on the information processing apparatus side. When the resolution of a display device connected to another device is larger, by making such a connection, the entire screen displayed on the information processing device can be displayed on the other device.

  The second image transmission means operates in the second connection mode, creates an image with the maximum resolution of the display device included in the display device information separately from the image displayed on the display device, and transmits the image via the communication device. Means for transmitting to other devices may be included.

  An image can be created in the information processing apparatus at the maximum resolution that can be displayed on the display device of the other apparatus, and transmitted to the other apparatus for display. As a result, even if the user is not familiar with the operation of the other device, the content held in the information processing device can be displayed on the display device of the other device regardless of the limitations of the display device of the information processing device. It is possible to provide an information processing apparatus that can be used on the other apparatus side with an easy operation while fully utilizing.

  Preferably, the display device information includes information related to a maximum resolution that can be displayed in a window in which an image transmitted from the information processing apparatus is displayed by the display device, and the predetermined resolution is equal to the maximum resolution of the window.

  When another device has a so-called PinP (Picture-in-Picture) function and displays an image from the information processing device in a window opened in a part of the screen, the image is displayed at the maximum resolution that can be displayed with the image. And send it to other devices. Since the entire window can be displayed, it is possible to realize an easy-to-view display that makes full use of the size of the window.

  More preferably, the information processing device is a virtual input for processing the predetermined signal as a virtual input to the information processing device in response to the communication device receiving the predetermined signal from another device. Processing means is further included.

  When receiving a predetermined signal from another device, the virtual input means processes it as a virtual input to the information processing device. Since such a means is provided, an appropriate process for an operation performed on the other device side can be performed on the information processing device side. In addition to the operation on the information processing device side, the other device side It is possible to realize an efficient display using even the operations performed in.

  The information processing apparatus further includes an input means for generating an operation instruction signal for the information processing apparatus in response to a user operation, and a predetermined process in response to the operation signal generated by the input means. Operation executing means for executing. The virtual input means includes means for converting a virtual input obtained from a predetermined signal received via the communication device into an operation instruction signal that can be processed by the input means and supplying the operation instruction signal to the operation execution means.

  More preferably, the virtual input processing means functions in both the first operation mode and the second operation mode.

  Preferably, in the second operation mode, the virtual input processing means responds to reception of a predetermined signal from another device and uses the predetermined signal as a virtual input to the second image transmission means. Means for processing. The second image transmission means executes processing using the virtual input given by the means for processing, and creates an image different from the image displayed on the display device according to the execution result. Means for transmitting to other devices via the communication device.

  More preferably, the information processing apparatus further responds to the fact that the display device information is not received from the other apparatus within a predetermined time from the transmission of the inquiry by the inquiry means, and the first and second images Means for stopping the transmission means;

  If there is no device that responds to the inquiry by the inquiry means, there is no other device nearby. In such a case, by stopping the first and second image transmission units, it is possible to realize a process that is completed only by the information processing apparatus without performing unnecessary processes.

  More preferably, the display device information further includes data format information for specifying a data format that can be displayed by another device. The information processing apparatus further responds to the fact that the data format of the content data for displaying the image on the display device by the processing execution means matches the data format that can be displayed by another device. And a means for transmitting the content data to another device via the communication device.

  In the case of content in a data format that can be processed on the other apparatus side, it is more reasonable to process the data on the other apparatus side than to expand the data on the information processing apparatus side. By transmitting the content data to the other device side and processing it on the other device side, it is not necessary to use resources on the information processing device side wastefully.

  An information processing system according to a second aspect of the present invention is an information processing system including first and second information processing apparatuses. The first information processing device includes a display device, a processing execution unit for executing a process involving display of an image on the display device, and an image displayed on the display device by the processing execution unit. The first connection mode and the second connection mode during the connection with the second information processing apparatus via the communication apparatus and the communication apparatus that communicates with the second information processing apparatus to share with the second information processing apparatus. For automatically switching the connection mode with the second information processing apparatus between the first and the second information processing apparatus, and the second operation is performed in the first connection mode, and the image displayed on the display apparatus is transmitted to the second through the communication apparatus. The first image transmission means for transmitting to the information processing apparatus and the second information that operates in the second connection mode and creates an image separately from the image displayed on the display apparatus, via the communication apparatus Second image transmission means for transmitting to the processing device. . The second information processing device includes a communication device for communicating with the communication device of the first information processing device, a display device, and the communication device of the first information processing device. In response to reception of the image transmitted from the first or second image transmission means by the communication device of the second information processing apparatus, for displaying the image on the display device of the second information processing apparatus Means.

  An information processing method according to a third aspect of the present invention includes a display device, a process execution unit for executing a process involving display of an image on the display device, and an image displayed on the display device by the process execution unit. In order to share with another device, a communication device that communicates with the other device, and the first connection mode and the second connection mode are connected to the other device via the communication device. Mode switching means for automatically switching the connection mode with another device between them, and operating in the first connection mode, and transmitting the image displayed on the display device to the other device via the communication device A first image transmission means for performing the operation and a second operation for operating in the second connection mode and generating an image separately from the image displayed on the display device and transmitting the image to another device via the communication device. Information processing apparatus including information transmission means It is a management method. In this method, a process executing unit executes a process involving display of an image on a display device, a communication device establishes communication with another device, and a mode switching unit includes a communication device. A step of automatically switching a connection mode between the first apparatus and the second connection mode between the first apparatus and the second apparatus during connection with the other apparatus via the first image transmission means; A step of transmitting an image displayed on the display device to another device via the communication device in response to the switching of the connection mode to the first connection mode, and a second image transmission means comprising: In response to the switching of the connection mode to the connection mode of 2, the image is generated separately from the image displayed on the display device and transmitted to another device via the communication device.

  When the computer program according to the fourth aspect of the present invention is executed by a computer connected to a display device and provided with a communication device, the computer program operates as one of the information processing devices described above. is there.

  A recording medium according to the fifth aspect of the present invention is a computer-readable recording medium on which the above-described computer program is recorded.

  As described above, according to the present invention, the connection mode is automatically switched between the first connection mode and the second connection mode. By using this information processing device, the user is unaware of switching between a state in which the same screen as the display screen of the information processing device is displayed on another device and a state in which another screen is displayed. In both cases, the connection mode is automatically switched. As a result, even if the user is not familiar with the operation of the other device, the content held in the information processing device can be easily operated on the other device side regardless of the limitations of the display device of the information processing device. An information processing apparatus that can be used can be provided.

It is a schematic block diagram of the data cooperation system which concerns on the 1st Embodiment of this invention. It is a figure which shows the external appearance of the mobile telephone as a content server which comprises the data cooperation system shown in FIG. It is a figure which shows the external appearance of the remote control of TV receiver which functions as a client in the data cooperation system shown in FIG. It is a figure which shows the transition of the display screen of TV receiver and a mobile telephone in 1st Embodiment. It is a figure which shows the transition of the display screen of TV receiver and a mobile telephone in 1st Embodiment. It is a figure which shows the transition of the display screen of TV receiver and a mobile telephone in 1st Embodiment. It is a figure which shows the transition of the display screen of TV receiver and a mobile telephone in 1st Embodiment. It is a figure which shows the transition of the display screen of TV receiver and a mobile telephone in 1st Embodiment. It is a block diagram which shows schematic structure of the mobile telephone shown in FIG. It is a block diagram which shows schematic structure of TV receiver shown in FIG. It is a figure which shows typically communication between the programs which implement | achieve the photography application performed with the data cooperation system 30 shown in FIG. It is a flowchart which shows the control structure of the photography application performed with a mobile telephone. It is a flowchart which shows the control structure of the program which transmits a screen to TV receivers from programs, such as a photography application, performed with a mobile telephone. It is a flowchart which shows the control structure of the program for receiving the screen from a mobile telephone performed with TV receiver. It is a flowchart which shows the control structure of the program which implement | achieves the switching process of the stand-alone / cooperation operation performed with the photograph application which shows a flowchart in FIG. It is a flowchart which shows the control structure of the program performed for the photograph application of a mobile telephone, a screen transmission program, and a photograph display with TV receiver. It is a figure which shows the data format of the various messages transmitted and received by the data cooperation system 30 shown in FIG.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description and drawings, the same components are denoted by the same reference numerals. Their functions and names are also the same. Therefore, detailed description thereof will not be repeated.

  Referring to FIG. 1, a data linkage system 30 according to an embodiment of the present invention has a large display screen 46 and has a function of receiving content from another server and displaying it on the display screen 46. And a mobile phone 44 that holds content and functions as a content server for the TV receiver 40. A remote control 42 is attached to the TV receiver 40. In the present embodiment, basically, a photograph stored in the mobile phone 44 is displayed on the display screen 46 of the TV receiver 40 by operating the mobile phone 44. However, once the content held by the mobile phone 44 is displayed on the TV receiver 40, the remote controller 42 can be used for the operation.

  The feature of this embodiment is that the content held in the mobile phone 44 can be displayed and viewed on the display screen 46 of the TV receiver 40 and the function of playing back the content held in the mobile phone 44 is a TV. Even when it does not exist in the receiver 40, the content can be displayed on the display screen 46 by using the function of the mobile phone 44. Furthermore, the mobile phone 44 and the TV receiver 40 have the same screen. The screen sharing mode for sharing the screen and the screen that the mobile phone 44 should display on the TV receiver 40 separately from the screen of the mobile phone 44 are created as virtual screens, and the virtual screen mode for displaying them separately is automatically switched. It is in.

  Referring to FIG. 2, mobile phone 44 includes a first casing 52 in which a key such as a numeric keypad is disposed, and a second casing that is foldably attached to first casing 52 and has a liquid crystal screen 54. 50.

  The keys arranged in the first casing 52 include function keys 62, 64, 68 and 70, a cursor key 66 arranged at the center thereof, and an enter key 72 arranged at the center of the cursor key 66. Including. Of the function keys, the function key 68 is used as a return key for returning the display to the previous state, and the function key 70 is used as an end key for ending the program.

  The keys arranged in the first housing 52 further include a numeric keypad 60 used for inputting numbers, alphabets, and various symbols. In the following description, a photo display program (referred to as “photo application”) that displays a photo held in the mobile phone 44 on the TV receiver 40 is taken as an example. In this photo application, the cursor key 66, Function keys 68 and 70 and enter key 72 are used.

  Referring to FIG. 3, the remote controller 42 is also provided with similar keys. Although these keys are ordinary ones used for controlling the TV receiver 40, only the keys used for controlling the photo application will be described here. The keys used for controlling the photo application include four arrow keys 80, a determination key 82 arranged at the center of these arrow keys 80, an end key 84, and a return key 86. Of these keys, the arrow key 80 is the cursor key 66 shown in FIG. 2, the enter key 82 is the enter key 72 shown in FIG. 2, the end key 84 is the function key 70 shown in FIG. 2, and the return key 86 is the FIG. Correspond to the function keys 68 shown in FIG.

<Screen transition>
Before explaining the configuration of the TV receiver 40 and the mobile phone 44 to realize the data linkage system 30, an operation by a user when displaying a photograph held in the mobile phone 44 on the TV receiver 40; The display transition of the TV receiver 40 and the mobile phone 44 associated therewith will be described with reference to FIGS. In these figures, (A) shows the display screen 46 of the TV receiver 40, and (B) shows the liquid crystal screen 54 of the mobile phone 44.

  Referring to FIG. 4, when the photo application is activated, a television image is displayed on display screen 46, and a so-called standby screen of mobile phone 44 is displayed on liquid crystal screen 54. Here, when the user presses the enter key 72 of the mobile phone 44, the screen changes to the state of FIG.

  Referring to FIG. 5, in cellular phone 44 according to the present embodiment, when determination key 72 is pressed on the standby screen, function selection is displayed on liquid crystal screen 54 of cellular phone 44 as shown in FIG. A screen is displayed. On the function selection screen, various functions that can be executed by the mobile phone 44 are displayed in a list format. When the user moves the focus 100 on the list using the cursor key 66 and presses the enter key 72 when the desired function is focused, execution of the function is started. Here, as shown in FIG. 5B, it is assumed that “photo application” is selected and the enter key 72 is pressed. The “photo application” is an application (photo display application) program that performs processing for displaying a photo held in the mobile phone 44 on the display screen 46 of the TV receiver 40 as described above.

  At this point, the TV image is still displayed on the display screen 46 of the TV receiver 40. When the enter key 72 is pressed while the “photo application” is selected, the display screen 46 and the liquid crystal screen 54 transition to the state shown in FIG.

  Referring to FIG. 6B, a list of photo thumbnails held on the mobile phone 44 is displayed on the liquid crystal screen 54, and the top thumbnail is in focus. At this time, as shown in FIG. 6A, the liquid crystal screen 54 of the mobile phone 44 is displayed as it is in the center of the display screen 46 of the TV receiver 40. Generally, the display screen 46 of the TV receiver 40 has a higher resolution than the liquid crystal screen 54 of the mobile phone 44. In the present embodiment, the liquid crystal screen 54 is displayed as it is in the center of the display screen 46, and the other areas are displayed in a predetermined color (for example, black).

  Here, it is assumed that the user moves the focus using the cursor key 66 and presses the enter key 82 when the desired photograph is focused. At this time, since the liquid crystal screen 54 of the mobile phone 44 is displayed as it is on the display screen 46 of the TV receiver 40, the movement of the focus is reflected on the display screen 46 as it is. Here, as described above, it is also possible to move the focus and select a photo using the remote controller 42.

  When the user presses the enter key 72 of the mobile phone 44 when the desired photo thumbnail is in focus, the screen changes as follows.

  With reference to FIG. 7B, an animation in which the selected photograph 122 slides upward is displayed on the liquid crystal screen 54 as indicated by an arrow. In this animation, the photograph 122 slides out from the upper end of the liquid crystal screen 54. 7A, at this time, on the display screen 46, the same photo 120 as the photo 122 is displayed from the lower end of the display screen 46 in synchronization with the slide of the photo 122 from the upper end of the liquid crystal screen 54. An animation that gradually moves toward the center of 46 is displayed. With the above animation, it is possible to give the user a feeling as if the photograph 122 stored in the mobile phone 44 is transferred as the photograph 120 into the TV receiver 40.

  The animation display of the photograph 120 and the photograph 122 ends when the photograph 120 reaches the center of the display screen 46. The state is shown in FIG. Referring to FIG. 8A, at this point, the photograph is displayed as a photograph 130 over the entire display screen 46 of the TV receiver 40. As shown in FIG. 8B, on the liquid crystal screen 54 of the mobile phone 44, the same photo 132 as the photo 130 is displayed at the center of the liquid crystal screen 54.

<Hardware configuration>
The configurations of the TV receiver 40 and the mobile phone 44 constituting the data linkage system 30 will be described below.

  Referring to FIG. 9, mobile phone 44 includes a CPU (Central Processing Unit) 150 that performs various processes by executing various programs, and a microphone that is connected to CPU 150 and performs voice input during a voice call. IEEE802.11g connected to the TV receiver 40 and the audio input unit 152, which is connected to the CPU 150, and the audio output unit 154, which is also connected to the CPU 150, and is connected to the CPU 150 and the CPU 150. Operation connected to the CPU 150, including a communication unit 164 for performing wireless communication in conformity with the function, and the function keys 62, 64, 68 and 70, the cursor key 66, the enter key 72, the numeric keypad 60, and the like shown in FIG. Unit 170 and CPU 150, which will be described later. A storage unit 168 for storing various data required for the program and program control is connected to the CPU 150, and a temporary memory 166 to be used when CPU 150 executes a program. When the CPU 150 executes a program, it is normal to display an image on the liquid crystal screen 54. The user can operate the mobile phone 44 by interactive operation using the liquid crystal screen 54 and various keys.

  The mobile phone 44 further includes an antenna 156 for transmitting and receiving radio waves to and from a base station of the telephone telephone network, a mobile phone program connected to the CPU 150 and executed by the CPU 150, and wireless communication using the antenna 156. A telephone network communication unit 158 for realizing a mobile phone function using the voice input unit 152 and the voice output unit 154, a mobile line unit 160 connected to the CPU 150 for controlling the line connection of the mobile phone, A display unit 162 having a liquid crystal screen 54 connected to the CPU 150 for displaying various types of information.

  The structure of the program for realizing the above-described movement of the photo application in the mobile phone 44 will be described later.

  On the other hand, referring to FIG. 10, TV receiver 40 is connected to CPU 200 and CPU 200, connected to CPU 200 and communication unit 208 for performing wireless communication with communication unit 164 shown in FIG. 9. An infrared light receiving unit 202 for receiving an infrared signal from the remote control 42, converting it into an electrical signal and giving it to the CPU 200, and a program executed by the CPU 200 and various data necessary for executing the program are connected to the CPU 200. A storage unit 216 for storing and a temporary storage 212 connected to the CPU 200 and temporarily storing data when the CPU 200 executes various programs are included.

  The TV receiver 40 is further connected to an antenna 204 for receiving a broadcast signal, and the antenna 204 and the CPU 200. Based on control from the CPU 200, the TV receiver 40 receives a signal having a predetermined frequency from the broadcast signal received by the antenna 204. A tuner 206 for selecting, demodulating, and providing to the CPU 200 as a TV video signal, a display unit 210 having a display screen 46 connected to the CPU 200 and displaying an image signal supplied from the CPU 200, and connected to the CPU 200. And an audio output unit 214 such as a speaker for converting the audio signal output by the CPU 200 into audio and outputting the audio.

<Program structure>
The program for realizing the data linkage system 30 is composed of three programs in the present embodiment. Referring to FIG. 11, these programs are executed by the CPU 150 of the mobile phone 44 and the photo application 230 that displays the photos stored in the mobile phone 44 on the liquid crystal screen 54 and the CPU 150 of the mobile phone 44. The photo selected by the photo application 230 is displayed on the display screen 46 using the function of the TV receiver 40, or the screen generated by the photo application 230 is transmitted to the TV receiver 40 and displayed on the display screen 46. The TV receiver 40 is stored in the mobile phone 44 for receiving instructions from the TV receiver 40 and performing processing corresponding to the instructions. Screen transmission program 232 for executing a service for displaying the displayed content, and TV receiver 40 Based on instructions and data received by the screen transmission program 232 and executed by the CPU 200, the photograph data stored in the mobile phone 44 is received and displayed on the display screen 46 using the function of the TV receiver 40. A screen for displaying the screen for the TV receiver 40 transmitted from the transmission program 232 on the display screen 46 or for transmitting an instruction given from the user by the remote control 42 to the screen transmission program 232 Client program 240.

  The screen transmission program 232 is remotely connected to a PC (Personal Computer) such as a terminal service server and a remote desktop function installed in Windows (registered trademark), a VNC (Virtual Network Computing) function, and the like. It can be realized by using the technology for doing this. In the following description, it will be described how this embodiment is realized using such a technique.

<Photo application 230 of mobile phone 44>
Referring to FIG. 12, the photo application 230 executed on the mobile phone 44 has the following control structure. Prior to the start of execution of this program, procedures such as setting of wireless communication connection by IEEE802.11g are completed between the mobile phone 44 and the TV receiver 40, and the mobile phone 44 transmits a screen. It is assumed that the program 232 and the client program 240 in the TV receiver 40 are activated. The program shown in FIG. 12 starts executing when the photograph application is focused on the screen shown in FIG. 5B and the enter key 72 is pressed.

  Referring to FIG. 12, this program displays a list of photos (thumbnail list) stored in mobile phone 44 on liquid crystal screen 54, and following step 260, transmits a screen sharing start instruction to the screen. Step 262 to be transmitted to the program 232 and whether or not the display device information (resolution, color information, information on decodable data format, etc.) of the TV receiver 40 is received from the screen transmission program 232 in response to the screen sharing start instruction. Based on this, a step of performing a process of switching the operation mode between the stand-alone mode in which the mobile phone 44 operates alone and the cooperative operation mode in which a photograph is displayed on the display screen 46 of the TV receiver 40 in cooperation with the TV receiver 40. H.264. Since the TV receiver 40 transmits display device information of the TV receiver 40 in response to the screen sharing start instruction, the screen sharing start instruction also has a function of inquiring display device information with respect to surrounding devices.

  When there are no clients such as the TV receiver 40 around, the screen transmission program 232 cannot communicate with these clients. Since the display device information is information related to the display performance of the client, this information will not return from the screen transmission program 232 to the photo application 230 if there is no client. In such a case, the mobile phone 44 operates in a stand-alone mode. When a display device (for example, the TV receiver 40) serving as a client exists in the surroundings, information regarding the display performance is given to the photo application 230 via the screen transmission program 232 as display device information. In step 264, based on such information, it is determined whether to operate the mobile phone 44 in a stand-alone manner or in a cooperative operation mode.

  In step 264, the program further secures a virtual screen memory in the temporary storage 166 shown in FIG. 9 for use in causing the TV receiver 40 to display content that cannot be processed by the TV receiver 40, and step 266. Step 268 of issuing an instruction to the screen transmission program 232 to set the operation mode of the TV receiver 40 to the screen sharing mode. This instruction is given to the client program 240 via the screen transmission program 232 as described later, and the client program 240 enters an operation mode in which the display screen 46 is shared with the mobile phone 44. As will be described later, a virtual screen to be displayed on the TV receiver 40 is drawn in the virtual screen memory and transmitted to the TV receiver 40. The virtual screen memory is secured in such a size that a screen with the maximum resolution of the display capability of the TV receiver 40 obtained according to the display device information can be drawn.

  In step 268, the program further determines whether or not the user has selected a photo to be displayed (step 270). If selected, the program performs a process for displaying the photo on the liquid crystal screen 54. 272 and after completion of execution of step 272 and when it is determined in step 270 that a photograph has not been selected, whether or not a return operation has been executed by the user (function of the mobile phone 44 shown in FIG. 2) Step 274 for branching the flow of control according to the determination result, and execution when it is determined in step 274 that the return operation has been performed, whether or not the key 68 or the return key 86 shown in FIG. Step 278 which performs processing for transitioning from the currently displayed photo screen to the photo list display screen and returns control to step 268; Including.

  This program is further executed when the determination result in step 274 is “NO”, whether or not the user has performed an end operation (whether the function key 70 shown in FIG. 2 or the end key 84 shown in FIG. 3 has been pressed). Whether or not) and branching the flow of control according to the determination result, and in response to the determination of YES in step 276, the virtual screen memory secured in step 268 is released. The process of transmitting the instruction to end the screen sharing to the screen transmission program 232, and the screen of the function list of the mobile phone 44 (the screen shown in FIG. 5B) is changed to the display of the liquid crystal screen 54. And step 280 for terminating the execution of the program. If the determination result in step 270 is NO (no photo is determined), control skips step 272 and proceeds to step 274. If the decision result in the step 276 is NO (no return operation and no end operation), the control returns to the step 270. Therefore, unless a return operation or an end operation is performed, the processing of step 270 and step 272 is repeatedly executed.

<Screen transmission program 232 of mobile phone 44>
Referring to FIG. 13, screen transmission program 232 has the following control structure. This program is activated, for example, when the mobile phone 44 is turned on, and then operates resident in the memory.

  This program is executed when waiting for receiving a screen sharing start instruction from another program (for example, the photo application 230), and when it is determined in step 300 that the screen sharing start instruction is received, and screen transmission is started. Step 302 for sending a notification to a nearby display device, and following step 302, when a screen transmission connection request 328 is received from a nearby display device, the display device information of that display device is sent to the photo application 230. Otherwise, do nothing 304. When the display device does not exist nearby, the following screen sharing process is not executed, and the control returns to step 300 although not shown in FIG. The screen transmission connection request transmitted from the display device includes the display device information described above. Here, it is assumed that the TV receiver 40 exists in the vicinity, and hereinafter, communication between the mobile phone 44 and the TV receiver 40 will be described as an example.

  FIG. 17A shows a data format of a screen transmission connection request 328 that may be transmitted from the client program 240 after step 302. Referring to FIG. 17A, the screen transmission connection request 328 includes information regarding the resolution, color information, and decodable data of the display device on which the client program 240 is operating, in addition to a normal communication header and the like. . For example, a value such as 1920 pixels × 1080 pixels is transmitted as the resolution. As the color information, information about color gradation expression capability such as “24-bit color” is transmitted. As information about the data that can be decoded, the client program 240 is operating and displaying the display device such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), MPEG (Moving Picture Experts Group) -2, MP4, etc. There is a file extension to identify possible formats.

  Referring again to FIG. 13, this program is further executed following step 304 when a screen transmission connection request is received from the client program 240 of the TV receiver 40, and is connected to the TV receiver 40. Step 305 for performing a connection process for establishing the connection, and subsequent to Step 305, it is determined whether or not the remote control key 330 has been transmitted from the TV receiver 40 according to the user's instruction. Step 306 for branching the flow and executed when it is determined in step 306 that the transmission of the remote control key 330 has been received, and the remote control key is converted to generate a key input signal of the mobile phone 44 corresponding to the remote control key. And the step 308 of passing to the system of the mobile phone 44.

  The connection process performed in step 305 is the same as the first connection process in remote login. In this connection process, as part of the negotiation for communication, details regarding the resolution of the screen transmitted between the server and the client using the display devices are determined. In the present embodiment, in the connection process of step 305, information for establishing a connection is exchanged, and the screen transmission program is included in the screen transmission connection request 328 at the maximum resolution of the display device on which the client program 240 is operating. 232 and the client program 240 are connected. Assuming that the display screen 46 of the TV receiver 40 has a higher resolution than the liquid crystal screen 54 of the mobile phone 44, the mobile phone can be connected as shown in FIG. The entire 44 liquid crystal screens 54 are displayed on the display screen 46, and an area where no video is displayed remains around the liquid crystal screen 54. However, by making such a connection, when creating the virtual screen described above, the screen can be created with the maximum resolution of the display screen 46 and transmitted to the TV receiver 40 for display. A display that fully utilizes the display function of the receiver 40 can be realized.

  FIG. 17B shows the format of remote control key 330 that may be received prior to step 306. Here, the communication header and the like are omitted. Referring to FIG. 17B, remote control key 330 has “signal type” indicating the type related to the operated key, and “action” indicating what operation the user has performed on the key. including. For example, when the user presses the enter key 82 (see FIG. 3) of the remote controller 42, the remote control key 330 in which the value “determine” is stored as the signal type, the value “down” is stored as the action is stored in the client program. 240 is transmitted to the screen transmission program 232. When the key is pressed once and then released, the action value becomes “up”.

  The key input signal passed to the photo application 230 in step 308 of FIG. 13 is generated in response to the operation of the operator such as the function key 68, the function key 70, the cursor key 66, and the enter key 72 of the first housing 52. It is treated as equivalent to a key input signal indicating an operation instruction, and is used in determinations such as steps 274 and 276 shown in FIG. That is, the key input signal converted from the remote control key 330 can be considered as a virtual input to the mobile phone 44. In this embodiment, since the key input signal is passed to the system of the mobile phone 44, when a key other than the key used in the photo application is operated, the key is effectively handled and what is the photo application? Another predetermined process may be executed by the CPU 150 or the like.

  The program further determines whether or not a screen sharing mode setting instruction has been received from the photo application 230 after the processing of steps 306 and 308, and branches the flow of control according to the determination result; And a step 312 of setting the screen sharing mode in the data linkage system 30 to “shared screen mode” when it is determined that the sharing mode setting instruction has been received.

  Here, there are two types of connection modes (this is referred to as “screen sharing mode”) for displaying screens in the data linkage system 30. One is the shared screen mode already described, and the other is the virtual screen mode. In the shared screen mode, the display on the liquid crystal screen 54 of the mobile phone 44 is displayed as it is on the display screen 46 of the TV receiver 40. In the virtual screen mode, the screen for the liquid crystal screen 54 is displayed inside the mobile phone 44. A screen for the display screen 46 of the TV receiver 40 is generated separately, and the display screen 46 of the TV receiver 40 is passed to the TV receiver 40 and displayed on the display screen 46. As will be described later. These modes are automatically switched.

  This program is further executed following the processing of step 310 and step 312, and the mobile phone 44 is displayed on the display screen 46 of the TV receiver 40 in conjunction with the corresponding processing of step 272 and the client program 240 shown in FIG. Step 314 for executing a photo display process for displaying a photo stored in the screen, and a screen to be displayed on the display screen 46 drawn by the photo application 230 following step 314 (in the shared screen mode, the screen of the liquid crystal screen 54 is displayed). Step 316 for branching the flow of control according to the determination result, and detecting the updated part when the screen is updated. Step 318 and determine whether a screen request has been received from the client program 240 And step 320, when it is determined that it has received the screen request, and a step 322 of transmitting the differential screen of before and after the update to the client program 240. When it is determined in step 316 that there is no screen update, when it is determined in step 320 that there is no screen request, and when the processing in step 322 is completed, the control proceeds to step 324.

  In step 324, it is determined whether or not an end instruction (transmitted in step 280 in FIG. 11) is received from the photo application 230. If it is determined that it has been received, a screen transmission end notification is sent in step 326 to the client program 240. , The communication with the client program 240 is disconnected, and the control returns to step 300. If an end instruction has not been received, control returns to step 306.

<Client program 240 of TV receiver 40>
When the TV receiver 40 is turned on, the client program 240 of the TV receiver 40 starts up, stays resident in the memory, and waits for a screen transmission start notification from the screen transmission program 232 (step 302 in FIG. 13). . Referring to FIG. 14, client program 240 waits until it receives a screen transmission start notification from screen transmission program 232 of a nearby content server (mobile phone 44 in the present embodiment), and a screen transmission start notification. Display device information (maximum resolution, color information, decodable data information, etc.) of the display device (TV receiver 40 in this embodiment) that is executed when the client program 240 is operating is received. Steps 342 for transmitting the display device information to the screen transmission program 232, Steps 344 for performing connection processing with the screen transmission program 232 for transmitting the screen transmission start notification following Step 342, and Step 344 Subsequently, it is determined whether or not there has been a remote control key input from the remote control 42 (STEP 346), the remote control key 330 is transmitted to the screen transmission program 232 if there is, and is executed following the steps 346 and 348. Step 350 for performing a photo display process for displaying the photograph stored in the telephone 44 on the display screen 46 of the TV receiver 40, and a screen for the display screen 46 of the TV receiver 40 are displayed following step 350. And step 352 for performing a screen request transmission to the transmission program 232 to the screen transmission program 232.

  Further, following this step 352, this program determines whether or not screen data to be displayed on the display screen 46 has been received from the screen transmission program 232 (step 354), and when received, the screen data is used to display the display screen 46. Step 356 for updating the display of, and subsequent to Steps 354 and 356, it is determined whether or not an end notification has been received from the screen transmission program 232 (Step 358), and when received, communication with the screen transmission program 232 is performed. , And after changing the state of the TV receiver 40 to a state in which a TV image is being displayed, the process returns to step 360. If it is determined in step 358 that an end notification has not been received, control returns to step 346 and the processing after step 346 is repeated.

<Standalone / cooperation operation switching process on mobile phone 44>
FIG. 15 shows a more detailed flowchart of the program for realizing step 264 of FIG. Referring to FIG. 15, this program includes step 380 that determines whether or not display device information has been received from screen transmission program 232, and if received, does nothing and ends the process. Here, if display device information is received within a predetermined time (for example, within 3 seconds), it is determined that there is a display device in which the client program 240 is operating nearby, and if not, there is no such device nearby. judge.

  When the display device information is received, there is a display device in which the client program 240 is operating nearby. Here, nothing is done and the process returns to the process of step 266 shown in FIG.

  When it is determined in step 380 that the display device information is not received, there is no display device in which the client program 240 is operating nearby. In this case, the mobile phone 44 has to display a photograph stand alone. Therefore, it is determined in step 382 whether or not a photograph to be displayed has been determined by the user's operation. The program further draws the photo on the liquid crystal screen 54 of the mobile phone 44 when it is determined at step 382 that the photo has been determined, and when it is determined at step 382 that the photo has not been determined. Or after completion of step 384, when it is determined in step 386 that the function key 68 has been operated in step 386 for branching the flow of control according to whether the return key (function key 68) has been operated, Step 388 for displaying the photo list screen stored in the telephone 44 (FIG. 6B), and when it is determined in Step 386 that the function key 68 has not been operated, and after completion of Step 388 It is determined whether or not the key (function key 70) is operated, and when it is operated, the process is terminated in step 390. Including. If it is determined in step 390 that the end key has not been operated, control returns to step 382.

<Photo display processing>
The photo display processing performed in step 272 in FIG. 12, step 314 in FIG. 13 and step 350 in FIG. 14 displays the photos stored in the mobile phone 44 in cooperation with each other on the display screen 46 of the TV receiver 40. It is a process to do. When the TV receiver 40 has a function for decoding the data of the selected photograph, the photograph is displayed on the display screen 46 of the TV receiver 40 by this processing. Otherwise, the photo data is drawn on the virtual screen in the mobile phone 44 by this processing. The virtual screen is transmitted to the TV receiver 40 at a later stage (step 322 in FIG. 13). Hereinafter, the control structure of the program for realizing the photo display processing will be described with reference to FIG. FIG. 16 shows details of step 272 in FIG. 12, step 314 in FIG. 13, and step 350 in FIG. 14 from the left.

  Referring to FIG. 16, the program executed in step 272 of photo application 230 includes step 410 of transmitting a virtual screen mode setting instruction to screen transmission program 232. The virtual screen mode setting instruction is to generate a display screen for the TV receiver 40 with the mobile phone 44 for content that cannot be decoded by the TV receiver 40, and transmit the display screen to the TV receiver 40 to transmit the content of the TV receiver 40. This is an instruction to set the operation mode of the screen transmission program 232 and the client program 240 so as to be displayed on the display screen 46.

  The program further includes a step 412 for starting the drawing of a photo animation as shown in FIG. This animation display is executed over a predetermined time (for example, 2 frames per second for 3 seconds).

  This program is further executed following step 412 to determine whether or not the picture data to be rendered can be decoded by the TV receiver 40 (that is, whether or not the picture data can be transmitted to the TV receiver 40). Step 414, which is executed when it is determined that data transmission is possible in Step 414, and the file path 422 to the file of the selected photo data is transmitted to the screen transmission program 232, and in Step 414 And step 416 for decoding the selected photo and rendering it on the virtual screen when it is determined that data transmission is impossible.

  An example of the data format of the file path 422 transmitted in step 418 is shown in FIG. Here, the communication header and the like are omitted. Referring to FIG. 17C, a file path 422 is text data in this embodiment, and indicates a path from the root to a file to be transmitted in a directory tree of data storage of the mobile phone 44.

  Note that the photograph drawn on the virtual screen in step 416 is not transmitted to the client program 240 in the process of FIG. The drawing data of the virtual screen is transmitted from the screen transmission program 232 to the client program 240 by processing at another location (steps 316 to 322 in FIG. 13) and displayed in step 356 in FIG.

  Referring to FIG. 16 again, this program is executed after the end of step 418 or step 416, and draws the selected photo in the maximum size that can be drawn on the liquid crystal screen 54 regardless of whether data transmission is possible or not. Step 420 to end the process.

  Referring to the flowchart in the center of FIG. 16, step 314 of the photo display process executed by the screen transmission program 232 in synchronization with the execution of the program shown in the left flowchart of FIG. It is determined whether or not a setting instruction has been received, and when it is determined that a virtual screen mode setting instruction has not been received, it is determined in step 440 that the process is immediately terminated, and in step 440 it is determined that a virtual screen mode setting instruction has been received. And step 442 for setting the screen transmission program 232 from the shared screen mode to the virtual screen mode. As described above, the shared screen mode is an operation mode in which the display contents of the display screen 46 of the TV receiver 40 and the liquid crystal screen 54 of the mobile phone 44 are made common as shown in FIG. The virtual screen mode is an operation mode in which the content of the display screen 46 is created separately from the content displayed on the liquid crystal screen 54 on the mobile phone 44 side, and transmitted to the TV receiver 40 for display.

  Referring to FIG. 16, the process executed in step 314 of screen transmission program 232 is further executed following step 442 to determine whether a file path has been received from photo application 230 and has not been received. When it is determined that the file path has been received in step 444, the process is terminated. The operation mode of the screen transmission program 232 is set to the file transmission mode, and the given file path is set. Step 448 which transmits the file data (substance of photograph data) 450 accessed by the above to the client program 240 and ends the processing. The file transmission mode is an operation mode in which content data (here, photo data) is directly supplied to the client program 240 and is decoded and displayed on a display device. In the file transmission mode, the screen generated by the mobile phone 44 is not transmitted. As a result, the display of the TV receiver 40 is automatically switched from the same display as the display of the liquid crystal screen 54 of the mobile phone 44 to the display of the decoding result.

  An example of the format of the file data 450 transmitted to the TV receiver 40 in step 448 is shown in FIG. Here, the communication header and the like are omitted. Referring to FIG. 17D, the file data 450 includes a file name excluding directory information in the file path, and a point from which playback is to be started (playback point) in the case of a moving image or the like from the beginning. It includes a “reproduction point” indicated by the number of seconds and binary data that is the substance of the file. In this embodiment, the file name has an extension less than a period. On the client side, the file data format is determined based on this extension, and appropriate decoding is performed.

  Referring to the flowchart on the right side of FIG. 16, has the file for the photo display processing executed in step 350 of the client program 240 (FIG. 14) of the TV receiver 40 received file data from the screen transmission program 232? If it is determined that the file data has been received in step 460, the received file data is decoded and displayed on the display screen 46. And 462 to end the processing. In step 462, if a start point is designated in the received file data, reproduction is performed from that point. Even if the file data is being received and displayed, if it is determined in step 354 in FIG. 14 that the screen data has been received, the display of the file data is stopped and the received screen is displayed.

<Operation>
[When starting]
The data linkage system 30 described above for the configuration operates as follows. Referring to FIG. 1, it is assumed that the TV receiver 40 is powered on in advance and the client program 240 shown in FIG. 11 is activated. In this case, the client program 240 stands by until a screen transmission start notification is received in step 340 shown in FIG. On the other hand, it is assumed that the mobile phone 44 is also powered on and the screen transmission program 232 shown in FIGS. 11 and 13 is activated. The screen transmission program 232 waits until a screen sharing start instruction is received in step 300 of FIG. Hereinafter, both the photo application 230 and the screen transmission program 232 are executed by the CPU 150 (see FIG. 9). However, in order to distinguish the two as processes, the CPU 150 when executing the process of the photo application 230 is simply “photograph”. The application 150 ”is called, and the CPU 150 when the process of the screen transmission program 232 is being executed is simply called the“ screen transmission program 232 ”. The same applies to the client program 240.

  A user who has a mobile phone 44 displays a function list of the mobile phone 44 as shown in FIG. 5B, places the focus 100 on the “photo application”, and presses the enter key 72 (see FIG. 2). And Referring to FIG. 12, the photo application 230 is activated, and a photo list is displayed as shown in FIG. 6B (FIG. 12, step 260). Simultaneously with displaying the photo list, the photo application 230 transmits a screen sharing start instruction to the photo application 230 (step 262).

  Referring to FIG. 13, when receiving a screen sharing start instruction in step 300, photo application 230 sends a screen transmission start notification to client program 240 in step 302 through wireless communication using communication unit 164 (see FIG. 9). Send. Referring to FIG. 14, when client program 240 receives this screen transmission start notification via communication unit 208 shown in FIG. 10, control proceeds from step 340 to step 342. In step 342, the client program 240 further transmits a screen transmission connection request 328 (see FIG. 17A) to the screen transmission program 232 by wireless communication. Upon receiving this screen transmission connection request 328, the screen transmission program 232 extracts the display device information included in the screen transmission connection request 328 in step 304 of FIG. 13, and transmits it to the photo application 230 as display device information. Thereafter, the screen transmission program 232 connects the client program 240 and the display screen 46 of the TV receiver 40 at the maximum resolution (step 305 in FIG. 13 and step 344 in FIG. 14). Thereafter, the screen transmission program 232 proceeds to execute the processing in step 306 and the subsequent steps shown in FIG. The client program 240 moves to the execution of the processing after step 346 shown in FIG.

  On the other hand, the photo application 230 that has received the display device information from the screen transmission program 232 performs a stand-alone / cooperation operation switching process (FIG. 15) in step 264 of FIG. In this example, since the display device information is received from the screen transmission program 232, the determination result in step 380 of FIG. 15 is YES, nothing is performed, and the control returns to step 266 of FIG.

  In step 266, the photo application 230 reserves a virtual screen memory area necessary for drawing on the entire screen of the display screen 46 in the temporary storage 166 (see FIG. 9). The photo application 230 further transmits a screen sharing mode setting instruction to the screen transmission program 232 in step 268. The screen transmission program 232 repeatedly executes the processing from step 306 to step 324 in FIG. If a screen sharing mode setting instruction is transmitted from the photo application 230, YES is determined in step 310, and the screen transmission program 232 shares the TV receiver 40 and the mobile phone 44, that is, the screen transmission program 232 and the client program 240. Set to screen mode.

[When there is no operation in the photo list display]
If the user does not perform any operation in the state shown in FIG. 6B, the determination result in step 270 of FIG. 12 is NO, and the photo display processing (step 272 of FIG. 12) in the photo application 230 is not performed. . Therefore, nothing is performed in the photo display process of the screen transmission program 232 (step 312 in FIG. 13) and the photo display process of the client program 240 (step 350 in FIG. 14), and the TV image is also received on the liquid crystal screen 54 of the mobile phone 44. The display on the display screen 46 of the machine 40 does not change.

[When an operation to select a photo is performed]
Assume that the user focuses on the photograph in the state shown in FIG. 6B and presses the enter key 72 (see FIG. 2). Then, the determination result in step 270 of FIG. 12 is YES, and the photo display process (step 272) is performed by the photo application 230.

  Referring to FIG. 16, in the photo display process of photo application 230, a virtual screen mode setting instruction is transmitted to screen transmission program 232 (step 410). When the screen transmission program 232 receives this virtual screen mode setting instruction, the determination result in step 440 is YES, and the screen transmission program 232 is set to the virtual screen mode in step 442.

  In the meantime, the photo application 230 starts drawing an animation (FIG. 7), and whether or not the data of the designated photo can be transmitted to the TV receiver 40, that is, whether or not the TV receiver 40 can decode the photo data. It is determined whether or not (FIG. 16: step 414). This determination is based on information (see FIG. 17A) regarding data that can be decoded by the TV receiver 40 included in the display device information received immediately before step 264 in FIG. 12, and the file name of the selected photo. This is done by comparing the extension. If the TV receiver 40 cannot decode the photo data, the process of step 416 is executed. That is, the selected photograph is drawn on the virtual screen secured in the temporary storage 166 of the mobile phone 44. However, in this process, the virtual screen is not transmitted to the screen transmission program 232 and the client program 240.

  On the other hand, if the TV receiver 40 can decode the photo data, the process of step 418 is executed. That is, the file path 422 to the photo data file is transmitted to the screen transmission program 232. The subsequent operations of the screen transmission program 232 and the client program 240 will be described later.

  When these processes are completed, the screen transmission program 232 draws the photograph on the liquid crystal screen 54 of the mobile phone 44 (FIG. 16, step 420), and returns from this process to step 274 in FIG.

  Thereafter, the processes of Step 270 and Step 272 shown in FIG. 12 are repeated until the user performs a return operation or an end operation. The operations of the mobile phone 44 and the TV receiver 40 when a return operation or an end instruction is given will be described later.

  The operations of the photo application 230, the screen transmission program 232, and the client program 240 after the shared screen mode is set will be described below. In this case, there are a case where the selected photograph can be decoded by the TV receiver 40 and a case where it cannot be decoded. This will be described in order below.

-When a photograph can be decoded by the TV receiver 40 Referring to FIG. 12, in the photograph application 230, when step 272 is executed, a virtual screen mode setting instruction is transmitted to the screen transmission program 232 (FIG. 16: step 410). The screen transmission program 232 receives this instruction (FIG. 16: step 440). The determination result in step 440 of FIG. 16 is YES, and in step 442, the screen transmission program 232 is set to the virtual screen mode. The determination result in step 414 is YES, and the file path 422 is transmitted to the screen transmission program 232 in step 418. The determination in step 444 of the screen transmission program 232 is YES, and the operation mode of the mobile phone 44 is automatically changed from the virtual screen mode to the data transmission mode (step 446). Thereafter, the corresponding photo data is read from the storage unit 168 in the mobile phone 44 according to the file path 422, and a message of the file data 450 shown in FIG. 17D is assembled and transmitted to the client program 240 (step 448). The photo display process of the screen transmission program 232 ends.

  When the client program 240 of the TV receiver 40 receives this file data 450, the determination result in step 460 of FIG. 16 is YES. As a result, in step 462, the received photo data is decoded and displayed on the display screen 46. Here, it is known in advance from the display device information that the TV receiver 40 has a function of decoding this type of photographic data. When the decoded photo data is displayed, the photo display processing of the client program 240 ends. In this manner, when the TV receiver 40 has a function of decoding the data of the photograph selected by the user, the photograph data is transmitted to the TV receiver 40 and decoded by the TV receiver 40. Are displayed on the display screen 46.

-When the TV receiver 40 cannot decode the photo When the TV receiver 40 cannot decode the selected photo data, the photo is decoded by the photo display processing of the mobile phone 44 (photo application 230) and rendered on the virtual screen. Is done. However, this screen is not transmitted to the TV receiver 40 in the photo display process. Hereinafter, the process in that case will be described.

  If the TV receiver 40 cannot decode the photograph data, step 418 in FIG. 16 is not executed, and step 416 is executed. When a photograph is drawn on the virtual screen in step 416, the virtual screen changes. In that case, the determination result in step 316 of FIG. 13 is YES. Therefore, in step 318, the updated portion of the virtual screen is detected. Thereafter, the screen transmission program 232 waits for a screen update request from the client program 240 (YES in step 320), and transmits the difference image to the client program 240 (step 322).

  On the other hand, the client program 240 shown in FIG. 14 transmits a screen request transmission to the screen transmission program 232 after the photo display process (step 352 in FIG. 14). If screen data (image data of the difference screen transmitted in step 322 in FIG. 13) is transmitted from the screen transmission program 232 in response to this screen request, the determination is made in step 354 and the difference screen is followed in step 356. The display image on the display screen 46 is updated (step 356). As a result, even if an image decoded by the TV receiver 40 is displayed on the display screen 46, the screen is automatically replaced with a virtual screen. Thereafter, the client program 240 repeats the processing of step 346 to step 356 until it receives a screen sharing end notification from the screen transmission program 232.

[When back operation is performed]
If the user performs a return operation (YES in step 274 in FIG. 12), the screen transitions to the photo list display screen (FIG. 6B) (step 278), and the control returns to step 268 to return to the screen sharing mode. Is transmitted to the screen transmission program 232. Thereafter, the processing of step 270 and step 272 is repeated until an operation for returning or ending is performed.

[When exit operation is performed]
When the user operates the function key 70 (end key, see FIG. 2), the photo application 230, the screen transmission program 232, and the client program 240 operate as follows. Referring to FIG. 12, the determination result of step 276 of photo application 230 is YES. In step 280, the virtual screen memory secured in the temporary storage 166 of the mobile phone 44 is released. Further, an end instruction is transmitted from the photo application 230 to the screen transmission program 232. The liquid crystal screen 54 of the mobile phone 44 returns to the state shown in FIG.

  The screen transmission program 232 that has received the termination instruction (YES in step 324 in FIG. 13) executes the processing in step 326. That is, a screen sharing end notification is transmitted to the client program 240, and communication with the client program 240 is disconnected. Thereafter, the control of the screen transmission program 232 returns to step 300 and waits until the next screen sharing start is received from the photo application 230.

  When the client program 240 receives a screen sharing end notification from the screen transmission program 232 (YES in step 358 in FIG. 14), the client program 240 disconnects communication with the screen transmission program 232 and displays the display screen 46 of the TV receiver 40. The used screen display is stopped (step 360). As a result, the TV image received by the tuner 206 is displayed on the display screen 46 of the TV receiver 40. Control returns to step 340 and the client program 240 waits until a screen sharing start notification is transmitted from the screen transmission program 232.

<Possible modification>
In the embodiment described above, the wireless communication between the mobile phone 44 and the TV receiver 40 is based on IEEE 802.11g. However, the present invention is not limited to such an embodiment. For example, Bluetooth (registered trademark) or Zigbee (registered trademark) may be used. In the above-described embodiment, the mobile phone 44 and the TV receiver 40 perform direct wireless communication. However, either or both of them are connected to a wired LAN, and are similar to the device of the above-described embodiment. Can be realized. However, in this case, at least a wireless LAN access point for connecting the mobile phone 44 to the wired LAN is used.

  In the above embodiment, the case where the photo application program and the screen transmission program operate as separate processes has been described. In this way, by making the screen transmission program resident, the cooperation processing using the virtual screen as described above for various application programs can be realized using the same screen transmission program. However, the present invention is not limited to such an embodiment. When resources such as a memory are limited as in a mobile phone, a screen operation program such as a photo application program and a screen transmission program can be realized by a single program. In this case, when only the application program that does not require the cooperation operation is operated on the mobile phone 44, it is not necessary to operate the screen transmission program, and the operation of the mobile phone 44 can be lightened.

  In the embodiment described above, a photograph that is a still image held in an information processing terminal such as the mobile phone 44 is displayed on the display screen 46 of the TV receiver 40 in cooperation with a display device such as the TV receiver 40. An example is shown. However, the present invention is not limited to such an embodiment. The present invention can be applied to any content that can be converted into a still image that can be displayed on the screen. For example, since a moving image can be expressed by a set of still images, the above technique can be applied almost as it is. Even if it is a data format that can be displayed by a specific application, such as HTML (HyperText Markup Language) data and presentation data, if it is converted into a still image in a predetermined unit such as a page, as in the above embodiment, The present invention can be applied.

In the embodiment described above, when the screen transmission program 232 and the client program 240 are connected, the resolution (maximum resolution) of the entire screen of the display screen 46 of the TV receiver 40 on which the client program 240 is operating is used. ing. However, the present invention is not limited to such an embodiment. You can connect at a resolution below the maximum resolution. For example, when the TV receiver 40 has a so-called Picture-in-Picture (PinP) function and displays a screen sent exclusively from a mobile phone on a child screen opened on a part of the screen, It is desirable to connect the screen transmission program 232 and the client program 240 with the resolution of the child screen.
Further, in the above embodiment, when the remote control key is transmitted from the TV receiver 40 to the screen transmission program 232, the screen transmission program 232 operates in the screen sharing mode or the virtual screen mode. The key is passed to the system of the mobile phone 44 regardless of whether the remote control key functions effectively for the entire mobile phone 44. However, the present invention is not limited to such an embodiment. For example, the handling of the remote control keys at this time may be changed depending on whether the mobile phone 44 is in the screen sharing mode or the virtual screen mode. For example, when the mobile phone 44 is in the screen sharing mode, if a remote control key is received, a valid key is issued to the entire mobile phone 44, but in the virtual screen mode, key information is transmitted to the photo application program. In this case, a key that is valid only for the photo application program may be issued. By performing such processing, an operation on the button of the mobile phone 44 can be handled as an operation on only the display screen of the mobile phone 44, and an operation on the remote control 42 can be handled as an operation on the virtual screen.

  In the above-described embodiment, content data is transmitted from the screen transmission program 232 to the client program 240 after displaying a picture transmission animation. However, the present invention is not limited to such an embodiment. For example, in order to display content immediately after the animation display is completed, it is possible to display the content without boring the user by changing the program so that the content is transmitted during the animation display. . In particular, for moving image data, streaming transmission can be performed from the server that holds the content (the mobile phone 44 in the above embodiment), but streaming transmission is started prior to starting the display of the moving image. If sufficient data is stored in the buffer on the client side (the TV receiver 40 in the above embodiment), the moving image can be reproduced smoothly.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

30 Data Linkage System 40 TV Receiver 42 Remote Control 44 Mobile Phone 46 Display Screen 54 Liquid Crystal Screen 66 Cursor Keys 62, 64, 68, 70 Function Keys 72 Enter Key 80 Arrow Key 82 Enter Key 100 Focus 150, 200 CPU
162, 210 Display unit 164, 208 Communication unit 166, 212 Temporary storage 168, 216 Storage unit 170 Operation unit 202 Infrared light receiving unit 206 Tuner 230 Photo application 232 Screen transmission program 240 Client program 328 Screen transmission connection request 330 Remote control key 422 File path 450 file data

Claims (17)

A display device;
Processing execution means for executing processing involving display of an image on the display device;
A communication device that communicates with the other device in order to share the image displayed on the display device with the other device by the processing execution unit;
Means for automatically switching the connection mode between the other device between the first connection mode and the second connection mode during connection with the other device via the communication device;
First image transmission means for operating in the first connection mode and transmitting an image displayed on the display device to the other device via the communication device;
A second image transmission unit that operates in the second connection mode and generates an image separately from the image displayed on the display device and transmits the image to the other device via the communication device; Information processing device. The communication device
In response to being instructed to connect to another device, an inquiry means for inquiring the display device information related to the display performance of the display device of the other device to the other device,
In response to receiving the display device information from another device in response to the inquiry by the inquiry means, the other device that has transmitted the display device information so that screen data can be communicated at a predetermined resolution. A connection establishment means for establishing a connection with the device;
Means for setting a connection mode with the other apparatus to the first connection mode and activating the first image transmission means in response to establishment of a connection by the connection establishment means When,
2. The information processing apparatus according to claim 1, further comprising: means for transmitting the image to the other apparatus in response to receiving an image from one of the first and second image transmission means. . The communication device
In response to being instructed to connect to another device, an inquiry means for inquiring the display device information related to the display performance of the display device of the other device to the other device,
In response to receiving the display device information from another device in response to the inquiry by the inquiry means, the other device that has transmitted the display device information so that screen data can be communicated at a predetermined resolution. A connection establishment means for establishing a connection with the device;
Means for setting a connection mode with the other apparatus to the second connection mode and activating the second image transmission means in response to establishment of a connection by the connection establishment means When,
2. The information processing apparatus according to claim 1, further comprising: means for transmitting the image to the other apparatus in response to receiving an image from one of the first and second image transmission means. . The display device information includes information on a maximum resolution that can be displayed on the display device,
The information processing apparatus according to claim 2, wherein the predetermined resolution is a resolution having a predetermined relationship with a maximum resolution of the display device included in the display device information.   The information processing apparatus according to claim 2, wherein the predetermined resolution is equal to a maximum resolution of the display device included in the display device information.   The second image transmission means operates in the second connection mode, and creates an image of the maximum resolution of the display device included in the display device information separately from the image displayed on the display device. The information processing apparatus according to claim 3, comprising means for transmitting to the other apparatus via the communication apparatus. The display device information includes information on a maximum resolution that can be displayed in a window in which an image transmitted from the information processing device is displayed by the display device,
The information processing apparatus according to claim 2, wherein the predetermined resolution is equal to a maximum resolution of the window.   The communication device further includes virtual input processing means for processing the predetermined signal as a virtual input to the information processing device in response to the communication device receiving the predetermined signal from the other device. The information processing apparatus according to any one of claims 1 to 7. The information processing apparatus further includes:
In response to a user operation, input means for generating an operation instruction signal for the information processing device;
Operation executing means for executing a predetermined process in response to the operation signal generated by the input means,
The virtual input means converts the virtual input obtained from the predetermined signal received via the communication device into an operation instruction signal that can be processed by the input means, and gives the operation instruction signal to the operation execution means The information processing apparatus according to claim 8, comprising means for   The information processing apparatus according to claim 8 or 9, wherein the virtual input processing unit functions in both the first operation mode and the second operation mode. In response to receiving the predetermined signal from the other device in the second operation mode, the virtual input processing unit sends the predetermined signal to the second image transmission unit in a virtual manner. Including means for processing as input,
The second image transmission means executes a process using the virtual input given by the means for processing, and an image different from the image displayed on the display device according to the execution result The information processing apparatus according to claim 8, further comprising: means for creating and transmitting to the other apparatus via the communication apparatus.   In order to stop the first and second image transmission units in response to the display device information not being received by the connection establishment unit from another device within a predetermined time from the transmission of the inquiry by the inquiry unit. The information processing apparatus according to claim 2, further comprising: The display device information further includes data format information for specifying a data format that can be displayed by the other device,
The information processing apparatus further includes:
In response to the fact that the data format of the content data for displaying the image on the display device by the processing execution means matches the data format that can be displayed on the other device, the first and second The information processing apparatus according to claim 2, further comprising means for stopping image transmission means and transmitting the content data to the other device via the communication device. An information processing system including first and second information processing apparatuses,
The first information processing apparatus includes:
A display device;
Processing execution means for executing processing involving display of an image on the display device;
A communication device that communicates with the second information processing device in order to share the image displayed on the display device with the second information processing device by the processing execution unit;
During connection with the second information processing apparatus via the communication apparatus, the connection mode between the second information processing apparatus is automatically set between the first connection mode and the second connection mode. Means for switching;
First image transmission means for operating in the first connection mode and transmitting an image displayed on the display device to the second information processing device via the communication device;
Second image transmitting means that operates in the second connection mode, creates an image separately from the image displayed on the display device, and transmits the image to the second information processing device via the communication device Including
The second information processing apparatus
A communication device for communicating with the communication device of the first information processing device;
A display device;
The communication device of the second information processing apparatus uses the image transmitted from the first or second image transmission unit of the first information processing apparatus via the communication device of the first information processing apparatus. Means for displaying the image on the display device of the second information processing apparatus in response to receiving the information. A display device;
Processing execution means for executing processing involving display of an image on the display device;
A communication device that communicates with the other device in order to share the image displayed on the display device with the other device by the processing execution unit;
Mode switching means for automatically switching the connection mode with the other device between the first connection mode and the second connection mode during connection with the other device via the communication device; ,
First image transmission means for operating in the first connection mode and transmitting an image displayed on the display device to the other device via the communication device;
A second image transmission unit that operates in the second connection mode and generates an image separately from the image displayed on the display device and transmits the image to the other device via the communication device; An information processing method in the information processing apparatus,
The process executing means executing a process involving display of an image on the display device;
The communication device establishing communication with the other device;
The mode switching means automatically switches the connection mode with the other device between the first connection mode and the second connection mode while being connected to the other device via the communication device. Steps,
In response to the connection mode being switched to the first connection mode, the first image transmission means transmits an image displayed on the display device to the other device via the communication device. Steps,
In response to the switching of the connection mode to the second connection mode, the second image transmission means creates an image separately from the image displayed on the display device, and passes through the communication device. And transmitting to the other apparatus.   A computer program for causing a computer to operate as the information processing apparatus according to any one of claims 1 to 13 when executed by a computer having a display device connected thereto and a communication device. The computer-readable recording medium which recorded the computer program of Claim 16.
body.

Images