JP6274140B2 - COMMUNICATION SYSTEM, TERMINAL DEVICE, PROGRAM, SERVER DEVICE, AND INFORMATION PROCESSING METHOD - Google Patents

Description

  TECHNICAL FIELD The present invention relates to a technical field of a system in which a server device executes processing according to a request transmitted via a network and transmits at least one of video and audio indicating the execution result of this processing via the network. About.

  Conventionally, in a client-server model, server-based computing is known in which most processing is concentrated on the server side, and the client side accepts user operations and displays processing results. In this method, for example, the client device transmits an operation signal corresponding to an operation input from the user to the server device. The server device executes processing according to the operation signal, and distributes video corresponding to the result to the client device by streaming or the like.

  Patent Document 1 discloses a technique for improving the response from the server device to the client device. In this technique, the server device divides screen information in response to a request from the client device into a plurality of blocks, and among the plurality of blocks, a block corresponding to an input from the user is preferentially transmitted to the client device.

JP 2008-210113 A

  However, in the technique described in Patent Document 1, a request is transmitted from the client device to the server device and a video is transmitted from the server device to the client device every time the user performs an operation. Since such communication between the client device and the server device always occurs, there is a problem that the response to the user operation is reduced.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a communication system that enables server-based computing with a shortened response to a user operation.

In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that the video transmitted from a server device that generates at least one of video data and audio data indicating an execution result of an application program via a network. In a communication system in which at least one of data and audio data is received by an output device, a terminal device that requests the server device to execute the application program via the network, the server device, and a second server device And the output device, the terminal device is definition information indicating a definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, Processing information indicating processing according to a user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by a device and second processing information indicating execution of the application program by the server device; and Based on the definition information stored in the storage means, display control means for displaying an operation screen corresponding to the application program on the display means, input means for inputting a user operation, and stored in the temporary storage means A determination unit configured to determine the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the input unit from the definition information; and When the first process information is determined, a process execution unit that executes a process indicated by the first process information; and When the second processing information is determined, request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device, and the request transmitted by the request information transmitting means. Switching information transmitted from the server device in response to execution of the application program by the server device based on information, and identifying the switching destination definition information as switching information related to the switching destination definition information and switching information receiving means for receiving the definition information identification information, the definition information identification information transmitting means for transmitting the definition information identification information received by said switching information receiving unit to the second server device, the second server device Definition information receiving means for receiving the definition information transmitted from the definition information received by the definition information receiving means Switching means for switching the definition information used for determining the processing information by the determination means to the definition information stored in the temporary storage means, and the server device includes: Program storage means for storing an application program, request information receiving means for receiving the request information from the terminal device, and stored in the program storage means based on the request information received by the request information receiving means A program execution unit that executes the application program, and the definition information identification information that identifies the definition information of the switching destination according to the execution of the application program by the program execution unit is transmitted to the terminal device as the switching information. Switching information transmission means and the program execution means Generating means for generating at least one of video data and audio data indicating an execution result of the application program; and at least one of the video data and the audio data generated by the generating means as the output device The second server device includes a definition information storage unit that stores a plurality of the definition information and definition information identification information for identifying the definition information in association with each other, and the terminal device. Definition information identification information receiving means for receiving the transmitted definition information identification information, and the definition received by the definition information identification information receiving means among the plurality of definition information stored in the definition information storage means comprising a definition information transmitting means for transmitting the definition information corresponding to the information identification information to the terminal device, wherein the output device, before A receiving unit that receives at least one of the video data and audio data transmitted from a server device, and outputs an execution result indicated by at least one of the video data and audio data received by the receiving unit. Output control means for outputting by the means.

Invention according to claim 2, in the communication system according to claim 1, wherein the terminal apparatus further comprises a non-volatile memory means for storing a plurality of said definition information, said switching means, the definition of the switching destination When the information is stored in the nonvolatile storage unit, the definition information stored in the nonvolatile storage unit is stored in the temporary storage unit, and the definition used for determining the processing information by the determination unit The information is switched to the definition information stored in the temporary storage means.

The invention according to claim 3 is the communication system according to claim 2 , wherein the second server device receives inquiry information indicating an inquiry about the definition information from the terminal device, and the inquiry. When the inquiry information is received by the information receiving means, out of the plurality of definition information stored in the definition information storage means, definition information that is newer than the definition information stored in the terminal apparatus is sent to the terminal apparatus. A second definition information transmitting means for transmitting, wherein the terminal device periodically transmits the inquiry information to the second server device, and the inquiry information is transmitted from the second server device to the inquiry information. a second definition information reception means for receiving the definition information transmitted in response, before being received by the second definition information receiving means In definition information, and further comprising a updating means for updating the definition information stored in the nonvolatile storage means.
According to a fourth aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating the execution result of the application program via a network. In a communication system in which the output device receives either of the terminal device, the server device, the second server device, and the output device that request the server device to execute the application program via the network. The terminal device is definition information indicating a definition of a user interface corresponding to the application program, and indicates operation information indicating the user operation and processing corresponding to the user operation for each predetermined user operation. Processing information that is executed by the terminal device Temporary storage means for storing definition information associated with any one of first processing information to be displayed and second processing information to indicate execution of the application program by the server device, and nonvolatile storage for storing a plurality of the definition information Based on the definition information stored in the temporary storage means, a display control means for displaying an operation screen corresponding to the application program on the display means, an input means for inputting a user operation, Determination to determine the first process information or the second process information as the process information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit And a process execution for executing the process indicated by the first process information when the first process information is determined by the determination means And when the second processing information is determined by the determining means, request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device, and the request information transmitting means Switching information transmitted from the server device in response to execution of the application program by the server device based on the request information transmitted by the server device, wherein switching information related to the switching destination definition information is received. Information receiving means and switching means for switching definition information used for determining processing information by the determining means based on the switching information received by the switching information receiving means, the definition information of the switching destination being the If stored in the nonvolatile storage means, the switching destination definition information stored in the nonvolatile storage means Switching means for switching definition information stored in the temporary storage means and used to determine processing information by the determination means to the definition information stored in the temporary storage means, and inquiry information indicating an inquiry about the definition information Received by the second definition information receiving means, the second definition information receiving means for receiving the definition information transmitted from the second server apparatus, and the second definition information receiving means. Updating means for updating the definition information stored in the nonvolatile storage means with the definition information, and the server device stores the request information from a program storage means for storing the application program and the terminal device. Based on the request information received by the request information receiving means and the request information receiving means, the program is received. Program execution means for executing the application program stored in the storage means, switching information transmission means for transmitting the switching information to the terminal device in response to execution of the application program by the program execution means, and program execution Generating means for generating at least one of video data and audio data indicating an execution result of the application program by the means, and outputting at least one of the video data and the audio data generated by the generating means Transmitting means for transmitting to the device, wherein the second server device stores definition information storage means for storing a plurality of the definition information, inquiry information receiving means for receiving the inquiry information from the terminal device, and the inquiry The inquiry by the information receiving means Second definition that transmits definition information newer than the definition information stored in the terminal device among the plurality of definition information stored in the definition information storage means to the terminal device Information transmitting means, wherein the output device receives from the server device at least one of the video data and audio data, and the video data received by the receiving means. And output control means for outputting the execution result indicated by at least one of the audio data by the output means.
According to a fifth aspect of the present invention, in the communication system according to the fourth aspect, the second server device includes definition information transmission means for transmitting the definition information stored in the definition information storage means to the terminal device. The terminal device further includes definition information receiving means for receiving the definition information transmitted by the definition information transmitting means of the second server device, and the switching means is received by the definition information receiving means. The definition information is stored in the temporary storage unit, and the definition information used for determining the processing information by the determination unit is switched to the definition information stored in the temporary storage unit.
According to a sixth aspect of the present invention, in the communication system according to the fourth aspect, the server device further includes a definition information storage unit that stores a plurality of the definition information, and the switching information transmission unit includes the definition information. Of the plurality of definition information stored in the storage means, the definition information of the switching destination according to the execution of the application program by the program execution means is transmitted as the switching information, and the switching means is the switching information receiving means The definition information received by the storage unit is stored in the temporary storage unit, and the definition information used for determining the processing information by the determination unit is switched to the definition information stored in the temporary storage unit.
The invention according to claim 7 is the communication system according to any one of claims 1 to 6, wherein the definition information includes scrolling of the operation screen displayed on the display means and in the operation screen. Switching between at least one of the first processing information indicating processing including any movement of an indicia indicating a user operation target and the video data and the audio data transmitted from the server device to the output device And the second process information indicating a process involving.
According to an eighth aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating the execution result of the application program via a network. In a communication system in which either one of the output devices receives an output device, the output device includes a terminal device that requests the server device to execute the application program via the network, the server device, and the output device. Definition information indicating a definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, 1st process information which shows the process performed by the said terminal device And temporary storage means for storing definition information associated with any one of the second processing information indicating execution of the application program by the server device, and based on the definition information stored in the temporary storage means The display control means for displaying the operation screen corresponding to the application program on the display means, the input means for inputting the user operation, and the definition information stored in the temporary storage means, using the input means Determining means for determining the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation performed, and when the first processing information is determined by the determination means, When the process execution means for executing the process indicated by the first process information and the second process information are determined by the determination means, Request information transmitting means for transmitting request information indicating a processing request indicated by the processing information to the server apparatus, and the application program is executed by the server apparatus based on the request information transmitted by the request information transmitting means. Based on the switching information received from the switching information receiving means, which is switching information transmitted from the server device according to the switching information receiving means for receiving the switching information related to the definition information of the switching destination, Switching means for switching definition information used for determining processing information by the determining means, and the server device stores program information for storing the application program, and request information for receiving the request information from the terminal device Based on the request information received by the receiving means and the request information receiving means Program execution means for executing the application program stored in the program storage means, and switching information transmission means for transmitting the switching information to the terminal device in accordance with execution of the application program by the program execution means. A generating unit that generates at least one of video data and audio data indicating an execution result of the application program by the program executing unit; and at least one of the video data and the audio data generated by the generating unit Transmitting means for transmitting one to the output device, wherein the output device receives at least one of the video data and audio data transmitted from the server device, and the receiving means The received video data and Output control means for causing the output means to output an execution result indicated by at least one of the voice data, and the definition information includes a scroll of the operation screen displayed on the display means and a user in the operation screen. Switching between at least one of the first processing information indicating processing including any one of movements of indicia indicating an operation target and the video data and the audio data transmitted from the server device to the output device. And the second process information indicating the accompanying process.
The invention according to claim 9 is the communication system according to claim 8, wherein a plurality of definition information storage means for storing the definition information and the definition information stored in the definition information storage means to the terminal device. Definition information transmitting means for transmitting, and further comprising: a definition information receiving means for receiving the definition information transmitted from the second server apparatus, wherein the switching means comprises: The definition information received by the definition information receiving means is stored in the temporary storage means, and the definition information used for determining the processing information by the determination means is switched to the definition information stored in the temporary storage means. It is characterized by that.
According to a tenth aspect of the present invention, in the communication system according to the eighth aspect, the server device further includes a definition information storage unit that stores a plurality of the definition information, and the switching information transmission unit includes the definition information. Of the plurality of definition information stored in the storage means, the definition information of the switching destination according to the execution of the application program by the program execution means is transmitted as the switching information, the terminal device, the switching means, The definition information received by the switching information receiving unit is stored in the temporary storage unit, and the definition information used for determining processing information by the determination unit is switched to the definition information stored in the temporary storage unit. It is characterized by.
An eleventh aspect of the present invention is the communication system according to any one of the eighth to tenth aspects, wherein the terminal device further includes a nonvolatile storage unit that stores a plurality of the definition information, and the switching unit. If the definition information of the switching destination is stored in the nonvolatile storage means, the definition information of the switching destination stored in the nonvolatile storage means is stored in the temporary storage means, and the processing by the determination means The definition information used for determining the information is switched to the definition information stored in the temporary storage means.

According to a twelfth aspect of the present invention, in the communication system according to any one of the first to eleventh aspects, the request information transmitting means is activated when the second processing information indicates activation of an application program. The request information including application program identification information for identifying the application program is transmitted, and the program storage means stores a plurality of application programs and application program identification information for identifying the application programs in association with each other, and executes the program The means activates an application program corresponding to the application program identification information included in the request information among the plurality of application programs stored in the program storage means, and the switching information transmission means If the application program is activated by the program execution unit, and transmits the switching information on the switching destination definition information corresponding to the application program started.

According to a thirteenth aspect of the present invention, in the communication system according to any one of the first to eleventh aspects, the switching information transmitting unit is configured such that a running application program is executed by the program executing unit. When the internal state of the running application program changes, the switching information related to the definition information of the switching destination corresponding to the changed internal state is transmitted among a plurality of definition information corresponding to the running application program It is characterized by doing.

According to a fourteenth aspect of the present invention, in the communication system according to any one of the first to thirteenth aspects, the terminal device further includes attribute information storage means for storing attribute information indicating an attribute of the terminal device. The switching means switches to definition information corresponding to the attribute indicated by the attribute information stored in the attribute information storage means.

According to a fifteenth aspect of the present invention, at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the terminal device included in the communication system in which the output device receives either of the definition information indicating the definition of the user interface corresponding to the application program, the operation information indicating the user operation for each predetermined user operation; One of processing information indicating processing according to the user operation, first processing information indicating processing executed by the terminal device, and second processing information indicating execution of the application program by the server device Temporary memory that stores definition information associated with Display control means for causing the display means to display an operation screen corresponding to the application program based on the definition information stored in the temporary storage means, input means for inputting a user operation, and the temporary storage means Determining means for determining the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input means from the definition information stored in When the first process information is determined by the determining means, a process executing means for executing the process indicated by the first process information; and when the second process information is determined by the determining means, the second process information Request information transmission means for transmitting request information indicating a request for the processing indicated by the server apparatus to the server device, and based on the request information transmitted by the request information transmission means. Wherein a switching information transmitted from the server apparatus in response to the application program is executed by the server device, as the switching information on the definition information of the switching destination, the switching destination in response to execution of the application program Te Switching information receiving means for receiving definition information identifying information for identifying the definition information, the definition information identifying information received by the switching information receiving means, the definition information and definition information identifying information for identifying the definition information Definition information identification information transmitting means for transmitting to a second server device comprising definition information storage means for storing a plurality of association information, and from the second server device receiving the definition information identification information to the definition information storage means Receives definition information corresponding to the definition information identification information among a plurality of the definition information stored. Definition information receiving means for storing the definition information received by the definition information receiving means in the temporary storage means, and the definition information used for determining the processing information by the determining means is stored in the temporary storage means. Switching means for switching to the definition information .
According to a sixteenth aspect of the present invention, at least one of the video data and the audio data transmitted from the server device that generates at least one of the video data and the audio data indicating the execution result of the application program via the network. In the terminal device included in the communication system in which the output device receives either of the definition information indicating the definition of the user interface corresponding to the application program, the operation information indicating the user operation for each predetermined user operation; One of processing information indicating processing according to the user operation, first processing information indicating processing executed by the terminal device, and second processing information indicating execution of the application program by the server device Temporary memory that stores definition information associated with A non-volatile storage unit that stores a plurality of the definition information, a display control unit that displays an operation screen corresponding to the application program on a display unit based on the definition information stored in the temporary storage unit, The first processing information as processing information corresponding to the operation information indicating the user operation input by using the input unit from the input unit to which a user operation is input and the definition information stored in the temporary storage unit Alternatively, a determination unit that determines the second processing information, and a process execution unit that executes a process indicated by the first processing information when the first processing information is determined by the determination unit, and the determination unit Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the two processing information is determined; Switching information transmitted from the server device in response to execution of the application program by the server device based on the request information transmitted by the information transmitting means, wherein switching information relating to the definition information of the switching destination is Switching information receiving means for receiving, and switching means for switching definition information used for determining processing information by the determining means based on the switching information received by the switching information receiving means, wherein the definition of the switching destination When the information is stored in the nonvolatile storage unit, the definition information stored in the nonvolatile storage unit is stored in the temporary storage unit, and the definition used for determining the processing information by the determination unit Switching means for switching information to the definition information stored in the temporary storage means; and Inquiry information that indicates an inquiry from a second server device that receives the inquiry information periodically from inquiry information transmission means that transmits a plurality of the definition information to a second server device that stores definition information storage means, Second definition information receiving means for receiving definition information newer than definition information stored in the terminal device among the plurality of definition information stored in the definition information storage means, and the second definition information receiving means Updating means for updating the definition information stored in the non-volatile storage means with the definition information received by the method.
According to a seventeenth aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating an execution result of the application program via a network. In the terminal device included in the communication system in which the output device receives either of the definition information indicating the definition of the user interface corresponding to the application program, the operation information indicating the user operation for each predetermined user operation; One of processing information indicating processing according to the user operation, first processing information indicating processing executed by the terminal device, and second processing information indicating execution of the application program by the server device Temporary memory that stores definition information associated with Display control means for causing the display means to display an operation screen corresponding to the application program based on the definition information stored in the temporary storage means, input means for inputting a user operation, and the temporary storage means Determining means for determining the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input means from the definition information stored in When the first process information is determined by the determining means, a process executing means for executing the process indicated by the first process information; and when the second process information is determined by the determining means, the second process information Request information transmission means for transmitting request information indicating a request for the processing indicated by the server apparatus to the server device, and based on the request information transmitted by the request information transmission means. Switching information transmitted from the server device in response to execution of the application program by the server device, the switching information receiving means for receiving switching information related to the definition information of the switching destination, and the switching information reception Switching means for switching definition information used for determining processing information by the determining means based on the switching information received by the means, and the definition information is displayed on the display means. The first processing information indicating processing including any of scrolling and movement of an indicia indicating a user operation target in the operation screen, and the video data and the audio transmitted from the server device to the output device And the second processing information indicating processing involving switching of at least one of the data.

According to an eighteenth aspect of the present invention, at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating an execution result of the application program. In a program executed by a terminal device included in a communication system in which one of the output devices receives the information, definition information indicating a definition of a user interface corresponding to the application program, the user operation for each predetermined user operation Operation information indicating processing and processing information indicating processing according to the user operation, first processing information indicating processing executed by the terminal device and second processing indicating execution of the application program by the server device A constant associated with one of the information An operation corresponding to the application program based on the definition information stored in the temporary storage means in a computer of the terminal device comprising a temporary storage means for storing information and an input means for inputting a user operation From the display control step of displaying a screen on a display means, and the definition information stored in the temporary storage means, the first processing information corresponding to the operation information indicating the user operation input using the input means A determination step for determining processing information or the second processing information; a processing execution step for executing processing indicated by the first processing information when the first processing information is determined by the determination step; and a determination step. When the second processing information is determined, request information indicating a processing request indicated by the second processing information is transmitted to the server device. A request information transmission step for transmission, and switching information transmitted from the server device in response to execution of the application program by the server device based on the request information transmitted by the request information transmission step. Switching information receiving step for receiving definition information identification information for identifying the switching destination definition information in accordance with execution of the application program, and switching information receiving step received as the switching information related to the switching destination definition information, and the switching information receiving step A definition information identification information transmission step for transmitting the definition information identification information to a second server device comprising a definition information storage means for storing a plurality of the definition information and the definition information identification information for identifying the definition information in association with each other; From the second server device that has received the definition information identification information, Of the plurality of definition information stored in the definition information storage means, a definition information reception step for receiving definition information corresponding to the definition information identification information, and the definition information received by the definition information reception step. And a switching step of switching definition information stored in the temporary storage means and used for determination of processing information in the determination step to the definition information stored in the temporary storage means .
According to a nineteenth aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating the execution result of the application program via a network. In a program executed by a terminal device included in a communication system in which one of the output devices receives the information, definition information indicating a definition of a user interface corresponding to the application program, the user operation for each predetermined user operation Operation information indicating processing and processing information indicating processing according to the user operation, first processing information indicating processing executed by the terminal device and second processing indicating execution of the application program by the server device A constant associated with one of the information Stored in the temporary storage means in a computer of the terminal device comprising: temporary storage means for storing information; nonvolatile storage means for storing a plurality of the definition information; and input means for inputting a user operation. Based on the definition information, a display control step for displaying an operation screen corresponding to the application program on a display means, and the user input using the input means from the definition information stored in the temporary storage means When the first process information or the second process information is determined as process information corresponding to operation information indicating an operation, and the first process information is determined by the determination step, the first process information is When the second process information is determined by the process execution step for executing the process shown in FIG. A request information transmission step of transmitting request information indicating a request for processing indicated by the server device to the server device, and the application program being executed by the server device based on the request information transmitted by the request information transmission step. In response to the switching information transmitted from the server device, the switching information receiving step for receiving switching information relating to the definition information of the switching destination, and the determination based on the switching information received by the switching information receiving step A switching step of switching definition information used for determination of processing information by step, wherein the switching destination stored in the nonvolatile storage means when the definition information of the switching destination is stored in the nonvolatile storage means The definition information is stored in the temporary storage means, and the process according to the determination step A switching step for switching definition information used for determining information to the definition information stored in the temporary storage means, and a definition for periodically storing inquiry information indicating an inquiry about the definition information, and storing a plurality of the definition information An inquiry information transmission step for transmitting to a second server device comprising information storage means, and the terminal among the plurality of definition information stored in the definition information storage means from the second server device that has received the inquiry information A second definition information receiving step for receiving definition information newer than the definition information stored in the apparatus, and the definition information received in the second definition information receiving step, the definition stored in the nonvolatile storage means And an update step for updating the information.
According to a twentieth aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating an execution result of the application program via a network. In a program executed by a terminal device included in a communication system in which one of the output devices receives the information, definition information indicating a definition of a user interface corresponding to the application program, the user operation for each predetermined user operation Operation information indicating processing and processing information indicating processing according to the user operation, first processing information indicating processing executed by the terminal device and second processing indicating execution of the application program by the server device A constant associated with one of the information An operation corresponding to the application program based on the definition information stored in the temporary storage means in a computer of the terminal device comprising a temporary storage means for storing information and an input means for inputting a user operation From the display control step of displaying a screen on a display means, and the definition information stored in the temporary storage means, the first processing information corresponding to the operation information indicating the user operation input using the input means A determination step for determining processing information or the second processing information; a processing execution step for executing processing indicated by the first processing information when the first processing information is determined by the determination step; and a determination step. When the second processing information is determined, request information indicating a processing request indicated by the second processing information is transmitted to the server device. A request information transmission step for transmission, and switching information transmitted from the server device in response to execution of the application program by the server device based on the request information transmitted by the request information transmission step. A switching information receiving step for receiving switching information relating to the definition information of the switching destination, and a switching for switching definition information used for determining processing information by the determining step based on the switching information received by the switching information receiving step. And the definition information includes a process including any one of scrolling of the operation screen displayed on the display means and movement of an indicia indicating a user operation target in the operation screen. 1 processing information and the video data transmitted from the server device to the output device. And the second processing information indicating processing involving switching of at least one of the audio data and the audio data.

According to a twenty-first aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating an execution result of the application program via a network. In the server device included in the communication system in which the output device receives either of them, a program storage unit that stores an application program executed in response to a request from the terminal device via the network, and corresponds to the application program Definition information indicating a definition of a user interface, operation information indicating the user operation and processing information indicating processing corresponding to the user operation for each predetermined user operation, the server via the network Application to the device A plurality of definition information associated with either one of first processing information indicating processing executed by a terminal device requesting execution of a gram and second processing information indicating execution of the application program by the server device is stored. Processing information corresponding to the operation information indicating the user operation input using the input means provided in the terminal device from the definition information storage means to be performed and the definition information stored in the temporary storage means provided in the terminal device. When the second processing information is determined, request information receiving means for receiving request information indicating a request for processing indicated by the second processing information from the terminal device, and the request received by the request information receiving means. Program execution means for executing an application program stored in the program storage means based on the information; and In accordance with the execution of the application program by the gram executing means, switching information related to the definition information of the switching destination of the definition information used for determining the processing information corresponding to the operation information indicating the user operation input using the input means. Switching information transmission means for transmitting to the terminal device, including definition information corresponding to execution of the application program among the plurality of definition information stored in the definition information storage means as switching destination definition information The switching information transmitting means for transmitting the switching information, the generating means for generating at least one of video data and audio data indicating the execution result of the application program by the program executing means, and the generating means, At least one of video data and audio data is Transmission means for transmitting to the output device, the definition information is an operation screen displayed on the display means provided in the terminal device based on the definition information stored in the temporary storage means, The first processing information indicating processing including any of scrolling of an operation screen corresponding to the application program and movement of an indicia indicating a user operation target in the operation screen, and transmission from the server device to the output device And the second processing information indicating processing involving switching of at least one of the video data and the audio data.

According to a twenty-second aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating the execution result of the application program via a network. In the information processing method in which the output device receives either of these, definition information indicating a definition of a user interface corresponding to the application program, the terminal device requesting the server device to execute the application program via the network The operation information indicating the user operation and the processing information indicating the process corresponding to the user operation for each predetermined user operation, the first process information indicating the process executed by the terminal device, and The application program by the server device A terminal device comprising: temporary storage means for storing definition information associated with any one of the second processing information indicating execution of a program; and input means for inputting a user operation. Based on the stored definition information, a display control step for displaying an operation screen corresponding to the application program on a display means, and the terminal device from the definition information stored in the temporary storage means, the input means A determination step of determining the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the terminal, and the terminal device performs the first step by the determination step. When the process information is determined, a process execution step for executing the process indicated by the first process information, and the terminal device are preceded by the determination step. When the second processing information is determined, a request information transmission step of transmitting request information indicating a request for processing indicated by the second processing information to the server device; and the server device receives the request information from the terminal device. A request information receiving step for receiving, and a program execution in which the server device executes the application program stored in a program storage means for storing the application program based on the request information received in the request information receiving step Definition information used for determining the processing information corresponding to operation information indicating a user operation input using the input unit in response to execution of the application program in the program execution step as the switching information about switching destination definition information A switching information transmission step for transmitting definition information identification information for identifying the switching destination definition information according to the execution of the application program in the program execution step to the terminal device, and the terminal device is transmitted from the server device. A switching information receiving step for receiving the definition information identification information, a definition information identification for identifying the definition information and the definition information, the definition information identifying information received by the terminal device in the switching information receiving step. A definition information identification information transmission step for transmitting to a second server device comprising a definition information storage means for storing a plurality of information in association with each other; and the definition information identification information transmitted from the terminal device by the second server device. The definition information identification information receiving step to be received and the second server device store the definition information storage means. A definition information transmission step of transmitting definition information corresponding to the definition information identification information received in the definition information identification information reception step among the plurality of definition information being performed, and the terminal device includes the second server A definition information receiving step for receiving the definition information transmitted from a device; and the terminal device stores the definition information received in the definition information receiving step in the temporary storage means ; A switching step of switching definition information used for determination to the definition information stored in the temporary storage means; and the server device includes at least video data and audio data indicating an execution result of the application program by the program execution step. A generating step for generating one of the server device and the server device A transmission step of transmitting at least one of the video data and the audio data generated by the generation step to the output device; and the output device transmits the video data and audio data transmitted from the server device. A reception step of receiving at least one of the output, and an output control step of causing the output device to output an execution result indicated by at least one of the video data and audio data received by the reception step; It is characterized by including.
According to a twenty-third aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating the execution result of the application program via a network. In the information processing method in which the output device receives either of these, definition information indicating a definition of a user interface corresponding to the application program, the terminal device requesting the server device to execute the application program via the network The operation information indicating the user operation and the processing information indicating the process corresponding to the user operation for each predetermined user operation, the first process information indicating the process executed by the terminal device, and The application program by the server device Temporary storage means for storing definition information associated with any one of the second processing information indicating execution of the program, non-volatile storage means for storing a plurality of the definition information, and input for user operation input A display control step for causing a display unit to display an operation screen corresponding to the application program based on the definition information stored in the temporary storage unit; and A determination step of determining the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the storage unit And the terminal device executes a process indicated by the first process information when the first process information is determined by the determining step. And when the second processing information is determined by the determining step, the terminal device transmits request information indicating a request for processing indicated by the second processing information to the server device, and A request information receiving step in which the server device receives the request information from the terminal device, and a program storage in which the server device stores the application program based on the request information received in the request information receiving step. A program execution step for executing the application program stored in the means, and operation information indicating a user operation input by the server device using the input means in response to the execution of the application program by the program execution step Definition used to determine the processing information corresponding to A switching information transmission step of transmitting switching information related to definition information of information switching destination to the terminal device, a switching information receiving step of receiving the switching information transmitted from the server device by the terminal device, and the terminal device; Is a switching step of switching definition information used for determination of processing information by the determination step based on the switching information received by the switching information reception step, wherein the definition information of the switching destination is the non-volatile storage If stored in the means, the definition information of the switching destination stored in the nonvolatile storage means is stored in the temporary storage means, and the definition information used for determining the processing information by the determining step is stored in the temporary storage. A switching step of switching to the definition information stored in the means, and the server device A generation step of generating at least one of video data and audio data indicating an execution result of the application program by the step, and at least one of the video data and the audio data generated by the server device in the generation step A transmission step of transmitting either of the video data and audio data transmitted from the server device; and a step of transmitting the output device to the output device; An output control step of outputting an execution result indicated by at least one of the video data and audio data received by the receiving step by an output means; and the terminal device periodically sends inquiry information indicating the inquiry of the definition information. A definition that stores a plurality of the definition information An inquiry information transmission step for transmitting to the second server device comprising an information storage means, an inquiry information reception step for the second server device to receive the inquiry information from the terminal device, and the second server device for the inquiry. When the inquiry information is received by the information receiving step, definition information newer than the definition information stored in the terminal device among the plurality of definition information stored in the definition information storage means is transmitted to the terminal device. A second definition information transmitting step for transmitting, a second definition information receiving step for the terminal device to receive the definition information transmitted from the second server device, and a terminal device for receiving the second definition information. Updating the definition information stored in the non-volatile storage means with the definition information received by , Including.
According to a twenty-fourth aspect of the present invention, at least one of the video data and the audio data transmitted from a server device that generates at least one of video data and audio data indicating an execution result of the application program via a network. In the information processing method in which the output device receives either of these, definition information indicating a definition of a user interface corresponding to the application program, the terminal device requesting the server device to execute the application program via the network The operation information indicating the user operation and the processing information indicating the process corresponding to the user operation for each predetermined user operation, the first process information indicating the process executed by the terminal device, and The application program by the server device A terminal device comprising: temporary storage means for storing definition information associated with any one of the second processing information indicating execution of a program; and input means for inputting a user operation. Based on the stored definition information, a display control step for displaying an operation screen corresponding to the application program on a display means, and the terminal device from the definition information stored in the temporary storage means, the input means A determination step of determining the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the terminal, and the terminal device performs the first step by the determination step. When the process information is determined, a process execution step for executing the process indicated by the first process information, and the terminal device are preceded by the determination step. When the second processing information is determined, a request information transmission step of transmitting request information indicating a request for processing indicated by the second processing information to the server device; and the server device receives the request information from the terminal device. A request information receiving step for receiving, and a program execution in which the server device executes the application program stored in a program storage means for storing the application program based on the request information received in the request information receiving step Definition information used for determining the processing information corresponding to operation information indicating a user operation input using the input unit in response to execution of the application program in the program execution step The switching information related to the definition information of the switching destination of A switching information transmission step for transmitting to the terminal device; a switching information receiving step for the terminal device to receive the switching information transmitted from the server device; and the terminal device received by the switching information receiving step. A switching step of switching definition information used to determine processing information in the determination step based on switching information; and the server device includes at least video data and audio data indicating an execution result of the application program in the program execution step. A generating step for generating any one of them; a transmission step for transmitting at least one of the video data and the audio data generated by the server device to the output device; and the output device, The video data transmitted from the server device A reception step of receiving at least one of data and audio data, and the output device causes an output unit to output an execution result received by the reception step and indicated by at least one of the video data and audio data An output control step, wherein the definition information indicates a process including any one of scrolling the operation screen displayed on the display means and moving an indicia indicating a user operation target in the operation screen. 1st process information and the said 2nd process information which shows the process accompanied by the switch of at least any one of the said video data and the said audio data transmitted to the said output device from the said server apparatus, It is characterized by the above-mentioned. .

According to each of the first, fifteenth , eighteenth, and twenty-second aspects of the present invention, among the processes corresponding to the user operation, the terminal apparatus does not transmit request information to the server apparatus for the processes executed by the terminal apparatus. Therefore, the response to the user operation can be shortened. Further, the terminal device can acquire definition information according to the execution of the application program from a second server device that is different from the server device that executes the application program.
According to the invention described in each of claims 4, 16, 19, and 23, among the processes corresponding to the user operation, the terminal apparatus does not transmit request information to the server apparatus for the processes executed by the terminal apparatus. Therefore, the response to the user operation can be shortened. Further, when the definition information is switched, the terminal device does not need to acquire the switching destination definition information from another device. Further, when the definition information is updated, the updated definition information can be stored in the terminal device before switching to the updated definition information.
According to the invention described in each of claims 8, 17, 20, 21, and 24, among the processes corresponding to the user operation, the terminal apparatus transmits request information to the server apparatus for the processes executed by the terminal apparatus. do not do. Therefore, the response to the user operation can be shortened. Further, the server device does not need to generate and transmit an operation screen whose display changes due to scrolling or movement of a sign.

According to the invention described in each of the fifth and ninth aspects, the terminal device can acquire the definition information from the second server device that is different from the server device that executes the application program.

According to the invention described in claims 6 and 10 , the terminal device can acquire the definition information corresponding to the execution of the application program from the server device that executes the application program.

According to the invention described in each of claims 2 and 11, when the definition information is switched, the terminal device does not need to acquire the definition information of the switching destination from another device.

According to the invention described in claim 3 , when the definition information is updated, the updated definition information can be stored in the terminal device before switching to the updated definition information.

According to the invention described in claim 12 , it is possible to switch to the user interface corresponding to the activated application program.

According to the thirteenth aspect of the present invention, it is possible to switch to the user interface corresponding to the internal state of the running application program.

According to invention of Claim 14 , it can switch to the user interface suitable for the attribute of the terminal device.

According to the seventh aspect of the present invention, the server device does not need to generate and transmit an operation screen whose display changes due to scrolling or movement of a sign.

(A) is a figure showing an example of outline composition of communications system S1 of one embodiment. (B) is a figure showing an example of outline composition of application server 1 of one embodiment. (C) is a diagram showing a schematic configuration example of the controller 2 of one embodiment. (A) is a figure which shows an example of UI definition corresponding to the operation screen for application program selection in case the controller 2 is a remote controller. (B) is a diagram showing an example of a list of application programs. (C) is a figure which shows the example of the external appearance of a remote controller, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen for music selection of the application program of karaoke in case the controller 2 is a remote controller. (B) is a figure which shows the example of the external appearance of a remote controller, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen during the performance of the karaoke of the application program of karaoke in case the controller 2 is a remote controller. (B) is a figure which shows the example of the external appearance of a remote controller, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen of the application program of a rhythm game in case the controller 2 is a remote controller. (B) is a figure which shows the example of the external appearance of a remote controller, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen for application program selection in case the controller 2 is a smart phone. (B) is a figure which shows the example of the external appearance of a smart phone, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen for music selection of the application program of karaoke in case the controller 2 is a smart phone. (B) is a figure which shows the example of the external appearance of a smart phone, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the creation screen of the application program of a rhythm game in case the controller 2 is a smart phone. (B) is a figure which shows the example of the external appearance of a smart phone, and an operation screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen for application program selection in case the integrated client 4 is a television receiver. (B) is a figure which shows the example of the external appearance of a television receiver, and a display screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen for music selection of the application program of karaoke in case the integrated client 4 is a television receiver. (B) is a figure which shows the example of the external appearance of a television receiver, and a display screen. (A) is a figure which shows an example of UI definition corresponding to the operation screen of the application program of a rhythm game in case the integrated client 4 is a television receiver. (B) is a figure which shows the example of the external appearance of a television receiver, and a display screen. It is a sequence diagram which shows an example of the process outline | summary of communication system S1. 3 is a flowchart illustrating a processing example of an application server 1. 3 is a flowchart showing a processing example of a controller 2. (A) is a figure showing an example of outline composition of communications system S2 of one embodiment. (B) is a diagram showing a schematic configuration example of the UI definition server 7 of one embodiment. It is a sequence diagram which shows an example of the process outline | summary of communication system S2. 6 is a flowchart illustrating an example of UI definition update processing in a controller 2;

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[1. First Embodiment]
[1-1. Configuration of communication system]
FIG. 1A is a diagram illustrating a schematic configuration example of the communication system S1 of the present embodiment. As shown in FIG. 1A, the communication system S1 includes an application server 1, a plurality of controllers 2, and a plurality of renderers 3. The application server 1 is an example of a server device of the present invention. The controller 2 is an example of a terminal device of the present invention. The renderer 3 is an example of the output device of the present invention. The application server 1, the controller 2, and the renderer 3 are each connected to the network 10. The network 10 is configured by, for example, the Internet.

  The application server 1 executes an application program in response to a request from the controller 2. Examples of application programs include application programs such as karaoke, fitness, and video games. The application server 1 generates at least one of video and audio indicating the execution result of the application program. Then, the application server 1 transmits at least one of the generated video and audio to the renderer 3 by streaming.

  The application server 1 is configured to be accessible to the application DB 5 and the UI definition DB 6. For example, the application server 1, the application DB 5, and the UI definition DB 6 may be connected by a network such as a LAN (Local Area Network). Alternatively, the application server 1 may include an application DB 5 and a UI definition DB 6. In the application DB 5, one or a plurality of application programs are registered. Each application program is stored in the application DB 5 in association with the application ID. The application ID is an example of application program identification information of the present invention. The application ID is identification information that identifies an application program. One or more UI definitions are registered in the UI definition DB 6. The UI definition is an example of definition information of the present invention. The UI definition is information indicating a definition of a user interface for operating or executing an application program. One or more UI definitions are stored in the UI definition DB 6 for each application program. The application server 1 transmits a UI definition corresponding to the application program to the controller 2 in accordance with the execution of the application program.

  The controller 2 is a terminal device that requests the application server 1 to execute an application program via the network 10. The controller 2 displays an operation screen corresponding to the application program on the display, based on the UI definition transmitted from the application server 1 in response to the execution of the application program. Further, when an operation by the user is input, the controller 2 executes a process corresponding to the operation based on the UI definition or requests the application server 1 to execute an application program corresponding to the operation. Examples of the controller 2 include a remote controller that can be connected to the network 10, a smartphone, a mobile phone, and a portable video game machine.

  The renderer 3 receives at least one of video and audio indicating the execution result of the application program from the application server 1. When the video indicating the execution result is received, the renderer 3 displays the received video on the display. When the voice indicating the execution result is received, the renderer 3 outputs the received voice through the speaker. Examples of the renderer 3 include a television receiver such as a smart TV, a personal computer, and a karaoke device.

  The user uses an application program by combining the controller 2 and the renderer 3. That is, the user operates the controller 2 to cause the application server 1 to execute the application program, and uses the renderer 3 to check the execution result of the application program.

  Note that the communication system S1 may or may not include the integrated client 4 having the functions of the controller 2 and the renderer 3. Examples of the integrated client 4 include a television receiver and a personal computer.

[1-2. Configuration of Application Server 1]
Next, the configuration of the application server 1 will be described with reference to FIG. FIG. 1B is a diagram illustrating a schematic configuration example of the application server 1 of the present embodiment. As shown in FIG. 1B, the application server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage unit 14, a communication unit 15, and the like. Composed. These components are connected to the bus 16. The communication unit 15 is connected to the network 10. The storage unit 14 is configured by, for example, a hard disk drive. The storage unit 14 stores, for example, music data, telop data, and video data. The music data is data indicating the performance sound of the karaoke music. The telop data is data indicating the lyrics of karaoke music. The video data is data indicating a video of karaoke. The music data, telop data, and video data are stored in the storage unit 14 in association with the music number, for example. The song number is an identification number for identifying a song of karaoke. The storage unit 14 may store at least one of the application DB 5 and the UI definition DB 6. The storage unit 14 stores an OS, an application server program, and the like. The application server program is a program that causes the CPU 11 as a computer to execute processing such as transmission of a UI definition, execution of an application program, and transmission of a video indicating an execution result of the application program. The application server program or the like may be downloaded from another server or the like via the network 10 or the like, for example. Alternatively, the application server program or the like may be recorded on a recording medium such as an optical disk, a magnetic tape, or a memory card and read into the storage unit 14 via a drive device.

[1-3. Configuration of controller 2]
Next, the configuration of the controller 2 will be described with reference to FIG. FIG. 1C is a diagram illustrating a schematic configuration example of the controller 2 of the present embodiment. As shown in FIG. 1C, the controller 2 includes a CPU 21, a ROM 22, a RAM 23, a storage unit 24, a communication unit 25, an input unit 26, a display unit 27, and the like. These components are connected to the bus 28. The communication unit 25 is connected to the network 10. The RAM 23 is an example of a temporary storage unit of the present invention. The temporary storage means is not limited to the RAM. For example, if the UI definition is temporarily stored without the purpose of saving, for example, a nonvolatile memory may be the temporary storage unit. Further, for example, when the controller 2 is powered off, the memory for deleting the stored UI definition may be a temporary storage unit. The input unit 26 is an input device for inputting a user operation. The input unit 26 may be configured by at least one of a button, a key, a touch panel, a keyboard, and the like, for example. The display unit 27 is a display configured by, for example, a liquid crystal panel, an organic EL panel, or the like. The storage unit 24 is an example of a nonvolatile storage unit of the present invention. The storage unit 24 is configured by, for example, a flash memory or a hard disk drive. The storage unit 24 stores controller attributes. The controller attribute is an example of attribute information of the present invention. The controller attribute is information indicating the attribute of the controller 2. For example, the controller attribute may be information indicating the type of the controller 2. For example, when the controller 2 is a remote controller, the controller attribute is set to “remote controller”, and when the controller 2 is a smartphone, the controller attribute is set to “smartphone”. The storage unit 24 stores an OS, a controller program, and the like. The controller program is a program that causes the CPU 21 as a computer to execute processes such as reception and storage of UI definitions, display of operation screens, execution of processes according to user operations, and transmission of requests to the application server 1. The controller program or the like may be downloaded from a server device such as the application server 1 via the network 10 or the like, for example. Alternatively, the controller program or the like may be recorded on a recording medium such as an optical disk, a magnetic tape, or a memory card and read into the storage unit 24 via a drive device. Alternatively, the controller program or the like may be installed in the controller 2 in advance.

[1-4. UI definition]
Next, an outline of UI definition and processing related to UI definition will be described.

  The UI definition is information indicating the definition of the user interface in the controller 2. Specifically, the UI definition defines the display contents of the operation screen and the processing corresponding to the user's operation. In the UI definition, for each user operation that can be accepted on the operation screen, operation information indicating a user operation and processing information indicating a process corresponding to the user operation are associated with each other. The process according to the user operation is a process indicated by the process information associated with the operation information indicating the user operation in the UI definition.

  The processing information includes local processing information indicating processing executed by the controller 2 and remote processing information indicating processing executed by the application server 1. The local processing information is an example of the first processing information of the present invention. The remote processing information is an example of second processing information of the present invention. The process executed by the controller 2 and the process executed by the application server 1 may not include overlapping processes, for example.

  The process executed by the controller 2 may be a process related to the operation screen, for example, or may be another process. The processing related to the operation screen may be, for example, scrolling of the operation screen, or may be to move the movement of the mark indicating the user operation target in the operation screen to a position corresponding to another component. Another process for updating at least a part of the operation screen may be used. The indicia may be, for example, a cursor, a pointer, a focus, or the like. The user operation target may be, for example, a button, a check box, an image, a character, an input area, a list, or the like. The sign may be a cursor in a text box. The cursor in this case indicates a position where a character is input by a user operation in the text box.

  Specifically, the process executed by the application server 1 is to execute an application program. For example, this process may be a process that involves switching of at least one of video and audio transmitted from the application server 1 to the renderer 3 as an execution result of the application program, or a process that does not involve such switching. It may be.

  The UI definition DB 6 stores one or more UI definitions that define a user interface corresponding to the application program for each application program. Each UI definition is stored in the UI definition DB 6 in association with the UI definition position information. The UI definition position information is an example of definition information identification information of the present invention. The UI definition position information is information for identifying the UI definition. For example, the UI definition location information may be a URL (Uniform Resource Locator). Since the UI definition exists separately from the application program, even if the user interface is changed, it is only necessary to change the UI definition, and there is no need to change the application program.

  When there are a plurality of UI definitions corresponding to one application program, for example, a plurality of UI definitions corresponding to the internal state of the application program are stored in the UI definition DB 6. The internal state may be, for example, the progress status or processing status of the application program. For example, in the case of a karaoke application program, a UI definition for selecting karaoke, a UI definition during karaoke performance, and the like may be stored in the UI definition DB 6.

  For each controller attribute, a UI definition corresponding to the controller attribute may be stored in the UI definition DB 6. Specifically, a UI definition corresponding to at least one of the input unit 26 and the display unit 27 of the controller 2 having the attribute indicated by the controller attribute is stored in the UI definition DB 6. For example, a UI definition for a remote controller, a UI definition for a smartphone, and the like may be stored in the UI definition DB 6. Even when a user interface is prepared for each controller attribute, it is only necessary to create a UI definition for each controller attribute, and it is not necessary to create an application program for each controller attribute.

  The UI definition may be stored in the UI definition DB 6 so that the corresponding application program, internal state, and controller attribute can be identified, for example. For example, these pieces of information may be identifiable by UI definition UI definition position information.

  The controller 2 stores the UI definition received from the application server 1 in the RAM 23. Then, the controller 2 causes the display unit 27 to display an operation screen based on the UI definition stored in the RAM 23. When a user operation is input using the input unit 26, the controller 2 performs processing information corresponding to the operation information indicating the input user operation from the UI definition stored in the RAM 23. Determine as information. At this time, local processing information or remote processing information is determined.

  When the local process information is determined, the controller 2 executes the process indicated by the local process information. As described above, this process may be a process related to an operation screen, for example.

  When the remote processing information is determined, the controller 2 transmits an application program execution request to the application server 1. The execution request is an example of request information of the present invention. This execution request is roughly classified into a request for starting an application program and a request for further processing for the application program being started. The application program execution request includes the application ID of the application program to be executed. This application ID is included in the remote processing information. The execution request may further include, for example, a controller attribute stored in the storage unit 24, or may not be included. The execution request may or may not include a parameter passed to the application program, for example.

  The application server 1 that has received the execution request from the controller 2 executes the application program based on the execution request. Specifically, the application server 1 executes the process indicated by the remote process information determined by the controller 2 by executing the application program indicated by the application ID included in the execution request. The application server 1 performs streaming transmission of at least one of video and audio indicating the execution result of the application program to the renderer 3.

  The application server 1 transmits a switching instruction indicating an instruction to switch the UI definition to the controller 2 in accordance with the execution of the application program. Specifically, the application server 1 transmits a switching instruction when the application program is activated or when the internal state of the application program changes due to execution of the activated application program. The switching instruction is an example of switching information of the present invention. The switching instruction includes a switching destination UI definition. Specifically, the application server 1 transmits a UI definition corresponding to the application program indicated by the application ID included in the received execution request. At this time, if the received execution request includes a controller attribute, the application server 1 transmits a UI definition corresponding to the controller attribute to the controller 2. In addition, a plurality of UI definitions corresponding to the internal state of the application program may be stored in the UI definition DB 6. In this case, when the application server 1 activates the application program, the application server 1 transmits a UI definition corresponding to the internal state immediately after activation to the controller 2. If the internal state of the application program has changed as a result of the execution of the application program being activated, a UI definition corresponding to the internal state after the change is transmitted to the controller 2. The UI definition corresponding to the internal state is a UI definition including remote processing information indicating processing that can be executed according to the internal state of the application. Depending on the situation or depending on the internal state of the application program, the application server 1 may not transmit the UI definition to the controller 2.

  The controller 2 switches the user interface based on the switching instruction received from the application server 1. Switching the user interface includes switching the operation screen and switching the UI definition used for determining processing information indicating processing to be executed. Specifically, the controller 2 overwrites the UI definition stored in the RAM 23 with the UI definition included in the switching instruction. Then, the controller 2 displays an operation screen on the display unit 27 based on the UI definition newly stored in the RAM 23.

  As described above, when the remote processing information is determined based on the UI definition, the controller 2 transmits an execution request to the application server 1. On the other hand, when the local processing information is determined, the controller 2 itself executes the processing indicated by the local processing information without transmitting an execution request to the application server 1. Therefore, in this case, since communication between the controller 2 and the application server 1 does not occur, the response to the user operation can be shortened.

  In conventional server-based computing, a server device performs streaming transmission of an image including an operation screen to a client device. Each time a user operation is performed, the client device transmits a request to the server device. Since the server device updates the video on the operation screen, for example, even when an operation for scrolling the operation screen or an operation for moving a mark on the operation screen is performed, the client device makes a request to the server device. Need to send. Accordingly, the response to the user operation decreases. Such a problem can be solved in the communication system S1.

  Further, the functions of the respective devices are simplified by using separate devices for the controller 2 for displaying the operation screen and the renderer 3 for outputting at least one of video and audio indicating the execution result of the application program. be able to. In addition, the processing load can be distributed. Note that, like the integrated client 4, the controller 2 and the renderer 3 may be configured integrally.

  The controller 2 may store the UI definition received from the application server 1 in the RAM 23 and store the UI definition in the storage unit 24. At this time, the controller 2 may store the UI definition in the storage unit 24 in association with the application ID or the like. The controller 2 that has determined the remote processing information as the processing information using the UI definition stored in the RAM 23 stores the UI definition corresponding to the application ID included in the remote processing information when the storage unit 24 stores the UI definition. The UI definition stored in the unit 24 may be stored in the RAM 23. Then, the controller 2 may switch the UI definition used for determining the processing information to the UI definition newly stored in the RAM 23. In this case, the controller 2 does not need to acquire a UI definition from the application server 1.

[1-5. Specific example of UI definition]
Next, specific examples of UI definitions and operation screens displayed based on the UI definitions will be described with reference to FIGS. In the following, details of an example of a UI definition described in XML will be described. The UI definition may be text data in a predetermined format, for example, or may be binary data in a predetermined format. The UI definition includes, for example, a UI definition identification unit, a display definition unit, and an input definition unit.

  The UI definition identifying unit is a part that defines a corresponding application program, an internal state, and a controller attribute. The UI definition identifying unit is indicated by a client tag, for example. The client tag includes a purpose attribute and a device attribute. The purpose attribute indicates the application program and the internal state. The device attribute indicates a controller attribute.

  The display definition part is a part that defines the operation screen, and the display definition part indicates the contents of the operation screen. The display definition unit defines processing when an element displayed on the operation screen is pressed. The display definition part is a part surrounded by, for example, a display start tag and a display end tag. The display definition unit includes at least one of a title tag, a select-box tag, a text-box tag, a text tag, and the like. The text enclosed by the title start tag and title end tag indicates the title displayed on the operation screen. The select-box tag is a tag that defines one or a plurality of buttons displayed on the operation screen. One or more item tags may be included between the select-box start tag and the select-box end tag. The item tag is a tag that defines one button. The item tag includes a face attribute. The face attribute indicates the text displayed on the button. A behavior tag is described between the item start tag and the item end tag. The behavior tag is a tag that defines a process to be executed by the controller 2 when the button is pressed. The behavior tag is an example of processing information of the present invention. The behavior tag includes a domain attribute, a target attribute, and a control attribute. The domain attribute indicates whether the device that performs processing is the application server 1 or the controller 2. When the domain attribute is “remote”, it indicates that the application server 1 executes the application program. The behavior tag whose domain attribute is “remote” is an example of remote processing information. When the domain attribute is “local”, it indicates that the controller 2 executes processing. The behavior tag whose domain attribute is “local” is an example of local processing information. The target attribute indicates the processing target. When the domain attribute is “remote”, the URL of the application program to be executed is set as the target attribute. The URL of the application program is an example of an application ID. When the domain attribute is “local”, the name attribute or the data attribute of the tag that defines the element to be processed among the elements displayed on the operation screen is set as the target attribute. The control attribute indicates the content of the process to be executed. When the domain attribute is “local”, the control attribute indicates processing executed by the controller 2. When the domain attribute is “remote”, the control attribute indicates a parameter that the controller 2 transmits to the application server 1 together with the URL of the application program. The text-box tag is a tag that defines a text box displayed on the operation screen. The user can input characters, numbers, etc. in the text box. The text enclosed by the text start tag and the text end tag indicates the text displayed on the operation screen.

  The input definition unit defines a process when an operation by the user is input. The display definition part is a part surrounded by an input start tag and an input end tag, for example. The display definition unit includes an operation element tag. The operation element tag is a tag that defines an element that receives an operation among operable elements such as a key, a button, and a touch panel constituting the input unit 26. For example, in the case of a cross button, a cross-key tag is described. For example, in the case of a selection key, a select-key tag is described. The operation element tag is an example of operation information of the present invention. A behavior tag is described between the operation element start tag and the operation element end tag. The behavior tag is a tag that defines a process that is executed when an operation is performed on an element indicated by the operation element tag. The configuration of the behavior tag described in the input definition section is the same as the configuration of the behavior tag described in the display definition section.

[1-5-1. For remote controller]
FIG. 2A is a diagram illustrating an example of a UI definition corresponding to an operation screen for selecting an application program when the controller 2 is a remote controller. FIG. 2B is a diagram illustrating an example of a list of application programs. FIG. 2C is a diagram illustrating an example of the appearance and operation screen of the remote controller. As illustrated in FIG. 2C, the remote controller includes, for example, a cross key 261 and a selection key 262 as components of the input unit 26. The user can press the cross key 261 in any of the up, down, left, and right directions.

  In FIG. 2A, a title tag and a select-box tag are described in the display definition section. The select-box tag in FIG. 2A includes a src attribute. As the src attribute, the file name of the application program list is set. As shown in FIG. 2B, the application program list describes the application name and the URL for identifying the application program for each application program. When the controller 2 requests the application server 1 to execute the application program, the controller 2 transmits any URL described in the list of application programs to the application server 1.

  In the case of the UI definition shown in FIG. 2A, an operation screen as shown in FIG. 2C is displayed on the display unit 27 of the remote controller. For example, the display unit 27 displays buttons indicating the title “application selection” and the application name described in the application program list.

  In FIG. 2 (A), a plurality of cross-key tags and select-key tags are described in the input definition section. The cross-key tag corresponds to the cross key 261, and the select-key tag corresponds to the selection key 262. Each cross-key tag has a class attribute, for example. The class attribute indicates the direction in which the cross key 261 is pressed. The behavior tag corresponding to the cross-key tag allows the controller 2 to move the cursor displayed at the position of the currently selected button among the buttons of the application program list to the position of one of the upper and lower buttons. Indicates. The behavior tag corresponding to the select-key tag indicates that the controller 2 causes the application server 1 to execute the application program by transmitting the URL of the application program corresponding to the currently selected button to the application server 1. .

  FIG. 3A is a diagram illustrating an example of a UI definition corresponding to an operation screen for music selection of a karaoke application program when the controller 2 is a remote controller. FIG. 3B is a diagram illustrating an example of the appearance and operation screen of the remote controller.

  In FIG. 3A, the display definition section includes a text-box tag corresponding to a text box into which a music number is input, an item tag corresponding to numeric buttons 0 to 9, an item tag corresponding to a music selection button, and a session. A text tag for displaying the number is described. Each behavior tag of the number buttons 0 to 9 indicates that the controller 2 adds a number corresponding to the number button pressed on the operation screen to the text box. The behavior tag of the music selection button indicates that the controller 2 transmits the music number input in the text box to the application server 1, and the application server 1 executes the karaoke application program. The application server 1 that has received the music number from the controller 2 starts the music performance process corresponding to the music number by executing the karaoke application program. In this performance processing, the application server 1 generates karaoke content based on the music data, telop data, and video data stored in the storage unit 14. The karaoke content is data in which karaoke performance sound and video including lyrics telop are multiplexed. The application server 1 performs streaming distribution of karaoke content as video and audio indicating the result of performance processing. Further, the application server 1 transmits a UI definition corresponding to the operation screen during the performance of karaoke to the controller 2. The session number is identification information for identifying a session for connecting to an application program started in the application server 1. In the case of the UI definition shown in FIG. 3A, an operation screen as shown in FIG. 3B is displayed on the display unit 27 of the remote controller. As shown in FIG. 3B, a text box, numeric buttons 0 to 9, a music selection button, and a session number are displayed.

  In FIG. 3A, the input definition section describes a plurality of cross-key tags and one select-key tag. The behavior tag corresponding to the cross-key tag indicates that the controller 2 moves the cursor displayed at the position of the currently selected button among the buttons displayed on the operation screen to the position of any one of the upper, lower, left and right buttons. Indicates moving. The behavior tag corresponding to the select-key tag indicates that the controller 2 executes processing when the currently selected button is pressed. In this case, in the display definition unit, the process indicated by the behavior tag corresponding to the item tag of the pressed button is executed.

  FIG. 4A is a diagram illustrating an example of a UI definition corresponding to an operation screen during karaoke performance of a karaoke application program when the controller 2 is a remote controller. FIG. 4B is a diagram illustrating an example of the appearance and operation screen of the remote controller.

  4A, an item tag corresponding to buttons for performing various operations during performance and a text tag for displaying a session number are described in the display definition section. As operations during performance, for example, the key of the music is raised, the key is lowered, the key is returned to neutral, the tempo of music is increased, the tempo is decreased, the tempo is returned to neutral, and the performance is stopped. The behavior tag corresponding to these item tags indicates that the controller 2 transmits a command corresponding to the pressed button to the application server 1, and the application server 1 executes the karaoke application program. The application server 1 that has received the command from the controller 2 executes processing corresponding to the command such as key change, tempo change, and karaoke performance stop by executing the karaoke application program. When changing the key or changing the tempo, the application server 1 generates karaoke content and performs streaming delivery based on the changed key or tempo. In the case of stopping the performance of karaoke, the application server 1 stops the generation and distribution of karaoke content, and transmits a UI definition for music selection to the controller 2, for example. In the case of the UI definition shown in FIG. 4A, an operation screen as shown in FIG. 4B is displayed on the display unit 27 of the remote controller. As shown in FIG. 4A, buttons and session numbers for performing various operations during performance are displayed. The input definition unit shown in FIG. 4A is the same as the input definition unit shown in FIG.

  FIG. 5A is a diagram illustrating an example of a UI definition corresponding to the operation screen of the application program of the rhythm game when the controller 2 is a remote controller. FIG. 5B is a diagram illustrating an example of an appearance and an operation screen of the remote controller. The rhythm game is a video game in which, for example, blue, red, and green buttons are pressed in a timely manner according to the video on the game screen displayed on the display of the renderer 3.

  In FIG. 5A, the display definition section describes item tags corresponding to blue, red, and green buttons, and a text tag for displaying a session number. The behavior tag of each of the blue, red, and green buttons indicates that the controller 2 transmits the color information of the button pressed on the operation screen to the application server 1, and the application server 1 executes the rhythm game application program. Indicates. The application server 1 that has received the color information from the controller 2 generates a video of the game screen in accordance with the received color information, the timing at which the color information is received, etc. by executing the application program of the rhythm game, and renders the generated video as a renderer. Stream to 3 In the case of the UI definition shown in FIG. 5A, an operation screen as shown in FIG. 5B is displayed on the display unit 27 of the remote controller. As shown in FIG. 5B, blue, red, and green buttons and session numbers are displayed.

  In FIG. 5A, an input definition section describes a plurality of cross-key tags and select-key tags. In the behavior tag corresponding to the cross-key tag, the controller 2 moves the cursor displayed at the position of the currently selected button among the buttons displayed on the operation screen to the position of either the left or right button. Indicates that The behavior tag corresponding to the select-key tag indicates that the controller 2 executes processing when the currently selected button is pressed. In this case, in the display definition unit, the process indicated by the behavior tag corresponding to the item tag of the pressed button is executed.

[1-5-2. For smartphones]
FIG. 6A is a diagram illustrating an example of a UI definition corresponding to an operation screen for selecting an application program when the controller 2 is a smartphone. FIG. 6B is a diagram illustrating an example of the appearance and operation screen of the smartphone. For example, the smartphone includes a touch panel in which an input unit 26 and a display unit 27 are integrally configured.

  The display definition unit shown in FIG. 6A is the same as the display definition unit shown in FIG. In the case of the UI definition shown in FIG. 6A, an operation screen as shown in FIG. 6B is displayed on the display unit 27 of the remote controller. In FIG. 6A, the input definition part describes a tap tag corresponding to a tap operation on the touch panel. The behavior tag corresponding to the tap tag indicates that the controller 2 causes the application server 1 to execute the application program by transmitting the URL of the application program corresponding to the tapped button to the application server 1.

  FIG. 7A is a diagram illustrating an example of a UI definition corresponding to an operation screen for music selection of a karaoke application program when the controller 2 is a smartphone. FIG. 7B is a diagram illustrating an example of the appearance and operation screen of the smartphone.

  The display definition unit shown in FIG. 7A is the same as the display definition unit shown in FIG. In the case of the UI definition shown in FIG. 7A, an operation screen as shown in FIG. 7B is displayed on the display unit 27 of the remote controller. In FIG. 7A, the input definition section describes a tap tag corresponding to a tap operation on the touch panel. The behavior tag corresponding to the tap tag indicates that the controller 2 executes processing corresponding to the tapped button. In this case, in the display definition unit, the process indicated by the behavior tag corresponding to the item tag of the tapped button is executed.

  FIG. 8A is a diagram illustrating an example of a UI definition corresponding to an operation screen of an application program for a rhythm game when the controller 2 is a smartphone. FIG. 8B is a diagram illustrating an example of an appearance and an operation screen of a smartphone.

  The display definition unit shown in FIG. 8A is the same as the display definition unit shown in FIG. In the case of the UI definition shown in FIG. 8A, an operation screen as shown in FIG. 8B is displayed on the display unit 27 of the remote controller. In FIG. 8A, the input definition section describes a tap tag corresponding to a tap operation on the touch panel. The behavior tag corresponding to the tap tag indicates that the controller 2 executes processing corresponding to the tapped button. In this case, in the display definition unit, the process indicated by the behavior tag corresponding to the item tag of the tapped button is executed.

[1-5-3. For TV receiver]
FIG. 9A is a diagram illustrating an example of a UI definition corresponding to an operation screen for selecting an application program when the integrated client 4 is a television receiver. FIG. 9B illustrates an example of an appearance and a display screen of a television receiver. In the case of a television receiver, for example, an operation can be performed using a television remote controller attached to the television receiver. This television remote control includes, for example, numeric buttons, direction buttons, OK buttons, blue, red, and green buttons. In the case of a television receiver as the integrated client 4, screens 271 and 272 are displayed on the display unit 27. The screen 271 displays a video showing the execution result of the application program. A screen 272 is an operation screen.

  The display definition unit shown in FIG. 9A is the same as the display definition unit shown in FIG. In the case of the UI definition shown in FIG. 9A, a screen 272 as shown in FIG. 9B is displayed as an operation screen on the display unit 27 of the television receiver. In FIG. 2A, a plurality of direction-key tags corresponding to direction buttons and an ok-key tag corresponding to an OK button are described in the input definition section. The behavior tag corresponding to the direction-key tag allows the controller 2 to move the cursor displayed at the position of the currently selected button among the buttons of the application program list to the position of one of the upper and lower buttons. Indicates. The behavior tag corresponding to the ok-key tag indicates that the controller 2 causes the application server 1 to execute the application program by transmitting the URL of the application program corresponding to the currently selected button to the application server 1. .

  FIG. 10A is a diagram illustrating an example of a UI definition corresponding to a music selection operation screen of a karaoke application program when the integrated client 4 is a television receiver. FIG. 10B is a diagram illustrating an example of an appearance and a display screen of a television receiver.

  In FIG. 10A, the display definition unit describes a text-box tag corresponding to a text box into which a music number is input, a text tag for displaying a session number, and the like. In the case of the UI definition shown in FIG. 10A, a screen 272 as shown in FIG. 10B is displayed on the display unit 27 of the television receiver. As shown in FIG. 10B, a text box, a session number, and the like are displayed.

  In FIG. 10A, the input definition section describes num-key tags and OK-key tags corresponding to the numeric buttons 0 to 9, respectively. The behavior tag corresponding to the num-key tag indicates that the controller 2 adds a number corresponding to the pressed number button to the text box. The behavior tag corresponding to the ok-key tag indicates that the controller 2 transmits the music number input in the text box to the application server 1, and the application server 1 executes the karaoke application program.

  FIG. 11A is a diagram illustrating an example of a UI definition corresponding to an operation screen of an application program for a rhythm game when the integrated client 4 is a television receiver. FIG. 11B illustrates an example of an appearance and a display screen of a television receiver. In FIG. 11A, the display definition section describes a text tag and the like for displaying a session number. In the case of the UI definition shown in FIG. 11A, a screen 272 as shown in FIG. 11B is displayed on the display unit 27 of the television receiver. As shown in FIG. 11B, a session number and the like are displayed.

  In FIG. 11A, the input definition section describes color-key tags corresponding to the blue, red, and green buttons. The behavior tag corresponding to the color-key tag indicates that the controller 2 transmits color information corresponding to the pressed button to the application server 1, and the application server 1 executes the application program of the rhythm game.

[1-6. Operation of Communication System S1]
Next, the operation of the communication system S1 of the present embodiment will be described with reference to FIGS. FIG. 12 is a sequence diagram illustrating an example of a processing outline of the communication system S1.

  As shown in FIG. 12, the user performs an operation for starting the controller program on the controller 2 (step S1). Then, the controller 2 transmits an application selection UI definition request to the application server 1 (step S2). The application selection UI definition request includes a controller attribute stored in the storage unit 24 of the controller 2. Upon receiving the application selection UI definition request, the application server 1 transmits a UI definition for application program selection corresponding to the controller attribute included in the application selection UI definition request to the controller 2 (step S3).

  The controller 2 that has received the UI definition stores the UI definition in the RAM 23 and displays an operation screen for selecting an application program on the display unit 27 based on the UI definition (step S4). Specifically, the controller 2 causes the display unit 27 to display an operation screen having the content indicated by the display definition unit included in the UI definition.

  It is assumed that the user performs an operation of selecting, for example, a karaoke application program from the list of application programs displayed on the operation screen (step S5). Then, the controller 2 transmits an application activation request to the application server 1 (step S6). The application activation request is an execution request indicating a request for activation of the application program. The application activation request in this case includes the application ID of the karaoke application program and the controller attribute of the controller 2.

  Upon receiving the application activation request, the application server 1 activates a karaoke application program from the application DB 5 (step S7). At this time, the application server 1 stores the controller attribute included in the application activation request in the RAM 13 as information inside the karaoke application program. The internal state of the karaoke application program at the time of activation is, for example, a song selection waiting state. Next, the application server 1 transmits a UI definition switching instruction to the controller 2 (step S8). The switching instruction in this case is a UI definition corresponding to the controller attribute included in the application activation request among the UI definitions corresponding to the karaoke application program, and includes a UI definition corresponding to the activation of the karaoke application program. The UI definition corresponding to the activation of the karaoke application program is, for example, a UI definition for karaoke music selection. The switching instruction further includes a session number. The application server 1 generates a new session number for the activated application program.

  The controller 2 that has received the switching instruction displays an operation screen for karaoke music selection on the display unit 27 and displays the session number included in the switching instruction on the display unit 27 based on the UI definition included in the switching instruction. (Step S9).

  Thereafter, the user performs an activation operation on the renderer 3 (step S10). Next, the user inputs the session number displayed on the display unit 27 of the controller 2 to the renderer 3 (step S11). Then, the renderer 3 transmits a session connection request to the application server 1 (step S12). The session connection request includes the input session number, the IP address of the renderer 3 and the port number. Upon receiving the session connection request, the application server 1 transmits a connection request to the active karaoke application program that is a process corresponding to the session number included in the session connection request (step S13). The connection request includes the IP address and port number of the renderer 3. The application server 1 that executes the application program that has received the connection request performs streaming transmission of the video of the karaoke screen to the renderer 3 specified by the IP address and port number included in the connection request (step S14). The renderer 3 displays a karaoke screen based on the received video (step S15).

  The user operates the input unit 26 of the controller 2 to input a music number. The controller 2 determines local processing information as processing information indicating processing corresponding to the operation of inputting a music number from the UI definition for music selection. Accordingly, the controller 2 adds a number to the text box and updates the operation screen for music selection as a process indicating the local process information. Thereafter, the user performs an operation of pressing the music selection button (step S16). The controller determines remote processing information as processing information indicating processing corresponding to an operation of pressing the music selection button from the UI definition for music selection. Therefore, the controller 2 transmits a karaoke performance request to the application server 1 (step S17). The karaoke performance request is an execution request indicating a karaoke performance request. The karaoke performance request includes the session number and the input music number. The application server 1 transfers the karaoke performance request to the karaoke application program corresponding to the session number included in the karaoke performance request. The application server 1 that executes the karaoke application program that has received the karaoke performance request switches between video and audio to be streamed to the renderer 3. Specifically, the application server 1 stream-transmits the karaoke content to the renderer 3 while generating karaoke content corresponding to the music number included in the karaoke performance request. When the music number is received, the internal state of the karaoke content changes, for example, to a playing state. Also, the application server 1 transmits to the controller 2 a switching instruction including a UI definition corresponding to a controller attribute as internal information of a running karaoke application program among UI definitions for playing karaoke (step S1). S18). The controller 2 that has received the switching instruction causes the display unit 27 to display an operation screen for karaoke performance based on the UI definition included in the switching instruction (step S19).

  FIG. 13 is a flowchart illustrating a processing example of the application server 1. For example, when the application server 1 is activated and the application server 1 activates the application server program, the CPU 11 of the application server 1 executes the process shown in FIG.

  As illustrated in FIG. 13, the CPU 11 determines whether or not the application server 1 is stopped (step S21). At this time, if the CPU 11 determines that the application server 1 does not stop (step S21: NO), the CPU 11 proceeds to step S22. In step S <b> 22, the CPU 11 determines whether an application selection UI definition request is received from the controller 2. At this time, if the CPU 11 determines that an application selection UI definition request has not been received (step S22: NO), the process proceeds to step S24. On the other hand, if the CPU 11 determines that an application selection UI definition request has been received (step S22: YES), the CPU 11 proceeds to step S23.

  In step S23, the CPU 11 acquires from the UI definition DB 6 a UI definition for selecting an application program corresponding to the controller attribute included in the application selection UI definition request. Then, the CPU 11 transmits the acquired UI definition to the controller 2 and proceeds to step S24.

  In step S <b> 24, the CPU 11 determines whether an application activation request is received from the controller 2. At this time, if the CPU 11 determines that an application activation request has not been received (step S24: NO), the CPU 11 proceeds to step S27. On the other hand, if the CPU 11 determines that an application activation request has been received (step S24: YES), the CPU 11 proceeds to step S25.

  In step S25, the CPU 11 executes an application program corresponding to the application ID included in the application activation request from the application DB 5. In addition, the CPU 11 generates a session number. Next, the CPU 11 acquires from the UI definition DB 6 a UI definition corresponding to a combination of the application ID and controller attribute included in the application activation request and the internal state at the time of application program activation. Then, the CPU 11 transmits a switching instruction including the UI definition and the controller attribute to the controller 2 (step S26), and proceeds to step S27.

  In step S <b> 27, the CPU 11 determines whether a session connection request is received from the controller 2. At this time, if the CPU 11 determines that a session connection request has not been received (step S27: NO), the process proceeds to step S21. On the other hand, when it is determined that the session connection request has been received (step S27: YES), the CPU 11 proceeds to step S28. In step S28, the CPU 11 acquires the IP address and port number of the renderer 3 from the session connection request. Then, the CPU 11 transmits a connection request including the acquired IP address and port number to the application program, and proceeds to step S21. In step S21, when the CPU 11 determines that the application server 1 is to be stopped (step S21: YES), the CPU 11 ends the processing illustrated in FIG.

  FIG. 14 is a flowchart illustrating a processing example of the controller 2. For example, when the controller 2 starts the controller program, the CPU 21 of the controller 2 executes the process shown in FIG.

  In FIG. 14, the CPU 21 acquires a UI definition for selecting an application program from the application server 1 by transmitting an application selection UI definition request including a controller attribute stored in the storage unit 24 to the application server 1. (Step S31). Next, the CPU 21 stores the UI definition in the RAM 23, and displays an operation screen on the display unit 27 based on the stored UI definition (step S32). For example, when the UI definition includes a title start tag and a title end tag, the CPU 21 displays a title surrounded by these tags on the display unit 27. For example, when the UI definition includes a select-box tag, the CPU 21 acquires a plurality of application names from the file indicated by the src attribute of the select-box tag. Then, the CPU 21 displays a plurality of buttons including each of a plurality of application names on the display unit 27. For example, when the UI definition includes a select-box start tag and a select-box end tag, the CPU 21 acquires one or more item tags surrounded by these tags from the UI definition. Then, for each item tag, the CPU 21 displays a button including text indicated by the src attribute of the item tag on the display unit 27. For example, when the UI definition includes a text-box tag, the CPU 21 displays a text box on the display unit 27. For example, when the UI definition includes a text start tag and a text end tag, the CPU 21 displays the contents surrounded by these tags on the display unit 27. Next, the CPU 21 determines whether or not a user operation has been input from the input unit 26 (step S33). At this time, if the CPU 21 determines that no user operation has been input (step S33: NO), the CPU 21 proceeds to step S38. On the other hand, if the CPU 21 determines that an input of a user operation has been performed (step S33: YES), the CPU 21 proceeds to step S34.

  In step S <b> 34, the CPU 21 generates operation information indicating the input user operation, and acquires processing information corresponding to the operation information from the RAM 23. Then, the CPU 21 determines whether or not the acquired processing information is remote processing information. At this time, if the CPU 21 determines that the acquired process information is not remote process information (step S34: NO), the process proceeds to step S35. In step S35, the CPU 21 executes the process indicated by the local process information acquired as the process information, and changes the display of the operation screen. Next, the CPU 21 proceeds to step S38. On the other hand, if the CPU 21 determines that the acquired processing information is remote processing information (step S34: YES), the CPU 21 proceeds to step S36.

  In step S36, the CPU 21 acquires an application ID from the acquired remote processing information. Moreover, CPU21 acquires a parameter as needed based on remote processing information. Then, the CPU 21 transmits an execution request including the application ID, the parameter, and the controller attribute stored in the storage unit 24 to the application server 1. Next, the CPU 21 determines whether or not a switching instruction including a UI definition has been received from the application server 1 in accordance with the transmitted execution request (step S37). At this time, if the CPU 21 determines that the switching instruction has not been received (step S37: NO), the CPU 21 proceeds to step S38. On the other hand, if the CPU 21 determines that the switching instruction has been received (step S37: YES), the CPU 21 proceeds to step S32. In step S <b> 32, the CPU 21 switches the UI definition by overwriting the UI definition stored in the RAM 23 with the UI definition included in the switching instruction.

  In step S38, the CPU 21 determines whether or not the controller program ends. At this time, if the CPU 21 determines that the controller program does not end (step S38: NO), the CPU 21 proceeds to step S33. On the other hand, when it is determined that the controller program ends (step S38: YES), the CPU 21 ends the process shown in FIG.

[2. Second Embodiment]
[2-1. Configuration of communication system]
FIG. 15A is a diagram illustrating a schematic configuration example of the communication system S2 of the present embodiment. In FIG. 15A, elements similar to those in FIG. As shown in FIG. 15A, the communication system S2 includes an application server 1, a UI definition server 7, a plurality of controllers 2, and a plurality of renderers 3. The communication system S2 may further include an integrated client 4. The UI definition server 7 is an example of a second server device according to the present invention. The application server 1, UI definition server 7, controller 2, and renderer 3 are each connected to the network 10. The application server 1 is configured to be accessible to the application DB 5.

  The UI definition server 7 transmits the UI definition to the controller 2. The UI definition server 7 is configured to be accessible to the UI definition DB 6. For example, the UI definition server 7 and the UI definition DB 6 may be connected by a network such as a LAN. Alternatively, the UI definition server 7 may include a UI definition DB 6, for example. According to the communication system S2, application programs and UI definitions can be managed by separate server devices.

[2-2. Configuration of UI definition server 7]
Next, the configuration of the UI definition server 7 will be described with reference to FIG. FIG. 15B is a diagram illustrating a schematic configuration example of the UI definition server 7 of the present embodiment. As illustrated in FIG. 15B, the UI definition server 7 includes a CPU 71, a ROM 72, a RAM 73, a storage unit 74, a communication unit 75, and the like. These components are connected to the bus 76. The communication unit 75 is connected to the network 10. The storage unit 74 is configured by, for example, a hard disk drive. The UI definition DB 6 may be stored in the storage unit 74. The storage unit 74 stores an OS, a UI definition server program, and the like. The UI definition server program is a program that causes the CPU 71 as a computer to execute processing such as transmission of a UI definition. The UI definition server program or the like may be downloaded from another server or the like via the network 10 or the like, for example. Alternatively, the UI definition server program or the like may be recorded on a recording medium such as an optical disk, a magnetic tape, or a memory card and read into the storage unit 74 via a drive device.

[2-3. Operation of Communication System S2]
Next, the operation of the communication system S2 of the present embodiment will be described with reference to FIG. FIG. 16 is a sequence diagram illustrating an example of a processing outline of the communication system S2. In FIG. 16, processes similar to those in FIG. 12 are given the same reference numerals.

  As shown in FIG. 16, first, steps S1 to S7 are executed. Next, the application server 1 is a UI definition corresponding to the controller attribute included in the application activation request among the UI definitions corresponding to the activated karaoke application program, and the UI definition corresponding to the activation of the karaoke application program. UI definition position information is generated. Then, the application server 1 transmits a switching instruction including UI definition position information and a session number to the controller 2 (step S41). The controller 2 that has received the switching instruction acquires the UI definition position information from the switching instruction, and transmits a UI definition request including the UI definition position information to the UI definition server 7 (step S42). The UI definition server 7 that has received the UI definition request acquires from the UI definition DB 6 a UI definition for karaoke music selection corresponding to the UI definition position information included in the UI definition request. Then, the UI definition server 7 transmits the UI definition to the controller 2 (step S43). The controller 2 that has received the UI definition stores the UI definition in the RAM 23. Then, the controller 2 displays an operation screen for karaoke music selection on the display unit 27 based on the UI definition stored in the RAM 23 (step S9).

  Next, steps S10 to S17 are executed. Next, the application server 1 that executes the activated karaoke application program includes a UI definition corresponding to a controller attribute as internal information of the activated karaoke application program among the UI definitions for performing the karaoke performance. A switching instruction including the position information is transmitted to the controller 2 (step S44). The controller 2 that has received the switching instruction acquires the UI definition position information from the switching instruction, and transmits a UI definition request including the UI definition position information to the UI definition server 7 (step S45). The UI definition server 7 that has received the UI definition request acquires from the UI definition DB 6 a UI definition for karaoke performance corresponding to the UI definition position information included in the UI definition request. Then, the UI definition server 7 transmits the UI definition to the controller 2 (step S46). The controller 2 that has received the UI definition stores the UI definition in the RAM 23. Then, based on the UI definition stored in the RAM 23, the controller 2 causes the display unit 27 to display an operation screen for karaoke performance (step S19).

  The controller 2 may store the UI definition received from the UI definition server 7 in the RAM 23 and store the UI definition in the storage unit 24. At this time, the controller 2 may store the UI definition in the storage unit 24 in association with the UI definition position information or the like. The controller 2 that has received the switching instruction from the application server 1 stores the UI definition stored in the storage unit 24 when the UI definition corresponding to the UI definition position information included in the switching instruction is stored in the storage unit 24. May be stored in the RAM 23. Then, the controller 2 may switch the UI definition used for determining the processing information to the UI definition newly stored in the RAM 23. In this case, the controller 2 does not need to acquire a UI definition from the UI definition server 7.

[3. Third Embodiment]
The communication system according to the third embodiment described below may be, for example, the communication system S1 or the communication system S2. In any case of the communication systems S1 and S2, the application server 1 transmits to the controller 2 a switching instruction including UI definition position information of the UI definition of the switching destination.

  The controller 2 stores a plurality of UI definitions in the storage unit 24. The plurality of UI definitions stored in the storage unit 24 are UI definitions corresponding to the controller attributes stored in the storage unit 24. For example, a plurality of UI definitions may be recorded on a recording medium such as an optical disk, a magnetic tape, or a memory card and read into the storage unit 24 via a drive device. Alternatively, a plurality of UI definitions may be stored in the storage unit 24 in advance when the controller 2 is shipped. Alternatively, the controller 2 may acquire all UI definitions corresponding to the controller attributes from the application server 1 or the UI definition server 7 and store them in the storage unit 24.

  When the controller 2 receives the switching instruction from the application server, the controller 2 reads out the UI definition corresponding to the UI definition position information included in the switching instruction from the storage unit 24. Then, the controller 2 stores the read UI definition in the RAM 23. When any UI definition is already stored in the RAM 23, the controller 2 overwrites the UI definition stored in the RAM 23 with the UI definition read from the storage unit 24. Thereby, the controller 2 switches the UI definition for determining the process corresponding to the user operation. For this reason, the controller 2 does not need to acquire the switching destination UI definition from the UI definition server 7 or the application server 1 when switching the UI definition.

  In the UI definition DB 6, a new UI definition may be registered or the UI definition may be updated. Even if the controller 2 periodically transmits a UI definition update status query to the application server 1 or the UI definition server 7 connected to the UI definition DB 6 in order to store the new or updated UI definition in the storage unit 24. Good. The UI definition update status query is an example of query information of the present invention. Then, the controller 2 updates the UI definition stored in the storage unit 24 with the UI definition acquired from the application server 1 or the UI definition server 7. Thus, the updated UI definition can be stored in the storage unit 24 before the UI definition is required.

  FIG. 17 is a flowchart illustrating an example of UI definition update processing in the controller 2. As shown in FIG. 17, the CPU 21 determines whether or not it is time to update the UI definition (step S51). For example, the predetermined time of the day may be a time for updating the UI definition. Alternatively, for example, a predetermined time on a predetermined day of the week may be a time for updating the UI definition. When the CPU 21 determines that the time for updating the UI definition has not arrived (step S51: NO), the CPU 21 proceeds to step S56. On the other hand, if the CPU 21 determines that the time for updating the UI definition has arrived (step S51: YES), the CPU 21 proceeds to step S52.

  In step S52, the CPU 21 transmits a UI definition update status query to the UI definition server 7 (or application server 1). The UI definition update status inquiry may include, for example, a UI definition location information and last update date / time list of the UI definition stored in the storage unit 24 and a controller attribute. Based on the list included in the UI definition update status query, the UI definition server 7 selects the UI definition corresponding to the controller attribute included in the UI definition update status query among the UI definitions stored in the UI definition DB 6. UI definition position information of a UI definition that is newer than the UI definition that is not stored in the controller 2 and the UI definition that is stored in the controller 2 is acquired. Then, the UI definition server 7 transmits an inquiry response including the acquired UI definition position information to the controller 2.

  The CPU 21 determines whether there is a UI definition update based on the received inquiry response (step S53). At this time, when the UI definition position information is not included in the inquiry response, the CPU 21 determines that the UI definition has not been updated (step S53: NO). In this case, the CPU 21 proceeds to step S56. On the other hand, when the UI definition position information is included in the inquiry response, the CPU 21 determines that the UI definition has been updated (step S53: YES). In this case, the CPU 21 proceeds to step S54.

  In step S <b> 54, the CPU 21 acquires UI definition position information from the inquiry response, and transmits a UI definition request including the UI definition position information to the UI definition server 7. As a result, the CPU 21 acquires the UI definition from the UI definition server 7. Next, the CPU 21 updates the UI definition stored in the storage unit 24 (step S55). Specifically, the CPU 21 stores the UI definition acquired from the UI definition server 7 in the storage unit 24. At this time, if the UI definition of the same UI definition position information as the UI definition position information of the UI definition acquired from the UI definition server 7 is already stored in the storage unit 24, the CPU 21 is already stored in the storage unit 24. The existing UI definition is overwritten with the acquired UI definition. Next, the CPU 21 proceeds to step S56. In step S56, the CPU 21 determines whether or not the controller 2 is stopped. At this time, if the CPU 21 determines that the controller 2 does not stop (step S56: NO), the CPU 21 proceeds to step S51. On the other hand, when it is determined that the controller 2 is stopped (step S56: YES), the CPU 21 ends the UI definition update process.

  As described above, according to each of the embodiments described above, the controller 2 does not transmit a request to the application server 1 for the processes executed by the controller 2 among the processes corresponding to the user operation. Therefore, the response to the user operation can be shortened.

  Note that the process executed by the controller 2 is at least one of scrolling of the operation screen and movement of an indicia indicating the user operation target in the operation screen, and the process executed by the application server 1 is the operation screen. All the processes other than scrolling and moving the indicia indicating the user operation target in the operation screen may be performed. That is, in the UI definition, the process information indicating the scroll of the operation screen or the movement of the indicia indicating the user operation target in the operation screen is described only as the local process information, and the process information indicating processes other than these processes is It may be described only as remote processing information.

  For example, a plurality of areas for storing the UI definition may be set in the RAM 23. The controller 2 may store different UI definitions in each of the plurality of areas. In this case, for example, the RAM 23 may store a UI-defined address to be used. The controller 2 may display the operation screen and determine the processing information using the UI definition stored in the area indicated by this address. The controller 2 may switch the UI definition used for displaying the operation screen and determining the processing information by changing the UI definition address.

1 Application Server 2 Controller 3 Renderer 4 Integrated Client 5 Application DB
6 UI definition DB
7 UI definition server 10 Network 11, 21, 71 CPU
12, 22, 72 ROM
13, 23, 73 RAM
14, 24, 74 Storage unit 15, 25, 75 Communication unit 26 Input unit 27 Display unit S1, S2 Communication system

Claims (24)

Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the system,
A terminal device that requests the server device to execute the application program via the network;
The server device;
A second server device;
The output device;
With
The terminal device
Definition information indicating the definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Display control means for displaying on the display means an operation screen corresponding to the application program based on the definition information stored in the temporary storage means;
An input means for inputting a user operation;
The first process information or the second process information is determined as the process information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit. A determination means;
A process execution unit that executes a process indicated by the first process information when the first process information is determined by the determination unit;
Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining means;
Switching information transmitted from the server device in response to the application program being executed by the server device on the basis of the request information transmitted by the request information transmitting means, the switching relating to the definition information of the switching destination Switching information receiving means for receiving definition information identification information for identifying the switching destination definition information as information ;
Definition information identification information transmission means for transmitting the definition information identification information received by the switching information reception means to the second server device;
Definition information receiving means for receiving the definition information transmitted from the second server device;
Switching the definition information received by the definition information receiving means to be stored in the temporary storage means, and switching the definition information used for determination of processing information by the determination means to the definition information stored in the temporary storage means Means,
With
The server device
Program storage means for storing the application program;
Request information receiving means for receiving the request information from the terminal device;
Program execution means for executing the application program stored in the program storage means based on the request information received by the request information receiving means;
Switching information transmitting means for transmitting, as the switching information , the definition information identification information for identifying the switching destination definition information according to the execution of the application program by the program execution means;
Generating means for generating at least one of video data and audio data indicating an execution result of the application program by the program executing means;
Transmitting means for transmitting at least one of the video data and the audio data generated by the generating means to the output device;
With
The second server device is
A definition information storage means for storing a plurality of the definition information and definition information identification information for identifying the definition information in association with each other;
Definition information identification information receiving means for receiving the definition information identification information transmitted from the terminal device;
Definition information transmission means for transmitting, to the terminal device, definition information corresponding to the definition information identification information received by the definition information identification information reception means among the plurality of definition information stored in the definition information storage means. When,
With
The output device is
Receiving means for receiving at least one of the video data and audio data transmitted from the server device;
Output control means for causing the output means to output an execution result indicated by at least one of the video data and audio data received by the receiving means;
A communication system comprising: The communication system according to claim 1,
The terminal device
A nonvolatile storage means for storing a plurality of the definition information;
When the definition information of the switching destination is stored in the nonvolatile storage unit, the switching unit stores the switching destination definition information stored in the nonvolatile storage unit in the temporary storage unit, and determines the determination. A communication system characterized by switching definition information used for determining processing information by means to the definition information stored in the temporary storage means. The communication system according to claim 2,
The second server device is
Inquiry information receiving means for receiving inquiry information indicating an inquiry of the definition information from the terminal device;
When the inquiry information is received by the inquiry information receiving means, out of the plurality of definition information stored in the definition information storage means, definition information that is newer than the definition information stored in the terminal device is used as the terminal. Second definition information transmitting means for transmitting to the device;
Further comprising
The terminal device
Inquiry information transmitting means for periodically transmitting the inquiry information to the second server device;
Second definition information receiving means for receiving the definition information transmitted in response to the inquiry information from the second server device;
Updating means for updating the definition information stored in the nonvolatile storage means with the definition information received by the second definition information receiving means;
A communication system, further comprising: Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the system,
A terminal device that requests the server device to execute the application program via the network;
The server device;
A second server device;
The output device;
With
The terminal device
Definition information indicating the definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Non-volatile storage means for storing a plurality of the definition information;
Display control means for displaying on the display means an operation screen corresponding to the application program based on the definition information stored in the temporary storage means;
An input means for inputting a user operation;
The first process information or the second process information is determined as the process information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit. A determination means;
A process execution unit that executes a process indicated by the first process information when the first process information is determined by the determination unit;
Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining means;
Switching information transmitted from the server device in response to the application program being executed by the server device on the basis of the request information transmitted by the request information transmitting means, the switching relating to the definition information of the switching destination Switching information receiving means for receiving information;
Switching means for switching definition information used for determining processing information by the determining means based on the switching information received by the switching information receiving means, wherein the definition information of the switching destination is stored in the nonvolatile storage means. If stored, the switching destination definition information stored in the non-volatile storage unit is stored in the temporary storage unit, and the definition information used for determining the processing information by the determination unit is stored in the temporary storage unit. Switching means for switching to the stored definition information;
Inquiry information transmitting means for periodically transmitting inquiry information indicating an inquiry of the definition information to the second server device;
Second definition information receiving means for receiving the definition information transmitted from the second server device;
Updating means for updating the definition information stored in the nonvolatile storage means with the definition information received by the second definition information receiving means;
With
The server device
Program storage means for storing the application program;
Request information receiving means for receiving the request information from the terminal device;
Program execution means for executing the application program stored in the program storage means based on the request information received by the request information receiving means;
Switching information transmitting means for transmitting the switching information to the terminal device in response to execution of the application program by the program executing means;
Generating means for generating at least one of video data and audio data indicating an execution result of the application program by the program executing means;
Transmitting means for transmitting at least one of the video data and the audio data generated by the generating means to the output device;
With
The second server device is
Definition information storage means for storing a plurality of the definition information;
Inquiry information receiving means for receiving the inquiry information from the terminal device;
When the inquiry information is received by the inquiry information receiving means, out of the plurality of definition information stored in the definition information storage means, definition information that is newer than the definition information stored in the terminal device is used as the terminal. Second definition information transmitting means for transmitting to the device;
With
The output device is
Receiving means for receiving at least one of the video data and audio data transmitted from the server device;
Output control means for causing the output means to output an execution result indicated by at least one of the video data and audio data received by the receiving means;
A communication system comprising: The communication system according to claim 4,
The second server device is
Further comprising definition information transmission means for transmitting the definition information stored in the definition information storage means to the terminal device,
The terminal device
A definition information receiving means for receiving the definition information transmitted by the definition information transmitting means of the second server device;
The switching means stores the definition information received by the definition information receiving means in the temporary storage means, and the definition information used for determining processing information by the determination means is stored in the temporary storage means. A communication system characterized by switching to definition information. The communication system according to claim 4,
The server device
Further comprising definition information storage means for storing a plurality of the definition information,
The switching information transmission means transmits the switching destination definition information according to the execution of the application program by the program execution means as the switching information among the plurality of definition information stored in the definition information storage means,
The switching means stores the definition information received by the switching information receiving means in the temporary storage means, and the definition information used for determining processing information by the determination means is stored in the temporary storage means. A communication system characterized by switching to definition information. The communication system according to any one of claims 1 to 6,
The definition information includes the first processing information indicating processing including any of scrolling of the operation screen displayed on the display unit and movement of an indicia indicating a user operation target in the operation screen; and the server And a second communication information indicating a process involving switching of at least one of the video data and the audio data transmitted from the apparatus to the output apparatus. Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the system,
A terminal device that requests the server device to execute the application program via the network;
The server device;
The output device;
With
The terminal device
Definition information indicating the definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Display control means for displaying on the display means an operation screen corresponding to the application program based on the definition information stored in the temporary storage means;
An input means for inputting a user operation;
The first process information or the second process information is determined as the process information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit. A determination means;
A process execution unit that executes a process indicated by the first process information when the first process information is determined by the determination unit;
Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining means;
Switching information transmitted from the server device in response to the application program being executed by the server device on the basis of the request information transmitted by the request information transmitting means, the switching relating to the definition information of the switching destination Switching information receiving means for receiving information;
Based on the switching information received by the switching information receiving means, switching means for switching definition information used for determining processing information by the determining means;
With
The server device
Program storage means for storing the application program;
Request information receiving means for receiving the request information from the terminal device;
Program execution means for executing the application program stored in the program storage means based on the request information received by the request information receiving means;
Switching information transmitting means for transmitting the switching information to the terminal device in response to execution of the application program by the program executing means;
Generating means for generating at least one of video data and audio data indicating an execution result of the application program by the program executing means;
Transmitting means for transmitting at least one of the video data and the audio data generated by the generating means to the output device;
With
The output device is
Receiving means for receiving at least one of the video data and audio data transmitted from the server device;
Output control means for causing the output means to output an execution result indicated by at least one of the video data and audio data received by the receiving means;
With
The definition information includes the first processing information indicating processing including any of scrolling of the operation screen displayed on the display unit and movement of an indicia indicating a user operation target in the operation screen; and the server And a second communication information indicating a process involving switching of at least one of the video data and the audio data transmitted from the apparatus to the output apparatus. The communication system according to claim 8,
Definition information storage means for storing a plurality of the definition information;
Definition information transmission means for transmitting the definition information stored in the definition information storage means to the terminal device;
A second server device comprising:
The terminal device
A definition information receiving means for receiving the definition information transmitted from the second server device;
The switching means stores the definition information received by the definition information receiving means in the temporary storage means, and the definition information used for determining processing information by the determination means is stored in the temporary storage means. A communication system characterized by switching to definition information. The communication system according to claim 8,
The server device
Further comprising definition information storage means for storing a plurality of the definition information,
The switching information transmission means transmits the switching destination definition information according to the execution of the application program by the program execution means as the switching information among the plurality of definition information stored in the definition information storage means,
The terminal device
The switching means stores the definition information received by the switching information receiving means in the temporary storage means, and the definition information used for determining processing information by the determination means is stored in the temporary storage means. A communication system characterized by switching to definition information. The communication system according to any one of claims 8 to 10,
The terminal device
A nonvolatile storage means for storing a plurality of the definition information;
When the definition information of the switching destination is stored in the nonvolatile storage unit, the switching unit stores the switching destination definition information stored in the nonvolatile storage unit in the temporary storage unit, and determines the determination. A communication system characterized by switching definition information used for determining processing information by means to the definition information stored in the temporary storage means. The communication system according to any one of claims 1 to 11,
The request information transmission means transmits the request information including application program identification information for identifying the application program to be activated when the second processing information indicates activation of the application program,
The program storage means stores a plurality of application programs and application program identification information for identifying the application programs in association with each other,
The program execution means activates an application program corresponding to the application program identification information included in the request information among a plurality of application programs stored in the program storage means,
The switching information transmitting unit transmits the switching information related to the switching destination definition information corresponding to the activated application program when the application program is activated by the program executing unit. . The communication system according to any one of claims 1 to 11,
The switching information transmission unit includes a plurality of definitions corresponding to the running application program when an internal state of the running application program changes as a result of the running application program being executed by the program execution unit. A communication system, wherein the switching information related to the switching destination definition information corresponding to the changed internal state is transmitted. The communication system according to any one of claims 1 to 13,
The terminal device
Attribute information storage means for storing attribute information indicating the attribute of the terminal device;
The switching means switches to definition information corresponding to an attribute indicated by the attribute information stored in the attribute information storage means. Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the terminal device included in the system,
Definition information indicating the definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Display control means for displaying on the display means an operation screen corresponding to the application program based on the definition information stored in the temporary storage means;
An input means for inputting a user operation;
Determination to determine the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit Means,
A process execution unit that executes a process indicated by the first process information when the first process information is determined by the determination unit;
Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining means;
Switching information transmitted from the server device in response to the application program being executed by the server device on the basis of the request information transmitted by the request information transmitting means, the switching relating to the definition information of the switching destination As information, switching information receiving means for receiving definition information identification information for identifying definition information of the switching destination according to execution of the application program;
To a second server device comprising definition information storage means for storing a plurality of the definition information identification information received by the switching information reception means in association with the definition information and definition information identification information for identifying the definition information Definition information identification information transmitting means for transmitting;
Definition information receiving means for receiving definition information corresponding to the definition information identification information from among the plurality of definition information stored in the definition information storage means from the second server device that has received the definition information identification information. When,
Switching the definition information received by the definition information receiving means to be stored in the temporary storage means, and switching the definition information used for determination of processing information by the determination means to the definition information stored in the temporary storage means Means,
A terminal device comprising: Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the terminal device included in the system,
Definition information indicating the definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Non-volatile storage means for storing a plurality of the definition information;
Display control means for displaying on the display means an operation screen corresponding to the application program based on the definition information stored in the temporary storage means;
An input means for inputting a user operation;
Determination to determine the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit Means,
A process execution unit that executes a process indicated by the first process information when the first process information is determined by the determination unit;
Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining means;
Switching information transmitted from the server device in response to the application program being executed by the server device on the basis of the request information transmitted by the request information transmitting means, the switching relating to the definition information of the switching destination Switching information receiving means for receiving information;
Switching means for switching definition information used for determining processing information by the determining means based on the switching information received by the switching information receiving means, wherein the definition information of the switching destination is stored in the nonvolatile storage means. If stored, the switching destination definition information stored in the non-volatile storage unit is stored in the temporary storage unit, and the definition information used for determining the processing information by the determination unit is stored in the temporary storage unit. Switching means for switching to the stored definition information;
Inquiry information transmission means for periodically sending inquiry information indicating the inquiry of the definition information to a second server device comprising definition information storage means for storing a plurality of the definition information;
Second definition information received from the second server device that has received the inquiry information, the definition information that is newer than the definition information stored in the terminal device among the plurality of definition information stored in the definition information storage means; Definition information receiving means;
Updating means for updating the definition information stored in the nonvolatile storage means with the definition information received by the second definition information receiving means;
A terminal device comprising: Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the terminal device included in the system,
Definition information indicating the definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Display control means for displaying on the display means an operation screen corresponding to the application program based on the definition information stored in the temporary storage means;
An input means for inputting a user operation;
Determination to determine the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit Means,
A process execution unit that executes a process indicated by the first process information when the first process information is determined by the determination unit;
Request information transmitting means for transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining means;
Switching information transmitted from the server device in response to the application program being executed by the server device on the basis of the request information transmitted by the request information transmitting means, the switching relating to the definition information of the switching destination Switching information receiving means for receiving information;
Based on the switching information received by the switching information receiving means, switching means for switching definition information used for determining processing information by the determining means;
With
The definition information includes the first processing information indicating processing including any of scrolling of the operation screen displayed on the display unit and movement of an indicia indicating a user operation target in the operation screen; and the server And a second processing information indicating processing involving switching of at least one of the video data and the audio data transmitted from the device to the output device. Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In a program executed by a terminal device included in the system,
Definition information indicating a definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
An input means for inputting a user operation;
A computer of the terminal device comprising:
Based on the definition information stored in the temporary storage means, a display control step for causing the display means to display an operation screen corresponding to the application program;
Determination to determine the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit Steps,
A process execution step for executing a process indicated by the first process information when the first process information is determined by the determination step;
A request information transmission step of transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining step;
Switching information transmitted from the server device in response to the application program being executed by the server device based on the request information transmitted in the request information transmission step, the switching relating to the definition information of the switching destination As information, a switching information receiving step for receiving definition information identification information for identifying the definition information of the switching destination according to the execution of the application program;
To a second server device comprising definition information storage means for storing a plurality of the definition information identification information received in the switching information reception step in association with the definition information and definition information identification information for identifying the definition information A definition information identification information transmission step to be transmitted;
A definition information receiving step of receiving definition information corresponding to the definition information identification information from among the plurality of definition information stored in the definition information storage means from the second server device that has received the definition information identification information. When,
Switching the definition information received in the definition information receiving step in the temporary storage means, and switching the definition information used for determining the processing information in the determination step to the definition information stored in the temporary storage means Steps,
A program characterized by having executed. Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In a program executed by a terminal device included in the system,
Definition information indicating a definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
Non-volatile storage means for storing a plurality of the definition information;
An input means for inputting a user operation;
A computer of the terminal device comprising:
Based on the definition information stored in the temporary storage means, a display control step for causing the display means to display an operation screen corresponding to the application program;
Determination to determine the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit Steps,
A process execution step for executing a process indicated by the first process information when the first process information is determined by the determination step;
A request information transmission step of transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining step;
Switching information transmitted from the server device in response to the application program being executed by the server device based on the request information transmitted in the request information transmission step, the switching relating to the definition information of the switching destination A switching information receiving step for receiving information;
Based on the switching information received in the switching information receiving step, the switching step switches definition information used for determining processing information in the determination step, and the switching destination definition information is stored in the nonvolatile storage means. If stored, the definition information of the switching destination stored in the non-volatile storage unit is stored in the temporary storage unit, and the definition information used for determining the processing information in the determination step is stored in the temporary storage unit. A switching step for switching to the stored definition information;
A query information transmission step of periodically transmitting query information indicating the query of the definition information to a second server device including a plurality of definition information storage units for storing the definition information;
Second definition information received from the second server device that has received the inquiry information, the definition information that is newer than the definition information stored in the terminal device among the plurality of definition information stored in the definition information storage means; A definition information receiving step;
An update step of updating the definition information stored in the nonvolatile storage means with the definition information received in the second definition information receiving step;
A program characterized by having executed. Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In a program executed by a terminal device included in the system,
Definition information indicating a definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Temporary storage means for storing definition information associated with any one of first processing information indicating processing executed by the terminal device and second processing information indicating execution of the application program by the server device;
An input means for inputting a user operation;
A computer of the terminal device comprising:
Based on the definition information stored in the temporary storage means, a display control step for causing the display means to display an operation screen corresponding to the application program;
Determination to determine the first processing information or the second processing information as processing information corresponding to operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit Steps,
A process execution step for executing a process indicated by the first process information when the first process information is determined by the determination step;
A request information transmission step of transmitting request information indicating a request for processing indicated by the second processing information to the server device when the second processing information is determined by the determining step;
Switching information transmitted from the server device in response to the application program being executed by the server device based on the request information transmitted in the request information transmission step, the switching relating to the definition information of the switching destination A switching information receiving step for receiving information;
A switching step of switching definition information used for determination of processing information by the determination step based on the switching information received by the switching information reception step;
And execute
The definition information includes the first processing information indicating processing including any of scrolling of the operation screen displayed on the display unit and movement of an indicia indicating a user operation target in the operation screen; and the server And a second process information indicating a process involving switching of at least one of the video data and the audio data transmitted from the apparatus to the output apparatus. Communication in which an output device receives at least one of the video data and the audio data transmitted via a network from a server device that generates at least one of video data and audio data indicating the execution result of the application program In the server device included in the system,
Program storage means for storing an application program executed in response to a request from the terminal device via the network;
Definition information indicating a definition of a user interface corresponding to the application program, operation information indicating the user operation for each predetermined user operation, and processing information indicating a process corresponding to the user operation, Either one of first processing information indicating processing executed by a terminal device that requests the server device to execute the application program via the network and second processing information indicating execution of the application program by the server device; Definition information storage means for storing a plurality of definition information associated with
The second processing information is determined as processing information corresponding to the operation information indicating the user operation input using the input unit provided in the terminal device from the definition information stored in the temporary storage unit provided in the terminal device. A request information receiving unit configured to receive request information indicating a request for processing indicated by the second processing information from the terminal device;
Program execution means for executing an application program stored in the program storage means based on the request information received by the request information receiving means;
Switching information related to definition information for switching the definition information used for determining the processing information corresponding to the operation information indicating the user operation input using the input unit according to the execution of the application program by the program execution unit. Switching information transmission means for transmitting to the terminal device, among the plurality of definition information stored in the definition information storage means, definition information according to execution of the application program is used as switching destination definition information Switching information transmitting means for transmitting the switching information including:
Generating means for generating at least one of video data and audio data indicating an execution result of the application program by the program executing means;
Transmitting means for transmitting at least one of the video data and the audio data generated by the generating means to the output device;
With
The definition information is an operation screen displayed on the display means included in the terminal device based on the definition information stored in the temporary storage means, and scrolls the operation screen corresponding to the application program, and At least one of the first processing information indicating processing including any movement of an indicia indicating a user operation target in the operation screen, and the video data and the audio data transmitted from the server device to the output device; The server apparatus characterized by including the said 2nd process information which shows the process accompanying any one switching. Information that the output device receives at least one of the video data and the audio data transmitted from the server device that generates at least one of the video data and the audio data indicating the execution result of the application program via the network In the processing method,
A terminal device that requests execution of the application program to the server device via the network, and is definition information indicating a definition of a user interface corresponding to the application program, and for each predetermined user operation, the user Operation information indicating an operation and process information indicating a process corresponding to the user operation, the first process information indicating a process executed by the terminal device, and a second indicating the execution of the application program by the server device A terminal device comprising temporary storage means for storing definition information associated with any one of the processing information and input means for inputting a user operation is stored in the definition information stored in the temporary storage means. Based on this, an operation screen corresponding to the application program is displayed on the display means. And a display control step,
The terminal device uses the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit. A decision step for determining processing information;
A process execution step of executing a process indicated by the first process information when the terminal apparatus determines the first process information by the determining step;
A request information transmission step for transmitting, to the server device, request information indicating a request for processing indicated by the second processing information when the terminal device determines the second processing information by the determining step;
A request information receiving step in which the server device receives the request information from the terminal device;
A program execution step in which the server device executes the application program stored in program storage means for storing the application program based on the request information received in the request information reception step;
Switching destination of definition information used by the server device to determine the processing information corresponding to operation information indicating a user operation input using the input unit in accordance with the execution of the application program in the program execution step. A switching information transmission step of transmitting definition information identification information for identifying the switching destination definition information according to the execution of the application program by the program execution step to the terminal device as the switching information related to the definition information;
A switching information receiving step in which the terminal device receives the definition information identification information transmitted from the server device;
The terminal device includes definition information storage means for storing a plurality of the definition information identification information received in the switching information reception step in association with the definition information and definition information identification information for identifying the definition information. A definition information identification information transmission step to be transmitted to the second server device;
A definition information identification information receiving step in which the second server device receives the definition information identification information transmitted from the terminal device;
Definition in which the second server device transmits definition information corresponding to the definition information identification information received in the definition information identification information reception step among the plurality of definition information stored in the definition information storage unit An information transmission step;
A definition information receiving step in which the terminal device receives the definition information transmitted from the second server device;
The terminal device stores the definition information received in the definition information receiving step in the temporary storage unit, and the definition information used for determining processing information in the determination step is stored in the temporary storage unit Switching step to switch to definition information;
A generating step in which the server device generates at least one of video data and audio data indicating an execution result of the application program in the program executing step;
A transmission step in which the server device transmits at least one of the video data and the audio data generated by the generation step to the output device;
A receiving step in which the output device receives at least one of the video data and audio data transmitted from the server device;
An output control step in which the output device causes the output means to output an execution result indicated by at least one of the video data and audio data received in the reception step;
An information processing method comprising: Information that the output device receives at least one of the video data and the audio data transmitted from the server device that generates at least one of the video data and the audio data indicating the execution result of the application program via the network In the processing method,
A terminal device that requests execution of the application program to the server device via the network, and is definition information indicating a definition of a user interface corresponding to the application program, and for each predetermined user operation, the user Operation information indicating an operation and process information indicating a process corresponding to the user operation, the first process information indicating a process executed by the terminal device, and a second indicating the execution of the application program by the server device A terminal device comprising: temporary storage means for storing definition information associated with any one of processing information; non-volatile storage means for storing a plurality of the definition information; and input means for inputting a user operation Is based on the definition information stored in the temporary storage means. A display control step of displaying on a display unit an operation screen corresponding to the program,
The terminal device uses the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit. A decision step for determining processing information;
A process execution step of executing a process indicated by the first process information when the terminal apparatus determines the first process information by the determining step;
A request information transmission step for transmitting, to the server device, request information indicating a request for processing indicated by the second processing information when the terminal device determines the second processing information by the determining step;
A request information receiving step in which the server device receives the request information from the terminal device;
A program execution step in which the server device executes the application program stored in program storage means for storing the application program based on the request information received in the request information reception step;
Switching destination of definition information used by the server device to determine the processing information corresponding to operation information indicating a user operation input using the input unit in accordance with the execution of the application program in the program execution step. A switching information transmission step of transmitting switching information related to the definition information to the terminal device;
A switching information receiving step in which the terminal device receives the switching information transmitted from the server device;
The terminal device is a switching step of switching definition information used for determination of processing information by the determination step based on the switching information received by the switching information reception step, and the definition information of the switching destination is the switching information When stored in the nonvolatile storage means, the definition information of the switching destination stored in the nonvolatile storage means is stored in the temporary storage means, and the definition information used for determining the processing information by the determination step is A switching step for switching to the definition information stored in the temporary storage means;
A generating step in which the server device generates at least one of video data and audio data indicating an execution result of the application program in the program executing step;
A transmission step in which the server device transmits at least one of the video data and the audio data generated by the generation step to the output device;
A receiving step in which the output device receives at least one of the video data and audio data transmitted from the server device;
An output control step in which the output device causes the output means to output an execution result indicated by at least one of the video data and audio data received in the reception step;
An inquiry information transmitting step in which the terminal device periodically transmits inquiry information indicating an inquiry of the definition information to a second server device including a plurality of definition information storage means for storing the definition information;
An inquiry information receiving step in which the second server device receives the inquiry information from the terminal device;
When the inquiry information is received by the inquiry information receiving step, the second server device uses the definition information stored in the terminal device among the plurality of definition information stored in the definition information storage means. A second definition information transmission step of transmitting new definition information to the terminal device;
A second definition information receiving step in which the terminal device receives the definition information transmitted from the second server device;
An update step in which the terminal device updates the definition information stored in the nonvolatile storage means with the definition information received in the second definition information receiving step;
An information processing method comprising: Information that the output device receives at least one of the video data and the audio data transmitted from the server device that generates at least one of the video data and the audio data indicating the execution result of the application program via the network In the processing method,
A terminal device that requests execution of the application program to the server device via the network, and is definition information indicating a definition of a user interface corresponding to the application program, and for each predetermined user operation, the user Operation information indicating an operation and process information indicating a process corresponding to the user operation, the first process information indicating a process executed by the terminal device, and a second indicating the execution of the application program by the server device A terminal device comprising temporary storage means for storing definition information associated with any one of the processing information and input means for inputting a user operation is stored in the definition information stored in the temporary storage means. Based on this, an operation screen corresponding to the application program is displayed on the display means. And a display control step,
The terminal device uses the first processing information or the second processing information as the processing information corresponding to the operation information indicating the user operation input using the input unit from the definition information stored in the temporary storage unit. A decision step for determining processing information;
A process execution step of executing a process indicated by the first process information when the terminal apparatus determines the first process information by the determining step;
A request information transmission step for transmitting, to the server device, request information indicating a request for processing indicated by the second processing information when the terminal device determines the second processing information by the determining step;
A request information receiving step in which the server device receives the request information from the terminal device;
A program execution step in which the server device executes the application program stored in program storage means for storing the application program based on the request information received in the request information reception step;
Switching destination of definition information used by the server device to determine the processing information corresponding to operation information indicating a user operation input using the input unit in accordance with the execution of the application program in the program execution step. A switching information transmission step of transmitting switching information related to the definition information to the terminal device;
A switching information receiving step in which the terminal device receives the switching information transmitted from the server device;
A switching step of switching definition information used for determining processing information by the determination step based on the switching information received by the switching information receiving step;
A generating step in which the server device generates at least one of video data and audio data indicating an execution result of the application program in the program executing step;
A transmission step in which the server device transmits at least one of the video data and the audio data generated by the generation step to the output device;
A receiving step in which the output device receives at least one of the video data and audio data transmitted from the server device;
An output control step in which the output device causes the output means to output an execution result indicated by at least one of the video data and audio data received in the reception step;
Including
The definition information includes the first processing information indicating processing including any of scrolling of the operation screen displayed on the display unit and movement of an indicia indicating a user operation target in the operation screen; and the server An information processing method comprising: the second processing information indicating processing involving switching of at least one of the video data and the audio data transmitted from a device to the output device.

Images