US20160001177A1 - Image generation system and image generation-purpose program - Google Patents

Image generation system and image generation-purpose program Download PDF

Info

Publication number
US20160001177A1
US20160001177A1 US14/769,215 US201314769215A US2016001177A1 US 20160001177 A1 US20160001177 A1 US 20160001177A1 US 201314769215 A US201314769215 A US 201314769215A US 2016001177 A1 US2016001177 A1 US 2016001177A1
Authority
US
United States
Prior art keywords
server
terminal
image
unit
side image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14/769,215
Inventor
Tadashi TSUSHIMA
Yoshiki Kashitani
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
FUZZ Inc
Original Assignee
FUZZ Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by FUZZ Inc filed Critical FUZZ Inc
Assigned to FUZZ, INC. reassignment FUZZ, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KASHITANI, YOSHIKI, TSUSHIMA, TADASHI
Publication of US20160001177A1 publication Critical patent/US20160001177A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
    • A63F13/35—Details of game servers
    • A63F13/355—Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
    • A63F13/35—Details of game servers
    • A63F13/358—Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51—Indexing; Data structures therefor; Storage structures
    • G06F17/3028—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H04N21/23103—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion using load balancing strategies, e.g. by placing or distributing content on different disks, different memories or different servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44209—Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4781—Games
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/637—Control signals issued by the client directed to the server or network components
    • H04N21/6373—Control signals issued by the client directed to the server or network components for rate control, e.g. request to the server to modify its transmission rate
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/658—Transmission by the client directed to the server
    • H04N21/6582—Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number

Definitions

  • the present invention relates to an image generation system and an image generation-purpose program, and more particularly, is suitably used for an image generation system serving to perform an image generation process by distribution through a terminal and a server and to synthesize and display an image generated on the terminal and an image generated on the server in the terminal.
  • a Cloud gaming service for operating a game on a server and streaming delivering a game screen and a sound to a terminal via a line.
  • operating information input on a terminal side is transmitted to the server and an image process such as a calculation or rendering based thereon is executed on the server side. Then, a game screen or a sound obtained as a result of the execution is streaming delivered to the terminal.
  • a terminal on a player side does not require an environment such as a storage medium for storing a game program and data and a high graphic processing performance. For this reason, by preparing these high performance operating environments on the server side, it is possible to comfortably execute a game at various terminals other than a private game machine or a high-end personal computer.
  • the terminal As one means for solving the problem, it is supposed to carry out image rendering by distribution through the terminal and the server and to synthesize an image generated at the terminal and an image delivered from the server and display the synthesized image.
  • image rendering By generating at least a part of images on the terminal side, it is possible to decrease a data volume of an image to be generated on the server side, that is, a data volume of an image to be delivered from the server to the terminal.
  • a background image is generated in a drawing engine of a server and is encoded in a video encoding device, and the encoded image is transferred to a user terminal through a network.
  • the user terminal decodes the image by a video decoding device and synthesizes the decoded image with a near view image generated in a drawing device by an image synthesizing device, and displays the synthesized image on a display.
  • an image management server device retrieves high correlation image data having a high correlation with image data corresponding to metadata transmitted from a client terminal out of registered image data stored in an image data storage unit, and transmits the high correlation image data obtained as a result of the retrieval to the client terminal through a network.
  • the client terminal generates synthesized image data by using the high correlation image data received from the image management server device and the image data corresponding to the meta data transmitted to the image management server device.
  • a certain latency time occurs before the images are synthesized on the terminal if the residual images are not delivered from the server.
  • the latency time produces a problem in that a response time is increased till display of the synthesized image on a screen and response performance of the screen display cannot be enhanced to such a sufficient degree as to cause the user to enjoy a game comfortably.
  • the present invention has been made to solve the problem and has an object to enhance response performance of screen display by shortening a latency time on a terminal side which is required for synchronously performing image synthesis in the case in which a process for generating a display image to be displayed on a terminal is carried out by distribution through the terminal and a server.
  • a process for generating a display image to be displayed on a terminal is performed by distribution through a terminal and a server, and control is carried out to execute either a distribution process for synthesizing a terminal-side image and a server-side image to generate a display image or a single process for generating the server-side image itself as the display image depending on a result of the decision.
  • a communication speed in transmission of the server-side image from the server to the terminal is stored as a history in a communication history storage unit.
  • a data volume of the server-side image to be generated by the server is adjusted based on the history of the communication speed stored in the communication history storage unit.
  • a data volume of a server-side image is adjusted so as not to increase the communication time based on the history of the communication speed when the server-side image is transmitted to the terminal in the past. Consequently, it is possible to shorten the latency time as greatly as possible at the terminal side required for synchronously synthesizing the terminal-side image and the server-side image. Thus, it is possible to enhance response performance of screen display.
  • FIG. 1 is a block diagram showing an example of a network structure of an image generation system according to the present embodiment.
  • FIG. 2 is a block diagram showing an example of a functional structure of a terminal according to the present embodiment.
  • FIG. 3 is a block diagram showing an example of a functional structure of a server according to the present embodiment.
  • FIG. 4 is a flowchart showing an example of an operation of an image generation system according to the present embodiment.
  • FIG. 1 is a block diagram showing an example of a network structure of an image generation system according to the present embodiment.
  • the image generation system includes a terminal 100 to be used by a player of Cloud gaming, and a server 200 for streaming delivering a game screen or a sound which is to be used for the Cloud gaming.
  • the terminal 100 and the server 200 are configured to be connectable to each other through a communication network such as internet 300 .
  • FIG. 2 is a block diagram showing an example of a functional structure of the terminal 100 according to the present embodiment.
  • FIG. 3 is a block diagram showing an example of a functional structure of the server 200 according to the present embodiment.
  • the terminal 100 includes, as a functional structure thereof, a delivery request unit 11 , a terminal-side image generation unit 12 , a server-side image receipt unit 13 , an image synthesis unit 14 , a display control unit 15 , a decision unit 16 and a control unit 17 , a communication speed detection unit 18 , a record unit 19 , a data volume adjustment unit 20 , and a communication history storage unit 30 .
  • the server 200 includes, as a functional structure thereof, a delivery request receipt unit 21 , a server-side image generation unit 22 , and a server-side image transmission unit 23 .
  • the respective blocks 11 to 20 in the terminal 100 are implemented by an image generation-purpose program incorporated into a game application (hereinafter referred to as a terminal-side game application) to be previously installed into the terminal 100 , for example.
  • the respective blocks 11 to 20 actually include a CPU, an RAM, an ROM and the like in a computer and are implemented by an operation of an image generation-purpose program stored in a recording medium such as the RAM, the ROM, a hard disk or a semiconductor memory.
  • functions of the respective blocks 21 to 23 in the server 200 are implemented by an image generation-purpose program incorporated into a game application (hereinafter referred to as a server-side game application) which is previously installed into the server 200 , for example.
  • the respective blocks 21 to 23 actually include a CPU, an RAM, an ROM and the like in a computer and are implemented by an operation of an image generation-purpose program stored in a recording medium such as the RAM, the ROM, a hard disk or a semiconductor memory.
  • the Cloud gaming is executed in cooperation of the terminal-side game application and the server-side game application.
  • the image generation system serves to perform a process for generating a display image to be displayed on the terminal 100 by distribution through the terminal 100 and the server 200 and to synthesize an image generated on the terminal 100 (which will be hereinafter referred to as a terminal-side image) and an image generated on the server 200 (which will be hereinafter referred to as a server-side image) and display the synthesized image in the terminal 100 .
  • the image may be a moving image or a static image.
  • the delivery request unit 11 of the terminal 100 gives the server 200 a request for delivering an image to be displayed on a screen of the terminal 100 depending on a state of progress of the Cloud gaming.
  • the delivery request unit 11 gives the server 200 a request for delivering a necessary image.
  • the terminal-side image generation unit 12 performs a process for generating a terminal-side image as necessary when the delivery request unit 11 gives the server 200 a request for delivering an image.
  • “As necessary” means the case in which the terminal-side image and the server-side image are synthesized and the synthesized image is displayed in the terminal 100 .
  • the image synthesis is not always carried out but a display image is constituted by only the server-side image received from the server 200 in some cases. Control as to whether to perform the image synthesis is carried out by the decision unit 16 and the control unit 17 , which will be described later.
  • the terminal 100 does not have a high performance graphic processing environment like the server 200 .
  • a letter image which does not require very high quality is generated by the terminal-side image generation unit 12 .
  • a body image other than the letter image that is, an image such as a background or a game character which is required to have high quality is generated by the server-side image generation unit 22 .
  • the server-side image receipt unit 13 receives, from the server 200 , a server-side image generated by the server-side image generation unit 22 .
  • the image synthesis unit 14 synthesizes a terminal-side image generated by the terminal-side image generation unit 12 and a server-side image received by the server-side image receipt unit 13 , thereby generating a display image.
  • a timing for completing the process for generating the terminal-side image by the terminal-side image generation unit 12 is not always coincident with a timing for completing the process for receiving the server-side image generated and transmitted by the server 200 through the server-side image receipt unit 13 . Accordingly, the image synthesis unit 14 needs to synchronously synthesize the terminal-side image and the server-side image with each other.
  • the server 200 has the high performance graphic processing environment, it generates and transmits the server-side image which has a higher quality level and a larger data volume than the terminal-side image. In many cases, therefore, the image synthesis unit 14 waits to carry out the image synthesis while the server-side image is received by the server-side image receipt unit 13 after the terminal-side image is generated by the terminal-side image generation unit 12 .
  • the data volume of the server-side image is adjusted to shorten a latency time for synchronization as greatly as possible, which will be described later. Consequently, the display image is switched smoothly so that a player can enjoy a game comfortably without feeling stress.
  • the display control unit 15 displays a display image on the screen of the terminal 100 .
  • the display image may be constituted by synthesizing the terminal-side image generated by the terminal-side image generation unit 12 and the server-side image received by the server-side image receipt unit 13 or may be constituted by only the server-side image received by the server-side image receipt unit 13 .
  • the decision unit 16 decides whether the process for generating a display image is performed by distribution through the terminal 100 and the server 200 . In other words, the decision unit 16 decides whether the terminal-side image and the server-side image are synthesized to generate a display image or whether the display image is generated by only the server-side image.
  • the decision unit 16 decides whether the process for generating a display image is to be performed by distribution through the terminal 100 and the server 200 depending on whether a letter image is included in the display image to be generated from now. In other words, it is decided that the distribution process for the image generation is to be performed if the letter image is included in the display image, and that the distribution process for the image generation is not to be performed if the letter image is not included.
  • the control unit 17 controls to execute either a distribution process or a single process depending on a result of the decision made by the decision unit 16 .
  • the distribution process serves to operate the terminal-side image generation unit 12 and the image synthesis unit 14 to synthesize the terminal-side image and the server-side image, thereby generating a display image.
  • the single process serves to generate the display image from only the server-side image received by the server-side image receipt unit 13 without operating the terminal-side image generation unit 12 and the image synthesis unit 14 .
  • the control unit 17 controls the operations of the delivery request unit 11 , the communication speed detection unit 18 , the record unit 19 and the data volume adjustment unit 20 depending on the result of the decision made by the decision unit 16 .
  • the communication speed detection unit 18 detects a communication speed of the internet 300 for connecting the terminal 100 and the server 200 to each other. For example, the communication speed detecting unit 18 detects a communication speed in receipt of the server-side image from the server 200 by the server-side image receipt unit 13 . It is sufficient to detect the communication speed in at least execution of the single process for generating, as the display image, the server-side image received by the server-side image receipt unit 13 .
  • the record unit 19 stores the communication speed detected by the communication speed detection unit 18 as a history in the communication history storage unit 30 when executing the single process under control of the control unit 17 .
  • the communication history storage unit 30 stores, as a history, a communication speed in the execution of the single process to transmit the server-side image from the server 200 to the terminal 100 .
  • the data volume adjustment unit 20 determines a quality level of the server-side image to be generated by the server 200 based on the history of the communication speed stored in the communication history storage unit 30 , thereby adjusting the data volume of the server-side image in the execution of the distribution process. More specifically, the data volume adjustment unit 20 adjusts a data volume in which a required processing time in the transmission of the server-side image at the lowest one of the communication speeds stored in the communication history storage unit 30 is within a predetermined time (a time-out time).
  • the time-out time is a time required for switching the display image without a player feeling stress, and is predetermined every type of the display image to be generated depending on the state of progress of the game.
  • the time-out time is previously incorporated as information in the program of the terminal-side game application, for example.
  • the data volume adjustment unit 20 adjusts the quality level of the server-side image to have such a data volume as to complete the image delivery within one second.
  • the quality level of the server-side image is specified depending on an image compression rate, for example.
  • the data volume adjustment unit 20 notifies the delivery request unit 11 of the quality level of the server-side image thus adjusted.
  • the delivery request unit 11 notifies the server 200 of the quality level together with the image delivery request described above.
  • a time required from the transmission of the delivery request of the image to the server 200 by the delivery request unit 11 to the transmission of the server-side image as a response is a total of the time required for generating the server-side image and the time required for transmitting the server-side image. Therefore, it is preferable to adjust the data volume (quality level) of the server-side image in such a manner that the total time is within the time-out time.
  • the delivery request receipt unit 21 of the server 200 receives an image delivery request transmitted from the terminal 100 .
  • the server-side image generation unit 22 performs the process for generating a server-side image upon receipt of the image delivery request by the delivery request receipt unit 21 .
  • the server-side image transmission unit 23 transmits, to the terminal 100 , the server-side image generated by the server-side image generation unit 22 .
  • the server-side image generation unit 22 generates a server-side image in accordance with a quality level when it is notified of the quality level together with the image delivery request.
  • the server side image generation unit 22 when the notice of the quality level is not given, the server side image generation unit 22 generates the server-side image on a predetermined quality level (a level to achieve high image quality), for example.
  • a predetermined quality level a level to achieve high image quality
  • FIG. 4 is a flowchart showing an example of an operation of the image generation system according to the present embodiment having the structure described above. The flowchart shown in FIG. 4 is started when the terminal side game application installed in the terminal 100 is activated.
  • the decision unit 16 of the terminal 100 decides whether a scene for switching a display image occurs depending on the state of progress of the Cloud gaming (Step S 1 ). If the scene for image switching does not occur, the progress of the game is exactly continued and it is consecutively decided whether the scene for switching an image occurs.
  • the decision unit 16 decides whether the process for generating a display image is performed by distribution through the terminal 100 and the server 200 depending on whether a letter image is included in a display image to be generated from now (Step S 2 ).
  • the decision unit 16 decides that the distribution process is performed, a notice of the decision is given to the control unit 17 .
  • the data volume adjustment unit 20 determines a quality level of the server-side image to be generated in the server 200 (a body image other than the letter image) based on the history of the communication speed stored in the communication history storage unit 30 under control of the control unit 17 (Step S 3 ). More specifically, the data volume adjustment unit 20 determines the quality level of the server-side image to have such a data volume that a processing time required for transmitting the server-side image from the server 200 to the terminal 100 at the lowest one of the communication speeds stored in the communication history storage unit 30 is within a time-out time.
  • the delivery request unit 11 transmits an image delivery request to the server 200 together with information about the quality level determined by the data volume adjustment unit 20 (Step S 4 ). Furthermore, the terminal-side image generation unit 12 performs the process for generating a terminal-side image (a letter image) under the control of the control unit 17 (Step S 5 ).
  • the delivery request unit 11 transmits an image delivery request to the server 200 without the information about the quality level related to the server-side image (Step S 6 ).
  • the delivery request receipt unit 21 of the server 200 receives the image delivery request transmitted from the terminal 100 in the Step S 4 or the Step S 6 (Step S 7 ).
  • the server-side image generation unit 22 performs the process for generating a server-side image upon receipt of the image delivery request (Step S 8 ).
  • the server-side image generation unit 22 generates the server-side image in accordance with the quality level when a notice of the information about the quality level is given together with the image delivery request.
  • the server-side image transmission unit 23 transmits, to the terminal 100 , the server-side image generated by the server-side image generation unit 22 (Step S 9 ).
  • the server-side image transmitted by the server-side image transmission unit 23 is received by the server-side image receipt unit 13 (Step S 10 ).
  • the decision unit 16 decides whether a distribution process is being executed (Step S 11 ). If the distribution process is being executed, the image synthesis unit 14 synthesizes the terminal-side image generated by the terminal-side image generation unit 12 in the Step S 5 and the server-side image received by the server-side image receipt unit 13 in the Step S 10 , thereby generating a display image (Step S 12 ). Thereafter, the display control unit 15 displays the display image generated by the image synthesis unit 14 on the screen of the terminal 100 (Step S 13 ).
  • the communication speed detection unit 18 detects a communication speed in the receipt of the server-side image from the server 200 by the server-side image receipt unit 13 under the control of the control unit 17 (Step S 14 ). Subsequently, the record unit 19 stores the communication speed detected by the communication speed detection unit 18 as a history in the communication history storage unit 30 (Step S 15 ). Then, the display control unit 15 displays, as a display image, the server-side image itself received by the server-side image receipt unit 13 in the Step S 10 on the screen of the terminal 100 (Step S 13 ).
  • Step S 16 decides whether the game is ended. In other words, the decision unit 16 decides whether the game is over depending on the progress of the game or an instruction for ending the game is given by a user's operation. If the game is not ended, the process is returned to the Step S 1 . On the other hand, if the game is ended, the process of the flowchart shown in FIG. 4 is ended.
  • the process for generating the display image is performed by distribution through the terminal 100 and the server 200 and whether the process is singly carried out in the server 200 . If the single process is performed, the communication speed in the receipt of the server-side image in the terminal 100 is detected and stored as a history in the communication history storage unit 30 . On the other hand, if the distribution process is performed, the data volume (quality level) of the server-side image to be generated in the server 200 is adjusted based on the history of the communication speed stored in the communication history storage unit 30 .
  • the data volume of the server-side image is adjusted as so not to increase a communication time based on the history of the communication speed in the transmission of the server-side image from the server 200 to the terminal 100 in the past. Consequently, it is possible to shorten a latency time on the terminal 100 side required for synchronously synthesizing the terminal-side image and the server-side image as greatly as possible. Thus, it is possible to enhance response performance of screen display in the switching of the display image.
  • the terminal-side game application may be downloaded from the server 200 into the terminal 100 in the activation of the game.
  • the delivery request unit 11 transmits the image delivery request to the server 200 together with a notice for giving the result of the decision made by the decision unit 16 .
  • the decision unit 16 and the control unit 17 are provided in the terminal 100 in the embodiment, furthermore, these may be provided in the server 200 .
  • the decision unit 16 of the server 200 decides whether the distribution process is performed. If the distribution process is carried out, the terminal 100 is notified of the decision. Then, the generation and transmission of the server-side image in the server 200 and the generation of the terminal-side image in the terminal 100 are performed concurrently, and the process for synthesizing both of the images is performed in the terminal 100 .
  • the data volume of the server-side image is adjusted in such a manner that the processing time required for transmitting the server-side image at the lowest one of the communication speeds stored in the communication history storage unit 30 is within the time-out time in the execution of the distribution process. If an actual communication speed at that time is further lower than the lowest communication speed in the past, however, the server-side image cannot be transmitted within the time-out time.
  • the terminal side image may be displayed when the time-out time is reached in the terminal 100 .
  • the reason is that the following is considered. More specifically, a player feels less stress by displaying at least a letter image as compared with a state in which the player waits for the transmission of the server-side image even if the time-out time passes and no image is displayed during all that time. If the letter image is displayed, the player can be informed of at least meaning of the screen even if there is no other body image.
  • the terminal-side image is set to be the letter image and the server-side image is set to be the other body image in the embodiment, moreover, the present invention is not restricted thereto.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Information Transfer Between Computers (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Processing Or Creating Images (AREA)

Abstract

When a process of generating a displayed image to be displayed on a terminal (100) is performed only by use of a server (200), a communication speed at which a server-side image (displayed image) is transmitted from the server (200) to the terminal (100) is stored, as a history, into a communication history storage unit (30). When the performance of the process of generating the displayed image is distributed between the terminal (100) and the server (200), the data volume of the server-side image generated at the server (200) is adjusted on the basis of the history of the communication speed stored in the communication history storage unit (30). In this way, the communication time can be prevented from becoming long, and the latency time required at the terminal (100) for synchronizing the terminal-side image with the server-side image and performing an image combination thereof can be possibly shortened.

Description

    TECHNICAL FIELD
  • The present invention relates to an image generation system and an image generation-purpose program, and more particularly, is suitably used for an image generation system serving to perform an image generation process by distribution through a terminal and a server and to synthesize and display an image generated on the terminal and an image generated on the server in the terminal.
    • BACKGROUND ART
  • Conventionally, there is provided a Cloud gaming service for operating a game on a server and streaming delivering a game screen and a sound to a terminal via a line. In the Cloud gaming service, operating information input on a terminal side is transmitted to the server and an image process such as a calculation or rendering based thereon is executed on the server side. Then, a game screen or a sound obtained as a result of the execution is streaming delivered to the terminal.
  • According to the Cloud gaming service, a terminal on a player side does not require an environment such as a storage medium for storing a game program and data and a high graphic processing performance. For this reason, by preparing these high performance operating environments on the server side, it is possible to comfortably execute a game at various terminals other than a private game machine or a high-end personal computer.
  • In the case of the Cloud gaming service, however, a data volume of an image to be delivered from the server to the terminal is increased. For this reason, in the case in which a line traffic is increased, a communication is delayed. Therefore, a response time is increased until image data on a result of a calculation corresponding to some operation performed by a user through the terminal is returned from the server and is displayed on a screen after the user performs the operation. Thus, there is a problem in that the user cannot enjoy a game comfortably.
  • As one means for solving the problem, it is supposed to carry out image rendering by distribution through the terminal and the server and to synthesize an image generated at the terminal and an image delivered from the server and display the synthesized image. By generating at least a part of images on the terminal side, it is possible to decrease a data volume of an image to be generated on the server side, that is, a data volume of an image to be delivered from the server to the terminal.
  • The technique itself for synthesizing and displaying an image generated on a server and an image generated on a terminal is conventionally known (for example, see Patent Documents 1 and 2).
      • Patent Document 1: Japanese Laid-Open Patent Publication No. 1996-69545
      • Patent Document 2: Japanese Laid-Open Patent Publication No. 2008-67290
  • Referring to the technique described in the Patent Document 1, a background image is generated in a drawing engine of a server and is encoded in a video encoding device, and the encoded image is transferred to a user terminal through a network. The user terminal decodes the image by a video decoding device and synthesizes the decoded image with a near view image generated in a drawing device by an image synthesizing device, and displays the synthesized image on a display.
  • Referring to the technique described in the Patent Document 2, an image management server device retrieves high correlation image data having a high correlation with image data corresponding to metadata transmitted from a client terminal out of registered image data stored in an image data storage unit, and transmits the high correlation image data obtained as a result of the retrieval to the client terminal through a network. The client terminal generates synthesized image data by using the high correlation image data received from the image management server device and the image data corresponding to the meta data transmitted to the image management server device.
    • DISCLOSURE OF THE INVENTION
  • In the case in which a process for generating an image to be displayed on a terminal is performed by distribution through the terminal and a server, however, a process time required for generating an image on the terminal side is different from a process time required for generating an image on the server side and delivering the generated image to the terminal. Therefore, it is necessary to synchronously synthesize images. In order to take synchronization, it is necessary to conform to either of the processing time on the terminal side (an image generation time) and the processing time on the server side (an image generation time+a communication time) which is later.
  • For this reason, even if a part of the images are completely generated on the terminal side, for example, a certain latency time occurs before the images are synthesized on the terminal if the residual images are not delivered from the server. The latency time produces a problem in that a response time is increased till display of the synthesized image on a screen and response performance of the screen display cannot be enhanced to such a sufficient degree as to cause the user to enjoy a game comfortably.
  • The present invention has been made to solve the problem and has an object to enhance response performance of screen display by shortening a latency time on a terminal side which is required for synchronously performing image synthesis in the case in which a process for generating a display image to be displayed on a terminal is carried out by distribution through the terminal and a server.
  • In order to solve the problem, in the present invention, it is decided whether a process for generating a display image to be displayed on a terminal is performed by distribution through a terminal and a server, and control is carried out to execute either a distribution process for synthesizing a terminal-side image and a server-side image to generate a display image or a single process for generating the server-side image itself as the display image depending on a result of the decision. In the execution of the single process, a communication speed in transmission of the server-side image from the server to the terminal is stored as a history in a communication history storage unit. In the execution of the distribution process, moreover, a data volume of the server-side image to be generated by the server is adjusted based on the history of the communication speed stored in the communication history storage unit.
  • According to the present invention having the structure described above, in the case in which the process for generating the display image to be displayed on the terminal is performed by distribution through the terminal and the server, a data volume of a server-side image is adjusted so as not to increase the communication time based on the history of the communication speed when the server-side image is transmitted to the terminal in the past. Consequently, it is possible to shorten the latency time as greatly as possible at the terminal side required for synchronously synthesizing the terminal-side image and the server-side image. Thus, it is possible to enhance response performance of screen display.
    • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram showing an example of a network structure of an image generation system according to the present embodiment.
  • FIG. 2 is a block diagram showing an example of a functional structure of a terminal according to the present embodiment.
  • FIG. 3 is a block diagram showing an example of a functional structure of a server according to the present embodiment.
  • FIG. 4 is a flowchart showing an example of an operation of an image generation system according to the present embodiment.
    •  BEST MODE FOR CARRYING OUT THE INVENTION
  • An embodiment according to the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an example of a network structure of an image generation system according to the present embodiment.
  • As shown in FIG. 1, the image generation system according to the present embodiment includes a terminal 100 to be used by a player of Cloud gaming, and a server 200 for streaming delivering a game screen or a sound which is to be used for the Cloud gaming. The terminal 100 and the server 200 are configured to be connectable to each other through a communication network such as internet 300.
  • FIG. 2 is a block diagram showing an example of a functional structure of the terminal 100 according to the present embodiment. Moreover, FIG. 3 is a block diagram showing an example of a functional structure of the server 200 according to the present embodiment.
  • As shown in FIG. 2, the terminal 100 according to the present embodiment includes, as a functional structure thereof, a delivery request unit 11, a terminal-side image generation unit 12, a server-side image receipt unit 13, an image synthesis unit 14, a display control unit 15, a decision unit 16 and a control unit 17, a communication speed detection unit 18, a record unit 19, a data volume adjustment unit 20, and a communication history storage unit 30. As shown in FIG. 3, moreover, the server 200 according to the present embodiment includes, as a functional structure thereof, a delivery request receipt unit 21, a server-side image generation unit 22, and a server-side image transmission unit 23.
  • Functions of the respective blocks 11 to 20 in the terminal 100 are implemented by an image generation-purpose program incorporated into a game application (hereinafter referred to as a terminal-side game application) to be previously installed into the terminal 100, for example. In other words, the respective blocks 11 to 20 actually include a CPU, an RAM, an ROM and the like in a computer and are implemented by an operation of an image generation-purpose program stored in a recording medium such as the RAM, the ROM, a hard disk or a semiconductor memory.
  • On the other hand, functions of the respective blocks 21 to 23 in the server 200 are implemented by an image generation-purpose program incorporated into a game application (hereinafter referred to as a server-side game application) which is previously installed into the server 200, for example. In other words, the respective blocks 21 to 23 actually include a CPU, an RAM, an ROM and the like in a computer and are implemented by an operation of an image generation-purpose program stored in a recording medium such as the RAM, the ROM, a hard disk or a semiconductor memory.
  • In the present embodiment, the Cloud gaming is executed in cooperation of the terminal-side game application and the server-side game application. During the execution of the Cloud gaming, the image generation system according to the present embodiment serves to perform a process for generating a display image to be displayed on the terminal 100 by distribution through the terminal 100 and the server 200 and to synthesize an image generated on the terminal 100 (which will be hereinafter referred to as a terminal-side image) and an image generated on the server 200 (which will be hereinafter referred to as a server-side image) and display the synthesized image in the terminal 100. In the present embodiment, the image may be a moving image or a static image.
  • A mechanism for image synthesis will be described below in detail. The delivery request unit 11 of the terminal 100 gives the server 200 a request for delivering an image to be displayed on a screen of the terminal 100 depending on a state of progress of the Cloud gaming. When there is generated a scene for switching the screen depending on a game operation by a player or when there is generated a scene for causing a program to automatically switch the screen depending on a result of execution of the game, typically, the delivery request unit 11 gives the server 200 a request for delivering a necessary image.
  • The terminal-side image generation unit 12 performs a process for generating a terminal-side image as necessary when the delivery request unit 11 gives the server 200 a request for delivering an image. “As necessary” means the case in which the terminal-side image and the server-side image are synthesized and the synthesized image is displayed in the terminal 100. In the present embodiment, the image synthesis is not always carried out but a display image is constituted by only the server-side image received from the server 200 in some cases. Control as to whether to perform the image synthesis is carried out by the decision unit 16 and the control unit 17, which will be described later.
  • Herein, the terminal 100 does not have a high performance graphic processing environment like the server 200. For this reason, a letter image which does not require very high quality is generated by the terminal-side image generation unit 12. On the other hand, in the server 200 having the high performance graphic processing environment, a body image other than the letter image, that is, an image such as a background or a game character which is required to have high quality is generated by the server-side image generation unit 22.
  • The server-side image receipt unit 13 receives, from the server 200, a server-side image generated by the server-side image generation unit 22. The image synthesis unit 14 synthesizes a terminal-side image generated by the terminal-side image generation unit 12 and a server-side image received by the server-side image receipt unit 13, thereby generating a display image.
  • A timing for completing the process for generating the terminal-side image by the terminal-side image generation unit 12 is not always coincident with a timing for completing the process for receiving the server-side image generated and transmitted by the server 200 through the server-side image receipt unit 13. Accordingly, the image synthesis unit 14 needs to synchronously synthesize the terminal-side image and the server-side image with each other.
  • Although the server 200 has the high performance graphic processing environment, it generates and transmits the server-side image which has a higher quality level and a larger data volume than the terminal-side image. In many cases, therefore, the image synthesis unit 14 waits to carry out the image synthesis while the server-side image is received by the server-side image receipt unit 13 after the terminal-side image is generated by the terminal-side image generation unit 12.
  • In the present embodiment, however, the data volume of the server-side image is adjusted to shorten a latency time for synchronization as greatly as possible, which will be described later. Consequently, the display image is switched smoothly so that a player can enjoy a game comfortably without feeling stress.
  • The display control unit 15 displays a display image on the screen of the terminal 100. As described above, the display image may be constituted by synthesizing the terminal-side image generated by the terminal-side image generation unit 12 and the server-side image received by the server-side image receipt unit 13 or may be constituted by only the server-side image received by the server-side image receipt unit 13.
  • The decision unit 16 decides whether the process for generating a display image is performed by distribution through the terminal 100 and the server 200. In other words, the decision unit 16 decides whether the terminal-side image and the server-side image are synthesized to generate a display image or whether the display image is generated by only the server-side image.
  • In the present embodiment, the decision unit 16 decides whether the process for generating a display image is to be performed by distribution through the terminal 100 and the server 200 depending on whether a letter image is included in the display image to be generated from now. In other words, it is decided that the distribution process for the image generation is to be performed if the letter image is included in the display image, and that the distribution process for the image generation is not to be performed if the letter image is not included.
  • It is possible to decide, based on information incorporated previously in the terminal-side game application, whether the letter image is included in the display image to be generated from now depending on the state of progress of the game, for example. Alternatively, it is also possible to make a decision by separating an executive instruction related to generation of a letter image from an executive instruction related to generation of a body image other than the letter image to build the terminal-side game application and confirming whether this executive instruction form is obtained. The decision method described above is only illustrative and the present invention is not restricted to this way.
  • The control unit 17 controls to execute either a distribution process or a single process depending on a result of the decision made by the decision unit 16. The distribution process serves to operate the terminal-side image generation unit 12 and the image synthesis unit 14 to synthesize the terminal-side image and the server-side image, thereby generating a display image. Moreover, the single process serves to generate the display image from only the server-side image received by the server-side image receipt unit 13 without operating the terminal-side image generation unit 12 and the image synthesis unit 14. Furthermore, the control unit 17 controls the operations of the delivery request unit 11, the communication speed detection unit 18, the record unit 19 and the data volume adjustment unit 20 depending on the result of the decision made by the decision unit 16.
  • The communication speed detection unit 18 detects a communication speed of the internet 300 for connecting the terminal 100 and the server 200 to each other. For example, the communication speed detecting unit 18 detects a communication speed in receipt of the server-side image from the server 200 by the server-side image receipt unit 13. It is sufficient to detect the communication speed in at least execution of the single process for generating, as the display image, the server-side image received by the server-side image receipt unit 13.
  • The record unit 19 stores the communication speed detected by the communication speed detection unit 18 as a history in the communication history storage unit 30 when executing the single process under control of the control unit 17. In other words, the communication history storage unit 30 stores, as a history, a communication speed in the execution of the single process to transmit the server-side image from the server 200 to the terminal 100.
  • The data volume adjustment unit 20 determines a quality level of the server-side image to be generated by the server 200 based on the history of the communication speed stored in the communication history storage unit 30, thereby adjusting the data volume of the server-side image in the execution of the distribution process. More specifically, the data volume adjustment unit 20 adjusts a data volume in which a required processing time in the transmission of the server-side image at the lowest one of the communication speeds stored in the communication history storage unit 30 is within a predetermined time (a time-out time).
  • Herein, the time-out time is a time required for switching the display image without a player feeling stress, and is predetermined every type of the display image to be generated depending on the state of progress of the game. The time-out time is previously incorporated as information in the program of the terminal-side game application, for example. Alternatively, it is also possible to have table information in addition to the program and to store, in the table information, the time-out time for each type of the display image.
  • In the case in which the communication speed is expressed in a bit per second (bps) and the time-out time is set to be one second, the data volume adjustment unit 20 adjusts the quality level of the server-side image to have such a data volume as to complete the image delivery within one second. The quality level of the server-side image is specified depending on an image compression rate, for example. The data volume adjustment unit 20 notifies the delivery request unit 11 of the quality level of the server-side image thus adjusted. The delivery request unit 11 notifies the server 200 of the quality level together with the image delivery request described above.
  • In fact, a time required from the transmission of the delivery request of the image to the server 200 by the delivery request unit 11 to the transmission of the server-side image as a response is a total of the time required for generating the server-side image and the time required for transmitting the server-side image. Therefore, it is preferable to adjust the data volume (quality level) of the server-side image in such a manner that the total time is within the time-out time.
  • The delivery request receipt unit 21 of the server 200 receives an image delivery request transmitted from the terminal 100. The server-side image generation unit 22 performs the process for generating a server-side image upon receipt of the image delivery request by the delivery request receipt unit 21. The server-side image transmission unit 23 transmits, to the terminal 100, the server-side image generated by the server-side image generation unit 22.
  • Herein, the server-side image generation unit 22 generates a server-side image in accordance with a quality level when it is notified of the quality level together with the image delivery request. On the other hand, when the notice of the quality level is not given, the server side image generation unit 22 generates the server-side image on a predetermined quality level (a level to achieve high image quality), for example. Alternatively, it is also possible to generate the server-side image on a quality level set to have such a data volume that a processing time required for detecting a communication speed in the receipt of the image delivery request at the server 200 side and transmitting the server-side image from the server 200 to the terminal 100 at the communication speed is within a time-out time.
  • FIG. 4 is a flowchart showing an example of an operation of the image generation system according to the present embodiment having the structure described above. The flowchart shown in FIG. 4 is started when the terminal side game application installed in the terminal 100 is activated.
  • First of all, the decision unit 16 of the terminal 100 decides whether a scene for switching a display image occurs depending on the state of progress of the Cloud gaming (Step S1). If the scene for image switching does not occur, the progress of the game is exactly continued and it is consecutively decided whether the scene for switching an image occurs.
  • On the other hand, when deciding that the scene for switching a display image occurs, the decision unit 16 decides whether the process for generating a display image is performed by distribution through the terminal 100 and the server 200 depending on whether a letter image is included in a display image to be generated from now (Step S2). Herein, if the decision unit 16 decides that the distribution process is performed, a notice of the decision is given to the control unit 17.
  • The data volume adjustment unit 20 determines a quality level of the server-side image to be generated in the server 200 (a body image other than the letter image) based on the history of the communication speed stored in the communication history storage unit 30 under control of the control unit 17 (Step S3). More specifically, the data volume adjustment unit 20 determines the quality level of the server-side image to have such a data volume that a processing time required for transmitting the server-side image from the server 200 to the terminal 100 at the lowest one of the communication speeds stored in the communication history storage unit 30 is within a time-out time.
  • Then, the delivery request unit 11 transmits an image delivery request to the server 200 together with information about the quality level determined by the data volume adjustment unit 20 (Step S4). Furthermore, the terminal-side image generation unit 12 performs the process for generating a terminal-side image (a letter image) under the control of the control unit 17 (Step S5).
  • On the other hand, if the decision unit 16 decides, in the Step S2, that the letter image is not included in the display image to be generated from now and the server 200 singly performs the process for generating a display image, the delivery request unit 11 transmits an image delivery request to the server 200 without the information about the quality level related to the server-side image (Step S6).
  • The delivery request receipt unit 21 of the server 200 receives the image delivery request transmitted from the terminal 100 in the Step S4 or the Step S6 (Step S7). The server-side image generation unit 22 performs the process for generating a server-side image upon receipt of the image delivery request (Step S8). At this time, the server-side image generation unit 22 generates the server-side image in accordance with the quality level when a notice of the information about the quality level is given together with the image delivery request. Then, the server-side image transmission unit 23 transmits, to the terminal 100, the server-side image generated by the server-side image generation unit 22 (Step S9).
  • In the terminal 100, the server-side image transmitted by the server-side image transmission unit 23 is received by the server-side image receipt unit 13 (Step S10). The decision unit 16 decides whether a distribution process is being executed (Step S11). If the distribution process is being executed, the image synthesis unit 14 synthesizes the terminal-side image generated by the terminal-side image generation unit 12 in the Step S5 and the server-side image received by the server-side image receipt unit 13 in the Step S10, thereby generating a display image (Step S12). Thereafter, the display control unit 15 displays the display image generated by the image synthesis unit 14 on the screen of the terminal 100 (Step S13).
  • On the other hand, if the decision unit 16 decides that the distribution process is not being executed in the Step S11, that is, a single process is being executed, the communication speed detection unit 18 detects a communication speed in the receipt of the server-side image from the server 200 by the server-side image receipt unit 13 under the control of the control unit 17 (Step S14). Subsequently, the record unit 19 stores the communication speed detected by the communication speed detection unit 18 as a history in the communication history storage unit 30 (Step S15). Then, the display control unit 15 displays, as a display image, the server-side image itself received by the server-side image receipt unit 13 in the Step S10 on the screen of the terminal 100 (Step S13).
  • Thereafter, the decision unit 16 decides whether the game is ended (Step S16). In other words, the decision unit 16 decides whether the game is over depending on the progress of the game or an instruction for ending the game is given by a user's operation. If the game is not ended, the process is returned to the Step S1. On the other hand, if the game is ended, the process of the flowchart shown in FIG. 4 is ended.
  • As described above in detail, in the present embodiment, when it is necessary to switch the image to be displayed on the terminal 100 depending on the progress of the game, it is decided whether the process for generating the display image is performed by distribution through the terminal 100 and the server 200 and whether the process is singly carried out in the server 200. If the single process is performed, the communication speed in the receipt of the server-side image in the terminal 100 is detected and stored as a history in the communication history storage unit 30. On the other hand, if the distribution process is performed, the data volume (quality level) of the server-side image to be generated in the server 200 is adjusted based on the history of the communication speed stored in the communication history storage unit 30.
  • According to the present embodiment thus configured, in the case in which the process for generating the display image to be displayed on the terminal 100 is performed by distribution through the terminal 100 and the server 200, the data volume of the server-side image is adjusted as so not to increase a communication time based on the history of the communication speed in the transmission of the server-side image from the server 200 to the terminal 100 in the past. Consequently, it is possible to shorten a latency time on the terminal 100 side required for synchronously synthesizing the terminal-side image and the server-side image as greatly as possible. Thus, it is possible to enhance response performance of screen display in the switching of the display image.
  • Although the description has been given to the example in which the terminal-side game application is previously installed into the terminal 100 in the embodiment, the present invention is not restricted thereto. For example, the terminal-side game application may be downloaded from the server 200 into the terminal 100 in the activation of the game. Moreover, it is also possible to download a script describing a necessary executive instruction for image display on that scene from the server 200 to the terminal 100 every scene for switching the display image depending on the progress of the game.
  • Although the description has been given to the example in which the terminal 100 is provided with the communication speed detection unit 18, the record unit 19, the data volume adjustment unit 20 and the communication history storage unit 30 in the embodiment, moreover, these may be provided in the server 200. In this case, the delivery request unit 11 transmits the image delivery request to the server 200 together with a notice for giving the result of the decision made by the decision unit 16.
  • Although the description has been given to the example in which the decision unit 16 and the control unit 17 are provided in the terminal 100 in the embodiment, furthermore, these may be provided in the server 200. In this case, when the scene for switching the display image occurs depending on the progress of the game, the decision unit 16 of the server 200 decides whether the distribution process is performed. If the distribution process is carried out, the terminal 100 is notified of the decision. Then, the generation and transmission of the server-side image in the server 200 and the generation of the terminal-side image in the terminal 100 are performed concurrently, and the process for synthesizing both of the images is performed in the terminal 100.
  • In the embodiment, moreover, the data volume of the server-side image is adjusted in such a manner that the processing time required for transmitting the server-side image at the lowest one of the communication speeds stored in the communication history storage unit 30 is within the time-out time in the execution of the distribution process. If an actual communication speed at that time is further lower than the lowest communication speed in the past, however, the server-side image cannot be transmitted within the time-out time.
  • In this case, only the terminal side image may be displayed when the time-out time is reached in the terminal 100. The reason is that the following is considered. More specifically, a player feels less stress by displaying at least a letter image as compared with a state in which the player waits for the transmission of the server-side image even if the time-out time passes and no image is displayed during all that time. If the letter image is displayed, the player can be informed of at least meaning of the screen even if there is no other body image.
  • Although the description has been given to the example in which the terminal-side image is set to be the letter image and the server-side image is set to be the other body image in the embodiment, moreover, the present invention is not restricted thereto. For example, it is also possible to generate, as the terminal-side image, a background image having no motion in a serial moving image and to generate, as the server-side image, a game character image having a motion or the like.
  • In addition, the embodiment is only illustrative for concreteness to carry out the present invention and the technical scope of the present invention should not be thereby construed to be restrictive. In other words, the present invention can be carried out in various configurations without departing from the gist or main features thereof.
    • EXPLANATION OF DESIGNATION
  • 11 delivery request unit
  • 12 terminal side image generation unit
  • 13 server side image receipt unit
  • 14 image synthesis unit
  • 15 display control unit
  • 16 decision unit
  • 17 control unit
  • 18 communication speed detection unit
  • 19 record unit
  • 20 data volume adjustment unit
  • 21 delivery request receipt unit
  • 22 server side image generation unit
  • 23 server side image transmission unit
  • 30 communication history storage unit
  • 100 terminal
  • 200 server

Claims (8)

1. An image generation system for performing a process for generating a display image to be displayed on a terminal by distribution through the terminal and a server and for synthesizing a terminal-side image generated on the terminal and a server-side image generated on the server in the terminal and displaying the synthesized image, the system comprising:
a decision unit for deciding whether the process for generating a display image is performed by distribution through the terminal and the server;
a control unit for controlling to execute either a distribution process for synthesizing the terminal-side image and the server-side image to generate the display image or a single process for generating the server-side image as the display image depending on a result of the decision made by the decision unit;
a communication history storage unit for storing, as a history, a communication speed in execution of the single process to transmit the server-side image from the server to the terminal; and
a data volume adjustment unit for adjusting a data volume of the server-side image based on the history of the communication speed stored in the transmission history storage unit when executing the distribution process to generate the server-side image.
2. The image generation system according to claim 1, wherein the data volume adjustment unit performs adjustment into such a data volume that a processing time required for transmitting the server-side image at the lowest one of the communication speeds stored in the communication history storage unit is within a predetermined time.
3. The image generation system according to claim 2, wherein the predetermined time is a predetermined time-out time for each type of the display image.
4. The image generation system according to claim 1, wherein the terminal-side image generated on the terminal in the execution of the distribution process is a letter image included in the display image, and the server-side image generated on the server in the execution of the distribution process is an image other than the letter image, and
the decision unit decides whether the process for generating a display image is performed by distribution through the terminal and the server depending on whether the letter image is included in the display image.
5. An image generation system for performing a process for generating a display image to be displayed on a terminal by distribution through the terminal and a server and for synthesizing and displaying a terminal-side image generated on the terminal and a server-side image generated on the server in the terminal,
wherein the terminal comprises:
a delivery request unit for requesting delivery of an image;
a terminal-side image generation unit for performing a process for generating the terminal-side image;
a server-side image receipt unit for receiving the server-side image;
an image synthesis unit for synthesizing the terminal-side image generated by the terminal-side image generation unit and the server-side image received by the server-side image receipt unit, thereby generating the display image;
a display control unit for displaying the display image;
a decision unit for deciding whether the process for generating a display image is performed by distribution through the terminal and the server;
a control unit for controlling to execute either a distribution process for operating the terminal-side image generation unit and the image synthesis unit to generate the display image or a single process for generating, as the display image, the server-side image received by the server-side image receipt unit without operating the terminal-side image generation unit and the image synthesis unit depending on a result of the decision made by the decision unit;
a communication speed detection unit for detecting a communication speed of a line for connecting the terminal and the server to each other;
a record unit for storing the communication speed detected by the communication speed detection unit as a history in a communication history storage unit when receiving the server-side image by the server-side image receipt unit in the execution of the single process; and
a data volume adjustment unit for adjusting a data volume of the server-side image based on the history of the communication speed stored in the communication history storage unit in the execution of the distribution process, and
the server comprises:
a delivery request receipt unit for receiving a delivery request of an image transmitted from the terminal;
a server-side image generation unit for performing a process for generating the server-side image so as to have a data volume adjusted by the data volume adjustment unit when receiving the delivery request by the delivery request receipt unit; and
a server-side image transmission unit for transmitting, to the terminal, the server-side image generated by the server-side image generation unit.
6. An image generation system for performing a process for generating a display image to be displayed on a terminal by distribution through the terminal and a server and for synthesizing and displaying a terminal-side image generated on the terminal and a server-side image generated on the server in the terminal,
wherein the terminal comprises:
a delivery request unit for requesting delivery of an image;
a terminal-side image generation unit for performing a process for generating the terminal-side image;
a server-side image receipt unit for receiving the server-side image;
an image synthesis unit for synthesizing the terminal-side image generated by the terminal-side image generation unit and the server-side image received by the server-side image receipt unit, thereby generating the display image;
a display control unit for displaying the display image;
a decision unit for deciding whether the process for generating a display image is performed by distribution through the terminal and the server; and
a control unit for controlling to execute either a distribution process for operating the terminal-side image generation unit and the image synthesis unit to generate the display image or a single process for generating, as the display image, the server-side image received by the server-side image receipt unit without operating the terminal-side image generation unit and the image synthesis unit depending on a result of the decision made by the decision unit, and
the server comprises:
a delivery request receipt unit for receiving a delivery request of an image transmitted from the terminal;
a server-side image generation unit for performing a process for generating the server-side image when receiving the delivery request by the delivery request receipt unit;
a server-side image transmission unit for transmitting, to the terminal, the server-side image generated by the server-side image generation unit;
a communication speed detection unit for detecting a communication speed of a line for connecting the terminal and the server to each other;
a record unit for storing the communication speed detected by the communication speed detection unit as a history in a communication history storage unit when transmitting the server-side image by the server-side image transmission unit in the execution of the single process; and
a data volume adjustment unit for adjusting a data volume of the server-side image based on the history of the communication speed stored in the communication history storage unit in the execution of the distribution process, and
the server-side image generation unit performs the process for generating the server-side image so as to have a data volume adjusted by the data volume adjustment unit when receiving the delivery request by the delivery request receipt unit.
7. An image generation system for performing a process for generating a display image to be displayed on a terminal by distribution through the terminal and a server and for synthesizing and displaying a terminal-side image generated on the terminal and a server-side image generated on the server in the terminal,
wherein the terminal comprises:
a delivery request unit for requesting delivery of an image;
a terminal-side image generation unit for performing a process for generating the terminal-side image;
a server-side image receipt unit for receiving the server-side image;
an image synthesis unit for synthesizing the terminal-side image generated by the terminal-side image generation unit and the server-side image received by the server-side image receipt unit, thereby generating the display image; and
a display control unit for displaying the display image, and
the server comprises:
a delivery request receipt unit for receiving a delivery request of an image transmitted from the terminal;
a server-side image generation unit for performing a process for generating the server-side image when receiving the delivery request by the delivery request receipt unit;
a server-side image transmission unit for transmitting, to the terminal, the server-side image generated by the server-side image generation unit;
a decision unit for deciding whether the process for generating a display image is performed by distribution through the terminal and the server;
a control unit for controlling to execute either a distribution process for generating the terminal-side image and the server-side image separately and synthesizing them by the image synthesis unit, thereby generating the display image or a single process for generating the display image by only the server-side image depending on a result of the decision made by the decision unit;
a communication speed detection unit for detecting a communication speed of a line for connecting the terminal and the server to each other;
a record unit for storing the communication speed detected by the communication speed detection unit as a history in a communication history storage unit when transmitting the server-side image by the server-side image transmission unit in the execution of the single process; and
a data volume adjustment unit for adjusting a data volume of the server-side image based on the history of the communication speed stored in the communication history storage unit in the execution of the distribution process, and
the server-side image generation unit performs the process for generating the server-side image so as to have a data volume adjusted by the data volume adjustment unit when receiving the delivery request by the delivery request receipt unit.
8. An image generation-purpose program to be applied to an image generation system for performing a process for generating a display image to be displayed on a terminal by distribution through the terminal and a server and for synthesizing a terminal-side image generated on the terminal and a server-side image generated on the server in the terminal and displaying the synthesized image, the program causing a computer to function as:
decision means for deciding whether the process for generating a display image is performed by distribution through the terminal and the server;
control means for controlling to execute either a distribution process for synthesizing the terminal-side image and the server-side image to generate the display image or a single process for generating the server-side image as the display image depending on a result of the decision made by the decision means;
record means for storing, as a history in a communication history storage unit, a communication speed in execution of the single process to transmit the server-side image from the server to the terminal; and
a data volume adjustment unit for adjusting a data volume of the server-side image based on the history of the communication speed stored in the transmission history storage unit when executing the distribution process to generate the server-side image.
US14/769,215 2013-07-22 2013-07-22 Image generation system and image generation-purpose program Abandoned US20160001177A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2013/069799 WO2015011762A1 (en) 2013-07-22 2013-07-22 Image generation system and image generation-purpose program

Publications (1)

Publication Number Publication Date
US20160001177A1 true US20160001177A1 (en) 2016-01-07

Family

ID=50749993

Family Applications (1)

Application Number Title Priority Date Filing Date
US14/769,215 Abandoned US20160001177A1 (en) 2013-07-22 2013-07-22 Image generation system and image generation-purpose program

Country Status (4)

Country Link
US (1) US20160001177A1 (en)
EP (1) EP3026919A4 (en)
JP (1) JP5481606B1 (en)
WO (1) WO2015011762A1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9821222B1 (en) 2014-11-14 2017-11-21 Amazon Technologies, Inc. Coordination of content presentation operations
US9839843B1 (en) * 2014-11-14 2017-12-12 Amazon Technologies, Inc. Coordination of content presentation operations
US10158700B1 (en) 2014-11-14 2018-12-18 Amazon Technologies, Inc. Coordination of content presentation operations
CN111797689A (en) * 2017-04-28 2020-10-20 创新先进技术有限公司 Vehicle loss assessment image acquisition method and device, server and client
US10974142B1 (en) 2019-10-01 2021-04-13 Sony Interactive Entertainment Inc. Synchronization and offset of VSYNC between cloud gaming server and client
US11344799B2 (en) 2019-10-01 2022-05-31 Sony Interactive Entertainment Inc. Scene change hint and client bandwidth used at encoder for handling video frames after a scene change in cloud gaming applications
US11395963B2 (en) 2019-10-01 2022-07-26 Sony Interactive Entertainment Inc. High speed scan-out of server display buffer for cloud gaming applications
US11420118B2 (en) 2019-10-01 2022-08-23 Sony Interactive Entertainment Inc. Overlapping encode and transmit at the server
US11539960B2 (en) 2019-10-01 2022-12-27 Sony Interactive Entertainment Inc. Game application providing scene change hint for encoding at a cloud gaming server

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6161645B2 (en) * 2015-01-20 2017-07-12 株式会社サミーネットワークス GAME CONTROL PROGRAM, GAME CONTROL DEVICE, AND GAME CONTROL METHOD
JP2016131756A (en) * 2015-01-20 2016-07-25 株式会社サミーネットワークス Game server, game control program, and game control method
KR101745808B1 (en) * 2015-12-31 2017-06-28 엔에이치엔엔터테인먼트 주식회사 Method and system for processing module modulization of server-side logic
JP7250721B2 (en) * 2020-03-23 2023-04-03 グリー株式会社 Computer program, server device, terminal device, and method
CN113419802B (en) * 2021-06-21 2022-08-05 网易（杭州）网络有限公司 Atlas generation method and apparatus, electronic device and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030158957A1 (en) * 2002-01-23 2003-08-21 Ali Abdolsalehi Interactive internet browser based media broadcast
US20100306813A1 (en) * 2009-06-01 2010-12-02 David Perry Qualified Video Delivery
US20120004041A1 (en) * 2008-12-15 2012-01-05 Rui Filipe Andrade Pereira Program Mode Transition

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3632705B2 (en) * 1994-08-31 2005-03-23 ソニー株式会社 Interactive image providing method, server device, providing method, user terminal, receiving method, image providing system, and image providing method
JP2006121267A (en) * 2004-10-20 2006-05-11 Sharp Corp Continuous reproduction system
JP2008067290A (en) 2006-09-11 2008-03-21 Victor Co Of Japan Ltd Image display system, client terminal, and image management server device
JP4695583B2 (en) * 2006-12-11 2011-06-08 株式会社ドワンゴ Display device, comment display method, and program
JP5359952B2 (en) * 2010-03-23 2013-12-04 三菱電機株式会社 High-bandwidth video transmission system and method using multiple public lines

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030158957A1 (en) * 2002-01-23 2003-08-21 Ali Abdolsalehi Interactive internet browser based media broadcast
US20120004041A1 (en) * 2008-12-15 2012-01-05 Rui Filipe Andrade Pereira Program Mode Transition
US20100306813A1 (en) * 2009-06-01 2010-12-02 David Perry Qualified Video Delivery

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9821222B1 (en) 2014-11-14 2017-11-21 Amazon Technologies, Inc. Coordination of content presentation operations
US9839843B1 (en) * 2014-11-14 2017-12-12 Amazon Technologies, Inc. Coordination of content presentation operations
US10158700B1 (en) 2014-11-14 2018-12-18 Amazon Technologies, Inc. Coordination of content presentation operations
US10729976B1 (en) 2014-11-14 2020-08-04 Amazon Technologies, Inc. Coordination of content presentation operations
US10792564B1 (en) 2014-11-14 2020-10-06 Amazon Technologies, Inc. Coordination of content presentation operations
CN111797689A (en) * 2017-04-28 2020-10-20 创新先进技术有限公司 Vehicle loss assessment image acquisition method and device, server and client
US10974142B1 (en) 2019-10-01 2021-04-13 Sony Interactive Entertainment Inc. Synchronization and offset of VSYNC between cloud gaming server and client
US11020661B2 (en) 2019-10-01 2021-06-01 Sony Interactive Entertainment Inc. Reducing latency in cloud gaming applications by overlapping reception and decoding of video frames and their display
US11110349B2 (en) 2019-10-01 2021-09-07 Sony Interactive Entertainment Inc. Dynamic client buffering and usage of received video frames for cloud gaming
US11235235B2 (en) 2019-10-01 2022-02-01 Sony Interactive Entertainment Inc. Synchronization and offset of VSYNC between gaming devices
US11344799B2 (en) 2019-10-01 2022-05-31 Sony Interactive Entertainment Inc. Scene change hint and client bandwidth used at encoder for handling video frames after a scene change in cloud gaming applications
US11395963B2 (en) 2019-10-01 2022-07-26 Sony Interactive Entertainment Inc. High speed scan-out of server display buffer for cloud gaming applications
US11420118B2 (en) 2019-10-01 2022-08-23 Sony Interactive Entertainment Inc. Overlapping encode and transmit at the server
US11446572B2 (en) 2019-10-01 2022-09-20 Sony Interactive Entertainment Inc. Early scan-out of server display buffer at flip-time for cloud gaming applications
US11458391B2 (en) 2019-10-01 2022-10-04 Sony Interactive Entertainment Inc. System and method for improving smoothness in cloud gaming applications
US11524230B2 (en) 2019-10-01 2022-12-13 Sony Interactive Entertainment Inc. Encoder tuning to improve tradeoffs between latency and video quality in cloud gaming applications
US11539960B2 (en) 2019-10-01 2022-12-27 Sony Interactive Entertainment Inc. Game application providing scene change hint for encoding at a cloud gaming server
US11865434B2 (en) 2019-10-01 2024-01-09 Sony Interactive Entertainment Inc. Reducing latency in cloud gaming applications by overlapping receive and decode of video frames and their display at the client

Also Published As

Publication number Publication date
WO2015011762A1 (en) 2015-01-29
JP5481606B1 (en) 2014-04-23
EP3026919A4 (en) 2017-04-05
JPWO2015011762A1 (en) 2017-03-02
EP3026919A1 (en) 2016-06-01

Similar Documents

Publication Publication Date Title
US20160001177A1 (en) Image generation system and image generation-purpose program
CN113347405B (en) Scaling related method and apparatus
CN110213616B (en) Video providing method, video obtaining method, video providing device, video obtaining device and video providing equipment
US9814979B2 (en) Data provision system, provision apparatus, execution apparatus, control method, and recording medium
CN111882626A (en) Image processing method, apparatus, server and medium
EP2151970B1 (en) Processing and supplying video data
US20160101356A1 (en) System And Method For Delivering Media Over Network
US9843825B1 (en) Distributed and synchronized media switching
US9420324B2 (en) Content isolation and processing for inline video playback
EP2908547A1 (en) Information-processing device, information-processing system, information-processing program, and moving image data transmission/reception method
US11652864B2 (en) Method and apparatus for transmitting resources and non-transitory storage medium
US20200351525A1 (en) Communication apparatus, communication method, and computer-readable storage medium
JP6710201B2 (en) Intelligent streaming of media content
US20200259880A1 (en) Data processing method and apparatus
EP3930335A1 (en) Resource transmission method and terminal
JP6560696B2 (en) Client, program and method for controlling segment reception of data
US8724691B2 (en) Transcoding video data
US9973562B2 (en) Split processing of encoded video in streaming segments
CN111182334B (en) Data processing method, server, terminal, and storage medium
US10356159B1 (en) Enabling playback and request of partial media fragments
US10433023B1 (en) Heuristics for streaming live content
US10313759B1 (en) Enabling playback and request of partial media fragments
JP6513054B2 (en) Client device of content delivery system, method and program for acquiring content
US12081448B2 (en) Reducing decode delay at a client device
WO2019058111A1 (en) Graphics streaming

Legal Events

Date Code Title Description
AS Assignment

Owner name: FUZZ, INC., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TSUSHIMA, TADASHI;KASHITANI, YOSHIKI;REEL/FRAME:036380/0353

Effective date: 20150811

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION