KR102562174B1 - System and method for providing cloud game service - Google Patents

Abstract

According to an embodiment, a cloud game service is provided that transmits a stream of a play video of a cloud game played by a user of a user terminal and metadata about a sound of a cloud game corresponding to the stream of the play video of the cloud game to the user terminal. and a user terminal that acquires the sound of the cloud game based on the device and the metadata, matches the acquired sound of the cloud game with a stream of a play video of the cloud game, and outputs the matched sound.

Description

Cloud game service providing system and cloud game service providing method {SYSTEM AND METHOD FOR PROVIDING CLOUD GAME SERVICE}

The following embodiments relate to a cloud game service providing system and a cloud game service providing method.

An online game is a computer game implemented so that a computer game can operate on a wired or wireless network, and an online game player can play a game with another person at a remote location through a wired or wireless network.

A role-playing game is a computer game in which a game user assumes the role of a character appearing in a game program and performs it directly. As you travel around the world and perform various missions, you can feel the interest as if you were the main character in the game.

In particular, MMORPG (Massively Multi-player Online Role Playing Game) games in which multiple users participate simultaneously and perform various missions (quests) on a game map implemented in a virtual space are gaining popularity.

Cloud gaming is a kind of cloud service, meaning a service that immediately streams games to terminals that request game use while storing them on a server. Depending on the streaming method, it can be divided into cloud games based on 'video streaming' and cloud games based on 'file streaming (pixel streaming)'.

According to an embodiment, a cloud game service providing system in which a cloud game service providing device may transmit a stream of a play video of a cloud game and metadata about sound of a cloud game corresponding to the stream of the play video of the cloud game to a user terminal. and a cloud game service providing method.

In addition, according to another embodiment, the cloud game service providing device transmits the stream of the play video of the cloud game and the sound data of the sound of the cloud game corresponding to the user terminal as metadata, thereby reducing the amount of computation and traffic of the cloud game service providing device. It is possible to provide a cloud game service providing system and a cloud game service providing method capable of reducing

Further, according to another embodiment, a cloud game service providing system and a cloud game service providing method may be provided in which a user terminal may determine a sound of a cloud game from a sound source stored in the user terminal based on metadata.

In addition, according to another embodiment, the volume of the sound of the cloud game and the stereo spatial feeling of the sound of the cloud game are based on the location information of the character matching the user terminal and the location information of the object matching the sound of the cloud game. It is possible to provide a cloud game service providing system and a cloud game service providing method capable of calculating .

In addition, according to another embodiment, when at least one of location information of a character matched with a user terminal and location information of an object matched with a sound of a cloud game is changed, the cloud game service providing device returns to the user terminal. A cloud game service providing system capable of transmitting modified metadata and a cloud game service providing method may be provided.

Further, according to another embodiment, the user terminal modifies the sound of the cloud game based on at least one of modified positional information of a character matched with the user terminal and modified positional information of an object matched to the sound of the cloud game. It is possible to provide a cloud game service providing system and a cloud game service providing method capable of calculating the modified stereo spatial effect of the volume and sound of the cloud game.

In addition, according to another embodiment, the user terminal can generate object speech corresponding to the object dialogue information using a deep neural network based on the object information matching the object dialogue information and the object dialogue information. A system for providing a cloud game service and a method for providing a cloud game service may be provided.

According to an embodiment, a cloud game service is provided that transmits a stream of a play video of a cloud game played by a user of a user terminal and metadata about a sound of a cloud game corresponding to the stream of the play video of the cloud game to the user terminal. and a user terminal that acquires the sound of the cloud game based on the device and the metadata, matches the acquired sound of the cloud game with a stream of a play video of the cloud game, and outputs the matched sound.

In addition, the meta data may include time stamp information representing start time and end time of reproduction of the sound of the cloud game in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and sound of the cloud game. It may include at least one of location information of an object matching with and sound source information matching the sound of the cloud game.

In addition, the user terminal determines the sound of the cloud game to be output based on the meta data from a sound source stored in the user terminal, and matches the sound of the cloud game and location information of a character matched with the user terminal. The volume of the sound of the cloud game and the stereo spatial feeling of the sound of the determined cloud game are calculated based on the location information of the object, and the volume of the sound of the cloud game and the stereo spatial feeling of the sound of the cloud game are calculated. Based on this, the determined sound of the cloud game may be corrected, and the corrected sound of the cloud game may be matched with a play video stream of the cloud game and output at a time matching the time stamp information.

In addition, the cloud game service providing device may, when at least one of the location information of a character matching the user terminal and the location information of an object matching the sound of the cloud game is changed, modify the meta to the user terminal. data is transmitted, and the modified metadata is location information of at least one of location information of a character matching the user terminal and location information of an object matching the sound of the cloud game in the stream of the play video of the cloud game. may include at least one of modified time stamp information indicating a changed time, modified location information of a character matching the user terminal, and modified location information of an object matching the sound of the cloud game.

In addition, the user terminal determines the sound of the cloud game to be output based on the meta data from a sound source stored in the user terminal, and matches the sound of the cloud game and location information of a character matched with the user terminal. The volume of the sound of the cloud game and the stereo spatial feeling of the sound of the determined cloud game are calculated based on the location information of the object, and the volume of the sound of the cloud game and the stereo spatial feeling of the sound of the cloud game are calculated. corrects the sound of the determined cloud game based on, outputs the corrected sound of the cloud game by matching it with the play video stream of the cloud game at a time matching the time stamp information, and matches the user terminal The corrected volume of the corrected cloud game sound being output and the corrected volume of the corrected cloud game being output based on at least one of the corrected location information of the character and the corrected location information of the object matching the sound of the cloud game. The corrected stereo spatial feeling of sound is recalculated, and the corrected sound of the cloud game being output based on the recalculated modified volume of the cloud game sound and the recalculated modified stereo spatial feeling of the cloud game sound and the modified sound of the cloud game is matched with the play video stream of the cloud game at a time matching the modified time stamp information and outputted.

In addition, the sound of the cloud game includes a dialogue of at least one object included in the cloud game, and the meta data includes the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game. Includes at least one of timestamp information indicating, location information of a character matching the user terminal, location information of an object matching the line of the object, object information matching the line of the object, and line information of the object. can do.

In addition, the user terminal generates a dialogue voice of the object corresponding to the dialogue information of the object by using a deep neural network based on object information matching the dialogue of the object and the dialogue information of the object. Based on the position information of the character matching the terminal and the position information of the object matching the dialogue of the object, the volume of the dialogue voice of the generated object and the stereo spatial feeling of the dialogue speech of the generated object are calculated, and the calculated The generated object's dialogue voice is corrected based on the volume of the object's dialogue voice and the calculated stereo spatial feeling of the object's dialogue voice, and the corrected object's dialogue voice is transferred to the cloud at the time matched with the time stamp information. It can be matched with the stream of game play video and output.

In addition, the cloud game service providing device, when at least one of the location information of a character matching the user terminal and the location information of an object matching the line of the object is changed, the modified metadata to the user terminal and the modified metadata is the location information of at least one of location information of a character matching the user terminal and location information of an object matching the lines of the object in the stream of the play video of the cloud game is changed. It may include at least one of modified timestamp information indicating time, modified positional information of a character matching the user terminal, and modified positional information of an object matching the lines of the object.

In addition, the user terminal generates a dialogue voice of the object corresponding to the dialogue information of the object by using a deep neural network based on object information matching the dialogue of the object and the dialogue information of the object. Based on the position information of the character matching the terminal and the position information of the object matching the dialogue of the object, the volume of the dialogue voice of the generated object and the stereo spatial feeling of the dialogue speech of the generated object are calculated, and the calculated The generated object's dialogue voice is corrected based on the volume of the object's dialogue voice and the calculated stereo spatial feeling of the object's dialogue voice, and the corrected object's dialogue voice is transferred to the cloud at the time matched with the time stamp information. Matching and outputting the stream of the game play video, and correcting the correction being output based on at least one of the corrected location information of a character matching the user terminal and the corrected location information of an object matching the lines of the object. The corrected volume of the object's dialogue voice and the modified stereo spatial feeling of the corrected object's dialogue voice being output are recalculated, and the recalculated volume of the object's dialogue voice and the recalculated object's dialogue voice are calculated. The dialogue voice of the corrected object being output is modified based on the modified stereo spatial feeling, and the dialogue voice of the corrected object is matched with the stream of the play video of the cloud game at the time matching the modified time stamp information. can be printed out.

According to another embodiment, in the cloud game service providing method performed by a cloud game service providing system including a user terminal and a cloud game service providing device, the cloud game service providing device plays a cloud game played by a user of a user terminal. An operation of transmitting to the user terminal metadata about the play video stream and the sound of the cloud game corresponding to the stream of the play video of the cloud game, and the user terminal converts the sound of the cloud game based on the metadata. and matching the acquired sound of the cloud game with a stream of the play video of the cloud game and outputting the sound.

In addition, the meta data may include time stamp information representing start time and end time of reproduction of the sound of the cloud game in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and sound of the cloud game. It may include at least one of location information of an object matching with and sound source information matching the sound of the cloud game.

In addition, the operation of the user terminal acquiring the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched image, the user terminal Determining the sound of the cloud game to be output based on the meta data from a sound source stored in the user terminal, location information of a character matching the user terminal and the object matching the sound of the cloud game An operation of calculating the volume of the sound of the determined cloud game and the stereo spatial effect of the sound of the determined cloud game based on the location information, and the user terminal calculates the volume of the calculated sound of the cloud game and the calculated sound of the cloud game Correcting the sound of the determined cloud game based on stereo spatial feeling and outputting the corrected sound of the cloud game by matching it with a stream of play video of the cloud game at a time when the user terminal matches the time stamp information Actions may be included.

In addition, the operation of the cloud game service providing device transmitting metadata about the sound of the cloud game corresponding to the play video of the cloud game to the user terminal includes location information of a character matched with the user terminal and the cloud game. and transmitting modified metadata to the user terminal when at least one of the positional information of an object that matches the sound of is changed, and the modified metadata is stored in the stream of the play video of the cloud game. Modified time stamp information indicating a time when at least one of location information of a character matched with the user terminal and location information of an object matched with the sound of the cloud game is changed, and modification of a character matched with the user terminal. It may include at least one of modified location information and modified location information of an object that matches the sound of the cloud game.

In addition, the operation of the user terminal acquiring the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched image, the user terminal Determining the sound of the cloud game to be output based on the meta data from a sound source stored in the user terminal, location information of a character matching the user terminal and the object matching the sound of the cloud game An operation of calculating the volume of the sound of the determined cloud game and the stereo spatial effect of the sound of the determined cloud game based on the location information, and the user terminal calculates the volume of the calculated sound of the cloud game and the calculated sound of the cloud game Correcting the sound of the determined cloud game based on stereo spatial feeling, matching the corrected sound of the cloud game with a stream of the play video of the cloud game at a time when the user terminal matches the time stamp information, and outputting the corrected sound. operation, the corrected sound of the cloud game being output based on at least one of corrected positional information of a character matched with the user terminal by the user terminal and modified positional information of an object matched to the sound of the cloud game. operation of recalculating the corrected volume of and the corrected stereo spatial feeling of the corrected sound of the cloud game being output, and the user terminal of the recalculated volume of the recalculated sound of the cloud game and the recalculated cloud game An operation of modifying the corrected sound of the cloud game being output based on the corrected stereo spatial feeling of sound, and the corrected sound of the cloud game at the time when the user terminal matches the corrected time stamp information of the cloud game. It may include an operation of matching and outputting a stream of play video.

In addition, the sound of the cloud game includes a dialogue of at least one object included in the cloud game, and the meta data includes the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game. Includes at least one of timestamp information indicating, location information of a character matching the user terminal, location information of an object matching the line of the object, object information matching the line of the object, and line information of the object. can do.

In addition, the operation of the user terminal acquiring the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched image, the user terminal An operation of generating object information matching the object's dialogue and speech of the object corresponding to the dialogue information of the object by using a deep neural network based on the dialogue information of the object, the user terminal matching the user terminal Calculating the volume of the dialogue voice of the generated object and the stereo spatial feeling of the dialogue voice of the generated object based on the position information of the character to be played and the position information of the object matching the dialogue of the object, the user terminal An operation of correcting the dialogue voice of the generated object based on the volume of the dialogue voice of the calculated object and the stereo spatial feeling of the calculated dialogue voice of the object, and the correction at the time when the user terminal matches the time stamp information It may include an operation of outputting a matched dialogue voice of the object with a stream of the play video of the cloud game.

In addition, the operation of the cloud game service providing device transmitting metadata about the sound of the cloud game corresponding to the play video of the cloud game to the user terminal, the location information of the character matching the user terminal and the object and transmitting modified metadata to the user terminal when at least one of the positional information of an object matching the dialogue is changed, wherein the modified metadata is configured as described above in the stream of the play video of the cloud game. Modified time stamp information indicating the time when at least one of the location information of the character matching the user terminal and the location information of the object matching the line of the object is changed, and the modified location of the character matching the user terminal It may include at least one of information and modified location information of an object that matches the dialogue of the object.

In addition, the operation of the user terminal acquiring the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched image, the user terminal An operation of generating object information matching the object's dialogue and speech of the object corresponding to the dialogue information of the object by using a deep neural network based on the dialogue information of the object, the user terminal matching the user terminal An operation of calculating the volume of the dialogue voice of the generated object and the stereo spatial feeling of the dialogue voice of the generated object based on the position information of the character to be played and the position information of the object matching the sound of the cloud game, the user terminal Correcting the dialogue voice of the generated object based on the volume of the calculated dialogue voice of the object and the stereo spatial feeling of the calculated dialogue voice of the object, the correction at the time when the user terminal matches the timestamp information Matching and outputting the dialogue voice of one object with the play video stream of the cloud game, the user terminal and the modified position information of the character matching the user terminal and the modified position of the object matching the dialogue of the object An operation of recalculating the corrected volume of the dialogue voice of the corrected object being output and the modified stereo spatial feeling of the dialogue voice of the corrected object being output based on at least one of the information; An operation of correcting the corrected object's dialogue voice being output based on the corrected volume of the object's dialogue voice and the recalculated stereo spatial feeling of the object's dialogue voice, and the user terminal providing the corrected timestamp information and An operation of matching and outputting the dialogue voice of the modified object with the stream of the play video of the cloud game at the matched time.

According to an embodiment, the cloud game service providing device can transmit a stream of a play video of a cloud game and metadata about a sound of a cloud game corresponding to the stream of the play video of the cloud game to a user terminal.

In addition, the cloud game service providing device transmits sound data for the sound of the cloud game corresponding to the stream of the play video of the cloud game as metadata to the user terminal, thereby reducing the amount of computation and traffic of the cloud game service providing device. there is

In addition, there is an effect that the user terminal can determine the sound of the cloud game from the sound source stored in the user terminal based on the meta data.

In addition, there is an effect that the user terminal can calculate the volume of the sound of the cloud game and the stereo spatial feeling of the sound of the cloud game based on the position information of the character matched with the user terminal and the position information of the object matched with the sound of the cloud game. .

In addition, when at least one of the location information of a character matching the user terminal and the location information of an object matching the sound of the cloud game is changed, the cloud game service providing device can transmit modified metadata to the user terminal It works.

In addition, the modified volume of the sound of the cloud game and the sound of the cloud game based on at least one of modified location information of a character matched by the user terminal with the user terminal and modified location information of an object matched to the sound of the cloud game. There is an effect that can calculate the modified stereo spatial feeling of .

In addition, there is an effect in that the user terminal can generate object dialogue voice corresponding to the object dialogue information using a deep neural network based on the object information matching the object dialogue and the object dialogue information.

1 is a diagram illustrating a cloud game service providing system according to an embodiment.
2 is a diagram illustrating a configuration of a user terminal according to an embodiment.
FIG. 3 is a diagram illustrating how a user terminal matches and outputs speech of an object generated based on metadata with a stream of play video of a cloud game according to an embodiment.
4 is a diagram illustrating how a user terminal modifies a sound of a cloud game based on meta data and modified meta data according to an embodiment.
FIG. 5 is a diagram illustrating how a user terminal modifies speech of an object based on meta data and modified meta data according to an exemplary embodiment.
6 is a flowchart illustrating a cloud game service providing method according to an embodiment.
7 is a flowchart illustrating a cloud game service providing method according to another embodiment.

Specific structural or functional descriptions of the embodiments disclosed herein are merely illustrated for the purpose of describing the embodiments, and the embodiments may be implemented in various forms and are limited to the embodiments described herein. It doesn't work.

Embodiments are intended to be exemplified in the drawings and described in detail herein since they can have various changes and can have various forms. However, this is not intended to limit the embodiments to specific disclosure forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the embodiments.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to embodiments, a first component may be named a second component, and similarly The second component may also be referred to as the first component.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Other expressions describing the relationship between elements, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", etc., should be interpreted similarly.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, terms such as "comprise" or "have" are intended to indicate that the described feature, number, step, operation, component, part, or combination thereof exists, but that one or more other features or numbers are present. However, it should be understood that it does not preclude the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

In the following description, the same identification symbol means the same configuration, and unnecessary redundant descriptions and descriptions of known technologies will be omitted.

Hereinafter, by describing preferred embodiments with reference to the accompanying drawings, it will be described in detail.

1 is a diagram illustrating a cloud game service providing system according to an embodiment.

Referring to FIG. 1 , a cloud game service providing system according to an embodiment may include a user terminal 100, a cloud game service providing device 200, and a game server 210.

According to an embodiment, the user terminal 100 may receive game play information of a game executed in the cloud game service providing device 200 or the game server 210 .

According to an embodiment, the user terminal 100 may receive a stream of a play video of a game executed in the cloud game service providing device 200 or the game server 210 .

According to an embodiment, the user terminal 100 may receive a stream of play video of a cloud game transmitted by the cloud game service providing device 200 .

According to an embodiment, the user terminal 100 may receive metadata about a sound of a cloud game corresponding to a stream of a play video of a cloud game transmitted by the device 200 for providing a cloud game service. At this time, the meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and object information matched with the sound of the cloud game. It may include at least one of location information and sound source information that matches the sound of a cloud game, but information that meta data may include is not limited thereto.

According to an embodiment, the user terminal 100 may obtain the sound of the cloud game based on the received metadata. In this case, the sound of the cloud game may include at least one of BGM (Back Ground Music), sound effect, and vibration included in the cloud game, but the sound of the cloud game may include is not limited thereto.

According to an embodiment, the sound of a cloud game played in the cloud game may be stored in the user terminal 100 .

According to an embodiment, the user terminal 100 may determine the sound of the cloud game to be output based on sound source information that matches the sound of the cloud game included in meta data in the stored sound sources.

According to an embodiment, the user terminal 100 determines the volume of the sound of the cloud game and the cloud game based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the sound of the cloud game. The stereo spatial feeling of the sound can be calculated.

According to an embodiment, the user terminal 100 may correct the sound of the cloud game determined based on the calculated volume of the sound of the cloud game and stereo spatial feeling of the sound of the cloud game.

According to an embodiment, the user terminal 100 calculates the volume of the sound of the cloud game based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the sound of the cloud game. A preset weight may be assigned to one of the calculated stereo spatial effects of the sound of the cloud game, and the determined sound of the cloud game may be corrected in consideration of the assigned weight. At this time, the preset weight is determined based on the in-game environment in which the game play video is generated (for example, when the game play is performed in a cave, it is determined to give a weight to the stereo spatial feeling of the sound of the cloud game, which is matched with the user terminal) If the distance between the character and the object that matches the sound of the cloud game in the game is very close, it may be determined to give a weight to the volume of the sound of the cloud game, etc.), but the criteria for determining the preset weight is not limited thereto. no.

According to an embodiment, the user terminal 100 may match and output the sound of the cloud game corrected at the time that matches the playback start time of the timestamp information included in the metadata with the stream of the play video of the cloud game. .

According to an embodiment, while the sound of the cloud game and the stream of the play video of the cloud game are matched and output, the user manipulates the user terminal 100 (eg, the user manipulates the keyboard to move the position of the character in the game). ), when at least one of location information of a character matched with the user terminal and location information of an object matched with the sound of the cloud game is changed by the user terminal 100, the cloud game service providing device 200 Transmitted modified metadata can be received. At this time, the modified metadata is a modified time indicating a time when at least one of the location information of a character matching the user terminal and the location information of an object matching the sound of the cloud game in the stream of the play video of the cloud game is changed. It may include at least one of stamp information, modified location information of a character that matches the user terminal, and modified location information of an object that matches the sound of a cloud game, but the information that the modified metadata can include is limited to this. it is not going to be

According to an embodiment, the user terminal 100 is based on at least one of modified location information of a character matching the user terminal included in the modified metadata and modified location information of an object matching the sound of the cloud game. The corrected volume of the corrected cloud game sound being output and the corrected stereo spatial feeling of the corrected cloud game sound being output may be recalculated.

According to an embodiment, the user terminal 100 may modify the corrected sound of the cloud game being output based on the corrected volume of the recalculated cloud game sound and the modified stereo spatial feeling of the recalculated cloud game sound. there is.

According to an embodiment, the user terminal 100 outputs recalculated based on the corrected location information of the character matching the user terminal included in the modified metadata and the corrected location information of the object matching the sound of the cloud game. A pre-set weight is given to either the volume of the sound of the cloud game being corrected and the stereo spatial feeling of the sound of the sound of the cloud game being output, which is recalculated, and the sound of the corrected cloud game being output is modified in consideration of the given weight. can At this time, the preset weight is determined based on the in-game environment in which the play video of the game is generated (for example, when the game play is performed in a cave, it is determined to give a weight to the stereo spatial feeling of the sound of the cloud game being output, and the user terminal and If the distance between the matched character and the object matched with the sound of the cloud game is very close, it may be determined to give a weight to the volume of the sound of the cloud game, etc.), but the criteria for determining the preset weight is not limited thereto. no.

According to an embodiment, the user terminal 100 may match and output the sound of the cloud game modified at the time matched with the modified time stamp information included in the modified metadata with the stream of the play video of the cloud game.

According to an embodiment, the user terminal 100 may receive metadata about a sound of a cloud game corresponding to a stream of a play video of a cloud game transmitted by the device 200 for providing a cloud game service. At this time, the meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and object information matched with the sound of the cloud game. It may include at least one of location information, object information matching the object's dialogue, and object's dialogue information, but the information that the meta data may include is not limited thereto. In addition, the form of the dialogue information of the object may be a text format, but the format of the dialogue information of the object is not limited thereto.

According to an embodiment, the user terminal 100 may obtain the sound of the cloud game based on the received metadata. In this case, the sound of the cloud game may include speech of at least one object included in the cloud game, but the sound that the sound of the cloud game may include is not limited thereto.

According to an embodiment, the user terminal 100 is a deep neural network (Deep Neural Networks, hereinafter referred to as a "deep neural network") based on object information that matches the object's dialogue included in metadata and the object's dialogue information. It is possible to generate an object's dialogue voice corresponding to the dialogue information of the object included in the stream of the play video of the cloud game. In this case, the deep neural network may be a long short-term memory (LSTM, hereinafter referred to as “long short-term memory”) or a neural vocoder that has been trained, but the deep neural network is not limited thereto.

According to an embodiment, the user terminal 100 is based on the positional information of the character matching the user terminal and the positional information of the object matching the dialogue of the object included in the meta data, the volume of the speech of the object and the dialogue of the object. It is possible to calculate the stereo spatial feeling of voice.

According to an embodiment, the user terminal 100 may correct the dialogue voice of the generated object based on the calculated volume of the dialogue voice of the object and the stereo spatial feeling of the dialogue voice of the object.

According to an embodiment, the user terminal 100 calculates the volume and calculation of the dialogue voice of the object calculated based on the positional information of the character matching the user terminal and the positional information of the object matching the dialogue of the object included in the metadata. Preset weights may be assigned to any one of the stereo spatial effects of an object's dialogue voice, and the determined object's dialogue voice may be corrected in consideration of the assigned weight. At this time, the preset weight is determined based on the in-game environment in which the game play video is generated (for example, when the game play is performed in a cave, it is determined to give a weight to the stereo spatial feeling of the dialogue voice of the object, which is matched with the user terminal If the distance between the character and the object that matches the dialogue of the object is very close, it may be determined to give a weight to the volume of the dialogue voice of the object, etc.), but the criteria for determining the preset weight is not limited thereto.

According to an embodiment, the user terminal 100 may match and output the corrected dialogue voice of the object with the play video stream of the cloud game at a time that matches the reproduction start time of the timestamp information included in the metadata. .

According to an embodiment, the user's manipulation of the user terminal 100 (for example, the user manipulates the keyboard to move the position of the character in the game) while matching and outputting a stream of dialogue voice of the object and a play video of the cloud game. ) When at least one of the location information of the character matching the user terminal and the location information of the object matching the line of the object is changed by the user terminal 100, the cloud game service providing device 200 transmits can receive modified metadata. At this time, the modified metadata is a modified time stamp indicating a time when at least one of location information of a character matching a user terminal and location information of an object matching an object's dialogue in the stream of a play video of a cloud game is changed. It may include at least one of information, modified positional information of a character that matches the user terminal, and modified positional information of an object that matches the dialogue of the object, but the information that the modified metadata may include is not limited thereto. no.

According to an embodiment, the user terminal 100 outputs based on at least one of modified location information of a character matching the user terminal included in the modified metadata and modified location information of an object matching the dialogue of the object. It is possible to recalculate the corrected volume of the dialogue voice of the object being corrected and the stereo spatial feeling of the dialogue voice of the object being output.

According to an embodiment, the user terminal 100 may modify the corrected object dialogue voice being output based on the recalculated corrected volume of the object's dialogue voice and the modified stereo spatial feeling of the recalculated object dialogue voice. there is.

According to an embodiment, the user terminal 100 is recalculated based on the corrected positional information of the character matching the user terminal included in the modified metadata and the modified positional information of the object matching the line of the object, and outputting the recalculated information. A pre-set weight is given to either the volume of the corrected object's dialogue voice or the recalculated stereo spatial feeling of the corrected object's dialogue voice being output, and the corrected object's dialogue voice being output can be modified in consideration of the weight given. there is. At this time, the preset weight is determined based on the in-game environment in which the game play video is generated (for example, when the game play is performed in a cave, it is determined to give a weight to the stereo spatial feeling of the dialogue voice of the object being output, and the user terminal and If the distance between the matched character and the object's line and the matched object's line is very close, it may be determined to give a weight to the volume of the line voice of the object), but the criteria for determining the preset weight is not limited thereto. .

According to an embodiment, the user terminal 100 may match and output the dialogue voice of the object modified at the time that matches the modified time stamp information included in the modified metadata with the stream of the play video of the cloud game.

According to an embodiment, the user terminal 100 refers to a terminal capable of transmitting and receiving various data using a communication intermediary device including an AP according to a user's key manipulation, a smart phone, a foldable smart phone, a tablet PC ( Tablet PC), laptop (Laptop), personal computer (PC: Personal Computer), smart phone (Smart Phone), personal portable information terminal (PDA: Personal Digital Assistant), and mobile communication terminal (Mobile Communication Terminal). However, the user terminal 100 is not limited thereto.

According to an embodiment, the user terminal 100 is a terminal that performs voice or data communication using an AP and a communication network, and a memory for storing a program or protocol for communicating with an external device via the AP and a communication network, a corresponding It refers to a terminal equipped with a microprocessor for calculating and controlling by executing a program. That is, the user terminal 100 can be any terminal as long as it can communicate using an AP and a communication network, and is a broad concept including any communication computing device such as a notebook computer, a mobile communication terminal, and a PDA.

According to an embodiment, the user terminal 100 is preferably manufactured in a form having a touch screen to perform input/output through a user's manipulation, and can drive a game program through an input according to a user's manipulation. .

According to an embodiment, the user terminal 100 preferably includes at least one of a video game terminal, an arcade terminal, a portable terminal, a PC terminal, and a wired or wireless terminal, and a predetermined key input provided in the terminal is provided. The means may include at least one of a keyboard, a mouse, a game control unit, an analog unit, and a touch screen.

According to one embodiment, user terminal 100 includes user equipment, mobile, PC capable of wireless communication, cell phone, kiosk, cellular phone, cellular, cellular terminal, subscriber unit, subscriber station, mobile station, terminal, remote station, PDA. , a remote terminal, an access terminal, a user agent, a portable device having a wireless connection function, and any device capable of using a wireless access mechanism, such as a wireless modem, but is not limited thereto. User terminal 100 may also be referred to as a terminal or a peer.

According to an embodiment, the cloud game service providing device 200 may distribute, operate, and manage cloud games.

According to an embodiment, the cloud game service providing device 200 is synchronously connected to the user terminal 100, and the cloud game service providing device 200 provides the cloud game service to the communicatively connected user terminal 100. can provide

According to an embodiment, the cloud game service providing device 200 may proceed with a membership sign-up procedure to store and manage various types of information of users subscribed as members.

According to an embodiment, the cloud game service providing device 200 may be communicatively connected to the game server 210 .

According to an embodiment, the cloud game service providing device 200 may determine the game server 210 to be accessed based on user terminal user identification information (eg, cloud game service providing device access ID, platform access ID, etc.) there is.

According to an embodiment, the cloud game service providing device 200 responds to a game play provision request for a game operated in the game server 210 and provides a game operated in the game server 210 provided by the game server 210. The game play for can be provided.

According to an embodiment, the cloud game service providing apparatus 200 may transmit an acquired input of a user terminal user related to a game provided by the game server 210 to the game server 210 .

According to an embodiment, the cloud game service providing device 200 may progress game play in the game server 210 based on a user terminal user's input.

According to an embodiment, the cloud game service providing apparatus 200 may transmit a play video stream of a cloud game played by a user of the user terminal to the user terminal 100 .

According to an embodiment, the cloud game service providing device 200 transmits sound data about the sound of the cloud game corresponding to the stream of the play video of the cloud game to the user terminal 100 as metadata, thereby providing the cloud game service providing device. It is possible to reduce the amount of computation and traffic of

According to an embodiment, the cloud game service providing apparatus 200 may generate metadata about the sound of a cloud game corresponding to a stream of a play video of a cloud game played by a user terminal user.

According to an embodiment, the cloud game service providing device 200 does not transmit the sound of the cloud game matched with the play video stream of the cloud game, and transmits the play video stream of the cloud game. Meta data about the sound of the cloud game corresponding to the stream may be transmitted to the user terminal 100 .

According to an embodiment, while the sound of the cloud game and the stream of the play video of the cloud game are matched and output, by manipulation of the user terminal user (eg, the user manipulates the keyboard to move the position of the character in the game) When at least one of location information of a character matching a user terminal and location information of an object matching a line of an object is changed, the cloud game service providing device 200 transmits modified metadata to the user terminal 100. can transmit

According to an embodiment, the game server 210 may transmit a game requested by the cloud game service providing device 200 or the user terminal 100 to the cloud game service providing device 200 or the user terminal 100 .

According to an embodiment, the game server 210 may transmit a stream of play video of a cloud game played by a user terminal user to the cloud game service providing device 200 or the user terminal 100 .

According to an embodiment, the game server 210 may generate metadata about the sound of the cloud game corresponding to the stream of the play video of the cloud game played by the user terminal user.

According to an embodiment, the game server 210 may transmit metadata about the sound of the cloud game corresponding to the stream of the generated play video of the cloud game to the cloud game service providing device 200 or the user terminal 100. there is.

According to one embodiment, the cloud game service providing device 200 and the game server 210 may operate within the same platform.

According to an embodiment, the cloud game service providing device 200 and the game server 210 may exchange data using a network.

According to an embodiment, the cloud game service providing device 200 and the game server 210 may mutually exchange data using a platform.

According to an embodiment, the cloud game service providing apparatus 200 and the game server 210 may identify a user of a user terminal using one user identification information (eg, an ID for accessing a platform).

According to one embodiment, the cloud game service providing device 200 and the game server 210 may have the same configuration as a conventional web server or WAP server in terms of hardware. However, in terms of software, it may include a program module that is implemented through any language such as C, C++, Java, Visual Basic, or Visual C and performs various functions.

According to an embodiment, the cloud game service providing device 200 and the game server 210 are generally connected to an unspecified number of authentication clients and/or other servers through an open computer network such as the Internet, and the authentication client or other server It refers to a computer system that accepts work performance requests, derives and provides work results for them, and computer software (server programs) installed for them.

According to an embodiment, the cloud game service providing device 200 and the game server 210, in addition to the above-described server program, a series of applications (Application Program) and various databases (DB: Database, hereinafter referred to as "DB") built internally or externally depending on the case, should be understood as a broad concept. Therefore, the cloud game service providing apparatus 200 and the game server 210 classify member subscription information and various information and data about games, store and manage them in a DB, which is the cloud game service providing apparatus 200. ) can be implemented inside or outside of.

According to one embodiment, the cloud game service providing device 200 and the game server 210 are DOS, Windows, Linux, UNIX, Macintosh, etc. in general server hardware. It can be implemented using server programs that are provided in various ways depending on the operating system of the operating system, and representative examples are CERN, NCSA, and APPACH used in Windows environment, IIS (Internet Information Server) and Unix environment. etc. can be used.

2 is a diagram illustrating a configuration of a user terminal according to an embodiment.

Referring to FIG. 2 , a user terminal according to an embodiment includes a processor 110 , an input/output interface module 120 and a memory 130 .

According to one embodiment, the processor 110, the input/output interface module 120, and the memory 130 included in the user terminal 100 are connected to each other and can transmit data to each other.

According to one embodiment, the processor 110 may execute programs or instructions stored in the memory 130 . At this time, an operating program (eg, OS) for operating the user terminal 100 may be stored in the memory 130 .

According to one embodiment, the processor 110 may execute a program for managing information about the user terminal 100 .

According to an embodiment, the processor 110 may execute a program for managing the operation of the user terminal 100 .

According to one embodiment, the processor 110 may execute a program for managing the operation of the input/output interface module 120 .

According to an embodiment, the processor 110 may obtain game play information of a game executed in a cloud game service providing device or a game server through the input/output interface module 120 .

According to an embodiment, the processor 110 may obtain a stream of game play video transmitted by a cloud game service providing device through the input/output interface module 120 .

According to an embodiment, the processor 110 may obtain metadata about the sound of the cloud game corresponding to the stream of the play video of the cloud game transmitted by the cloud game service providing device through the input/output interface module 120. . At this time, the meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and object information matched with the sound of the cloud game. It may include at least one of location information and sound source information that matches the sound of a cloud game, but information that meta data may include is not limited thereto.

According to an embodiment, the processor 110 may acquire the sound of the cloud game based on sound source information that matches the sound of the cloud game included in the acquired metadata. In this case, the sound of the cloud game may include at least one of BGM (Back Ground Music), sound effect, and vibration included in the cloud game, but the sound of the cloud game may include is not limited thereto.

According to an embodiment, the processor 110 may determine the sound of the cloud game to be output based on sound source information that matches the sound of the cloud game included in meta data in the sound source stored in the memory 130 . In this case, the stored sound source may be a sound source expressing a preset volume (eg, a medium volume among expressible sounds in a game) and a preset sound space (eg, an intermediate sound spatial feeling among expressible sounds on a game). The sound source is not limited thereto.

According to an embodiment, the processor 110 determines the location information of the character matched with the user terminal and the location information of the object matched with the sound of the cloud game included in the meta data, and the character matched with the user terminal and the cloud game. The distance between the sound and the matched object can be determined.

According to an embodiment, the processor 110 may calculate the volume of the sound of the cloud game based on the distance between the character matching the determined user terminal and the object matching the sound of the cloud game.

According to an embodiment, the processor 110 determines the location information of the character matched with the user terminal and the location information of the object matched with the sound of the cloud game included in the meta data, and the character matched with the user terminal and the cloud game. An environment where an object matching the sound exists in the game may be determined (eg, a cave in the game, a plain in the game, an inside of a dungeon in the game, etc.).

According to an embodiment, the processor 110 may perform an environment where a character matching the determined user terminal and an object matching the sound of the cloud game exist in the game (eg, a cave in the game, a plain in the game, and a dungeon in the game). interior, etc.), it is possible to calculate the stereo spatial feeling of the sound of the cloud game.

According to an embodiment, the processor 110 may correct the sound of the cloud game determined based on the calculated volume of the sound of the cloud game and stereo spatial feeling of the sound of the cloud game.

According to an embodiment, the character matching the user terminal and the sound of the cloud game determined based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the sound of the cloud game When the distance between objects is within a preset distance (eg, 1 m in game), the processor 110 may assign a weight to the calculated sound volume of the cloud game.

According to an embodiment, the character matching the user terminal and the sound of the cloud game determined based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the sound of the cloud game If the environment where the object exists in the game is a preset environment (eg, a cave in the game, a cavity in a dungeon in the game, etc.), the processor 110 may assign a weight to the calculated stereo spatial feeling of the sound of the cloud game. can

According to an embodiment, the processor 110 may correct the determined sound of the cloud game in consideration of a weight assigned to the volume of the calculated sound of the cloud game and a weight assigned to the stereo spatial effect of the calculated sound of the cloud game.

According to an embodiment, the processor 110 may output the sound of the cloud game corrected at a time that matches the playback time of the timestamp information included in the meta data by matching it with the play video stream of the cloud game.

According to an embodiment, while the sound of the cloud game and the stream of the play video of the cloud game are matched and output, location information of a character matched with the user terminal and cloud game by the user's input input to the input/output interface module 120 When at least one of the location information of the object matching the sound of is changed, the processor 110 may obtain modified metadata transmitted by the cloud game service providing device through the input/output interface module 120. At this time, the modified metadata is a modified time indicating a time when at least one of the location information of a character matching the user terminal and the location information of an object matching the sound of the cloud game in the stream of the play video of the cloud game is changed. It may include at least one of stamp information, modified location information of a character that matches the user terminal, and modified location information of an object that matches the sound of a cloud game, but the information that the modified metadata can include is limited to this. it is not going to be

According to an embodiment, the processor 110 is matched with the user terminal based on the modified location information of the character matching the user terminal included in the modified metadata and the modified location information of the object matching the sound of the cloud game. The distance between the character and the object matching the sound of the cloud game may be re-determined.

According to an embodiment, the processor 110 may recalculate the volume of the corrected sound of the cloud game being output based on the distance between the re-determined user terminal and the matching object and the sound of the cloud game.

According to an embodiment, the processor 110 is matched with the user terminal based on the modified location information of the character matching the user terminal included in the modified metadata and the modified location information of the object matching the sound of the cloud game. An environment (eg, a cave in a game, a plain in a game, an inside of a dungeon in a game, etc.) may be re-determined where an object matching the sound of a character and a cloud game exists in the game.

According to an embodiment, the processor 110 may perform an environment where a character matching the re-determined user terminal and an object matching the sound of the cloud game exist in the game (eg, a cave in the game, a plain in the game, and an object in the game). Inside the dungeon, etc.), it is possible to recalculate the stereo spatial feeling of the sound of the cloud game being output.

According to an embodiment, the processor 110 may correct the corrected sound of the cloud game being output based on the recalculated volume of the sound of the cloud game and the stereo spatial feeling of the sound of the cloud game.

According to an embodiment, the character matching the user terminal and the cloud re-determined based on the modified location information of the character matching the user terminal included in the modified metadata and the position information of the modified object matching the sound of the cloud game. When the distance between the sound of the game and the matched object is within a preset distance (eg, 1 m in the game), the processor 110 may assign a weight to the volume of the recalculated sound of the cloud game being output. .

According to an embodiment, the character matching the user terminal and the cloud re-determined based on the modified location information of the character matching the user terminal included in the modified metadata and the position information of the modified object matching the sound of the cloud game. If the environment where the object matching the sound of the game exists in the game is a preset environment (eg, a cave in the game, a cavity in a dungeon in the game, etc.), the processor 110 recalculates and outputs the corrected cloud. Weight can be assigned to the stereo spatial feeling of the sound of the game.

According to an embodiment, the processor 110 considers a weight assigned to the volume of the recalculated sound of the cloud game being output and a weight assigned to the stereo spatial feeling of the recalculated sound of the cloud game being output. You can modify the sound of the calibrated cloud game.

According to an embodiment, the processor 110 may output a matched sound of a cloud game modified at a time matching the modified time stamp information included in modified metadata with a stream of play video of the cloud game.

According to an embodiment, the processor 110 may obtain metadata about the sound of the cloud game corresponding to the stream of the play video of the cloud game transmitted by the cloud game service providing device through the input/output interface module 120. . At this time, the meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and object information matched with the sound of the cloud game. It may include at least one of location information, object information matching the object's dialogue, and object's dialogue information, but the information that the meta data may include is not limited thereto. In addition, the form of the dialogue information of the object may be a text format, but the format of the dialogue information of the object is not limited thereto.

According to one embodiment, the processor 110 may acquire the sound of the cloud game based on the acquired metadata. In this case, the sound of the cloud game may include speech of at least one object included in the cloud game, but the sound that the sound of the cloud game may include is not limited thereto.

According to an embodiment, the processor 110 uses a deep neural network based on object information that matches the object's dialogue included in the meta data and the object's dialogue information, so that the object included in the stream of the play video of the cloud game is displayed. A dialogue voice of an object corresponding to dialogue information may be generated. In this case, the deep neural network may be a long short-term memory (LSTM) that has been learned or a neural vocoder that has been learned, but the deep neural network is not limited thereto. In addition, the dialogue voice of the generated object is of the object created to express a preset volume (eg, a medium volume among expressible sounds in a game) and a preset sense of sound space (eg, an intermediate sound spatial sense among expressible sounds in a game). It may be a dialogue voice, but the spatial sense of volume and sound expressed by the dialogue voice of the generated object is not limited thereto.

According to an embodiment, the processor 110 is a character matched with the user terminal and the sound of the cloud game based on the position information of the character matched with the user terminal included in the metadata and the position information of the object matched with the lines of the object. It is possible to determine the distance between objects matching with .

According to an embodiment, the processor 110 may calculate the volume of the object's dialogue voice based on the distance between the character matching the determined user terminal and the object matching the sound of the cloud game.

According to an embodiment, the processor 110 may include the dialogues of characters and objects matched with the user terminal based on the positional information of the character and the positional information of the object matched with the dialogue of the object included in the meta data. An environment where the matched object exists in the game (eg, a cave in the game, a plain in the game, an inside of a dungeon in the game, etc.) may be determined.

According to an embodiment, the processor 110 may perform an environment in which a character matching the determined user terminal and an object matching the lines of the object exist in the game (eg, a cave in the game, a plain in the game, and an inside of a dungeon in the game). etc.), it is possible to calculate the stereo spatial feeling of the dialogue voice of the object.

According to an embodiment, the processor 110 may correct the dialogue voice of the generated object based on the calculated volume of the dialogue voice of the object and the stereo spatial feeling of the dialogue voice of the object.

According to an embodiment, between the character matching the user terminal and the object matching the dialogue of the object determined based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the dialogue of the object. When the distance of is within a preset distance (eg, 1 m in game), the processor 110 may assign a weight to the volume of the calculated dialogue voice of the object.

According to an embodiment, a character matching the user terminal and an object matching the dialogue of the object determined based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the dialogue of the object are determined. When the environment of the place in the game is a preset environment (eg, a cave in the game, a cavity in a dungeon in the game, etc.), the processor 110 may assign a weight to the stereo spatial feeling of the calculated dialogue voice of the object. .

According to an embodiment, the processor 110 may correct the dialogue voice of the created object in consideration of a weight assigned to the volume of the calculated dialogue voice of the object and a weight assigned to the stereo spatial effect of the calculated dialogue voice of the object.

According to an embodiment, the processor 110 may match the corrected object's dialogue voice with the play video stream of the cloud game at a time that matches the playback time of the timestamp information included in the meta data and output it.

According to an embodiment, while the stream of the speech of the object and the play video of the cloud game are matched and output, the user's input to the input/output interface module 120 determines the location information of the character matched with the user terminal and the object When at least one of the location information of the object matching the line is changed, the processor 110 may obtain modified metadata transmitted by the cloud game service providing device through the input/output interface module 120. At this time, the modified metadata is a modified time stamp indicating a time when at least one of location information of a character matching a user terminal and location information of an object matching an object's dialogue in the stream of a play video of a cloud game is changed. It may include at least one of information, modified positional information of a character that matches the user terminal, and modified positional information of an object that matches the dialogue of the object, but the information that the modified metadata may include is not limited thereto. no.

According to one embodiment, the processor 110 is a character matched with the user terminal based on the corrected positional information of the character matching the user terminal included in the modified metadata and the modified positional information of the object matching the lines of the object. It is possible to re-determine the distance between the object and the object matching the dialogue of the object.

According to an embodiment, the processor 110 may recalculate the volume of the voice of the corrected object's dialogue voice being output, based on the distance between the dialogue of the character and object matching the re-determined user terminal and the matching object.

According to one embodiment, the processor 110 is a character matched with the user terminal based on the corrected positional information of the character matching the user terminal included in the modified metadata and the modified positional information of the object matching the lines of the object. It is possible to re-determine an environment (eg, a cave in a game, a plain in a game, an inside of a dungeon in a game, etc.)

According to an embodiment, the processor 110 may perform an environment in which a character matching the re-determined user terminal and an object matching the lines of the object and the matching object exist in the game (eg, a cave in the game, a plain in the game, and a dungeon in the game). The stereo spatial feeling of the dialogue voice of the corrected object being output can be recalculated based on the interior, etc.).

According to an embodiment, the processor 110 may modify the calibrated dialogue voice of the object being output based on the recalculated volume of the dialogue voice of the object and the stereo spatial effect of the dialogue voice of the object.

According to an embodiment, the character and object matching the user terminal re-determined based on the modified location information of the character matching the user terminal included in the modified metadata and the location information of the modified object matching the line of the object. When the distance between the dialogue and the matched object is within a preset distance (eg, 1 m in the game), the processor 110 may assign a weight to the recalculated volume of the corrected dialogue voice of the object being output.

According to an embodiment, the character and object matching the user terminal re-determined based on the modified location information of the character matching the user terminal included in the modified metadata and the location information of the modified object matching the line of the object. If the environment where the object matching the dialogue exists in the game is a preset environment (eg, a cave in the game, a cavity in a dungeon in the game, etc.), the processor 110 recalculates and outputs the dialogue of the corrected object. A weight can be assigned to the stereo spatial feeling of the voice.

According to an embodiment, the processor 110 considers the recalculated weight given to the volume of the corrected object's dialogue voice being output and the recalculated weight given to the stereo spatial sense of the corrected object's dialogue voice being output. It is possible to modify the dialogue voice of the calibrated object.

According to an embodiment, the processor 110 may match the voice of the modified object with the stream of the play video of the cloud game at the time that matches the modified time stamp information included in the modified metadata, and output the same.

According to an embodiment, the input/output interface module 120 may be communicatively connected to a cloud game service providing device through a network.

According to an embodiment, the input/output interface module 120 includes metadata about a cloud game play video stream transmitted by a cloud game service providing device over a network and a cloud game sound corresponding to the cloud game play video stream. can receive

According to an embodiment, the input/output interface module 120 may transmit data to a cloud game service providing device through a network.

According to an embodiment, the input/output interface module 120 may receive data transmitted by a cloud game service providing device through a network.

According to one embodiment, the input/output interface module 120 may be communicatively connected to the game server through a network.

According to one embodiment, the input/output interface module 120 may transmit data to the game server through a network.

According to one embodiment, the input/output interface module 120 may receive data transmitted by the game server through a network.

According to an embodiment, the input/output interface module 120 may output a play video stream of a cloud game and a sound of a cloud game corresponding to the play video stream of the cloud game.

According to one embodiment, the input/output interface module 120 may obtain an input of a user terminal user.

According to one embodiment, the input/output interface module 120 may be provided integrally with the user terminal 100 .

According to one embodiment, the input/output interface module 120 may be provided separately from the user terminal 100 .

According to one embodiment, the input/output interface module 120 may be a separate device communicatively connected to the user terminal 100 .

According to one embodiment, the input/output interface module 120 may include a port (eg, USB port) for connection with an external device.

According to one embodiment, the input/output interface module 120 may include a monitor, touch screen, mouse, electronic pen, microphone, keyboard, speaker, earphone, headphone, or touch pad.

According to one embodiment, memory 130 may store sound sources.

According to an embodiment, the memory 130 may store speech dialogue of an object generated by the processor 110 .

According to one embodiment, the memory 130 may store data received by the input/output interface module 120 .

According to one embodiment, the memory 130 may store data transmitted by the input/output interface module 120 .

According to one embodiment, the memory 130 may store a user's input acquired by the input/output interface module 120 .

The term 'module' used herein denotes a logical unit, and it is obvious to those skilled in the art that the term 'module' is not necessarily a physically separated component.

FIG. 3 is a diagram illustrating how a user terminal matches and outputs speech of an object generated based on metadata with a stream of play video of a cloud game according to an embodiment.

Referring to FIG. 3 , according to an embodiment, the cloud game service providing device 300 transmits sound data for the sound of the cloud game corresponding to the stream 301 of the play video of the cloud game to the user terminal 310 as meta data. It is possible to reduce the amount of computation and traffic of the cloud game service providing device by transmitting it as data.

According to an embodiment, the cloud game service providing apparatus 300 may generate metadata 302 about the sound of the cloud game corresponding to the stream 301 of the play video of the cloud game played by the user terminal user. .

According to an embodiment, the cloud game service providing device 300 includes a stream 301 of a play video of a cloud game played by a user of a user terminal and a sound of a cloud game corresponding to a stream 301 of a play video of the cloud game. Meta data 302 for may be transmitted to the user terminal 310 .

According to an embodiment, the cloud game service providing device 300 may not transmit the cloud game sound corresponding to the stream 301 of the play video of the cloud game to the user terminal 310 .

According to an embodiment, the stream 301 of the play video of the cloud game transmitted from the cloud game service providing device 300 to the user terminal 310 and the metadata 302 of the sound of the cloud game corresponding thereto include the cloud game In the stream 301 of the play video, time stamp information representing the playback start time and playback end time of the object's line, the location information of the character matching the user terminal, the location information of the object matching the line of the object, and the line of the object However, the information that can be included in the stream 301 of the play video of the cloud game and the metadata 302 of the sound of the cloud game corresponding to it may be included. . In addition, the object information matching the lines of the object may be the name of a non-player character (NPC) in the game (for example, the name of a non-player character), but the object information matching the lines of the object is not limited thereto. In addition, the dialogue information of the object may be in text form, but the format of the dialogue information of the object is not limited thereto.

According to an embodiment, the user terminal 310 is a cloud game corresponding to a stream of a play video of a cloud game played by a user of the user terminal transmitted by the cloud game service providing device 300 and a stream of a play video of the cloud game. Meta data about the sound of can be received.

According to an embodiment, the user terminal 310 transmits metadata about the sound of the cloud game corresponding to the stream of the received play video of the cloud game (eg, object information matching the dialogue of the object and dialogue information of the object). Based on this, a cloud game sound (eg, object dialogue voice) may be generated using a deep neural network.

According to an embodiment, the user terminal 310 transmits the cloud game sound (eg, object dialogue voice) generated at the time matching the reproduction start time of the time stamp information included in the metadata to the play video of the cloud game. It can be output 311 by matching with the stream. At this time, the user terminal 310 displays the name of the NPC (Non-Player Character) in the game, which is object information matched with the object's dialogue (eg, the name of the non-player character) and the dialogue information of the object in a preset image (eg, a speech bubble). to output 311.

According to an embodiment, the user terminal 310 transmits the cloud game sound (eg, object dialogue voice) generated up to the time matching the reproduction end time of the time stamp information included in the metadata to the play video of the cloud game. It can be output 311 by matching with the stream.

4 is a diagram illustrating how a user terminal modifies a sound of a cloud game based on meta data and modified meta data according to an embodiment.

Referring to FIG. 4 , according to an embodiment, a user terminal corresponds to a stream 400 of a play video of a cloud game played by a user of the user terminal transmitted by a cloud game service providing device and a stream of a play video of the cloud game. Meta data 410 about the sound of the cloud game being played may be acquired. At this time, the meta data 410 includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and matching with the sound of the cloud game. At least one of location information of a target object and sound source information that matches the sound of a cloud game may be included, but information that may be included in the meta data 410 is not limited thereto.

According to an embodiment, the user terminal includes time stamp information 401 representing the playback start time of the cloud game sound in the stream of the play video of the cloud game included in the metadata 410 and time stamp information representing the playback end time. Based on step 402, the sound of the cloud game may be matched with the play video stream of the cloud game and then output.

According to an embodiment, the user terminal may determine the sound of the cloud game to be output based on the sound source information 411 that matches the sound of the cloud game included in the meta data 410 in the stored sound source.

According to an embodiment, the user terminal determines the sound of the cloud game determined based on the location information of the character matched with the user terminal included in the metadata 410 and the location information 412 of the object matched with the sound of the cloud game. It is possible to calculate the stereo spatial feeling of the sound of the volume and the determined cloud game.

According to an embodiment, the user terminal may correct the sound of the cloud game determined based on the volume of the calculated sound of the cloud game and the stereo spatial feeling of the calculated sound of the cloud game.

According to an embodiment, the user terminal corrects the time corresponding to the timestamp information 401 and 413 indicating the start time of playing the sound of the cloud game in the stream of the play video of the cloud game included in the metadata 410. The sound of the cloud game may be matched with the stream 400 of the play video of the cloud game and output 420 .

According to an embodiment, while the sound of the cloud game and the stream of the play video of the cloud game are matched and output (401, 402), the user's manipulation of the user terminal (eg, the user's keyboard to move the position of the character in the game) If at least one of the location information of a character matched with the user terminal and the location information of an object matched with the sound of the cloud game is changed by manipulating the user terminal, the user terminal receives the modified meta transmitted by the cloud game service providing device. Data 414 may be received.

According to an embodiment, when the user terminal receives the modified metadata 414 while the sound of the cloud game is being output (401, 402), the user terminal receives the cloud game included in the meta data 410 from the stored sound source. Sound source information 415 matching the sound of may not be newly determined.

According to an embodiment, the user terminal is being output based on the corrected positional information of the character matched with the user terminal included in the modified metadata 414 and the positional information 416 of the object matched with the sound of the cloud game. It is possible to recalculate the volume of the sound of one cloud game and the stereo spatial feeling of the sound of the cloud game that has been calibrated during output.

According to an embodiment, the user terminal may correct the corrected cloud game sound based on the volume of the recalculated and corrected cloud game sound and the stereo spatial feeling of the recalculated and corrected cloud game sound.

According to an embodiment, the user terminal receives at least one of location information of a character matching the user terminal and position information of an object matching the lines of the object in the stream of the play video of the cloud game included in the modified metadata 414. The sound of the cloud game modified in the modified time stamp information 417 indicating the time at which the location information of is matched with the stream 400 of the play video of the cloud game may be output (421).

According to an embodiment, the user terminal transmits the sound of the cloud game from the stream of the play video of the cloud game included in the meta data 410 to the time corresponding to the time stamp information 402 indicating the end time of playing the sound of the cloud game. can be output by matching with the stream of the play video of the cloud game.

FIG. 5 is a diagram illustrating how a user terminal modifies speech of an object based on meta data and modified meta data according to an exemplary embodiment.

Referring to FIG. 5 , according to an embodiment, a user terminal corresponds to a stream 500 of a play video of a cloud game played by a user of the user terminal transmitted by a cloud game service providing device and a stream of a play video of the cloud game. Meta data 510 for the sound of the cloud game to be played may be obtained. At this time, the meta data 510 includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matching with the user terminal, and matching with the lines of an object. At least one of object location information, object information that matches the object's dialogue, and object dialogue information may be included, but information that may be included in the meta data 510 is not limited thereto. In addition, the sound of the cloud game may include speech of objects included in the cloud game, but the sound of the cloud game may be included is not limited thereto.

According to an embodiment, the user terminal includes time stamp information 501 representing the reproduction start time of the object's dialogue voice in the stream of the play video of the cloud game included in the metadata 510 and time stamp information indicating the reproduction end time. Based on step 502, the speech of the object may be matched with the play video stream of the cloud game and then output.

According to an embodiment, the user terminal uses a deep neural network based on the object information 512 matching the object's dialogue included in the metadata 510 and the object's dialogue information 511 to provide the object's dialogue information ( 511) may generate a dialogue voice of an object corresponding to.

According to one embodiment, the user terminal is generated based on the location information of the character matched with the user terminal included in the metadata 510 and the location information 513 of the object matched with the dialogue of the object, which is generated based on the dialogue voice of the object. It is possible to calculate the volume and stereo spatial feeling of the dialogue voice of the created object.

According to an embodiment, the user terminal may correct the dialogue voice of the created object based on the volume of the dialogue voice of the calculated object and the stereo spatial feeling of the calculated dialogue voice of the object.

According to an embodiment, the user terminal corrects the time corresponding to the timestamp information (501, 514) indicating the start time of playback of the speech of the object in the stream of the play video of the cloud game included in the metadata 510. The speech of the object may be matched with the stream 500 of the play video of the cloud game and output 520 .

According to an embodiment, while the object's dialogue voice and the stream of the play video of the cloud game are matched and output (501, 502), the user's manipulation of the user terminal (eg, the user's keyboard to move the position of the character in the game, etc.) If at least one of the location information of the character matching the user terminal and the location information of the object matching the dialogue of the object is changed by manipulating the user terminal, the user terminal receives modified metadata transmitted by the cloud game service providing device. (515) can be received.

According to an embodiment, when the user terminal receives the modified meta data 515 while the speech of the object is being output (501, 502), the user terminal matches the dialogue of the object included in the meta data 510. Based on the object information 517 and the object's dialogue information 516, the object's dialogue voice corresponding to the object's dialogue information 516 may not be generated using a deep neural network.

According to an embodiment, the user terminal corrects the correction being output based on the modified location information of the character matching the user terminal included in the modified metadata 515 and the location information 518 of the object matching the lines of the object. The volume of the dialogue voice of the object and the stereo spatial feeling of the corrected dialogue voice of the object being output may be recalculated.

According to an embodiment, the user terminal may correct the corrected object's dialogue voice based on the recalculated volume of the corrected object's dialogue voice and the recalculated stereo spatial effect of the corrected object's dialogue voice.

According to an embodiment, the user terminal receives at least one of location information of a character matching the user terminal and location information of an object matching the dialogue of the object in the stream of the play video of the cloud game included in the modified metadata 515. The dialogue voice of the object modified in the modified time stamp information 519 indicating the time at which the location information of is matched with the stream 500 of the play video of the cloud game can be output (521).

According to an embodiment, the user terminal provides the object's dialogue voice until the time corresponding to the timestamp information 502 representing the end time of reproduction of the object's dialogue voice in the stream of play video of the cloud game included in the metadata 510. can be output by matching with the stream of the play video of the cloud game.

6 is a flowchart illustrating a cloud game service providing method according to an embodiment.

Referring to FIG. 6 , a cloud game service providing device transmits metadata about a stream of a play video of a cloud game played by a user of a user terminal and a sound of a cloud game corresponding to the stream of the play video of the cloud game to the user terminal. Transmit (600).

At this time, the meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and object information matched with the sound of the cloud game. It may include at least one of location information and sound source information that matches the sound of a cloud game, but information that meta data may include is not limited thereto.

The user terminal determines the sound of the cloud game to be output based on meta data from a sound source stored in the user terminal (610).

In this case, the sound of the cloud game played in the cloud game may be stored in the user terminal.

In addition, the stored sound source may be a sound source that expresses a preset volume (eg, a medium volume among expressible sounds in a game) and a preset sense of sound space (eg, an intermediate sound spatial sense among expressible sounds in a game). The sound source is not limited thereto.

The user terminal calculates the volume of the sound of the cloud game determined based on the positional information of the character matched with the user terminal and the positional information of the object matched to the sound of the cloud game and the stereo spatial feeling of the determined sound of the cloud game (620).

At this time, the user terminal is located between the character matched with the user terminal and the object matched with the sound of the cloud game based on the positional information of the character matched with the user terminal included in the metadata and the positional information of the object matched with the sound of the cloud game. distance can be determined.

Also, the user terminal may calculate the volume of the sound of the cloud game based on the distance between the determined character matching the user terminal and the object matching the sound of the cloud game.

In addition, the user terminal has a character matched with the user terminal and an object matched with the sound of the cloud game based on the position information of the character matched with the user terminal included in the metadata and the position information of the object matched with the sound of the cloud game. The environment of the place in the game (eg, a cave in the game, a plain in the game, inside a dungeon in the game, etc.) can be determined.

In addition, the user terminal is based on the environment where the character matching the determined user terminal and the object matching the sound of the cloud game exist in the game (eg, a cave in the game, a plain in the game, an inside of a dungeon in the game, etc.) The stereo spatial feeling of the sound of the cloud game can be calculated.

The sound of the cloud game determined based on the volume of the sound of the cloud game calculated by the user terminal and the stereo spatial feeling of the calculated sound of the cloud game is corrected (630).

At this time, the distance between the character matched with the user terminal and the object matched with the sound of the cloud game determined based on the location information of the character matched with the user terminal included in the metadata and the position information of the object matched with the sound of the cloud game. If is within a preset distance (eg, 1 m in the game), the user terminal may assign a weight to the calculated volume of the sound of the cloud game.

In addition, the character matching the user terminal and the object matching the sound of the cloud game determined based on the location information of the character matching the user terminal and the location information of the object matching the sound of the cloud game included in the metadata are displayed in the game. When the environment of a place in is a preset environment (eg, a cave in a game, a cavity in a dungeon in a game, etc.), the user terminal may assign a weight to the calculated stereo spatial feeling of the sound of the cloud game.

In addition, the user terminal may correct the determined sound of the cloud game in consideration of a weight assigned to the volume of the calculated sound of the cloud game and a weight assigned to the stereo space of the calculated sound of the cloud game.

The sound of the cloud game corrected at the time when the user terminal matches the time stamp information is matched with the play video stream of the cloud game and outputs (640).

At this time, while the sound of the cloud game and the stream of the play video of the cloud game are matched and output, at least one of location information of a character matched with the user terminal and location information of an object matched with the sound of the cloud game by a user's input. When the location information of is changed, the user terminal may obtain modified metadata transmitted by the cloud game service providing device. At this time, the modified metadata is a modified time indicating a time when at least one of the location information of a character matching the user terminal and the location information of an object matching the sound of the cloud game in the stream of the play video of the cloud game is changed. It may include at least one of stamp information, modified location information of a character that matches the user terminal, and modified location information of an object that matches the sound of a cloud game, but the information that the modified metadata can include is limited to this. it is not going to be

In addition, the user terminal is based on the modified location information of the character matching the user terminal included in the modified metadata and the modified location information of the object matching the sound of the cloud game, and the character matched with the user terminal and the sound of the cloud game. It is possible to re-determine the distance between objects matching with

Also, the user terminal may recalculate the volume of the corrected cloud game sound being output based on the distance between the re-determined user terminal matched character and the cloud game sound matched object.

In addition, the user terminal is based on the modified location information of the character matching the user terminal included in the modified metadata and the modified location information of the object matching the sound of the cloud game, and the character matched with the user terminal and the sound of the cloud game. An environment (eg, a cave in a game, a plain in a game, an inside of a dungeon in a game, etc.) may be re-determined where an object matching with exists in the game.

In addition, the user terminal is based on the environment where the character matching the re-determined user terminal and the object matching the sound of the cloud game exist in the game (eg, a cave in the game, a plain in the game, an inside of a dungeon in the game, etc.) It is possible to recalculate the stereo spatial feeling of the sound of the cloud game being output with .

Also, the user terminal may correct the corrected sound of the cloud game being output based on the recalculated volume of the sound of the cloud game and the stereo spatial feeling of the sound of the cloud game.

In addition, based on the modified location information of the character matching the user terminal included in the modified metadata and the location information of the modified object matching the sound of the cloud game, the character matching the user terminal and the sound of the cloud game are re-determined. When the distance between the matched objects is within a preset distance (eg, 1 m in the game), the user terminal may assign a weight to the volume of the recalculated and corrected sound of the cloud game being output.

In addition, based on the modified location information of the character matching the user terminal included in the modified metadata and the location information of the modified object matching the sound of the cloud game, the character matching the user terminal and the sound of the cloud game are re-determined. If the environment where the matching object exists in the game is a preset environment (eg, in-game cave, in-game dungeon cavity, etc.), the user terminal recalculates and outputs the corrected stereo spatial feeling of the sound of the cloud game. can be weighted.

In addition, the user terminal considers the recalculated weight given to the volume of the corrected cloud game sound being output and the recalculated weight given to the stereo space of the corrected cloud game sound being output, and the corrected sound of the cloud game being output. can be modified.

In addition, the user terminal may match and output the sound of the cloud game modified at the time that matches the modified time stamp information included in the modified metadata with the stream of the play video of the cloud game.

7 is a flowchart illustrating a cloud game service providing method according to another embodiment.

Referring to FIG. 7 , a cloud game service providing device transmits metadata about a stream of a play video of a cloud game played by a user of a user terminal and a sound of a cloud game corresponding to the stream of the play video of the cloud game to the user terminal. Transmit (700).

At this time, the meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and object information matched with the sound of the cloud game. It may include at least one of location information, object information matching the object's dialogue, and object's dialogue information, but the information that the meta data may include is not limited thereto. In addition, the form of the dialogue information of the object may be a text format, but the format of the dialogue information of the object is not limited thereto.

The user terminal generates a dialogue voice of the object corresponding to the dialogue information of the object by using a deep neural network based on the meta data (710).

At this time, the user terminal uses a deep neural network based on object information matching the object's dialogue included in the meta data and the object's dialogue information, and the object corresponding to the dialogue information of the object included in the stream of the play video of the cloud game. of speech can be generated. In this case, the deep neural network may be a long short-term memory (LSTM) that has been learned or a neural vocoder that has been learned, but the deep neural network is not limited thereto. In addition, the dialogue voice of the generated object is of the object created to express a preset volume (eg, a medium volume among expressible sounds in a game) and a preset sense of sound space (eg, an intermediate sound spatial sense among expressible sounds in a game). It may be a dialogue voice, but the spatial sense of volume and sound expressed by the dialogue voice of the generated object is not limited thereto.

The user terminal calculates the volume of the created object's dialogue voice and the stereo spatial feeling of the created object's dialogue voice based on the location information of the character matching the user terminal and the location information of the object matching the dialogue of the object (720) .

At this time, the user terminal is based on the location information of the character matched with the user terminal included in the metadata and the location information of the object matched with the lines of the object, between the character matched with the user terminal and the object matched with the sound of the cloud game. distance can be determined.

Also, the user terminal may calculate the volume of the dialogue voice of the object based on the distance between the character matching the determined user terminal and the object matching the sound of the cloud game.

In addition, based on the location information of the character matching the user terminal and the location information of the object matching the line of the object included in the meta data, the user terminal matches the character matching the user terminal and the object matching the line of the object in the game. It is possible to determine the environment (eg, a cave in the game, a plain in the game, inside a dungeon in the game, etc.)

In addition, the user terminal is an object based on the character and object matching the determined user terminal and the environment where the matching object exists in the game (eg, a cave in the game, a plain in the game, an inside of a dungeon in the game, etc.) It is possible to calculate the stereo spatial feeling of the dialogue voice of

The dialogue voice of the object generated based on the volume of the dialogue voice of the object calculated by the user terminal and the stereo spatial feeling of the calculated dialogue voice of the object is corrected (730).

At this time, the distance between the character matched with the user terminal and the object matched with the line of the object determined based on the location information of the character matching the user terminal included in the metadata and the location information of the object matching the line of the object is determined in advance. If it is within a set distance (eg, 1 m in the game), the user terminal may assign a weight to the volume of the calculated dialogue voice of the object.

In addition, a character matching the user terminal and an object matching the dialogue of the object determined based on the location information of the character matching the user terminal included in the meta data and the location information of the object matching the dialogue of the object exist in the game. When the environment of the destination is a preset environment (eg, a cave in a game, a cavity in a dungeon in a game, etc.), the user terminal may assign a weight to the stereo spatial feeling of the calculated dialogue voice of the object.

In addition, the user terminal may correct the dialogue voice of the generated object in consideration of the weight assigned to the volume of the calculated dialogue voice of the object and the weight assigned to the stereo spatial effect of the calculated dialogue voice of the object.

The user terminal outputs the corrected dialogue voice of the object at the time matched with the timestamp information by matching it with the play video stream of the cloud game (740).

At this time, while the speech of the object and the stream of the play video of the cloud game are matched and output, at least one of the location information of the character matching the user terminal and the location information of the object matching the dialog of the object by the user's input When the location information is changed, the user terminal may obtain modified metadata transmitted by the cloud game service providing device. At this time, the modified metadata is a modified time stamp indicating a time when at least one of location information of a character matching a user terminal and location information of an object matching an object's dialogue in the stream of a play video of a cloud game is changed. It may include at least one of information, modified positional information of a character that matches the user terminal, and modified positional information of an object that matches the dialogue of the object, but the information that the modified metadata may include is not limited thereto. no.

In addition, the user terminal matches the lines of characters and objects matched with the user terminal based on the modified location information of the character matching the user terminal included in the modified metadata and the modified location information of the object matching the lines of the object. It is possible to re-determine the distance between objects to be

In addition, the user terminal may recalculate the volume of the voice of the corrected dialogue of the object being output based on the distance between the dialogue of the character and object matching the re-determined user terminal and the matching object.

In addition, the user terminal matches the lines of characters and objects matched with the user terminal based on the modified location information of the character matching the user terminal included in the modified metadata and the modified location information of the object matching the lines of the object. It is possible to re-determine the environment where the object to be located exists in the game (eg, a cave in the game, a plain in the game, inside a dungeon in the game, etc.).

In addition, the user terminal is based on the environment where the character and object matching the re-determined user terminal and the matching object exist in the game (eg, in-game cave, in-game plain, in-game dungeon, etc. It is possible to recalculate the stereo spatial feeling of the dialog voice of the corrected object being output.

In addition, the user terminal may correct the corrected dialogue voice of the object being output based on the recalculated volume of the dialogue voice of the object and the stereo spatial feeling of the dialogue voice of the object.

In addition, based on the modified location information of the character matching the user terminal included in the modified metadata and the location information of the modified object matching the line of the object, which is re-determined based on the character matching the line of the object and the line of the character matched to the user terminal included in the modified metadata When the distance between the objects is within a preset distance (eg, 1 m in the game), the user terminal may assign a weight to the recalculated volume of the corrected object's dialogue voice being output.

In addition, based on the modified location information of the character matching the user terminal included in the modified metadata and the location information of the modified object matching the line of the object, which is re-determined based on the character matching the line of the object and the line of the character matched to the user terminal included in the modified metadata If the environment where the object exists in the game is a preset environment (eg, a cave in the game, a cavity in a dungeon in the game, etc.), the user terminal recalculates and weights the stereo spatial feeling of the calibrated object's dialogue voice during output. can be granted.

In addition, the user terminal considers the recalculated weight given to the volume of the calibrated object's dialogue voice during output and the recalculated weight assigned to the stereo spatial effect of the calibrated object's dialogue voice during output, and the dialogue voice of the calibrated object being output. can be modified.

In addition, the user terminal may match and output the dialogue voice of the modified object with the stream of the play video of the cloud game at the time that matches the modified time stamp information included in the modified metadata.

In the above, just because all the components constituting the embodiment have been described as being combined or operated as one, it is not necessarily limited to this embodiment. That is, within the scope of the purpose of the present embodiments, all of the components may be selectively combined with at least one and operated.

In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art.

Such a computer program may implement an embodiment of the present invention by being stored in a computer readable storage medium, read and executed by a computer. A storage medium of a computer program may include a magnetic recording medium, an optical recording medium, and the like.

In addition, terms such as "include", "comprise" or "have" described above mean that the corresponding component may be inherent unless otherwise stated, excluding other components. It should be construed as being able to further include other components.

All terms, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Commonly used terms, such as terms defined in a dictionary, should be interpreted as consistent with the meaning in the context of the related art, and unless explicitly defined in the present invention, they are not interpreted in an ideal or excessively formal meaning.

The methods disclosed herein include one or more actions or steps for achieving the methods described above. The method actions and/or steps may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions or steps is specified, the order and/or use of specific actions and/or steps may be modified without departing from the scope of the claims.

As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" means a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a , a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

As used herein, the term "determining" encompasses a wide variety of operations. For example, “determining” may include calculating, computing, processing, deriving, examining, looking up (eg, looking up in a table, database, or other data structure), ascertaining, and the like. . Also, “determining” may include receiving (eg, receiving information), accessing (accessing data in memory), and the like. Also, "determining" can include resolving, selecting, selecting, establishing, and the like.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100 ... user terminal
200...device for providing cloud game service
210...game server

Claims (18)

delete a cloud game service providing device that transmits a cloud game play video stream played by a user of a user terminal and metadata about sound of a cloud game corresponding to the cloud game play video stream to the user terminal; and
A user terminal that obtains the sound of the cloud game based on the meta data, matches the acquired sound of the cloud game with a stream of a play video of the cloud game, and outputs the matched sound;
The meta data includes time stamp information representing start time and end time of reproduction of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and matching with the sound of the cloud game. A system for providing a cloud game service including at least one of location information of a target object and sound source information matching the sound of the cloud game. According to claim 2,
The user terminal,
Determining the sound of the cloud game to be output based on the meta data in a sound source stored in the user terminal;
Calculate the volume of the determined sound of the cloud game and the stereo spatial feeling of the determined sound of the cloud game based on the positional information of the character matched with the user terminal and the positional information of the object matched to the sound of the cloud game;
Correcting the determined sound of the cloud game based on the volume of the calculated sound of the cloud game and the stereo spatial feeling of the calculated sound of the cloud game;
A system for providing a cloud game service that matches and outputs the corrected sound of the cloud game with a stream of play video of the cloud game at a time matching the time stamp information. According to claim 2,
The cloud game service providing device,
When at least one of location information of a character matching the user terminal and location information of an object matching the sound of the cloud game is changed, modifying metadata is transmitted to the user terminal;
The modified metadata,
Modified time stamp information indicating a time when at least one of location information of a character matching the user terminal and location information of an object matching the sound of the cloud game in the stream of the play video of the cloud game is changed, The cloud game service providing system including at least one of corrected positional information of a character matching the user terminal and modified positional information of an object matching the sound of the cloud game. According to claim 4,
The user terminal,
Determining the sound of the cloud game to be output based on the meta data in a sound source stored in the user terminal;
Calculate the volume of the determined sound of the cloud game and the stereo spatial feeling of the determined sound of the cloud game based on the positional information of the character matched with the user terminal and the positional information of the object matched to the sound of the cloud game;
Correcting the determined sound of the cloud game based on the volume of the calculated sound of the cloud game and the stereo spatial feeling of the calculated sound of the cloud game;
At a time matched with the time stamp information, the corrected sound of the cloud game is matched with a play video stream of the cloud game and output;
The corrected volume and output of the corrected sound of the cloud game being output based on at least one of corrected location information of a character that matches the user terminal and corrected location information of an object that matches the sound of the cloud game. recalculating the corrected stereo spatial feeling of the sound of the corrected cloud game;
Correcting the corrected sound of the cloud game being output based on the corrected volume of the recalculated cloud game sound and the modified stereo spatial feeling of the recalculated cloud game sound;
A system for providing a cloud game service that matches and outputs the sound of the modified cloud game with the play video stream of the cloud game at a time matching the modified time stamp information. a cloud game service providing device that transmits a cloud game play video stream played by a user of a user terminal and metadata about sound of a cloud game corresponding to the cloud game play video stream to the user terminal; and
A user terminal that acquires the sound of the cloud game based on the meta data, matches the obtained sound of the cloud game with a stream of a play video of the cloud game, and outputs the matched sound;
The sound of the cloud game includes speech of at least one object included in the cloud game;
The meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and matching with the dialogue of the object. A cloud game service providing system including at least one of location information of an object, object information matching the dialogue of the object, and dialogue information of the object. According to claim 6,
The user terminal,
Generating a dialogue voice of the object corresponding to the dialogue information of the object using a deep neural network based on object information matching the dialogue of the object and the dialogue information of the object;
Based on the location information of the character matching the user terminal and the location information of the object matching the dialog of the object, the volume of the dialogue voice of the generated object and the stereo spatial feeling of the dialogue voice of the generated object are calculated,
correcting the dialogue voice of the generated object based on the volume of the calculated dialogue voice of the object and the stereo spatial feeling of the calculated dialogue voice of the object;
A system for providing a cloud game service that matches and outputs the audio of the dialogue of the corrected object with the stream of the play video of the cloud game at a time matching the time stamp information. According to claim 6,
The cloud game service providing device,
When at least one of location information of a character matching the user terminal and location information of an object matching the line of the object is changed, modifying metadata is transmitted to the user terminal,
The modified metadata,
Modified time stamp information indicating a time when at least one of location information of a character matching the user terminal and location information of an object matching the line of the object in the stream of the play video of the cloud game is changed; A system for providing a cloud game service comprising at least one of modified positional information of a character matching a user terminal and modified positional information of an object matching a dialogue of the object. According to claim 8,
The user terminal,
Generating a dialogue voice of the object corresponding to the dialogue information of the object using a deep neural network based on object information matching the dialogue of the object and the dialogue information of the object;
Based on the location information of the character matching the user terminal and the location information of the object matching the dialog of the object, the volume of the dialogue voice of the generated object and the stereo spatial feeling of the dialogue voice of the generated object are calculated,
correcting the dialogue voice of the generated object based on the volume of the calculated dialogue voice of the object and the stereo spatial feeling of the calculated dialogue voice of the object;
At a time matching the timestamp information, the corrected object's dialogue voice is matched with the play video stream of the cloud game and output;
The corrected volume of the dialogue voice of the corrected object being output based on at least one of corrected positional information of a character matching the user terminal and modified positional information of an object matching the dialogue of the object and the corrected volume of the dialogue voice being output The corrected stereo spatial feeling of the dialogue voice of the corrected object is recalculated,
Correcting the corrected speech of the object being output based on the corrected volume of the recalculated speech of the object and the modified stereo spatial feeling of the recalculated speech of the object;
A system for providing a cloud game service that matches and outputs the dialogue voice of the modified object with the stream of the play video of the cloud game at a time matching the modified time stamp information. delete A cloud game service providing method performed by a cloud game service providing system including a user terminal and a cloud game service providing device,
transmitting, by the cloud game service providing apparatus, a stream of play video of a cloud game played by a user of a user terminal and metadata about a sound of a cloud game corresponding to the stream of the play video of the cloud game to the user terminal; and
The user terminal acquires the sound of the cloud game based on the metadata, matches the acquired sound of the cloud game with a stream of a play video of the cloud game, and outputs the matched sound;
The meta data includes time stamp information representing start time and end time of reproduction of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and matching with the sound of the cloud game. A cloud game service providing method comprising at least one of location information of a target object and sound source information matching the sound of the cloud game. According to claim 11,
The operation of the user terminal obtaining the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched sound,
determining, by the user terminal, the sound of the cloud game to be output based on the meta data from a sound source stored in the user terminal;
The user terminal calculates the volume of the determined sound of the cloud game and the stereo spatial feeling of the determined sound of the cloud game based on the positional information of the character matched with the user terminal and the positional information of the object matched to the sound of the cloud game. action;
correcting, by the user terminal, the determined sound of the cloud game based on the volume of the calculated sound of the cloud game and the stereo spatial feeling of the calculated sound of the cloud game; and
An operation of matching the corrected sound of the cloud game with the stream of the play video of the cloud game and outputting the corrected sound at the time when the user terminal matches the time stamp information.
A cloud game service providing method comprising a. According to claim 11,
The operation of the cloud game service providing device transmitting metadata about the sound of the cloud game corresponding to the play video of the cloud game to the user terminal,
Transmitting modified metadata to the user terminal when at least one of location information of a character matching the user terminal and location information of an object matching the sound of the cloud game is changed.
including,
The modified metadata,
Modified time stamp information indicating a time when at least one of location information of a character matching the user terminal and location information of an object matching the sound of the cloud game in the stream of the play video of the cloud game is changed, A cloud game service providing method comprising at least one of modified location information of a character matching the user terminal and modified location information of an object matching the sound of the cloud game. According to claim 13,
The operation of the user terminal obtaining the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched sound,
determining, by the user terminal, the sound of the cloud game to be output based on the meta data from a sound source stored in the user terminal;
The user terminal calculates the volume of the determined sound of the cloud game and the stereo spatial feeling of the determined sound of the cloud game based on the positional information of the character matched with the user terminal and the positional information of the object matched to the sound of the cloud game. action;
correcting, by the user terminal, the determined sound of the cloud game based on the volume of the calculated sound of the cloud game and the stereo spatial feeling of the calculated sound of the cloud game;
matching and outputting the corrected sound of the cloud game with the play video stream of the cloud game at a time when the user terminal matches the time stamp information;
Correction of the corrected sound of the cloud game being output based on at least one of corrected location information of a character matched with the user terminal and corrected location information of an object matched with the sound of the cloud game by the user terminal. recalculating the modified volume and the corrected stereo spatial feeling of the corrected sound of the cloud game being output;
modifying, by the user terminal, the corrected sound of the cloud game being output based on the corrected volume of the recalculated cloud game sound and the modified stereo spatial feeling of the recalculated cloud game sound; and
An operation of matching and outputting the modified sound of the cloud game with the play video stream of the cloud game at the time when the user terminal matches the modified time stamp information
A cloud game service providing method comprising a. A cloud game service providing method performed by a cloud game service providing system including a user terminal and a cloud game service providing device,
transmitting, by the cloud game service providing apparatus, a stream of play video of a cloud game played by a user of a user terminal and metadata about a sound of a cloud game corresponding to the stream of the play video of the cloud game to the user terminal; and
The user terminal acquires the sound of the cloud game based on the metadata, matches the acquired sound of the cloud game with a stream of a play video of the cloud game, and outputs the matched sound;
The sound of the cloud game includes speech of at least one object included in the cloud game;
The meta data includes time stamp information representing the playback start time and playback end time of the cloud game sound in the stream of the play video of the cloud game, location information of a character matched with the user terminal, and matching with the dialogue of the object. A method for providing a cloud game service including at least one of location information of an object, object information matching the dialogue of the object, and dialogue information of the object. According to claim 15,
The operation of the user terminal acquiring the sound of the cloud game based on the meta data, matching the obtained sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched sound,
generating, by the user terminal, a dialogue voice of the object corresponding to the dialogue information of the object by using a deep neural network based on object information matching the dialogue of the object and the dialogue information of the object;
The volume of the dialogue voice of the generated object and the stereo spatial feeling of the generated dialogue voice of the object based on the location information of the character matched by the user terminal with the user terminal and the position information of the object matching the dialogue of the object. the act of counting;
correcting, by the user terminal, the dialogue voice of the generated object based on the volume of the calculated dialogue voice of the object and the stereo spatial feeling of the calculated dialogue voice of the object; and
An operation of the user terminal matching the corrected object's dialogue voice with the play video stream of the cloud game at the time when the time stamp information is matched and outputting the matched
A cloud game service providing method comprising a. According to claim 15,
The operation of the cloud game service providing device transmitting metadata about the sound of the cloud game corresponding to the play video of the cloud game to the user terminal,
Transmitting modified metadata to the user terminal when at least one location information of the location information of a character matching the user terminal and the location information of an object matching the lines of the object is changed.
including,
The modified metadata,
Modified time stamp information indicating a time when at least one of location information of a character matching the user terminal and location information of an object matching the line of the object in the stream of the play video of the cloud game is changed; A cloud game service providing method comprising at least one of modified location information of a character matching a user terminal and modified location information of an object matching a line of the object. According to claim 17,
The operation of the user terminal obtaining the sound of the cloud game based on the meta data, matching the acquired sound of the cloud game with the stream of the play video of the cloud game, and outputting the matched sound,
generating, by the user terminal, a dialogue voice of the object corresponding to the dialogue information of the object by using a deep neural network based on object information matching the dialogue of the object and the dialogue information of the object;
Volume of dialogue voice of the generated object and stereo spatial sense of dialogue voice of the created object based on location information of a character matched by the user terminal with the user terminal and location information of an object matched to the sound of the cloud game. the operation of calculating ;
correcting, by the user terminal, the dialogue voice of the generated object based on the volume of the calculated dialogue voice of the object and the stereo spatial feeling of the calculated dialogue voice of the object;
matching the calibrated speech of the object with the stream of the play video of the cloud game at the time when the user terminal matches the time stamp information and outputting the matching;
The modified volume of the dialogue voice of the corrected object that the user terminal is outputting based on at least one of corrected positional information of a character matching the user terminal and modified positional information of an object matching the dialogue of the object. and recalculating a corrected stereo spatial feeling of the speech of the corrected object being output.
modifying, by the user terminal, the corrected speech of the object being output based on the corrected volume of the recalculated speech of the speech of the object and the modified stereo spatial feeling of the recalculated speech of the speech of the object; and
An operation of matching and outputting the dialogue voice of the modified object with the play video stream of the cloud game at the time when the user terminal matches the modified time stamp information
A cloud game service providing method comprising a.