KR102336067B1 - A compuer program for generating audio data corresponding to a game play situation - Google Patents

Abstract

In one embodiment of the present disclosure for solving the above problems, a computer program stored in a computer readable medium executable by one or more processors is disclosed. The computer program, when executed by one or more processors, causes the one or more processors to perform operations for generating voice data corresponding to an in-game play situation, the operations comprising: synthesizing voice based on the game data; It may include an operation of generating metadata for the purpose of the present invention, an operation of inputting the metadata and dialogue text into a speech synthesis model, and an operation of generating speech data synthesized through the speech synthesis model.

Description

{A COMPUER PROGRAM FOR GENERATING AUDIO DATA CORRESPONDING TO A GAME PLAY SITUATION}

The present disclosure relates to a game service, and more particularly, to a computer program for generating voice data corresponding to an in-game play situation.

Recently, with the rapid development of IT technology, the game industry is growing together. As the game industry grows, game companies that provide game services are also being created explosively. On the other hand, due to an increase in the number of game companies that provide game services, competition in the game service market is intensifying. Accordingly, game companies are continuously making efforts to secure the competitiveness of their respective games. In particular, game companies are continuously searching for factors that can arouse the interest of game users.

For example, in a Massively Multiplayer Online Role Playing Game (MMORPG), in which numerous simultaneous players perform missions or quests in the game, understand the game's story for the smooth progress of the game, and ) is important to do.

Here, game companies that provide game services provide the voices of characters in the game as the voices of various voice actors in order to secure the competitiveness of the games they provide. Specifically, game companies provide the voice of a character existing in the game as the voice of a voice actor when the game user guides the game user to the story or when the game user performs an in-game quest. For example, in a general game service, when a non-player character (NPC) existing in a game guides a game story, a pre-recorded voice of a voice actor is provided along with text. For another example, in a general game service, when an NPC existing in a game provides information related to quest progress, a pre-recorded voice of a voice actor is provided along with text.

However, if the vast amount of guidance voices in the game are all pre-recorded through the voice of a voice actor, there is a risk that the cost of providing voice will increase depending on the cost of hiring a voice actor and recording cost, and Difficulties may exist even when trying to modify it.

Accordingly, there may be a need in the art for a method for securing diversity in a method of generating a voice for guiding a story of a game or a voice for guiding a quest in a game.

Korean Patent Publication No. 2007-0078682

The present disclosure has been made in response to the above-described background technology, and is intended to provide a computer program for generating voice data corresponding to a play situation in a game.

In one embodiment of the present disclosure for solving the above problems, a computer program stored in a computer readable medium executable by one or more processors is disclosed. The computer program causes one or more processors to perform operations for generating voice data corresponding to an in-game play situation, and the operations include: generating metadata for voice synthesis based on the game data; It may include inputting data and dialogue text into a speech synthesis model and generating speech data synthesized through the speech synthesis model.

Alternatively, the metadata is attribute indication information for generating the voice data, and may include at least one of image vector information, style vector information, and voice identification information.

Alternatively, the attribute indication information includes at least one of attribute information of each of the one or more contents included in the game, relationship information between a plurality of objects existing in the game, and attribute information of each of the plurality of objects can do.

Alternatively, the image vector information includes instruction information for generating voice data generated based on a game image, and the style vector information includes predetermined instruction information for each of one or more contents included in the game. and the voice identification information may include information for identifying seed voice data for voice synthesis.

Alternatively, the dialogue text may include content information for generating the voice data, and may be predetermined based on each of one or more contents included in the game.

Alternatively, the method may further include, when receiving a voice output control signal from the client terminal, causing the client terminal to output voice data corresponding to the in-game play situation.

Alternatively, the operation of constructing training data for training the speech synthesis model based on the one or more seed speech data, text corresponding to each of the one or more seed speech data, and metadata associated with the one or more seed speech data may further include.

Alternatively, the speech synthesis model includes a dimensionality reduction submodel, a dimension restoration submodel, an attention module, and an audio output module, and is trained through training data including the metadata, training dialogue text, and training target voice data. can be

Alternatively, the speech synthesis model may include a learning spectrogram corresponding to the target speech data by using the learning input text including the metadata and the training dialogue text as an input of the dimension reduction sub-model, and the dimension reconstruction sub-model corresponding to the target speech data. (Spectrogram) can be learned to output.

Alternatively, the dimensionality reduction sub-model may take the metadata and the dialogue text as inputs, and output a language speech feature from the dialogue text.

Alternatively, the dimensional reconstruction submodel includes one or more recurrent neural networks (RNNs), wherein the one or more RNNs receive a spectrogram of a first timestamp as an input and output a spectrogram of a second timestamp. can

Alternatively, the attention module may generate association information between a phoneme of the dialogue text and a time step of the dimensional reconstruction submodel.

Alternatively, the speech output module may generate speech data synthesized using a speech reconstruction algorithm using a spectrogram that is an output of the dimensional reconstruction sub-model.

In another embodiment of the present disclosure, a method of generating voice data corresponding to an in-game play situation is disclosed. The method includes the steps of generating metadata for speech synthesis based on game data, inputting the metadata and dialogue text into a speech synthesis model, and generating speech data synthesized through the speech synthesis model. may include

In another embodiment of the present disclosure, a server for generating voice data corresponding to an in-game play situation is disclosed. The server includes a processor including one or more cores, a memory for storing program codes executable by the processor, and a network unit for transmitting and receiving data to and from a game server and a client terminal, wherein the processor includes a voice based on the game data Meta data for synthesis may be generated, the metadata and dialogue text may be input to a speech synthesis model, and synthesized speech data may be generated through the speech synthesis model.

The present disclosure may provide a computer program for generating voice data corresponding to an in-game play situation.

Various aspects are now described with reference to the drawings, wherein like reference numbers are used to refer to like elements collectively. In the following example, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It will be evident, however, that such aspect(s) may be practiced without these specific details.
1 shows a schematic diagram of a system for generating voice data corresponding to an in-game play situation according to an embodiment of the present disclosure;
2 is a block diagram of a server for generating voice data corresponding to an in-game play situation related to an embodiment of the present disclosure.
3 illustrates an exemplary flowchart for generating voice data corresponding to an in-game play situation related to an embodiment of the present disclosure.
4 illustrates an exemplary diagram for explaining voice data and text information that are differently determined in response to each in-game play situation related to an embodiment of the present disclosure.
5 illustrates an exemplary diagram for explaining voice data and text information that are differently determined in response to each in-game play situation related to an embodiment of the present disclosure.
6 is a block diagram illustrating a speech synthesis model according to an embodiment of the present disclosure.
7 is a detailed configuration diagram of a speech synthesis model related to an embodiment of the present disclosure.
8 is a schematic diagram illustrating a network function related to an embodiment of the present disclosure.
9 illustrates a module for implementing a method for generating voice data corresponding to an in-game play situation related to an embodiment of the present disclosure.
10 shows a simplified, general schematic diagram of an exemplary computing environment in which one embodiment of the present disclosure may be implemented, in conjunction with one embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific descriptions.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored therein. Components may communicate via a network such as the Internet with another system, for example, via a signal having one or more data packets (eg, data and/or signals from one component interacting with another component in a local system, distributed system, etc.) may communicate via local and/or remote processes depending on the data being transmitted).

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean one of the natural implicit substitutions. That is, X employs A; X employs B; or when X employs both A and B, "X employs A or B" may apply to either of these cases. It should also be understood that the term “and/or” as used herein refers to and includes all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the feature and/or element in question is present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements and/or groups thereof. Also, unless otherwise specified or unless it is clear from context to refer to a singular form, the singular in the specification and claims should generally be construed to mean “one or more”.

Those skilled in the art will further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or combinations of both. It should be recognized that they can be implemented with To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The scope of rights to the operations in the claims of the present disclosure is generated by the functions and features described in the respective operations, and unless a precedence of the order in each operation is specified, the scope of the claims is The order of description of each operation is not affected. For example, in a claim described as an action including action A and action B, even if action A is described before action B, the scope of rights is not limited to that action A must precede action B.

1 shows a schematic diagram of a system for generating voice data corresponding to an in-game play situation according to an embodiment of the present disclosure;

As shown in FIG. 1 , a system for generating voice data corresponding to an in-game play situation may include a voice data generating server 100 , a client terminal 10 , a game server 200 , and a communication network. . The components illustrated in FIG. 1 are exemplary, and additional components may be present or some of the components illustrated in FIG. 1 may be omitted.

The client terminal 10 in FIG. 1 may be related to a user who wants to access at least one of the game server 200 and the voice data generating server 100 .

The client terminal 10 in the present disclosure may be at least one terminal that allows a user to play a game. Also, the client terminal 10 may display a game screen or output a sound related to a game based on a signal transmitted from the game server 200 .

The client terminal 10 may mean any type of entity(s) in the system having a mechanism for communication with the voice data generating server 100 and the game server 200 . For example, such a client terminal 10 is a PC (personal computer), a notebook (note book), a mobile terminal (mobile terminal), a smart phone (smart phone), a tablet PC (tablet pc) and a wearable device (wearable device) and the like, and may include all types of terminals capable of accessing a wired/wireless network. Also, the client terminal 10 may include an arbitrary server implemented by at least one of an agent, an application programming interface (API), and a plug-in. In addition, the client terminal 10 may include an application source and/or a client application.

The client terminal 10 may access the game server 200 to play a game provided by the game server 200 , and may output visual and auditory effects corresponding to the user's game play. The game in the present specification may include any type of game, such as a mobile game, a web game, a VR game, a P2P game, an online/offline game, and the like.

Since the client terminal 10 has a display, it can receive a user's input related to game play and provide an output in any form to the user. In addition, since the client terminal 10 is provided with a sound output unit, it is possible to provide any form of output related to game play.

According to one embodiment of the present disclosure, the game server 200 and the voice data generation server 100 are any type of computer system, such as, for example, a microprocessor, a mainframe computer, a digital processor, a portable device and a device controller, and the like. or a computer device. Although not shown in FIG. 1 , the game server 200 and the voice data generating server 100 may include a memory and a processor.

The game server 200 may allow the client terminal 10 to play a game. The voice data generation server 100 may allow the client terminal 10 to output any type of voice data corresponding to an in-game play situation when the client terminal 10 accesses the game server 200 to play a game.

Although the game server 200 and the voice data generating server 100 are represented as separate entities in FIG. 1 , the voice data generating server 100 is included in the game server 200 according to an embodiment of the present disclosure. , a game play function and a voice data generation function corresponding to the game may be performed in the integrated server. In this example, when the function of the voice data generation server 100 is integrated into the game server 200 , the client terminal 10 may perform in-game voice data generation.

In addition, when the voice data generating server 100 and the game server 200 are separated, the client terminal may output the voice data generated through the voice data generating server 100 existing outside the game server 200 . . In this case, the voice data generation server 100 may communicate with the game server 200 to implement voice data generation related to game play.

According to an embodiment of the present disclosure, the voice data generation server 100 may distribute and process the model using at least one of a CPU, a GPGPU, and a TPU. In addition, in an embodiment of the present disclosure, the voice data generation server 100 may distribute and process the model together with other computing devices.

In the present specification, a network function may be used interchangeably with a sub-model, an artificial neural network, and a neural network. In the present specification, the network function may include one or more neural networks, and in this case, the output of the network function may be an ensemble of the outputs of the one or more neural networks.

In this specification, a model may include a network function. The model may include one or more network functions, in which case the output of the model may be an ensemble of outputs of the one or more network functions.

According to an embodiment of the present disclosure, a communication network is a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High (VDSL) Various wired communication systems such as Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) can be used.

In addition, the communication network presented in the present disclosure is CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier) -FDMA) and other systems can be used for various wireless communication systems. The techniques described in this disclosure may be used in the networks mentioned above, as well as in any form of other communication networks.

Hereinafter, a method in which the voice data generating server 100 provides voice data corresponding to in-game play situations of a plurality of users using a game service will be described in detail with reference to FIG. 2 .

2 is a block diagram of a server for generating voice data corresponding to an in-game play situation related to an embodiment of the present disclosure.

As shown in FIG. 2 , the voice data generation server 100 may include a network unit 110 , a memory 120 , and a processor 130 . The above-described components are exemplary, and the scope of the present disclosure is not limited to the above-described components. That is, additional components may be included or some of the above-described components may be omitted depending on implementation aspects for the embodiments of the present disclosure.

According to an embodiment of the present disclosure, the voice data generation server 100 transmits/receives data for generating voice data corresponding to in-game play with the client terminal 10 and the game server 200 and the network unit 110. may include

The network unit 110 according to an embodiment of the present disclosure includes a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), VDSL ( A variety of wired communication systems such as Very High Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) can be used.

In addition, the network unit 110 presented herein is CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA ( A variety of wireless communication systems can be used, such as Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit 110 may be configured regardless of its communication mode, such as wired and wireless, and may be composed of various communication networks such as a short-range network (PAN: Personal Area Network) and a local area network (WAN: Wide Area Network). can In addition, the network may be a well-known World Wide Web (WWW), and may use a wireless transmission technology used for short-range communication, such as infrared (IrDA) or Bluetooth (Bluetooth).

The techniques described herein may be used in the networks mentioned above, as well as in other networks.

According to an embodiment of the present disclosure, the memory 120 may store any type of information generated or determined by the processor 130 and any type of information received by the network unit 110 .

According to an embodiment of the present disclosure, the memory 120 is a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, a SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read (PROM) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The voice data generation server 100 may operate in relation to a web storage that performs a storage function of the memory 120 on the Internet. The description of the above-described memory is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 130 may be configured with one or more cores, and a central processing unit (CPU) of the voice data generation server 100, a general-purpose graphic processing unit (GPGPU: general) It may include a processor 130 for data analysis, deep learning, such as a purpose graphics processing unit) and a tensor processing unit (TPU). The processor 130 may read the computer program stored in the memory 120 to generate voice data corresponding to an in-game play situation according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the processor 130 may perform a calculation for learning the neural network. The processor 130 is configured to process input data for learning in deep learning (DL), extract features from input data, calculate errors, update weights of neural networks using back-propagation, etc. Calculations for training neural networks can be performed.

In addition, at least one of the CPU, GPGPU, and TPU of the processor 130 may process the training of the model. For example, the CPU and the GPGPU can learn the model together and process the operation on the voice data corresponding to the in-game play situation using the model. In addition, in an embodiment of the present disclosure, by using the processors 130 of a plurality of computing devices together, model learning and operation on voice data corresponding to an in-game play situation may be processed. In addition, the computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present disclosure, the processor 130 may typically process the overall operation of the voice data generating server 100 . The processor 130 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 120 . .

According to an embodiment of the present disclosure, the processor 130 may generate metadata for voice synthesis based on game data. Specifically, the processor 130 may receive, from the game server 200 , game data related to a game build, or image data related to a plurality of objects included in the game, and the game data and a plurality of objects related to the game build. Meta data for voice synthesis may be generated based on image data related to .

Meta data is attribute indication information for generating voice data, and may include image vector information, style vector information, and voice identification information. In this case, the attribute indication information is game-related information provided by the game server 200 , and includes attribute information of each of one or more contents included in the game, relationship information between a plurality of objects existing in the game, and attributes of each of the plurality of objects. It may include at least one piece of information among the information.

One or more contents included in the game means various contents that can be enjoyed by a plurality of users in the game, for example, the purpose of performing various plays existing in the game, performance effect, performance method, performance process, and performance result It may be classified based on at least one of. For example, one or more contents included in the game may mean a plurality of quests or a plurality of stages included in game play provided by the game server 200 to the client terminal 10 connected to the game server 200 . can The detailed description of one or more contents included in the above-described game is merely an example, and the present disclosure is not limited thereto.

The image vector information included in the metadata may include instruction information for generating voice data generated based on a game image. Specifically, the image vector information may be generated based on game images including a plurality of objects or one or more contents existing in a game provided by the game server 200 . Also, the image vector information may include instruction information for generating voice data.

For example, if the game image received from the game server 200 includes the first NPC, and the appearance image of the first NPC is of an elf race, the processor 130 determines that the first NPC included in the game image is female. Image vector information including instruction information for generating calm and high-tone feminine voice data may be generated based on the .

As another example, if the game image received from the game server 200 includes the second NPC, and the appearance image of the second NPC is of an orc race, the processor 130 determines that the second NPC included in the game image is male. Based on this, image vector information including instruction information for generating coarser and lower tone masculine voice data may be generated. The detailed description of the indication information included in the above-described image vector information is only an example, and the present disclosure is not limited thereto.

That is, the processor 130 may generate image vector information including instruction information for generating specific voice data based on the game image.

The style vector information included in the metadata may include predetermined indication information for each of one or more contents included in the game. Specifically, the style vector information may include instruction information for generating voice data corresponding to a game build predetermined by a game developer for each of one or more contents included in a game provided by the game server 200 .

For example, when the game build of the game provided by the game server 200 is pre-determined that the first quest has an overall gloomy and dark atmosphere, the processor 130 responds to information about the first quest included in the game build Accordingly, style vector information including instruction information for generating voice data of a slower tempo and low tone may be generated.

For another example, when the game build of the game provided by the game server 200 is predetermined with the second quest as an overall lively and lively atmosphere, the processor 130 responds to information about the second quest included in the game build Accordingly, style vector information including instruction information for generating faster tempo and higher tone voice data may be generated.

As another example, when the game build of the game provided by the game server 200 pre-determines the voice data for the first NPC included in the third quest in a calm and calm tone, the processor 130 is included in the game build Style vector information including instruction information for generating voice data having a calm and calm tone may be generated in response to the information on the third quest. The detailed description of the indication information included in the above-described style vector information is only an example, and the present disclosure is not limited thereto.

That is, the processor 130 may generate style vector information including predetermined instruction information for generating specific voice data.

The voice identification information included in the metadata may include information for identifying seed voice data for voice synthesis. More specifically, the processor 130 may acquire one or more seed voice data that is a basis for voice synthesis, and may match voice identification information to each of the one or more seed voice data. That is, the voice identification information may be information for identifying specific seed voice data among one or more seed voice data. For example, the first seed voice data may be obtained from the first voice actor, and voice identification information for identifying the first seed voice data may be 'voice_1'. In addition, the second seed voice data may be obtained from the second voice actor, and voice identification information for identifying the second seed voice data may be 'voice_2'. That is, the processor 130 may identify the voice of the voice actor, which is the basis for generating the voice data, through the voice identification information. The detailed description of the aforementioned seed voice data, voice actors, and voice identification information is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 130 may construct training data for training the speech synthesis model. More specifically, the processor 130 is configured to construct training data for training a speech synthesis model based on one or more seed voice data, text corresponding to each of the one or more seed voice data, and metadata associated with the one or more seed voice data. can

More specifically, the processor 130 is configured to train the speech synthesis model 600 based on one or more seed speech data, text corresponding to each of the one or more seed speech data, and image vector information associated with the one or more seed speech data. You can build training data. In addition, the processor 130 builds training data for training the speech synthesis model 600 based on one or more seed voice data, text corresponding to the one or more seed voice data, and style vector information associated with the one or more seed voice data. can do. That is, the processor 130 generates the speech synthesis model 600 based on at least one of image vector information and style vector information included in the metadata, one or more seed voice data, and text corresponding to one or more seed voice data. Learning data for learning can be built.

As a specific example, in response to the first game image including the first NPC, the predetermined first dialogue text is “I will keep the item”, and one or more seed voice data for the first dialogue text (the first seed voice) When the data, the second seed voice data, and the third voice data) are pre-acquired through the voices of different voice actors, the processor 130 performs the first image vector information corresponding to the first game image, the first dialogue text, and Learning data may be constructed through one or more seed voice data (first seed voice data, second seed voice data, and third voice data). In this case, the processor 130 may determine the voice identification information for the first to third seed voice data as 'voice_1', 'voice_2', and 'voice_3', respectively. In addition, the processor 130 may determine learning input data including first image vector information corresponding to the first game image, voice identification information, and first dialogue text, the first seed voice data to the third seed voice data can be determined as the learning target voice data to construct the learning data.

As another example, the second dialogue text predetermined in response to the second stage is “I do not have time”, and one or more seed voice data (the first seed voice data and the second seed voice data) for the second dialogue text is generated. When pre-obtained through voice actors different from each other, the processor 130 performs the second style vector information, the second dialogue text and one or more seed voice data (the first seed voice data and the second seed preset) corresponding to the second stage. Voice data) can be used to build learning data. In this case, the processor 130 may determine the voice identification information for each of the first seed voice data and the second seed voice data as 'voice_1' and 'voice_2'. In addition, the processor 130 may determine learning input data including predetermined second style vector information, voice identification information, and second dialogue text information in response to the second quest, the first seed voice data and the second seed Learning data may be constructed by determining voice data as learning target voice data. Specific descriptions of the aforementioned dialogue text, one or more seed voice data, image vector information, and style vector information are merely examples, and the present disclosure is not limited thereto.

That is, the learning input data may include learning metadata (learning image vector information and learning voice identification information, or learning style vector information and learning voice identification information) and learning dialogue text information. In addition, the learning target voice data may be one or more pre-obtained seed voice data.

According to an embodiment of the present disclosure, the processor 130 may train a speech synthesis model through training data. Specifically, the processor 130 may train the speech synthesis model 600 to output the training input data by taking the training input data as an input.

As shown in FIG. 6 , the speech synthesis model 600 may include a dimension reduction submodel 610 , a dimension reconstruction submodel 630 , an attention module 620 , and a speech output module 640 . A detailed description of how the processor 130 trains the speech synthesis model 600 through the training data will be described later with reference to FIGS. 6 and 7 .

The processor 130 may use the training input data as an input of the dimension reduction sub-model 610 to train the dimension restoration sub-model 630 to output a learning spectrogram corresponding to the target voice data. The spectrogram is for visualizing and grasping voice data, and may be a graph in which a difference in amplitude according to time and frequency changes is expressed as print density and/or display color. The dimension reduction sub-model 610 may be a model for extracting speech speech features (ie, embeddings) by inputting metadata and dialogue text as inputs. That is, the dimension reduction sub-model 610 receives learning input data including learning metadata and learning dialogue text from the processor 130 and designates a spectral feature vector column of the learning input data as an output to convert the input data into a spectrum Intermediate courses can be learned.

Also, the processor 130 may transmit the embeddings for the meta data and the dialogue text from the dimension reduction sub-model 610 to the attention module 620 . The attention module 620 may generate association information between the phoneme of the dialogue text and the time step of the dimension reconstruction sub-model. In other words, the processor 130 causes the attention module 620 including the neural network layer to input (ie, learning metadata and learning dialogue text) and output (ie, learning spectrogram) through the learning input data and the learning spectrogram. It is possible to learn the mapping relationship between

The dimension reconstruction submodel 630 includes one or more Recurrent Neural Networks (RNNs), and may be a model that outputs a spectrogram of a second timestamp by inputting a spectrogram of a first timestamp as an input through the one or more RNNs. have. In this case, the first time stamp may be earlier than the second time stamp. Specifically, the spectrogram predicted through the first RNN of the dimension reconstruction submodel 630 is transferred to the second RNN of the next time stamp, and the spectrogram predicted through the second RNN is transferred to the third RNN of the next time stamp. can be transmitted. In this process, the dimension reconstruction sub-model 630 may determine the text to be focused on through association information between the phoneme of the dialogue text from the attention module 620 and the time step of the dimension reconstruction sub-model. That is, the dimension restoration sub-model 630 determines the text to focus on through the attention module 620 in the process of repeatedly predicting the spectrogram through one or more RNNs, thereby learning to output the spectrogram for the unlearned text. can

The processor 130 may transmit a spectrogram that is an output of the dimension reconstruction submodel 630 to the voice output module 640 . The voice output module 640 may synthesize the spectrogram into voice data and output the synthesized voice data. In this case, the voice output module 640 may be implemented through a Griffin-Lim algorithm.

Through the learning process as described above, the voice synthesis model may output more natural synthesized voice data in response to meta information and dialogue text to reflect it in the game. Accordingly, game users do not feel awkward about the game guide voice, so their immersion in the game may increase, and the interest in the game may increase. In addition, it is possible to generate synthesized voice data corresponding to non-pre-trained dialogue text, so that voice data including intonation, tone, intention and emotion can be generated only with a relatively small amount of learning data.

According to an embodiment of the present disclosure, the processor 130 may input metadata and dialogue text to the speech synthesis model. The dialogue text includes content information for generating voice data, and may be predetermined based on each of one or more contents included in the game. Specifically, the processor 130 may input metadata including at least one of image vector information, style vector information, and voice identification information and a dialogue text corresponding to the metadata to the speech synthesis model.

As a more specific example, the first image vector information includes instruction information for generating calm, high-toned, feminine voice data corresponding to the appearance image of the first NPC, and the first voice identification information corresponds to the voice of the first voice actor. When the information on the corresponding 'Voice_1' is included, the processor 130 transmits a predetermined first dialogue text to the corresponding first NPC (eg, “Hello, nice to meet you!”), the first image vector information, and the first Voice identification information may be input to the speech synthesis model.

For another example, the second style vector information includes instructional information for generating voice data of a slower tempo and lower tone in response to a game build predetermined in which the second stage is generally gloomy and dark atmosphere, and the second voice If the identification information includes information on 'Voice_2' corresponding to the voice of the second voice actor, the processor 130 may perform a second pre-determined second dialogue text (eg, “whohoo, welcome”) in the second stage, the second The second style vector information and the second voice identification information may be input to the speech synthesis model. The detailed description of the instruction information, the voice identification information, and the dialogue text included in the above-described image vector information is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 130 may generate synthesized voice data through the voice synthesis model 600 . Specifically, the processor 130 may generate synthesized voice data by using metadata and dialogue text as inputs of the voice synthesis model 600 . Also, the processor 130 may pre-determine that the synthesized voice data be output corresponding to one or more contents included in the game and each of a plurality of objects.

As a specific example, as shown in FIG. 4 , when the client terminal 10 connects to the game server 200 and plays a game, a voice guidance about the game is provided to the character of the client terminal 10 The object may include a first character object 410 corresponding to a human race and a second character object 420 corresponding to an orc race.

The processor 130 may generate first image vector information including instruction information for generating serious, soft, and masculine voice data based on the image information on the first character object 410, and the first image Voice data capable of outputting first voice data generated based on the vector information, the first dialogue text 411 , and the first voice identification information (eg, the voice of the first voice actor) corresponding to the first character object 410 . can be determined in advance. That is, when the character of the client terminal 10 performs a play associated with the first character object 410 in the process of playing the game, the first voice data (ie, the solemn and soft masculine voice data) may be provided.

In addition, the processor 130 may generate second image vector information including instruction information for generating voice data of a lower tone and an angry tone based on the image information about the second character object 420 , , corresponding to the second voice data generated based on the second image vector information, the second dialogue text 421 and the second voice identification information (eg, the voice of the second voice actor) to the second character object 420 . This can be pre-determined as outputable voice data. That is, when the character of the client terminal 10 performs play associated with the second character object 420 in the process of playing the game, the second voice data (ie, lower tone and anger based on the second voice actor's voice) voice data in a full tone) may be provided. Specific description of the above-described character object, dialogue text, voice identification information, and voice data is merely an example, and the present disclosure is not limited thereto.

That is, when the character of the client terminal 10 performs a game play associated with each of the first character object 410 and the second character object 420 , the processor 130 performs different voice data corresponding to each play. can provide In other words, the processor 130 may provide voice data corresponding to a play situation performed by the character of the client terminal 10 .

For another example, as shown in FIG. 5 , when the client terminal 10 connects to the game server 200 and plays a game, the user character object 510 of the client terminal 10 is provided with information about the game. The object providing the guidance voice may be a third character object 520 that is a female NPC.

The processor 130 has a shy tone based on image information (ie, female) about the third character object 520 that is a female NPC and image information (ie, male) on the user character object 510 of the client terminal 10 . and third image vector information including instruction information for generating voice data of shameful emotion, the third image vector information, the third dialogue text 530 and the third voice identification information (eg, the third The third voice data generated based on the voice of a voice actor) may be pre-determined as outputable voice data corresponding to the third character object 520 . That is, the processor 130 may provide voice data in consideration of the relationship between objects as voice data corresponding to a play situation performed by the character of the client terminal 10 . Although the above-described example describes that the relationship between each object is based on gender, the present disclosure provides more diverse relationships that can be formed between each object (eg, a relationship or position, occupation according to the race of the player character and NPC associated with the game story). It is also possible to provide voice data corresponding to the upper and lower relationships of the upper and lower levels).

Therefore, there is no need to pre-record all of the vast amount of guidance voices in the game through the voice actors, resulting in a reduction in the cost of providing in-game guidance voices, which rises according to the cost of hiring voice actors and recording costs, thereby increasing the profitability of the game service. can be stocked In addition, in the case of recording a specific text through a voice actor, it may be difficult to modify it later, but through the above-described configurations of the present disclosure, parameters are adjusted to synthesize a voice with emotion, convert the speaker's voice, etc. Through various voice synthesis, it may be possible to modify the provided voice data corresponding to the game play situation.

According to an embodiment of the present disclosure, the processor 130 may cause the client terminal 10 to output voice data corresponding to an in-game play situation. Specifically, when receiving the voice output control signal from the client terminal 10 , the processor 130 may cause the client terminal 10 to output voice data corresponding to the in-game play situation. The voice output control signal is generated in response to the play of the character of the client terminal 10, for example, to be generated when the character of the client terminal 10 intends to perform or completes one or more contents existing in the game. can For another example, the character of the client terminal 10 outputs a voice in response to a linked play on an object that outputs voice data for providing game information to the user of the client terminal 10 among a plurality of objects existing in the game A control signal may be generated. The detailed description of the above-described audio output control signal is only an example, and the present disclosure is not limited thereto.

More specifically, when a game is updated through the game server 200 , the processor 130 may pre-store a plurality of synthesized voice data generated through the voice synthesis model 600 in a content delivery network (CDN). . The CDN configures a plurality of cache servers to correspond to the game server 200 , and when receiving a use request and/or a download request from the client terminal 10 , the CDN responds to the request from the cache server closest to the request site It can mean conveying information that In addition, the processor 130 may access the game server 200 through the CDN to transmit a plurality of voice data to each of the client terminals 10 performing a game. That is, the client terminal 10 may download a plurality of synthesized voice data generated through the voice synthesis model 600 through the CDN during game update. In other words, each of the client terminals 10 may download a plurality of voice data corresponding to each play situation in the game through the CDN when the game is updated, and the user performs the game on the game server 200 while playing the game. It is possible to output audio data corresponding to the play being played.

Accordingly, the client terminal 10 can download a plurality of synthesized voice data generated through the CDN, thereby solving a bottleneck that may occur in the process of downloading the corresponding voice data, and providing voice data quickly and stably can receive In addition, even if a specific cache server fails, since voice data can be downloaded from another cache server, the download may not be interrupted.

Also, when the client terminal 10 does not download a plurality of synthesized voice data generated through the voice synthesis model 600 , the processor 130 may transmit synthesized voice data in real time. Specifically, the processor 130 may identify whether synthesized voice data corresponding to the play performed by the client terminal 10 has been downloaded. In addition, when the synthesized voice data corresponding to the play performed by the client terminal 10 is not downloaded, the processor 130 may transmit the synthesized voice data in real time through the CDN.

For example, the client terminal 10 connects to the game server 200 to play a first stage, and the client terminal 10 receives first synthesized voice data corresponding to the first stage. If it is not downloaded, the processor 130 may identify that the first synthesized voice data is not downloaded to the client terminal 10 and transmit the first synthesized voice data to the client terminal 10 through the CDN. . The detailed description of the above stage and synthesized voice data is only an example, and the present disclosure is not limited thereto.

According to another embodiment of the present disclosure, the processor 130 may cause the client terminal 10 to output the synthesized voice data by providing a URL corresponding to the synthesized voice data to the client terminal 10 . have. More specifically, when the client terminal 10 intends to perform one or more contents included in the game, the processor 130 transmits a URL corresponding to synthesized voice data corresponding to each of the one or more contents to the client terminal 10 ) to cause the client terminal 10 to output synthesized voice data corresponding to each of the one or more contents.

For example, when the client terminal 10 wants to interact with the first NPC, the processor 130 provides the URL of the first voice data stored in advance in response to the first NPC to the client terminal 10 . In this case, the client terminal 10 may output the first voice data received from the processor 130 as a voice for interaction with the first NPC. The detailed description of the NPC and voice data described above is only an example, and the present disclosure is not limited thereto.

That is, the client terminal 10 receives the URL of the synthesized voice data corresponding to the play the user wants to perform from the processor 130 without the need to pre-store a plurality of synthesized voice data reflected in the game through downloading. It is possible to output audio data corresponding to the user's play.

3 illustrates an exemplary flowchart for providing voice data corresponding to an in-game play situation related to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the voice data generation server 100 may generate data data for voice synthesis based on game data ( 310 ).

According to an embodiment of the present disclosure, the voice data generation server 100 may input metadata and dialogue text into the voice synthesis model ( 320 ).

According to an embodiment of the present disclosure, the voice data generation server 100 may generate synthesized voice data through a voice synthesis model ( 330 ).

The order of the steps illustrated in FIG. 3 described above may be changed if necessary, and at least one or more steps may be omitted or added. That is, the above-described steps are merely an embodiment of the present disclosure, and the scope of the present disclosure is not limited thereto.

8 is a schematic diagram illustrating a network function related to an embodiment of the present disclosure.

Throughout this specification, the terms sub-model, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as “nodes”. These “nodes” may be referred to as “neurons”. A neural network is configured to include at least one or more nodes. Nodes (or neurons) constituting the neural networks may be interconnected by one or more “links”.

In the neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may be in an input node relationship in a relationship with another node, and vice versa. As described above, an input node-to-output node relationship may be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node sets values input to input nodes connected to the output node and links corresponding to the respective input nodes. An output node value may be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship in the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between the nodes and the links, and the value of a weight assigned to each of the links. For example, when the same number of nodes and links exist and there are two neural networks having different weight values between the links, the two neural networks may be recognized as different from each other.

A neural network may include one or more nodes. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance of n from the initial input node is You can configure n layers. The distance from the initial input node may be defined by the minimum number of links that must be passed to reach the corresponding node from the initial input node. However, the definition of such a layer is arbitrary for description, and the order of the layer in the neural network may be defined in a different way from the above. For example, a layer of nodes may be defined by a distance from the final output node.

The initial input node may mean one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. In addition, the hidden node may mean nodes constituting the neural network other than the first input node and the last output node. The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as progresses from the input layer to the hidden layer. can Also, in the neural network according to another embodiment of the present disclosure, the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes may be reduced as the number of nodes progresses from the input layer to the hidden layer. have. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes progresses from the input layer to the hidden layer. can The neural network according to another embodiment of the present disclosure may be a neural network in a combined form of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify the latent structures of data. In other words, it can identify the potential structure of photos, texts, videos, voices, and music (e.g., what objects are in the photos, what the text and emotions are, what the texts and emotions are, etc.) . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep trust network (DBN), a Q network, a U network, a Siamese network, and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

The neural network may be trained by at least one of supervised learning, unsupervised learning, and semi-supervised learning. The training of the neural network is to minimize the error in the output. In the training of a neural network, iteratively input the training data into the neural network, calculate the output of the neural network and the target error for the training data, and calculate the error of the neural network from the output layer of the neural network to the input layer in the direction to reduce the error. It is a process of updating the weight of each node in the neural network by backpropagation in the direction. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (ie, labeled learning data), and in the case of comparative learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning related to data classification may be data in which categories are labeled in each of the learning data. The labeled training data is input to the neural network, and an error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of comparison learning related to data classification, an error may be calculated by comparing the input training data with the neural network output. The calculated error is back propagated in the reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back propagation. The change amount of the connection weight of each node to be updated may be determined according to a learning rate. The computation of the neural network on the input data and the backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, in the early stage of learning of a neural network, a high learning rate can be used to enable the neural network to quickly obtain a certain level of performance, thereby increasing efficiency, and using a low learning rate at a later stage of learning can increase accuracy.

In the training of neural networks, in general, the training data may be a subset of real data (that is, data to be processed using the trained neural network), and thus, the error on the training data is reduced, but the error on the real data is reduced. There may be increasing learning cycles. Overfitting is a phenomenon in which errors on actual data increase by over-learning on training data as described above. For example, a phenomenon in which a neural network that has learned a cat by seeing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing errors in machine learning algorithms. In order to prevent such overfitting, various optimization methods can be used. In order to prevent overfitting, methods such as increasing training data, regulation, or dropout omitting a part of nodes in the network in the process of learning may be applied.

9 illustrates a module for implementing a method for generating voice data corresponding to an in-game play situation related to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, a computer program may be implemented by the following modules.

According to an embodiment of the present disclosure, the computer program includes a module 710 for generating metadata for speech synthesis based on game data, and a module for inputting the metadata and dialogue text into a speech synthesis model ( 720) and a module 730 for generating synthesized voice data through the voice synthesis model.

Alternatively, the metadata is attribute indication information for generating the voice data, and may include at least one of image vector information, style vector information, and voice identification information.

Alternatively, the attribute indication information includes at least one of attribute information of each of the one or more contents included in the game, relationship information between a plurality of objects existing in the game, and attribute information of each of the plurality of objects can do.

Alternatively, the image vector information includes instruction information for generating voice data generated based on a game image, and the style vector information includes predetermined instruction information for each of one or more contents included in the game. and the voice identification information may include information for identifying seed voice data for voice synthesis.

Alternatively, the dialogue text may include content information for generating the voice data, and may be predetermined based on each of one or more contents included in the game.

Alternatively, when receiving a voice output control signal from the client terminal, the module may further include a module for causing the client terminal to output voice data corresponding to the in-game play situation.

Alternatively, for constructing training data for training the speech synthesis model based on the one or more seed speech data, text corresponding to each of the one or more seed speech data, and metadata associated with the one or more seed speech data; It may further include a module.

Alternatively, the speech synthesis model includes a dimensionality reduction submodel, a dimension restoration submodel, an attention module, and an audio output module, and is trained through training data including the metadata, training dialogue text, and training target voice data. can be

Alternatively, the speech synthesis model may include a learning spectrogram corresponding to the target speech data by using the learning input text including the metadata and the training dialogue text as an input of the dimension reduction sub-model, and the dimension reconstruction sub-model corresponding to the target speech data. (Spectrogram) can be learned to output.

Alternatively, the dimensionality reduction sub-model may take the metadata and the dialogue text as inputs, and output a language speech feature from the dialogue text.

Alternatively, the dimensional reconstruction submodel includes one or more recurrent neural networks (RNNs), wherein the one or more RNNs receive a spectrogram of a first timestamp as an input and output a spectrogram of a second timestamp. can

Alternatively, the attention module may generate association information between a phoneme of the dialogue text and a time step of the dimensional reconstruction submodel.

Alternatively, the speech output module may generate speech data synthesized using a speech reconstruction algorithm using a spectrogram that is an output of the dimensional reconstruction sub-model.

According to an embodiment of the present disclosure, a module for implementing a method of generating voice data corresponding to an in-game play situation may be implemented by means, circuits or logic for implementing a computing program.

Those skilled in the art will further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be combined with electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

10 shows a simplified, general schematic diagram of an exemplary computing environment in which one embodiment of the present disclosure may be implemented, in conjunction with one embodiment of the present disclosure.

Although the present disclosure has been described above generally in the context of computer-executable instructions that may be executed on one or more computers, those skilled in the art will appreciate that the present disclosure may be implemented in combination with other program modules and/or in a combination of hardware and software. will be.

Generally, program modules include routines, procedures, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those skilled in the art will appreciate that the methods of the present disclosure can be applied to single-processor or multiprocessor computer systems, minicomputers, mainframe computers as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which is It will be appreciated that other computer system configurations may be implemented, including those that may operate in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and the computer-readable medium may include a computer-readable storage medium and a computer-readable transmission medium. Such computer-readable storage media includes volatile and nonvolatile media, removable and non-removable media. Computer readable storage media includes volatile and nonvolatile media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. A computer-readable storage medium may be RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device, or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store the desired information.

Computer-readable transmission media typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. and information delivery media. The term modulated data signal means a signal in which one or more of the characteristics of the signal is set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 1100 implementing various aspects of the disclosure is shown including a computer 1102 , the computer 1102 including a processing unit 1104 , a system memory 1106 , and a system bus 1108 . do. A system bus 1108 couples system components, including but not limited to system memory 1106 , to the processing device 1104 . The processing device 1104 may be any of a variety of commercially available processors. Dual processor and other multiprocessor architectures may also be used as processing unit 1104 .

The system bus 1108 may be any of several types of bus structures that may further interconnect a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112 . A basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, EEPROM, etc., the BIOS is the basic input/output system (BIOS) that helps transfer information between components within computer 1102, such as during startup. contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

The computer 1102 may also include an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA) - this internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes—a magnetic floppy disk drive (FDD) 1116 (eg, for reading from or writing to removable diskette 1118), and an optical disk drive 1120 (eg, a CD-ROM) for reading from, or writing to, disk 1122, or other high capacity optical media, such as DVD. The hard disk drive 1114 , the magnetic disk drive 1116 , and the optical disk drive 1120 are connected to the system bus 1108 by the hard disk drive interface 1124 , the magnetic disk drive interface 1126 , and the optical drive interface 1128 , respectively. ) can be connected to The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of computer readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will use zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other tangible computer-readable media such as etc. may also be used in the exemplary operating environment and any such media may include computer-executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored in the drive and RAM 1112 , including an operating system 1130 , one or more application programs 1132 , other program modules 1134 , and program data 1136 . All or portions of the operating system, applications, modules, and/or data may also be cached in RAM 1112 . It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 via one or more wired/wireless input devices, for example, a pointing device such as a keyboard 1138 and a mouse 1140 . Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are connected to the processing unit 1104 through an input device interface 1142 that is often connected to the system bus 1108, parallel ports, IEEE 1394 serial ports, game ports, USB ports, IR interfaces, and the like may be connected by other interfaces.

A monitor 1144 or other type of display device is also coupled to the system bus 1108 via an interface, such as a video adapter 1146 . In addition to the monitor 1144, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communications. Remote computer(s) 1148 may be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, and are generally Although including many or all of the components described, only memory storage device 1150 is shown for simplicity. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, eg, a wide area network (WAN) 1154 . Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, for example, the Internet.

When used in a LAN networking environment, the computer 1102 is coupled to the local network 1152 through a wired and/or wireless communication network interface or adapter 1156 . Adapter 1156 may facilitate wired or wireless communication to LAN 1152 , which LAN 1152 also includes a wireless access point installed therein for communicating with wireless adapter 1156 . When used in a WAN networking environment, the computer 1102 may include a modem 1158 , connected to a communication server on the WAN 1154 , or otherwise establishing communications over the WAN 1154 , such as over the Internet. have the means A modem 1158 , which may be internal or external and a wired or wireless device, is coupled to the system bus 1108 via a serial port interface 1142 . In a networked environment, program modules described for computer 1102 , or portions thereof, may be stored in remote memory/storage device 1150 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between the computers may be used.

The computer 1102 may be associated with any wireless device or object that is deployed and operates in wireless communication, for example, a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communication satellite, wireless detectable tag. It operates to communicate with any device or place, and phone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network or may simply be an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) makes it possible to connect to the Internet, etc. without a wired connection. Wi-Fi is a wireless technology such as cell phones that allows these devices, eg, computers, to transmit and receive data indoors and outdoors, ie anywhere within range of a base station. Wi-Fi networks use a radio technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks may operate in unlicensed 2.4 and 5 GHz radio bands, for example, at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band). have.

Those of ordinary skill in the art of the present disclosure will recognize that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein include electronic hardware, (convenience For this purpose, it will be understood that it may be implemented by various forms of program or design code (referred to herein as "software") or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. A person skilled in the art of the present disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as a departure from the scope of the present disclosure.

The various embodiments presented herein may be implemented as methods, apparatus, or articles of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory. devices (eg, EEPROMs, cards, sticks, key drives, etc.). Also, various storage media presented herein include one or more devices and/or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, wireless channels and various other media that can store, hold, and/or convey instruction(s) and/or data.

It is to be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. Based on design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The appended method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

Claims (15)

A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, causes the one or more processors to perform the following operations for outputting voice data corresponding to an in-game play situation, , the operations are:
generating metadata for voice synthesis based on game data related to game build and image data related to a plurality of objects included in the game;
inputting the metadata and dialogue text into a speech synthesis model; and
generating speech data synthesized through the speech synthesis model;
includes,
The metadata is attribute indication information for generating the voice data, and the attribute indication information includes attribute information of each of one or more contents included in the game, relationship information between a plurality of objects existing in the game, and the plurality of pieces of information. Including at least one piece of information among the attribute information of each object,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The metadata is
comprising at least one of image vector information, style vector information, and voice identification information,
A computer program stored on a computer-readable storage medium.

delete 3. The method of claim 2,
The image vector information is,
Includes instruction information for generating voice data generated based on the game image,
The style vector information is
including predetermined instructional information for each of the one or more contents included in the game; and
The voice identification information is
Containing information for identifying seed voice data for voice synthesis,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The dialogue text is
It includes content information for generating the voice data, and is predetermined based on each of the one or more contents included in the game,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
when receiving a voice output control signal from the client terminal, causing the client terminal to output voice data corresponding to the in-game play situation;
further comprising,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
constructing training data for training the speech synthesis model based on the at least one seed speech data, text corresponding to each of the at least one seed speech data, and metadata associated with the at least one seed speech data;
further comprising,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The speech synthesis model is
It includes a dimension reduction sub-model, a dimension restoration sub-model, an attention module and a voice output module, and is learned through learning data including the metadata, learning dialogue text, and learning target voice data,
A computer program stored on a computer-readable storage medium.
9. The method of claim 8,
The speech synthesis model is
By using the learning input text including the meta data and the learning dialogue text as an input of the dimension reduction sub-model, the dimension restoration sub-model is trained to output a learning spectrogram corresponding to the target speech data,
A computer program stored on a computer-readable storage medium.
9. The method of claim 8,
The dimensionality reduction sub-model is,
outputting language speech features from the dialogue text by inputting the metadata and the dialogue text as inputs;
A computer program stored on a computer-readable storage medium.
9. The method of claim 8,
The dimensional restoration sub-model is,
It contains one or more Recurrent Neural Networks (RNNs),
The one or more RNNs are
Outputting the spectrogram of the second timestamp by receiving the spectrogram of the first timestamp as an input,
A computer program stored on a computer-readable storage medium.
9. The method of claim 8,
The attention module is
generating association information between the phoneme of the dialogue text and the time step of the dimensional reconstruction sub-model,
A computer program stored on a computer-readable storage medium.
9. The method of claim 8,
The audio output module is
Generating speech data synthesized using a speech reconstruction algorithm using a spectrogram output from the dimensional reconstruction sub-model,
A computer program stored on a computer-readable storage medium.
A method of generating voice data corresponding to the in-game play situation,
generating metadata for voice synthesis based on game data related to game build and image data related to a plurality of objects included in the game;
inputting the meta data and the dialogue text into a speech synthesis model; and
generating speech data synthesized through the speech synthesis model;
includes, and
The metadata is attribute indication information for generating the voice data, and the attribute indication information includes attribute information of each of one or more contents included in the game, relationship information between a plurality of objects existing in the game, and the plurality of pieces of information. Including at least one piece of information among the attribute information of each object,
A method of generating voice data corresponding to in-game play situations.
A server for generating voice data corresponding to an in-game play situation, the server comprising:
a processor including one or more cores;
a memory storing program codes executable by the processor; and
a network unit for transmitting and receiving data to and from the game server and the client terminal;
including, and
Generate metadata for voice synthesis based on game data related to game build and image data related to a plurality of objects included in the game,
inputting the metadata and dialogue text into a speech synthesis model; and
Generates synthesized speech data through the speech synthesis model,
The metadata is attribute indication information for generating the voice data, and the attribute indication information includes attribute information of each of one or more contents included in the game, relationship information between a plurality of objects existing in the game, and the plurality of pieces of information. Including at least one piece of information among the attribute information of each object,
A server for generating voice data corresponding to the in-game play situation.

Images