JP6993034B1 - Content playback method and content playback system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a content reproduction method and a content reproduction system capable of expressing a user's appearance with the appearance of a user's favorite character. SOLUTION: An acquisition step for acquiring face image data including a character's face and emotion data indicating emotion, a first input data including a reference face image data and reference emotion data acquired in advance, and The first output data including the face of the same character as the character included in the reference face image data and including the first processed face image data different from the reference face image data is used as a set of data sets. Using a plurality of training data for 1 processing, the first processing database generated by machine learning is referred to, and the face image data after the first processing for the face image data and the emotion data acquired in the acquisition step is generated. A content reproduction method characterized by causing a computer to execute a first generation step. [Selection diagram] Fig. 2

Description

The present invention relates to a content reproduction method and a content reproduction system.

In recent years, when distributing videos, the user's appearance and voice should be the appearance and voice of the user's favorite character in order to protect the privacy of the distributor or to distribute with the distributor's favorite face and voice. There is a need for technology that can be expressed. Therefore, a technique for expressing a character's voice so that a conversation by a character is more natural than a conversation by a user is attracting attention. For example, the information processing system of Patent Document 1 is known. ..

In the technique described in Patent Document 1, the processor receives a selection signal for selecting a specific character via a client terminal of a user, and at the same time, transmits and receives a speech phrase of the specific character by a communication unit. Generates a converted message converted to the voice of a specific character based on the user's message. Further, the technique described in Patent Document 1 generates an utterance phrase of a specific character corresponding to a user's message, and returns the generated conversion message and the utterance phrase to the client terminal. Patent Document 1 describes a technique related to an information processing system that makes it possible to further enhance entertainment by allowing a user to experience a character by himself / herself.

Japanese Unexamined Patent Publication No. 2021-39370

Here, in Patent Document 1, a converted message converted into a voice of a specific character is generated based on the received message of the user. However, Patent Document 1 does not assume that the appearance of a specific character is reflected in the character conversion message. Therefore, in Patent Document 1, it is not possible to express the user's distribution with the appearance desired by the user. Therefore, there is a demand for a technique capable of expressing the user's appearance with the appearance of the user's favorite character.

Therefore, the present invention has been devised in view of the above-mentioned problems, and an object thereof is a content reproduction method capable of expressing the appearance of a user with the appearance of a character of the user's preference. And to provide a content reproduction system.

The content reproduction method according to the first invention includes an acquisition step of acquiring face image data including a character's face, emotion data indicating emotions, and text data, and previously acquired reference face image data and reference emotions. The first output data including the first input data including the data and the face of the same character as the character included in the reference face image data and including the first processed face image data different from the reference face image data. With reference to the first processing database generated by machine learning using a plurality of first processing training data having The first is based on the first generation step of generating the post-processed face image data, the first post-processed face image data generated by the first generation step , and the text data acquired by the acquisition step. A computer is made to execute a second generation step of generating a second post-processed face image data in which a part of the post-processed face image data is changed, and the second generation step is a pre-acquired first post-processing for reference. A plurality of second processing training data in which the second input data including the face image data and the reference text data and the second output data including the second processed face image data for reference are set as a set of data. Using, referring to the second processing database generated by machine learning, based on the first processed face image data generated by the first generation step and the text data acquired by the acquisition step. It is characterized in that the face image data after the second processing is generated .

The content reproduction method according to the second invention is the first invention, in which the acquisition step acquires voice quality data related to voice quality, and is based on the voice quality data, text data, and emotion data acquired by the acquisition step. , The expression of the character based on the voice processing step that generates the voice data indicating the voice of the character, the processed face image data generated by the generation step, and the voice data generated by the voice processing step. It is characterized in that a computer is further executed with an expression generation step for generating expression data indicating.

The content reproduction method according to the third invention is the second invention, in which the voice processing step includes a third input data including a previously acquired reference voice quality data, reference text data, and reference emotion data. Using a plurality of voice processing learning data having a third output data including reference voice data as a set of data, the voice processing database generated by machine learning was referred to, and the data was acquired by the acquisition step. It is characterized in that the voice data for the voice quality data, the text data, and the emotion data is generated.

The content reproduction method according to the fourth invention is a response model showing a correspondence relationship between the reference conversation text data acquired in advance and the response data to the reference conversation text data in any one of the first invention to the third invention. It is characterized in that the response data to the conversational sentence data input by the user is determined by reference, and the text data based on the determined response data is acquired.

The content reproduction system according to the fifth invention is an acquisition means for acquiring face image data including a character's face, emotion data indicating emotions, and text data, and previously acquired reference face image data and reference emotions. The first output data including the first input data including the data and the face of the same character as the character included in the reference face image data and including the first processed face image data different from the reference face image data. With reference to the first processing database generated by machine learning using a plurality of first processing training data having The first generation means based on the first generation means for generating the face image data after the first processing, the first post-processing face image data generated by the first generation means , and the text data acquired by the acquisition means. The second generation means for generating the second post-processed face image data in which a part of the processed face image data is changed is provided, and the second generation means is the first post-processed face image data for reference acquired in advance. And, using a plurality of second processing training data in which the second input data including the reference text data and the second output data including the reference second post-processed facial image data are set as a set of data. With reference to the second processing database generated by machine learning, the second processing based on the first processed face image data generated by the first generation means and the text data acquired by the acquisition means. It is characterized by generating face image data after processing .

According to the first to fourth inventions, the first processed facial image data for the facial image data and the emotional data is generated with reference to the first processing database. This makes it possible to generate face image data after the first processing that reflects the emotions of the user. As a result, the user's emotions are reflected, and the user's appearance can be expressed by the appearance of the user's favorite character. Further, referring to the second processing database, the second processed face image data is generated based on the first processed face image data and the text data. As a result, it is possible to generate highly accurate second-processed face image data suitable for the input text data, and it is possible to express a highly accurate character according to the conversation of the user.

In particular, according to the second invention, voice data is generated based on voice quality data, text data, and emotion data, and expression data is generated based on processed facial image data and voice data. As a result, it is possible to generate voice data that reflects the user's emotions, so that it is possible to express a character that can reflect the user's emotions.

In particular, according to the third invention, the voice data is generated based on the voice quality data, the text data, and the emotion data with reference to the voice processing database generated by machine learning. As a result, it is possible to generate highly accurate voice data that reflects the user's emotions, so that it is possible to express a character that can reflect the user's emotions.

According to the fourth invention, the response data for the conversational sentence data input by the user is determined with reference to the response model, and the text data based on the determined response data is acquired. As a result, the response data to the conversational sentence data input by the user can be automatically acquired, so that the character can be expressed according to the conversation of the user.

FIG. 1 is a schematic diagram showing an example of a content reproduction system according to the first embodiment. FIG. 2 is a schematic diagram showing an example of the operation of the content reproduction system according to the first embodiment. FIG. 3 is a schematic diagram showing an example of a learning method of the first processing database. FIG. 4 is a schematic diagram showing an example of the linkage of the first processing database. FIG. 5A is a schematic diagram showing an example of the configuration of the content playback device according to the first embodiment, and FIG. 5B is a schematic diagram showing an example of the function of the content playback device according to the first embodiment. be. FIG. 6 is a diagram showing an example of the operation of the content reproduction device according to the first embodiment. FIG. 7 is a schematic diagram showing an example of the operation of the content reproduction system according to the second embodiment. FIG. 8 is a schematic diagram showing an example of a learning method of the second processing database. FIG. 9 is a schematic diagram showing an example of the linkage of the second processing database. FIG. 10 is a diagram showing an example of the operation of the content reproduction device according to the second embodiment. FIG. 11 is a schematic diagram showing an example of the operation of the content reproduction system according to the third embodiment. FIG. 12 is a schematic diagram showing an example of a learning method of the first processing database in the third embodiment. FIG. 13 is a schematic diagram showing an example of the linkage of the third processing database. FIG. 14 is a diagram showing an example of the operation of the content reproduction device according to the third embodiment. FIG. 15 is a diagram showing an example of the operation of the content reproduction device according to the fourth embodiment. FIG. 16 is a schematic diagram showing an example of a learning method of a voice processing database. FIG. 17 is a schematic diagram showing an example of the linkage of the speech processing database. FIG. 18 is a diagram showing an example of the operation of the content reproduction device according to the fifth embodiment. FIG. 19 is a schematic diagram showing an example of the operation of the content reproduction system according to the sixth embodiment. FIG. 20 is a schematic diagram showing an example of a learning method of the first processing database according to the sixth embodiment. FIG. 21 is a schematic diagram showing an example of the linkage of the sixth processing database. FIG. 22 is a diagram showing an example of the operation of the content reproduction device according to the sixth embodiment.

Hereinafter, an example of the content reproduction system according to the embodiment to which the present invention is applied will be described with reference to the drawings.

(First Embodiment)
An example of the content reproduction system 100, the content reproduction device 1, and the learning method according to the first embodiment will be described with reference to the drawings. FIG. 1 is a schematic diagram showing an example of the content reproduction system 100 in the present embodiment. FIG. 2 is a schematic diagram showing an example of the operation of the content reproduction system 100 in the present embodiment.

<Content playback system 100>
The content reproduction system 100 is used to generate the face image data after the first processing with respect to the input arbitrary face image data and the emotion data. The content reproduction system 100 refers to, for example, a first processing database generated by machine learning using learning data, and generates face image data after the first processing with respect to face image data and emotion data.

The content reproduction system 100 includes, for example, a content reproduction device 1 as shown in FIG. The content reproduction system 100 may include, for example, at least one of a terminal 2 and a server 3. The content reproduction device 1 is connected to the terminal 2 and the server 3 via, for example, the communication network 4.

In the content reproduction system 100, for example, as shown in FIG. 2, the content reproduction device 1 acquires input data. For example, the content reproduction device 1 acquires input data. After that, the face image data after the first processing is generated with respect to the input data by referring to the database for the first processing.

The face image data is image data including the face of the character. The face image data is used, for example, when generating the face image data after the first processing output by the content reproduction system 100. The image data is data including an aggregate of a plurality of pixels. The face image data may be, for example, extracted from a moving image or may be moving image data. Further, the reference face image data is face image data used for the first processing learning data, and may be in the same format as the face image data.

The face image data may be acquired via, for example, the communication network 4. The face image data may show, for example, a face image captured by using a known image pickup device or the like, or may show, for example, a pseudo face image generated by a known technique. The face image data may be input by a user or the like via, for example, the content reproduction device 1.

Emotion data is data showing emotions. The emotional data may be, for example, text data showing emotions such as anger, joy, and sadness. Further, the emotional data may be text data or numerical data in which anger, joy, sadness, etc. are evaluated by three or more grades such as percentage.

The face image data after the first processing is face image data that includes the face of the same character as the character included in the face image data and is different from the face image data. The face image data after the first processing may include, for example, the face of the same character as the character included in the face image data, and may have a different facial expression, gesture, orientation, etc. from the face of the character in the face image data.

The above-mentioned "character" is a person or animal simulated to imitate a user, a person or animal simulated to imitate a real person or animal, or an animation or the like. It may be a pseudo-generated person or animal.

The first processing database is generated by machine learning. As the first processing database, for example, the first input data including the reference face image data and the reference emotion data and the first output data including the first processed face image data are set as a set of data sets. A trained model for generating the first output data from the first input data, which is generated by machine learning, is used by using a plurality of training data for processing.

As shown in FIG. 3, for example, the first processing database includes the first input data including the reference face image data and the reference emotion data, and the first output data including the first processed face image data. It is generated by machine learning using a plurality of first processing training data as a set of data sets.

The first processing database is generated using, for example, machine learning using a neural network as a model. The first processing database is generated by using machine learning using a neural network such as CNN (Convolution Neural Network) as a model, or any model may be used.

The first processing database has a degree of association between the first input data including, for example, reference face image data and reference emotion data, and the first output data including the first processed face image data. 1 Relationship is memorized. The degree of association indicates the degree of connection between the first input data and the first output data. For example, it can be determined that the higher the degree of association, the stronger the connection of each data. The degree of association may be indicated by three values or more (three stages or more) such as percentage, or may be indicated by two values (two stages).

For example, the association is constructed by the degree of connection between many-to-many information (plurality of first input data, pair, plurality of first output data). The association is appropriately updated in the process of machine learning, and indicates a function (classifier) optimized based on, for example, a plurality of first input data and a plurality of first output data. The first association may have a plurality of association degrees indicating the degree of connection between the data, for example. The degree of association can correspond to a weight variable, for example, when the database is constructed with a neural network.

Therefore, in the content reproduction system 100, for example, the first output data suitable for the first input data is selected by using the first association based on all the determination results of the classifier. As a result, it is possible to quantitatively select the first output data suitable for the first input data even when the first input data is the same as or similar to the first output data or dissimilar to the first output data. can.

The first association may indicate the degree of connection between the plurality of first output data and the plurality of first input data, for example, as shown in FIG. In this case, by using the first association, a plurality of first output data (“first processed face image data A” to “first processed face image data C” in FIG. 4) are obtained. It is possible to store the degree of relationship of the first input data (“face image data A + emotion data A” to “face image data C + emotion data C” in FIG. 4) in association with each other. Therefore, for example, a plurality of first input data can be associated with one first output data via the first association. As a result, it is possible to realize the selection of the first output data from various angles with respect to the first input data.

The first association has, for example, a plurality of association degrees for associating each first output data with each first input data. The degree of association is shown in three or more steps such as percentage, 10 steps, or 5 steps, and is shown by, for example, the characteristics of the line (for example, thickness). For example, the "face image data A + emotion data A" included in the first input data indicates a degree of association AA "73%" with the "first processed face image data A" included in the first output data. , The degree of association AB “12%” with the “first processed face image data B” included in the first output data is shown. That is, the "degree of association" indicates the degree of connection between each data, and for example, the higher the degree of association, the stronger the connection of each data.

Further, in the first internal representation database, at least one hidden layer may be provided between the first input data and the first output data, and machine learning may be performed. The above-mentioned degree of association is set in either one or both of the first input data and the hidden layer data, and this is the weighting of each data, and the output is selected based on this. Then, when this degree of association exceeds a certain threshold value, the output may be selected.

<Content playback device 1>
Next, an example of the content reproduction device 1 in the present embodiment will be described with reference to FIGS. 5 and 6. FIG. 5A is a schematic diagram showing an example of the configuration of the content reproduction device 1 in the present embodiment, and FIG. 5B is a schematic diagram showing an example of the function of the content reproduction device 1 in the present embodiment. be.

As the content reproduction device 1, for example, an electronic device such as a laptop (notebook) PC or a desktop PC is used. As shown in FIG. 5A, for example, the content playback device 1 stores a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and the like. A unit 104 and I / F 105 to 107 are provided. Each configuration 101 to 107 is connected by an internal bus 110.

The CPU 101 controls the entire content reproduction device 1. The ROM 102 stores the operation code of the CPU 101. The RAM 103 is a work area used when the CPU 101 operates. The storage unit 104 stores various information such as a database and learning target data. As the storage unit 104, for example, in addition to an HDD (Hard Disk Drive), a data storage device such as an SSD (Solid State Drive) is used. For example, the content reproduction device 1 may have a GPU (Graphics Processing Unit) (not shown).

The I / F 105 is an interface for transmitting and receiving various information to and from the terminal 2, the server 3, the website, etc., as needed, via the communication network 4. The I / F 106 is an interface for transmitting / receiving information to / from the input unit 108. For example, a keyboard is used as the input unit 108, and the user or the like of the content reproduction device 1 inputs various information, a control command of the content reproduction device 1, or the like via the input unit 108. The I / F 107 is an interface for transmitting and receiving various information to and from the display unit 109. The display unit 109 displays various information, contents, and the like stored in the storage unit 104. A display is used as the display unit 109, and for example, in the case of a touch panel type, it is provided integrally with the input unit 108. Further, a speaker may be used for the display unit 109.

FIG. 5B is a schematic diagram showing an example of the function of the content reproduction device 1. The content reproduction device 1 includes an acquisition unit 11, a processing unit 12, a generation unit 13, an output unit 14, and a storage unit 15, and may include, for example, a DB generation unit 16. It should be noted that each function shown in FIG. 5B is realized by the CPU 101 executing a program stored in the storage unit 104 or the like using the RAM 103 as a work area, even if it is controlled by, for example, artificial intelligence. good.

<< Acquisition unit 11 >>
The acquisition unit 11 acquires the face image data and the emotion data. The acquired data is used when generating the above-mentioned first post-processing facial image data. The acquisition unit 11 acquires, for example, the face image data and the emotion data input from the input unit 108, and also acquires the face image data and the emotion data from the terminal 2 or the like via, for example, the communication network 4. May be good. Further, the acquisition unit 11 may acquire data selected by the user from a plurality of face image data and emotion data acquired in advance.

The acquisition unit 11 may acquire, for example, the learning data used for generating the above-mentioned database. In addition to acquiring the learning data input from the input unit 108, for example, the acquisition unit 11 may acquire the learning data from the terminal 2 or the like via, for example, the communication network 4.

For example, as the first processing learning data used for generating the first processing database, past reference face image data and reference emotion data can be mentioned.

<< Processing unit 12 >>
The processing unit 12 refers to, for example, the first processing database, and generates the face image data after the first processing for the face image data and the emotion data.

<< Generation unit 13 >>
The generation unit 13 generates at least one pseudo data based on the face image data generated by the processing unit 12. The generation unit 13 generates pseudo data including voice and face image based on the first processed face image data generated by the processing unit 12, for example. By generating the pseudo data, it is possible to output the expression of the character that is not stored in the storage unit 15. The generation unit 13 may use a known technique when generating pseudo data.

<< Output unit 14 >>
The output unit 14 outputs various data. The output unit 14 may output, for example, the pseudo data generated by the generation unit 13. The output unit 14 outputs various data to the display unit 109 via the I / F 107, and also outputs various data to a plurality of terminals 2 and the like via, for example, the I / F 105.

<< Memory unit 15 >>
The storage unit 15 retrieves various data such as a database stored in the storage unit 104 as needed. The storage unit 15 stores various data acquired or generated by the configurations 11 to 14 and 16 in the storage unit 104 as needed.

<< DB generation unit 16 >>
The DB generation unit 16 generates a database by machine learning using a plurality of learning data. For machine learning, for example, a neural network or the like is used.

<Terminal 2>
The terminal 2 is owned by, for example, a user of a service using the content reproduction system 100, and is connected to the content reproduction device 1 via a communication network 4. The terminal 2 may indicate, for example, an electronic device that generates a database. As the terminal 2, for example, an electronic device such as a personal computer or a tablet terminal is used. The terminal 2 may have at least some of the functions of the content reproduction device 1, for example.

<Server 3>
The server 3 is connected to the content reproduction device 1 via the communication network 4. Various past data and the like are stored in the server 3, and various data are transmitted from the content reproduction device 1 as needed. The server 3 may have at least a part of the functions of the content reproduction device 1, for example, and may perform at least a part of the processing instead of the content reproduction device 1, for example. The server 3 stores, for example, at least a part of various data stored in the storage unit 104 of the content reproduction device 1, and may be used in place of the storage unit 104, for example.

<Communication network 4>
The communication network 4 is, for example, an Internet network or the like to which the content reproduction device 1 is connected via a communication circuit. The communication network 4 may be configured by a so-called optical fiber communication network. Further, the communication network 4 may be realized by a known communication technology such as a wireless communication network in addition to the wired communication network.

(First Embodiment: Operation of content reproduction system)
Next, an example of the operation of the content reproduction system 100 in the present embodiment will be described. FIG. 6 is a diagram showing an example of the operation of the content reproduction device according to the first embodiment.

<Acquisition step S110>
The acquisition step S110 acquires the face image data and the emotion data input by the user or the like. In the acquisition step S110, for example, the acquisition unit 11 acquires the face image data and the emotion data. The acquisition unit 11 may acquire the face image data and the emotion data from, for example, the terminal 2 or the like, or may acquire the face image data and the emotion data from the storage unit 104 via, for example, the storage unit 15. Further, in the acquisition step S110, for example, as face image data, face image data such as a moving image and voice data associated with the face image data may be acquired.

<First processing step S120>
The first processing step S120 refers to, for example, the database for the first processing, and generates the face image data after the first processing for the face image data and the emotion data acquired in the acquisition step S110. In the first processing step S120, for example, the first processing unit 121 refers to the first processing database and generates the first processed facial image data for the facial image data and the emotional data. In the first processing step S120, the generated face image data after the first processing may be stored in the storage unit 104, for example, via the storage unit 15. The generated face image data after the first processing may be transmitted to, for example, the server 3, another content reproduction device 1, or a plurality of user terminals 2. As the data to be generated, a plurality of data may be generated for one input data. This makes it possible to generate face image data after the first processing that reflects the emotions of the user. This makes it possible to express a character that can reflect the user's emotions. Further, the generation unit 13 may generate the face image data after the first processing in a pseudo manner. Further, the first processing step S120 may be processed by, for example, the first processing unit 121 included in the processing unit 12.

<Output step S130>
In the output step S130, for example, the output unit 14 outputs the face image data after the first processing acquired in the first processing step S120 to the display unit 109, the terminal 2, and the like.

By performing each of the steps described above, the operation of the content reproduction system 100 in the present embodiment is completed.

(Second Embodiment)
Hereinafter, the content reproduction system 100 to which the second embodiment of the present invention is applied will be described. The second embodiment of the present invention is different from the first embodiment in that the second post-processed face image data for the first post-processed face image data and the text data is generated. Further, the description of the configuration similar to that of the first embodiment will be omitted.

<Content playback system 100>
As shown in FIG. 7, the content reproduction system 100 has, as shown in FIG. 7, after the first processing for the face image data and the emotion data generated by referring to the first processing database, and after the second processing for the text data. It is used to generate facial image data. The content reproduction system 100 refers to, for example, a second processing database generated by machine learning using learning data, and generates a second post-processing face image data for the first post-processing face image data and text data. .. In such a case, the content reproduction system 100 generates, for example, the first post-processed face image data in which the eyes of the character included in the face image data are changed as the first post-processed face image data for the face image data and the emotion data. Further, referring to the second processing database, the generated first post-processing face image data and the second post-processing face image data in which the mouth of the character included in the first post-processing face image data for the text data is changed are obtained. May be generated.

The text data is used, for example, when generating face image data generated by the content reproduction system 100. The text data may be, for example, a conversational sentence input by the user, a conversational sentence that the character wants to speak, or the like. Text data is data represented by characters and character codes. The text data includes, for example, control characters that are data for controlling a device such as a monitor or a printer. The control character includes, for example, a line feed character representing a line feed and a tab (horizontal tab).

The text data may be extracted via, for example, the communication network 4, and the text data may be extracted by recognizing the voice. The text data may be input by a user or the like via, for example, the content reproduction device 1.

The face image data after the second processing is face image data obtained by changing a part of the face image data after the first processing. The face image data after the second processing may be, for example, a part of the image data such as the mouth of the character included in the face image data after the first processing changed.

The second processing database is preferably generated by machine learning, for example, but this is not the case. As the database for the second processing, for example, a set of the second input data including the first processed face image data for reference and the text data for reference and the second output data including the second processed face image data for reference. A trained model for generating the second output data from the second input data generated by machine learning is used by using a plurality of second processing training data as the data set of. In such a case, the method of generating the second processing database is different from the first processing database in that the input data is the second input data and the output is the second output data.

As shown in FIG. 8, for example, the second processing database includes a second input data including a reference first post-processing facial image data and a reference emotion data, and a second post-processing facial image data for reference. It is generated by machine learning using a plurality of second processing training data having two output data as a set of data.

The second processing database is generated using, for example, machine learning using a neural network as a model. The second processing database is generated by using machine learning using a neural network such as CNN (Convolution Neural Network) as a model, or any model may be used.

In the second processing database, for example, as shown in FIG. 9, a second association having a degree of association between the second input data and the second output data is stored. The degree of association indicates the degree of connection between the second input data and the second output data. For example, it can be determined that the higher the degree of association, the stronger the connection of each data. The degree of association may be indicated by three values or more (three stages or more) such as percentage, or may be indicated by two values (two stages).

(Second Embodiment: Operation of content reproduction system)
Next, an example of the operation of the content reproduction system 100 in the second embodiment will be described. FIG. 10 is a diagram showing an example of the operation of the content reproduction device according to the second embodiment.

<Acquisition step S210>
The acquisition step S210 acquires the face image data, the emotion data, and the text data input by the user or the like. In the acquisition step S210, for example, the acquisition unit 11 acquires face image data, emotion data, and text data. Further, in the acquisition step S210, for example, as text data, face image data such as a moving image and voice data associated with the face image data may be acquired, and the acquired voice data may be acquired by voice recognition. ..

<Second processing step S240>
The second processing step S240 refers to, for example, the database for the second processing, and the second processed face with respect to the first processed face image data generated by the first processing step S120 and the text data acquired in the acquisition step S210. Generate image data. In the second processing step S240, for example, the second processing unit 122 refers to the second processing database and generates the second processed face image data for the first processed face image data and the text data. In such a case, the second processing unit 122 analyzes the input first-processed face image data by a known image analysis technique, and a part of the character's face included in the first-processed face image data, for example, the mouth. The second processed facial image data may be generated by determining the above and changing the determined portion according to the text data. Further, in the second processing step S240, the face image data after the second processing may be generated without using the database for the second processing. This makes it possible to generate face image data after the second processing suitable for the input text data, and it is possible to express a character with high accuracy according to the conversation of the user. Further, the generation unit 13 may generate pseudo second-processed face image data. Further, the second processing step S240 may be processed by the second processing unit 122 included in the processing unit 12, for example, and connected to the first processing unit 121.

By performing each of the steps described above, the operation of the content reproduction system 100 in the present embodiment is completed.

(Third Embodiment)
Hereinafter, the content reproduction system 100 to which the third embodiment of the present invention is applied will be described. The third embodiment of the present invention is different from the first embodiment in that the face image data after the first processing for the face image data, the emotion data, and the text data is generated. Further, the description of the configuration similar to that of the first embodiment will be omitted.

<Content playback system 100>
The content reproduction system 100 is used to generate the face image data after the first processing with respect to the arbitrary face image data, the emotion data, and the text data input as shown in FIG. The content reproduction system 100 refers to, for example, a first processing database generated by machine learning using learning data, and generates face image data, emotion data, and first-processed face image data for text data.

The first processing database is generated by machine learning. As the first processing database, for example, the first input data including the reference face image data, the reference emotion data, and the reference text data, and the first output data including the reference first processed face image data are combined. A trained model for generating the first output data from the first input data generated by machine learning is used by using a plurality of first processing training data as a set of data sets. In such a case, the method of generating the first processing database is different from the first embodiment in that the reference text data is included in the first input data.

As shown in FIG. 12, for example, the first processing database includes first input data including reference first post-processing face image data, reference emotion data, and reference text data, and reference first post-processing face. It is generated by machine learning using a plurality of first processing training data having the first output data including image data as a set of data sets.

The second processing database is generated by using machine learning using a neural network as a model, for example, as shown in FIG. The second processing database is generated by using machine learning using a neural network such as CNN (Convolution Neural Network) as a model, or any model may be used.

In the first processing database, for example, the first association having a degree of association between the first input data and the first output data is stored. The degree of association indicates the degree of connection between the first input data and the first output data. For example, it can be determined that the higher the degree of association, the stronger the connection of each data. The degree of association may be indicated by three values or more (three stages or more) such as percentage, or may be indicated by two values (two stages).

(Third Embodiment: Operation of content reproduction system)
Next, an example of the operation of the content reproduction system 100 in the third embodiment will be described. FIG. 14 is a diagram showing an example of the operation of the content reproduction device according to the third embodiment.

<Acquisition step S310>
The acquisition step S310 acquires the face image data, the emotion data, and the text data input by the user or the like.

<First processing step S320>
The first processing step S320 refers to, for example, the database for the first processing, and generates the face image data after the first processing for the face image data, the emotion data, and the text data acquired in the acquisition step S310. In the first processing step S320, for example, the first processing unit 121 refers to the first processing database and generates the face image data, the emotion data, and the post-processed face image data for the text data. This makes it possible to generate face image data after the first processing suitable for the input text data, and it is possible to express a character with high accuracy according to the conversation of the user. Further, the generation unit 13 may generate the face image data after the first processing in a pseudo manner. Further, the first processing step S320 may be processed by, for example, the first processing unit 121 included in the processing unit 12.

By performing each of the steps described above, the operation of the content reproduction system 100 in the present embodiment is completed.

(Fourth Embodiment)
Hereinafter, the content reproduction system 100 to which the fourth embodiment of the present invention is applied will be described. The fourth embodiment of the present invention is different from the third embodiment in that it refers to a response model, determines a response to a conversational sentence input by the user, and acquires text data based on the determined response. Further, the description of the configuration similar to that of the third embodiment will be omitted.

<Content playback system 100>
The content reproduction system 100 refers to the response model, determines a response to the conversational sentence input by the user, and acquires text data based on the determined response. After that, the content reproduction system 100 generates the face image data after the first processing for the input arbitrary face image data, the emotion data, and the text data. The content reproduction system 100 refers to a response model showing a correspondence relationship between a reference conversation sentence acquired in advance and a response to the reference conversation sentence, determines a response to the conversation sentence input by the user, and determines the determined response. Get text data based on.

The response model may be a table in which the response to the conversational sentence input by the user is uniquely determined, for example, as shown in Table 1. In such a case, for example, when the user inputs the conversational sentence "I'm home", the conversational sentence "Welcome back" is determined as a reply. Further, the response model may determine a conversational sentence obtained by translating a Japanese conversational sentence input by the user into English as a response. The response model may also be generated by machine learning. In such a case, the response model is generated by machine learning using a plurality of learning data in which the reference conversation sentence is input and the response to the reference conversation sentence is output.

(Fourth Embodiment: Operation of content reproduction system)
Next, an example of the operation of the content reproduction system 100 in the fourth embodiment will be described. FIG. 15 is a diagram showing an example of the operation of the content reproduction device according to the fourth embodiment.

<Acquisition step S410>
The acquisition step S410 acquires the face image data, the emotion data, and the text data input by the user or the like.

<Response processing step S440>
The response processing step S440 refers to, for example, the response model, and generates the response text data for the text data acquired in the acquisition step S410. This makes it possible to generate text data based on a response suitable for the input text data, and it is possible to express a character with high accuracy according to the conversation of the user. Further, the response processing step S440 may be processed by, for example, the response processing unit 123 included in the processing unit 12.

By performing each of the steps described above, the operation of the content reproduction system 100 in the present embodiment is completed.

(Fifth Embodiment)
Hereinafter, the content reproduction system 100 to which the fifth embodiment of the present invention is applied will be described. The fifth embodiment of the present invention is different from the third embodiment in that voice data for sound quality data, emotion data, and text data is generated, and expression data for processed facial image data and voice data is generated. Further, the description of the configuration similar to that of the third embodiment will be omitted.

<Content playback system 100>
The content reproduction system 100 acquires face image data, emotion data, text data, and voice quality data related to voice quality, and generates voice data indicating the voice of the character based on the text data, emotion data, and sound quality data. .. After that, the content reproduction system 100 generates expression data indicating the expression of the character based on the generated processed face image data and the generated voice data.

Further, the content reproduction system 100 includes a third input data including a reference voice quality data acquired in advance, a reference text data, a reference emotion data, and a third output data including a reference voice data. Using a plurality of voice processing training data as a set of data sets, referring to the voice processing database generated by machine learning, voice data based on the acquired voice quality data, text data, and emotion data. May be generated.

Sound quality data is data indicating sound quality. The sound quality data is, for example, data showing an acoustic feature amount which is a feature of sound resonance. The acoustic feature quantity indicates, for example, fundamental frequency, spectral inclusion, aperiodicity index, spectrogram, voice size, cepstrum, word pronunciation, intonation, sound wave time delay, change in increase / decrease with time of voice, etc. Is. The sound quality data may be input by a user or the like via, for example, the content reproduction device 1.

The voice data is a coded voice. For example, the coding of voice is based on the pulse code modulation (PCM) method, which is represented as a bit string of a length determined by the number of quantization bits, sampling frequency, and time, and the density of voice waves is expressed in 1 bit. Some are based on the pulse density modulation (PDM) method, which samples at regular intervals.

The audio data may be based on audio extracted from moving image data, for example. As the voice data, for example, the data of the voice collected by using a known sound collecting device or the like may be shown, or, for example, a pseudo voice generated by a known technique may be shown.

The expression data is data showing the expression of the character composed of the image including the character and the voice of the character. Expressions include, for example, visual expressions, audio expressions, physical expressions, and the like. The visual expression is an expression that works on the visual sense, and includes gestures and facial expressions. Speech expressions are expressions that work on hearing, and include words, remarks, songs, and the like. Physical expressions are expressions that work on the sense of touch, such as body touch. The representation data may include pseudo data generated in a pseudo manner. Further, the expression data may be a moving image including a character.

The speech processing database is generated by machine learning. As a database for voice processing, for example, a third input data including reference sound quality data, reference emotion data, and reference text data, and a third output data including reference voice data are used as a set of data sets. A trained model for generating the third output data from the third input data generated by machine learning is used by using a plurality of training data for processing. In such a case, the method of generating the voice processing database is different from the first processing database in that the input data is the third input data and the output is the third output data.

As shown in FIG. 16, for example, the voice processing database has a third input data including reference sound quality data, reference emotion data, and reference text data, and a third output data including reference voice data. It is generated by machine learning using a plurality of learning data for voice processing as a set of data sets.

The speech processing database is generated using machine learning modeled on, for example, a neural network. The speech processing database is generated by using machine learning using a neural network such as CNN (Convolution Neural Network) as a model, or any model may be used.

As shown in FIG. 17, for example, the voice processing database stores a third association having a degree of association between the third input data and the third output data. The degree of association indicates the degree of connection between the third input data and the third output data. For example, it can be determined that the higher the degree of association, the stronger the connection of each data. The degree of association may be indicated by three values or more (three stages or more) such as percentage, or may be indicated by two values (two stages).

(Fifth Embodiment: Operation of content reproduction system)
Next, an example of the operation of the content reproduction system 100 in the fifth embodiment will be described. FIG. 18 is a diagram showing an example of the operation of the content reproduction device according to the fifth embodiment.

<Acquisition step S510>
The acquisition step S510 acquires the face image data, the emotion data, the text data, and the sound quality data input by the user or the like.

<Voice processing step S550>
The voice processing step S550 refers to, for example, a voice processing database, and generates voice data for the sound quality data, emotion data, and text data acquired in the acquisition step S310. In the voice processing step S550, for example, the voice processing unit 124 refers to the voice processing database and generates voice data for the sound quality data, the emotion data, and the text data. This makes it possible to generate voice data suitable for the input sound quality data, emotion data, and text data, and it is possible to express a character with high accuracy according to the user's conversation. Further, the voice processing step S550 may be processed by, for example, the voice processing unit 124 included in the processing unit 12.

<Expression generation step S560>
The expression generation step S560 generates expression data based on the generated voice data and the processed face image data. The processed face image data includes, for example, the first post-processed face image data or the second post-processed face image data. Further, the expression generation step S560 may be processed by, for example, the generation unit 13.

By performing each of the steps described above, the operation of the content reproduction system 100 in the present embodiment is completed.

(Sixth Embodiment)
Hereinafter, the content reproduction system 100 to which the sixth embodiment of the present invention is applied will be described. The sixth embodiment of the present invention is different from the first embodiment in that the face image data after the first processing for the face image data and the text data are generated. Further, the description of the configuration similar to that of the first embodiment will be omitted.

<Content playback system 100>
As shown in FIG. 19, the content reproduction system 100 is used to generate the face image data after the first processing for the face image data and the text data generated by referring to the database for the first processing. The content reproduction system 100 refers to, for example, a first processing database generated by machine learning using learning data, and generates face image data and first processed face image data for text data.

The first processing database is generated by machine learning, for example. As the first processing database, for example, the first input data including the reference face image data and the reference text data and the first output data including the reference first processed face image data are combined with a set of data sets. A trained model for generating the first output data from the first input data generated by machine learning is used by using a plurality of the training data for the first processing. In such a case, the method for generating the first processing database is different from the first processing database in the first embodiment in that the input data includes face image data and text data.

As shown in FIG. 20, for example, the first processing database includes first input data including reference face image data and reference text data, and first output data including reference first processed face image data. Is generated by machine learning using a plurality of first processing training data having the above as a set of data sets.

The first processing database is generated using, for example, machine learning using a neural network as a model. The first processing database is generated by using machine learning using a neural network such as CNN (Convolution Neural Network) as a model, or any model may be used.

In the first processing database, for example, as shown in FIG. 21, the first association having a degree of association between the first input data and the first output data is stored. The degree of association indicates the degree of connection between the first input data and the first output data. For example, it can be determined that the higher the degree of association, the stronger the connection of each data. The degree of association may be indicated by three values or more (three stages or more) such as percentage, or may be indicated by two values (two stages).

(Sixth Embodiment: Operation of content reproduction system)
Next, an example of the operation of the content reproduction system 100 in the sixth embodiment will be described. FIG. 22 is a diagram showing an example of the operation of the content reproduction device according to the second embodiment.

<Acquisition step S610>
The acquisition step S610 acquires the face image data and the text data input by the user or the like. In the acquisition step S610, for example, the acquisition unit 11 acquires the face image data and the text data. Further, the acquisition step S610 may acquire text data by acquiring face image data and voice data associated with the face image data as in a moving image and recognizing the acquired voice data by voice. ..

<First processing step S620>
The first processing step S620 refers to, for example, the database for the first processing, and generates the face image data after the first processing for the face image data acquired in the acquisition step S610 and the text data. In the first processing step S620, for example, the first processing unit 121 refers to the first processing database and generates the face image data and the first processed face image data for the text data. This makes it possible to generate face image data after the first processing suitable for the input text data, and it is possible to express a character according to the conversation of the user. Further, the generation unit 13 may generate the face image data after the first processing in a pseudo manner. Further, the first processing step S620 may be processed by, for example, the first processing unit 121 included in the processing unit 12.

By performing each of the steps described above, the operation of the content reproduction system 100 in the present embodiment is completed.

Although embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1: Content playback device 2: Terminal 3: Server 4: Communication network 10: Housing 11: Acquisition unit 12: Processing unit 13: Generation unit 14: Output unit 15: Storage unit 16: DB generation unit 100: Content playback system 101 : CPU
102: ROM
103: RAM
104: Storage unit 105: I / F
106: I / F
107: I / F
108: Input unit 109: Display unit 110: Internal bus 121: First processing unit 122: Second processing unit 123: Response processing unit 124: Voice processing unit S110: Acquisition step S120: First processing step S130: Output step S210: Acquisition step S220: First processing step S230: Output step S240: Second processing step S310: Acquisition step S320: First processing step S330: Output step S410: Acquisition step S420: First processing step S430: Output step S440: Response processing Step S510: Acquisition step S520: First processing step S530: Output step S550: Speech processing step S560: Expression generation step S610: Acquisition step S620: First processing step S630: Output step

Claims (5)

An acquisition step for acquiring face image data including a character's face, emotion data indicating emotions, and text data ,
The first input data including the reference face image data and the reference emotion data acquired in advance includes the face of the same character as the character included in the reference face image data, and is different from the reference face image data. The above-mentioned The first generation step of generating the face image data after the first processing for the face image data and the emotion data acquired by the acquisition step, and
After the second processing, a part of the post-processed face image data is changed based on the post-processed face image data generated by the first generation step and the text data acquired by the acquisition step. Let the computer perform the second generation step to generate the face image data ,
In the second generation step, the second input data including the first post-processed face image data for reference and the text data for reference acquired in advance, and the second output data including the second post-processed face image data for reference are included. With reference to the second processing database generated by machine learning using a plurality of second processing training data having To generate the face image data after the second processing based on the image data and the text data acquired in the acquisition step.
A content playback method characterized by. The acquisition step acquires voice quality data regarding voice quality and obtains voice quality data.
A voice processing step that generates voice data indicating the voice of the character based on the voice quality data, the text data, and the emotion data acquired in the acquisition step.
Further, an expression generation step for generating expression data indicating the expression of the character based on the second processed face image data generated by the second generation step and the voice data generated by the voice processing step. The content reproduction method according to claim 1 , wherein the data is executed by a computer. The voice processing step includes a set of a third input data including a reference voice quality data acquired in advance, a reference text data, a reference emotion data, and a third output data including the reference voice data. Using a plurality of voice processing training data as a data set, referring to the voice processing database generated by machine learning, the voice data acquired in the acquisition step, the text data, and the voice data for the emotion data. The content reproduction method according to claim 2 , wherein the data is generated. The acquisition step refers to a response model showing a correspondence relationship between the reference conversation data acquired in advance and the response data for the reference conversation data, determines the response data for the conversation data input by the user, and determines the response data. The content reproduction method according to any one of claims 1 to 3, wherein the text data based on the determined response data is acquired. An acquisition method for acquiring face image data including a character's face, emotion data indicating emotions, and text data ,
The first input data including the reference face image data and the reference emotion data acquired in advance includes the face of the same character as the character included in the reference face image data, and is different from the reference face image data. The above-mentioned The first generation means for generating the face image data after the first processing for the face image data and the emotion data acquired by the acquisition means , and the first generation means .
After the second processing, a part of the post-processed face image data is changed based on the post-processed face image data generated by the first generation means and the text data acquired by the acquisition means. With a second generation means to generate face image data
Prepare,
The second generation means includes second input data including a pre-acquired first post-processed face image data for reference, text data for reference, and second output data including second post-processed face image data for reference. The first post-processing face generated by the first generation means with reference to the second processing database generated by machine learning using a plurality of second processing training data having To generate the face image data after the second processing based on the image data and the text data acquired by the acquisition means.
A content playback system featuring.

Images