JP2022159632A - Learning method and content reproduction device - Google Patents

Abstract

To provide a learning method capable of representing characters in which multifaceted feelings of a user can be reflected and a content reproduction device.SOLUTION: A learning method includes an input data acquisition step, an output data acquisition step, a first internal representation database generation step, and a second internal representation database generation step. The input data acquisition step acquires user data of a user. The output data acquisition step acquires internal representation data indicating an internal representation of the user. The first internal representation database generation step generates a first internal representation database with the user data as first input data. The second internal representation database generation step generates second internal representation database with user data as second input data and with data of one kind or more included internal representation data and different from a kind of the first output data in the first internal representation database generation step as second output data.SELECTED DRAWING: Figure 14

Description

The present invention relates to a learning method and a content reproduction device.

2. Description of the Related Art In recent years, attention has been focused on a technique for generating a character that is similar in appearance, voice, and tastes to a user, generated by AI (Artificial Intelligence). For example, by turning the user's record into a character, just like taking pictures to record childbirth, Shichigosan, coming-of-age ceremonies, and weddings, the knowledge, skills, and even memories of the time can continue to live digitally. It becomes possible. At the same time, technology for reproducing character expressions to make conversations with characters feel more natural than conversations with users is attracting attention. Are known.

The technology described in Patent Document 1 is a trained neural network that converts video-audio data representing an utterance of a source speaker into video-audio data representing a target speaker who utters an utterance corresponding to the emotion of the source speaker. By storing the structure and converting the video data representing the utterance of the source speaker and the audio data of the source speaker into the video data and the audio data representing the utterance of the target speaker via the neural network, the video processing and the audio processing are performed. This technology relates to a speaker conversion device capable of performing speaker conversion by mutually using video data and audio data without requiring conversion processing by an expert or the like.

Japanese Patent Application Laid-Open No. 2020-91338

Here, in Patent Document 1, training data composed of a pair of audio-video data representing a user's utterance and audio-video data representing a character's utterance corresponding to the user's emotion is input to a neural network structure, and the character's output video and audio data. However, in Patent Document 1, as training data, audiovisual data representing a user's utterance and audiovisual data representing an utterance of a character corresponding to the user's emotion are used. This cannot be reflected in the audio data, resulting in video and audio data of the character that leaves a sense of incongruity compared to when talking with the user. For example, even if the user's emotion indicates anger, if the facial expression is a smile, in Patent Document 1, a smile is selected as the user's emotion. It is impossible to do so, and the video and audio data of the character that leaves a sense of incongruity is created. Therefore, there is a demand for a technique for reproducing character expressions that can reflect the user's multifaceted emotions.

SUMMARY OF THE INVENTION Accordingly, the present invention has been devised in view of the above-described problems, and its object is to provide a learning method and a content reproducing apparatus capable of reproducing character expressions that can reflect the user's multifaceted emotions. to provide.

A learning method according to a first aspect of the invention is a learning method for generating a database used to generate expression data representing an expression of a character, the learning method including text data describing information about a user and an image of the user. an input data acquisition step of acquiring user data including at least one of image data and audio data relating to the user's voice; self-recognition data indicating the user's self-recognition; and the user's priority for events. including two or more types of data among priority order data indicating a ranking, emotional expression data indicating an emotional expression of the user's event, and causal relationship data indicating a presumed causal relationship of the user's event. an output data obtaining step for obtaining internal representation data representing a user's internal representation; and user data obtained by said input data obtaining step as first input data, and at least one type of data included in said internal representation data. Using the first internal representation data as first output data, the first input data and the first output data as a set of first internal representation learning data, and using a plurality of the first internal representation learning data A first internal representation database generation step for generating a first internal representation database by machine learning; Second internal representation data, which is data of a different type from the first output data and is one or more types of data included in the internal representation data, is used as second output data, and the second input data and the second output data are combined. and a second internal representation database generating step of generating a second internal representation database by machine learning using a plurality of the second internal representation learning data as a set of second internal representation learning data. Characterized by

A learning method according to a second invention is, in the first invention, the first internal representation data generated using the first internal representation database, and the second internal representation data generated using the second internal representation database. The method further comprises an expression database generating step of generating an expression database for outputting expression data representing an expression of the character, using the expression data as an input.

A learning method according to a third invention is the first invention or the second invention, wherein the input data acquisition step comprises: the user data including any one or more of the image data including image format data relating to the content of the user's response to the question and the audio format data relating to the content of the user's response to the question; characterized by obtaining

A learning method according to a fourth invention is the learning method according to any one of the first to third inventions, wherein the user data has text feature amount data indicating characteristics of the text data, and the input data acquiring step comprises: It is characterized by including a text feature amount data acquisition step of acquiring the text feature amount data extracted based on the text data.

A learning method according to a fifth aspect is the learning method according to any one of the first to fourth aspects, wherein the user data has image feature amount data indicating features of the image data, and the input data obtaining step includes: It is characterized by including an image feature amount data acquisition step of acquiring the image feature amount data extracted based on the image data.

A learning method according to a sixth invention is the learning method according to any one of the first to fifth inventions, wherein the user data includes voice feature amount data indicating features of the voice data, and the input data obtaining step comprises: It is characterized by including an audio feature amount data acquisition step of acquiring the audio feature amount data extracted based on the audio data.

A content reproduction device according to a seventh aspect of the invention refers to the first internal representation database, the second internal representation database, and the representation database generated by the learning method of the second invention, and reproduces the character. A content reproduction device for outputting expression data, comprising: an acquisition unit for acquiring stimulus data including any one or more of arbitrary text data, image data, and audio data; and the first internal representation. a first internal representation processing unit for obtaining the first internal representation data corresponding to the stimulus data obtained by the obtaining unit by referring to the database for internal representation; and obtaining by the obtaining unit by referring to the second internal representation database. a second internal representation processing unit that acquires the second internal representation data corresponding to the stimulus data obtained; and first internal representation data generated by referring to the representation database and using the first internal representation database. and an expression processing unit that outputs the expression data corresponding to the second internal representation data generated using the second internal representation database.

According to the first to seventh inventions, the first internal representation database generating step generates the first internal representation database by machine learning using a plurality of first internal representation learning data, and the second internal representation database The step of generating a second internal representation database uses the second input data and the second output data as a set of second internal representation learning data, and performs machine learning using a plurality of second internal representation learning data to generate a second internal representation database. to generate Therefore, internal representation data including different types of data can be generated from one user data. This makes it possible to learn the user's emotions in a multifaceted manner. For example, the user's emotion is anger, but the user's facial expression is laughter. This makes it possible to express a character that can reflect the user's multifaceted emotions.

In particular, according to the second invention, an expression database is generated for outputting expression data representing character expressions by using internal representation data as input. As a result, since the expression of the character can be acquired, it is possible to learn the expression of the character in accordance with the multifaceted emotions of the user.

In particular, according to the third invention, the input data acquisition step acquires text data including text format data relating to the content of the user's answer to the question and image format data relating to the content of the user's answer to the question. User data including one or more of image data including data and voice format data regarding the content of the user's answer to the question is acquired. As a result, for example, it is possible to reflect answers to questions about the user's preferences and values in learning, and learning that is more suited to the user's characteristics is possible, so that the user's multifaceted emotions can be learned.

In particular, according to the fourth invention, the input data acquisition step acquires text feature amount data extracted based on the acquired text data. As a result, since it is possible to learn the tendency of the user's disposition from the acquired text data, more accurate learning becomes possible.

In particular, according to the fifth invention, the input data obtaining step obtains the image feature amount data extracted based on the obtained image data. As a result, since it is possible to learn the tendency of the user's disposition from the acquired image data, more accurate learning becomes possible.

In particular, according to the sixth invention, the input data acquisition step acquires speech feature amount data extracted based on the acquired speech data. As a result, since it is possible to learn the tendency of the user's disposition from the acquired voice data, more accurate learning becomes possible.

In particular, according to the seventh invention, the first internal representation database, the second internal representation database, and the representation database are referred to, and the representation data of the character corresponding to the stimulus data is output. As a result, it is possible to output character expression data that reflects the user's internal representation of the stimulus data, so that the user's multifaceted emotions can be reproduced by the character.

FIG. 1 is a schematic diagram showing an example of a content reproduction system according to the first embodiment. FIG. 2 is a schematic diagram showing an example of the operation of the content reproduction system according to the first embodiment. FIG. 3(a) is a schematic diagram showing an example of the learning method for the first internal representation database, and FIG. 3(b) is a schematic diagram showing an example of the learning method for the second internal representation database. FIG. 4(a) is a schematic diagram showing an example of a learning method for a representation database, and FIG. 4(a) is a schematic diagram showing an example of a sound learning model learning method. FIG. 5(a) is a schematic diagram showing an example of the learning method of the visual learning model, and FIG. 5(b) is a schematic diagram showing an example of the learning method of the text learning model. FIG. 6 is a schematic diagram showing an example of the first internal representation database. FIG. 7 is a schematic diagram showing an example of the second internal representation database. FIG. 8 is a schematic diagram showing an example of an expression database. FIG. 9 is a schematic diagram showing an example of a sound learning model. FIG. 10 is a schematic diagram showing an example of a visual learning model. FIG. 11 is a schematic diagram showing an example of a text learning model. FIG. 12(a) is a schematic diagram showing an example of the configuration of the content reproduction device according to the embodiment, and FIG. 12(b) is a schematic diagram showing an example of the functions of the content reproduction device according to the embodiment. (c) is a schematic diagram showing an example of a DB generator. FIG. 13 is a schematic diagram illustrating an example of a processing unit; FIG. 14 is a flow chart showing an example of a learning method according to the embodiment. FIG. 15 is a flow chart showing an example of the operation of the content reproduction system in the embodiment.

An example of a learning method, a content reproduction device, and a content reproduction system according to embodiments to which the present invention is applied will be described below with reference to the drawings.

(First embodiment)
An example of a content reproduction system 100, a content reproduction device 1, and a learning method according to the first embodiment will be described with reference to FIGS. 1 to 5. FIG. FIG. 1 is a schematic diagram showing an example of a content reproduction system 100 according to this embodiment. FIG. 2 is a schematic diagram showing an example of the operation of the content reproduction system 100 according to this embodiment. 3 to 5 are schematic diagrams showing an example of the learning method in this embodiment.

<Content reproduction system 100>
The content reproduction system 100 generates expression data representing an expression of a character in response to stimulus data including any one or more of input arbitrary text data, image data, and audio data. Used. The content reproduction system 100 refers to a database generated by, for example, machine learning using learning data, and generates any one or more of character voice data, image data, and text data for input stimulus data. Generate expression data containing

The content reproduction system 100 includes a content reproduction device 1 as shown in FIG. 1, for example. The content reproduction system 100 may include at least one of the terminal 2 and the server 3, for example. The content reproduction device 1 is connected to the terminal 2 and the server 3 via the communication network 4, for example.

In the content reproduction system 100, for example, as shown in FIG. 2, the content reproduction device 1 acquires stimulus data. For example, the content reproduction device 1 acquires stimulus data. After that, the content reproduction device 1 refers to the sound learning model to obtain sound feature amount data for the sound data included in the stimulus data, and to the visual learning model to obtain image feature amount data for the image data included in the stimulus data. The text learning model is referred to, and the text feature amount data corresponding to the text data included in the stimulus data are obtained. After that, the content reproduction device 1 refers to the first internal representation database, and refers to the first internal representation data corresponding to any one or more of the audio feature amount data, the image feature amount data, and the text feature amount data. Representation data and second internal representation data corresponding to at least one of speech feature amount data, image feature amount data, and text feature amount data with reference to a second internal representation database get. Then, based on the acquired first internal representation data and second internal representation data, the content reproduction device 1 refers to the representation database, and reproduces any one or more of the voice data, image data, and text data of the character. Generate expression data containing As a result, in the content reproduction system 100, by outputting the generated expression data, an optimal stimulus data corresponding to any one or more of input audio data, image data, and text data can be obtained. expression data can be reproduced.

The sound learning model is a model that outputs speech feature amount data from input speech data. A sound learning model may be generated, for example, by machine learning. The sound learning model uses, for example, a set of previously acquired past speech data and speech feature quantity data linked to the past speech data as learning data (speech feature quantity learning data), and a plurality of learning data A trained model constructed by machine learning using may be used.

A visual learning model is a model that outputs image feature amount data from input image data. A visual learning model may be generated, for example, by machine learning. The visual learning model uses, for example, a set of previously acquired past image data and image feature amount data linked to the past image data as learning data (image feature amount learning data), and a plurality of learning data A trained model constructed by machine learning using may be used.

A text learning model is a model that outputs text feature data from input text data. A text learning model may be generated, for example, by machine learning. The text learning model uses, for example, a set of previously acquired past text data and text feature data linked to the past text data as learning data (learning data for text features), and a plurality of learning data A trained model constructed by machine learning using may be used.

The first internal representation database is generated by machine learning. As the first internal representation database, for example, user data is used as input data, and one or more types of data (first internal representation data) included in the internal representation data are used as first output data, first input data and first output data is a set of learning data (first internal representation learning data), and a trained model for generating first output data from first input data is used, which is constructed by machine learning using the learning data. The first output data is first internal representation data used as first internal representation learning data. The first internal representation data includes internal representation data generated using the first internal representation database.

The second internal representation database differs from the first internal representation database in that the second output data is data of a different type from the first output data used in the first internal representation database. The second internal representation database is generated by machine learning. As the second internal representation database, for example, user data is used as second input data, one or more types of data (second internal representation data) contained in the internal representation data are used as second output data, and the second input data and the second A trained model for generating the second output data from the second input data, which is constructed by machine learning using the learning data with the output data as a set of learning data (second internal representation learning data), is used. be done. The second output data is second internal representation data used as second internal representation learning data. The second internal representation data includes internal representation data generated using the second internal representation database.

The expression database outputs expression data based on the input first internal representation data and second internal representation data. The representation database may be generated by machine learning, for example. As an expression database, for example, a pair of previously acquired first internal representation data and second internal representation data is used as third input data, expression data is used as third output data, and the third input data and third output data are As a set of learning data (learning data for expression), a trained model for generating third output data from third input data, constructed by machine learning using learning data, may be used.

The stimulus data is used, for example, in generating internal representational data output by the content reproduction system 100 . Stimulus data includes any one or more of arbitrary text data, image data, and audio data. The stimulus data may be, for example, only image data, or image data and audio data. The stimulus data includes text feature amount data extracted based on arbitrary text data, image feature amount data extracted based on arbitrary image data, and voice feature amount data extracted based on arbitrary voice data. may include any one or more of

The text data is used, for example, when generating internal representation data output by the content reproduction system 100 . Text data is data represented by character codes such as characters. Text data includes, for example, control characters, which are data for controlling devices such as monitors and printers. Control characters include, for example, line feed characters representing line breaks and tabs (horizontal tabs).

The text data includes data posted by a user stored in a server such as an SNS via the communication network 4, or data describing information about the user. Also, the text data may be extracted by recognizing voice data. The text data may be input by a user or the like via the content reproduction device 1 or the like, for example.

Audio data is used, for example, in generating internal representation data output by the content reproduction system 100 . Audio data is encoded audio. For audio coding, for example, one based on pulse code modulation (PCM), which is expressed as a bit string with a length determined by the number of quantization bits, sampling frequency, and time, and one that expresses the density of audio waves with 1 bit. Some are based on a pulse density modulation (PDM) method in which sampling is performed at regular intervals.

The audio data may be based on audio extracted from the video data, for example. The audio data may indicate, for example, data of audio collected using a known sound collection device or the like, or may indicate, for example, pseudo-audio generated by a known technique. The voice data includes data posted by a user stored in a server such as an SNS via the communication network 4, or data describing information about the user. The audio data may be input by a user or the like via the content reproduction device 1 or the like, for example.

The image data is used, for example, when generating internal representation data output by the content reproduction system 100 . Image data is data that includes an aggregate of a plurality of pixels. The image data may be, for example, extracted from a moving image, or may be moving image data.

The image data may be obtained via the communication network 4, for example. The image data may indicate an image captured using a known imaging device or the like, or may indicate a pseudo image generated by a known technique, for example. The image data includes data posted by a user stored in a server such as an SNS via the communication network 4, or data describing information about the user. The image data may be input by a user or the like via the content reproduction device 1 or the like, for example.

The text feature amount data is data indicating features included in the text data. The feature of the text may be, for example, the occurrence tendency of similar words and the meaning of the words, which is calculated based on the meaning of words and sentences obtained by morphological analysis of the text. It may also be a vector, function graph, or the like based on the meaning of words or sentences. Also, the text feature amount data may include meanings of words inferred from the content of the conversation. Also, the text feature amount data may be acquired using a known technique.

The audio feature amount data is data indicating audio features included in the audio data. The feature of speech includes an acoustic feature quantity that is a feature of the reverberation of sound, and a semantic feature quantity that accompanies the linguistic meaning of the speech and is not lost even if the speech is converted into text. Acoustic features include, for example, fundamental frequency, spectral envelope, aperiodicity index, spectrogram, loudness of speech, cepstrum, pronunciation of words, intonation, time delay of sound waves, change in increase or decrease over time of speech, etc. is. The semantic feature quantity indicates the tendency of the uttered word, word usage, and the like. Also, the semantic feature amount may be the same as the text feature amount. Also, the speech feature quantity data may include a semantic feature quantity obtained from the acoustic feature quantity. In this case, the semantic features may include homonyms determined by obtaining the semantic features of words from the accents of the words included in the acoustic features, for example. Also, the voice feature amount data may be acquired using a known technique.

The image feature amount data is data indicating features of an image. The feature of the image may be an imaging target recognized by image recognition, for example. Alternatively, data that appears in common from a plurality of images may be used. For example, if a smile of a user tends to be often seen in moving images of the user, the data that commonly appears may be the data that commonly appears of the user's smile described above. Further, the feature of the image may be, for example, the feature of the movement of the human eye. Also, the image feature amount data may be point cloud data based on an imaging target. The point cloud data characterizes the three-dimensional structure of the object to be imaged, and may be obtained by image analysis by SIFT (Scale-Invariant Feature Transform) or known imaging devices and processing techniques such as a 3D camera. The point cloud data may include, for example, curvature information based on the structure of the imaging target and position information. Curvature information and position information may be acquired by a known imaging device or processing technology. Also, the image feature amount data may be obtained using a known technique.

The user data is data including any one or more of text data describing information about the user, image data including the user's image, and voice data regarding the user's voice. The user data includes text feature amount data extracted based on text data describing information about the user, image feature amount data extracted based on image data describing information about the user, and information about the user. and audio feature amount data extracted based on the described audio data.

Text data describing information about the user is, for example, text data describing personal information such as the user's address and name, text data describing user preferences such as user preferences and memories, or text data describing user preferences such as user preferences and memories. Includes self-written text data. Also, the text data in which the information about the user is described may include data in text format regarding the content of the user's answer to the question about the user.

Image data containing an image of a user is image data containing an image of the user's whole body or a part of the body. Further, the image data including the image of the user may be data in image format regarding the content of the user's answer to the question about the user.

The audio data related to the user's voice is audio data in which the user's voice is recorded. Also, the voice data related to the user's voice may be voice format data related to the content of the user's answer to the user's question.

The internal representation data includes self-recognition data indicating the user's self-recognition, priority data indicating the user's priority for the event, emotional expression data indicating the user's emotional expression for the event, and estimation of the user's causal relationship to the event. and causal relationship data indicating and data including one or more types of data in.

The self-recognition data is data indicating the user's self-recognition. Self-awareness refers to feelings that arise from one's mission, role, or position in society. For example, when a person is in a position of leader in a group, self-awareness is the expression that should be taken as a leader. Self-awareness also includes, for example, feelings for not disturbing the atmosphere in a group. Self-awareness data includes, for example, joy, anger, sorrow, fun, and the like.

The priority data is data indicating the priority of user events. A user's event priority is a sequence of user's priorities for events. For example, if a user prioritizes self-awareness over empathy, it indicates that the user tends to express self-awareness. Examples of priority order data include self-recognition, modality, emotional expression, causal relationship, and the like.

Emotional expression data is data indicating the user's emotional expression for an event. The user's emotional expression to the event indicates how the user feels about the event. Emotional expression data includes, for example, joy, anger, sorrow, and fun.

Causal relationship data is data that indicates an estimated causal relationship to an event. Estimation of causality for an event indicates what kind of event the user associates with the event. For example, it refers to associating the event of traffic congestion with the event of an accident.

The representation data is data representing the representation of the character composed of an image containing the character and the voice of the character. Expressions include, for example, visual expressions, audio expressions, and physical expressions. Visual expressions are expressions that work on the sense of sight, and include gestures, facial expressions, and the like. The phonetic expression is an expression that works on hearing, and includes words, remarks, songs, and the like. Physical expressions are expressions that work on the sense of touch, such as body touch. The expression data may include simulated pseudo data.

It should be noted that the above-mentioned "user" may be an actual person or animal, or may be a simulated person or animal such as an animation.

In addition, the above-mentioned "character" is a person or animal that is pseudo-generated to imitate a user, or a person or animal that is pseudo-generated to imitate a real person or animal, as well as animation etc. Alternatively, a simulated person or animal may be used.

<Learning method>
The learning method according to the present embodiment is used when generating a database or a learning model that is used to generate expression data representing character expressions for input stimulus data. The database includes, for example, a first internal representation database, a second internal representation database, and an expression database. Learning models include, for example, sound learning models, visual learning models, and text learning models.

The learning method generates a first internal representation database, for example, as shown in FIG. 3(a). user data as first input data, one or more kinds of first internal representation data contained in internal representation data as first output data, and first input data and first output data as a set of first internal representation data A first internal representation database for generating first output data from first input data is generated by machine learning using internal representation learning data as learning data.

Also, the learning method generates a second internal representation database, for example, as shown in FIG. 3(b). user data as second input data, one or more types of second internal representation data contained in internal representation data as second output data, and second input data and second output data as a set of second internal representation data A second internal representation database for generating second output data from second input data is generated by machine learning using internal representation learning data as learning data. The learning method for the second internal representation database is different from the learning method for the first internal representation database in that data of a different type from the first output data used in the first internal representation database is used as the second output data. different.

Also, the learning method may generate an expression database, for example, as shown in FIG. 4(a). In the learning method, internal representation data including two or more types of data generated using the first internal representation database and the second internal representation database are input, and representation data representing character representation is output. Expression data for outputting expression data indicating a character's expression by inputting internal representation data including two or more types of data by machine learning using a plurality of expression learning data as a set of expression learning data A database may be generated.

Moreover, the learning method may generate a sound learning model, for example, as shown in FIG. 4(a). In the learning method, a set of past speech data obtained in advance and speech feature quantity data linked to the past speech data are used as speech feature quantity learning data, and a plurality of speech feature quantity learning data are used. A sound learning model may be generated by machine learning using speech data as input and for outputting speech feature amount data.

Moreover, the learning method may generate a visual learning model, for example, as shown in FIG. 5(a). In the learning method, a set of past image data obtained in advance and image feature amount data linked to the past image data are used as image feature amount learning data, and a plurality of image feature amount learning data are used. A visual learning model may be generated by machine learning using image data as input and outputting image feature data.

Also, the learning method may generate a text learning model, for example, as shown in FIG. 5(b). In the learning method, a set of previously obtained past text data and text feature amount data linked to the past text data are used as text feature amount learning data, and a plurality of text feature amount learning data are used. Machine learning may be used to generate a text learning model for inputting text data and outputting text feature amount data.

In the learning method, for example, machine learning using a neural network as a model is used to generate the database described above. The database described above may be generated using machine learning using a neural network such as CNN (Convolution Neural Network) as a model, or any other model may be used.

In the first internal representation database, for example, a first Associations are stored. The degree of association indicates the degree of connection between the first input data and the first output data. For example, it can be determined that the higher the degree of association, the stronger the connection between the data. The degree of association may be indicated by three or more values (three or more levels) such as percentage, or may be indicated by two values (two levels).

For example, the first association is constructed by the degree of connection between many-to-many information (plurality of first input data, vs. pluralities of first output data). The first association is appropriately updated in the process of machine learning, and indicates, for example, a function (classifier) optimized based on a plurality of first input data and a plurality of first output data. In addition, the first association may have, for example, a plurality of degrees of association indicating the degree of connection between each piece of data. The degree of association can correspond to a weight variable, for example when the database is built with neural networks.

Therefore, in the content reproduction system 100, the first output data suitable for the first input data is selected, for example, using the first association based on all the results determined by the classifier. As a result, the first output data suitable for the first input data can be quantitatively selected not only when the first input data is the same as or similar to the first output data, but also when the first output data is dissimilar. can.

The first association may indicate the degree of connection between the plurality of first output data and the plurality of first input data, as shown in FIG. 6, for example. In this case, by using the first association, for each of the plurality of first output data (“first output data A” to “first output data C” in FIG. 6), the plurality of first input data ( In FIG. 6, the degree of relationship between "first output data A" to "first output data C") can be linked and stored. Therefore, for example, a plurality of first input data can be associated with one first output data via the first association. As a result, it is possible to realize a multifaceted selection of the first output data for the first input data.

The first association has, for example, a plurality of degrees of association each linking each first output data and each first input data. The degree of association is indicated, for example, in three or more levels such as percentage, 10 levels, or 5 levels, and is indicated, for example, by line characteristics (such as thickness). For example, the “first output data A” included in the first output data indicates the degree of association AA “73%” with the “first output data A” included in the first input data, and the first input data shows the degree of association AB "12%" with "first output data B" included in . That is, the "relevance degree" indicates the degree of connection between each piece of data. For example, the higher the degree of association, the stronger the connection between each piece of data.

Further, the first internal representation database may be provided with at least one hidden layer between the first input data and the first output data, and may be machine-learned. The degree of association described above is set in either or both of the first input data and the hidden layer data, and this serves as weighting for each data, and output selection is performed based on this. Then, when the degree of association exceeds a certain threshold, the output may be selected.

In the second internal representation database, for example, a second Associations are stored. The degree of association indicates the degree of connection between the second input data and the second output data. For example, it can be determined that the higher the degree of association, the stronger the connection between the data. The degree of association may be indicated by three or more values (three or more levels) such as percentage, or may be indicated by two values (two levels).

For example, the second association is constructed by the degree of connection between many-to-many information (plurality of second input data, vs. pluralities of second output data). The second association is appropriately updated in the process of machine learning, and indicates, for example, a function (classifier) optimized based on a plurality of second input data and a plurality of second output data. Note that the second association may have, for example, a plurality of degrees of association indicating the degree of connection between each piece of data. The degree of association can correspond to a weight variable, for example when the database is built with neural networks.

For this reason, the content reproduction system 100 selects the second output data suitable for the second input data, for example, using the second association based on all the results determined by the classifier. Thereby, the second output data suitable for the second input data can be quantitatively selected not only when the second input data is the same as or similar to the second output data, but also when the second output data is dissimilar. can.

The second association may indicate the degree of connection between the plurality of second output data and the plurality of second input data, as shown in FIG. 7, for example. In this case, by using the second association, for each of the plurality of second output data (“second output data A” to “second output data C” in FIG. 7), the plurality of second input data ( In FIG. 7, the degree of relationship between "second output data A" to "second output data C") can be linked and stored. Therefore, for example, a plurality of second input data can be associated with one second output data via the second association. As a result, it is possible to realize a multifaceted selection of the second output data for the second input data.

The second association has, for example, a plurality of degrees of association each linking each second output data and each second input data. The degree of association is indicated, for example, in three or more levels such as percentage, 10 levels, or 5 levels, and is indicated, for example, by line characteristics (such as thickness). For example, the “second output data A” included in the second output data indicates the degree of association AA “73%” with the “second output data A” included in the second input data, and the second input data shows the degree of association AB "12%" with "second output data B" included in . That is, the "relevance degree" indicates the degree of connection between each piece of data. For example, the higher the degree of association, the stronger the connection between each piece of data.

Also, the second internal representation database may be provided with at least one hidden layer between the second input data and the second output data, and may be machine-learned. The degree of association described above is set in either or both of the second input data and the hidden layer data, and this serves as weighting for each data, and output selection is performed based on this. Then, when the degree of association exceeds a certain threshold, the output may be selected.

The expression database stores, for example, a third association having a degree of association between internal representation data (third input data) and expression data (third output data) including two or more types of data. The degree of association indicates the degree of connection between the third input data and the third output data. For example, it can be determined that the higher the degree of association, the stronger the connection between the data. The degree of association may be indicated by three or more values (three or more levels) such as percentage, or may be indicated by two values (two levels).

For example, the third association is constructed by the degree of connection between many-to-many information (a plurality of third input data, versus a plurality of third output data). The third relevance is appropriately updated in the process of machine learning, and indicates, for example, a function (classifier) optimized based on a plurality of third input data and a plurality of third output data. Note that the third association may have, for example, a plurality of degrees of association indicating the degree of connection between each piece of data. The degree of association can correspond to a weight variable, for example when the database is built with neural networks.

Therefore, in the content reproduction system 100, the third output data suitable for the third input data is selected, for example, using the third relevance based on all the results determined by the classifier. As a result, it is possible to quantitatively select the third output data suitable for the third input data not only when the third input data is the same as or similar to the third output data, but also when the third output data is dissimilar. can.

The third association may indicate the degree of connection between the plurality of third output data and the plurality of third input data, as shown in FIG. 8, for example. In this case, by using the third association, for each of the plurality of third output data (“third output data A” to “third output data C” in FIG. 8), the plurality of third input data ( In FIG. 8, the degree of relationship between "third output data A" to "third output data C") can be linked and stored. For this reason, a plurality of third input data can be associated with one third output data via the third association, for example. As a result, it is possible to realize multifaceted selection of the third output data for the third input data.

The third association has, for example, a plurality of degrees of association that link each third output data and each third input data. The degree of association is indicated, for example, in three or more levels such as percentage, 10 levels, or 5 levels, and is indicated, for example, by line characteristics (such as thickness). For example, the “third output data A” included in the third output data indicates the degree of association AA “73%” with the “third output data A” included in the third input data, and the third input data shows the degree of association AB "12%" with "third output data B" included in . That is, the "relevance degree" indicates the degree of connection between each piece of data. For example, the higher the degree of association, the stronger the connection between each piece of data.

Further, the representation database may be provided with at least one or more hidden layers between the third input data and the third output data for machine learning. The degree of association described above is set in either or both of the third input data and the hidden layer data, and this serves as weighting for each data, and output selection is performed based on this. Then, when the degree of association exceeds a certain threshold, the output may be selected.

The sound learning model stores, for example, speech data as input data, speech feature amount data as output data, and speech association having a degree of association between the input data and the output data. The degree of association indicates the degree of connection between the input data and the output data. For example, it can be determined that the higher the degree of association, the stronger the connection between the data. The degree of association may be indicated by three or more values (three or more levels) such as percentage, or may be indicated by two values (two levels).

For example, speech associations are constructed by the degree of connectivity between many-to-many information (multiple input data versus multiple output data). The phonetic associations are updated accordingly in the course of machine learning and represent optimized functions (classifiers) based on, for example, multiple input data and multiple output data. Note that the audio association may have, for example, a plurality of degrees of association indicating the degree of connection between each piece of data. The degree of association can correspond to a weight variable, for example when the database is built with neural networks.

Therefore, in the content reproduction system 100, output data suitable for input data is selected, for example, using audio association based on all the results determined by the classifier. This makes it possible to quantitatively select output data suitable for input data, not only when the input data is the same as or similar to the output data, but also when they are dissimilar.

The audio association may indicate the degree of connection between multiple output data and multiple input data, as shown in FIG. 9, for example. In this case, by using the speech association, for each of the plurality of output data ("speech feature amount data A" to "speech feature amount data C" in FIG. Data A” to “Voice Data C”) can be linked and stored. For this reason, multiple input data can be associated with one output data, for example, via audio association. This makes it possible to realize multifaceted selection of output data for input data.

The audio association has, for example, a plurality of degrees of association that link each output data and each input data. The degree of association is indicated, for example, in three or more levels such as percentage, 10 levels, or 5 levels, and is indicated, for example, by line characteristics (such as thickness). For example, "speech feature amount data A" included in the output data indicates an association degree AA of "73%" with "speech feature amount data A" included in the input data, and "speech feature amount data A" included in the input data Quantity Data B” and the degree of association AB “12%”. That is, the "relevance degree" indicates the degree of connection between each piece of data. For example, the higher the degree of association, the stronger the connection between each piece of data.

Also, the sound learning model may be machine-learned by providing at least one or more hidden layers between the input data and the output data. The degree of association described above is set in either or both of the input data and the hidden layer data, and this serves as weighting for each data, and output selection is performed based on this. Then, when the degree of association exceeds a certain threshold, the output may be selected.

The visual learning model stores, for example, image data as input data, image feature amount data as output data, and image association having a degree of association between the input data and the output data. The degree of association indicates the degree of connection between the input data and the output data. For example, it can be determined that the higher the degree of association, the stronger the connection between the data. The degree of association may be indicated by three or more values (three or more levels) such as percentage, or may be indicated by two values (two levels).

For example, image association is constructed by the degree of connection between many-to-many information (multiple input data, vs. multiple output data). The image association is updated as appropriate in the course of machine learning, and represents an optimized function (classifier) based on, for example, multiple input data and multiple output data. Note that the image association may have, for example, a plurality of association degrees indicating the degree of connection between each piece of data. The degree of association can correspond to a weight variable, for example when the database is built with neural networks.

For this reason, the content reproduction system 100 selects output data suitable for input data, for example, using image association based on all the results determined by the classifier. This makes it possible to quantitatively select output data suitable for input data, not only when the input data is the same as or similar to the output data, but also when they are dissimilar.

Image association may indicate the degree of connection between a plurality of output data and a plurality of input data, as shown in FIG. 10, for example. In this case, by using image associativity, a plurality of input data ("image The degree of relationship between data A” to “image data C”) can be linked and stored. For this reason, a plurality of pieces of input data can be associated with one piece of output data, for example, via image association. This makes it possible to realize multifaceted selection of output data for input data.

The image association has, for example, a plurality of degrees of association that link each output data and each input data. The degree of association is indicated, for example, in three or more levels such as percentage, 10 levels, or 5 levels, and is indicated, for example, by line characteristics (such as thickness). For example, "image feature amount data A" included in the output data indicates the degree of association AA "73%" between "image feature amount data A" included in the input data, and "image feature amount data A" included in the input data. Quantity Data B” and the degree of association AB “12%”. That is, the "relevance degree" indicates the degree of connection between each piece of data. For example, the higher the degree of association, the stronger the connection between each piece of data.

Also, the visual learning model may be machine-learned by providing at least one or more hidden layers between the input data and the output data. The degree of association described above is set in either or both of the input data and the hidden layer data, and this serves as weighting for each data, and output selection is performed based on this. Then, when the degree of association exceeds a certain threshold, the output may be selected.

The text learning model stores, for example, text data as input data, text feature data as output data, and text relevance having a degree of relevance between the input data and the output data. The degree of association indicates the degree of connection between the input data and the output data. For example, it can be determined that the higher the degree of association, the stronger the connection between the data. The degree of association may be indicated by three or more values (three or more levels) such as percentage, or may be indicated by two values (two levels).

For example, textual associations are constructed by the degree of connectivity between many-to-many information (multiple input data versus multiple output data). Text relevance is appropriately updated in the course of machine learning, and represents an optimized function (classifier) based on, for example, multiple input data and multiple output data. Note that the text relevance may have, for example, multiple degrees of relevance indicating the degree of connection between each piece of data. The degree of association can correspond to a weight variable, for example when the database is built with neural networks.

For this reason, the content reproduction system 100 selects output data suitable for input data, for example, using text relevance based on all the results determined by the classifier. This makes it possible to quantitatively select output data suitable for input data, not only when the input data is the same as or similar to the output data, but also when they are dissimilar.

Text relevance may indicate the degree of connection between a plurality of output data and a plurality of input data, as shown in FIG. 11, for example. In this case, by using text relevance, a plurality of input data ("text Data A” to “Text Data C”) can be linked and stored. For this reason, a plurality of input data can be associated with one output data, for example, via text association. This makes it possible to realize multifaceted selection of output data for input data.

Text relevance has, for example, a plurality of degrees of relevance that link each piece of output data and each piece of input data. The degree of association is indicated, for example, in three or more levels such as percentage, 10 levels, or 5 levels, and is indicated, for example, by line characteristics (such as thickness). For example, the "text feature amount data A" included in the output data indicates the degree of association AA "73%" with the "text feature amount data A" included in the input data, and the "text feature amount data A" included in the input data. Quantity Data B” and the degree of association AB “12%”. That is, the "relevance degree" indicates the degree of connection between each piece of data. For example, the higher the degree of association, the stronger the connection between each piece of data.

Also, the text learning model may be machine-learned by providing at least one or more hidden layers between the input data and the output data. The degree of association described above is set in either or both of the input data and the hidden layer data, and this serves as weighting for each data, and output selection is performed based on this. Then, when the degree of association exceeds a certain threshold, the output may be selected.

<Content playback device 1>
Next, an example of the content reproduction device 1 according to the present embodiment will be described with reference to FIGS. 12 and 13. FIG. FIG. 12(a) is a schematic diagram showing an example of the configuration of the content reproduction apparatus 1 according to this embodiment, and FIG. 12(b) is a schematic diagram showing an example of the functions of the content reproduction apparatus 1 according to this embodiment. be. FIG. 12C is a schematic diagram showing an example of the DB generator 16. As shown in FIG. FIG. 13 is a schematic diagram showing an example of the processing unit 12. As shown in FIG.

As the content reproduction device 1, for example, an electronic device such as a laptop (notebook) PC or desktop PC is used. For example, as shown in FIG. 12A, the content reproduction apparatus 1 includes a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and a storage unit. It has a unit 104 and I/Fs 105-107. Each configuration 101 - 107 is connected by an internal bus 110 .

The CPU 101 controls the content reproduction device 1 as a whole. ROM 102 stores the operation code of CPU 101 . A RAM 103 is a work area used when the CPU 101 operates. The storage unit 104 stores various types of information such as databases and data to be learned. As the storage unit 104, for example, a data storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) is used. Note that, for example, the content reproduction device 1 may have a GPU (Graphics Processing Unit) not shown.

The I/F 105 is an interface for transmitting/receiving various information to/from the terminal 2, the server 3, a website, etc. via the communication network 4 as necessary. The I/F 106 is an interface for transmitting/receiving information to/from the input unit 108 . A keyboard, for example, is used as the input unit 108 , and the user of the content reproduction apparatus 1 inputs various information, control commands for the content reproduction apparatus 1 , etc. via the input unit 108 . The I/F 107 is an interface for transmitting and receiving various information to and from the display unit 109 . The display unit 109 displays various types of information or content stored in the storage unit 104 . A display is used as the display unit 109 , and is provided integrally with the input unit 108 in the case of a touch panel type, for example. A speaker may be used for the display unit 109 .

FIG. 12(b) is a schematic diagram showing an example of the functions of the content reproduction device 1. As shown in FIG. The content reproduction device 1 includes an acquisition unit 11, a processing unit 12, a generation unit 13, an output unit 14, and a storage unit 15, and may have a DB generation unit 16, for example. The DB generation unit 16 has a first internal representation database generation unit 161, a second internal representation database generation unit 162, and a representation database generation unit 163, as shown in FIG. 12(c), for example. The functions shown in FIGS. 12B, 12C, and 13 are implemented by the CPU 101 using the RAM 103 as a work area and executing a program stored in the storage unit 104 or the like. It may be controlled by artificial intelligence or the like.

<<acquisition unit 11>>
Acquisition unit 11 acquires stimulation data. The acquired data is used when generating the expression data described above. Acquisition unit 11 acquires text data, image data, and voice data input from input unit 108, for example, and also acquires text data, image data, and voice data from terminal 2 or the like via communication network 4, for example. good too.

The acquisition unit 11 may acquire, for example, learning data used for generating various databases described above. The acquiring unit 11 acquires learning data input from the input unit 108, for example, and may also acquire learning data from the terminal 2 or the like via the communication network 4, for example.

For example, the first internal representation learning data used to generate the first internal representation database includes past user data and internal representation data. Further, for example, expression data can be given as learning data (learning data for expression) used for generating a database for expression.

<<processing unit 12>>
The processing unit 12 refers to, for example, the sound learning model, the visual learning model, the text learning model, the first internal representation database, the second internal representation database, and the representation database, and acquires representation data corresponding to the stimulus data.

The processing unit 12 has an audio processing unit 121, an image processing unit 122, and a text processing unit 123, which are connected to the acquisition unit 11, as shown in FIG. The processing unit 12 also includes a speech processing unit 121 , an image processing unit 122 , a first internal representation processing unit 124 connected to the text processing unit 123 , and a second internal representation processing unit 125 . The processing unit 12 also has a representation processing unit 126 connected to the first internal representation processing unit 124 and the second internal representation processing unit 125 .

The sound processing unit 121 refers to, for example, a sound learning model, and acquires sound feature amount data corresponding to sound data. The audio processing unit 121 selects, for example, the audio feature amount data having the highest audio correlation with the audio data as the first audio feature amount data. The feature amount data may be selected as the first audio feature amount data. Also, the number of audio feature amount data to be selected can be arbitrarily set.

The image processing unit 122 acquires image feature amount data corresponding to image data by referring to, for example, a visual learning model. The image processing unit 122 selects, for example, image feature amount data with the highest image relevance to image data as first image feature amount data, and also selects, for example, a plurality of images having a degree of relevance equal to or greater than a preset threshold value. The feature amount data may be selected as the first image feature amount data. Also, the number of image feature amount data to be selected can be arbitrarily set.

The text processing unit 123 acquires text feature amount data corresponding to text data, for example, by referring to a text learning model. The text processing unit 123 selects, for example, the text feature amount data with the highest text relevance to the text data as the first text feature amount data. The feature data may be selected as the first text feature data. Also, the number of selected text feature quantity data can be set arbitrarily.

The first internal representation processing unit 124 refers to, for example, the first internal representation database, and converts audio data including audio feature amount data, image data including image feature amount data, and text data including text feature amount data. One or more of the data are input, and one or more types of data (first internal representation data) included in the internal representation data corresponding to the input are obtained. The first internal representation processing unit 124 receives, for example, text data as input data, and acquires output data calculated by referring to the first internal representation database as first internal representation data.

The first internal representation processing unit 124 selects, for example, first internal representation data having the highest degree of first association with text data, and also selects first internal representation data having a degree of association equal to or greater than a preset threshold value, for example. You may choose. Also, the number of selected first internal representation data can be set arbitrarily.

The second internal representation processing unit 125 refers to, for example, the second internal representation database, and converts audio data including audio feature amount data, image data including image feature amount data, and text data including text feature amount data. One or more of the data are input, and one or more types of data (second internal representation data) included in the internal representation data corresponding to the input are obtained. The second internal representation processing unit 125 uses, for example, text data as input data, and acquires output data calculated by referring to the second internal representation database as second internal representation data.

The second internal representation processing unit 125 selects, for example, second internal representation data having the highest degree of second association with text data, and also selects second internal representation data having a degree of association greater than or equal to a preset threshold. You may choose. Also, the number of selected second internal representation data can be set arbitrarily.

The expression processing unit 126 refers to, for example, an expression database, receives the first internal representation data and the second internal representation data, and obtains representation data corresponding to the input. The expression processing unit 126 uses, for example, the self-recognition data included in the first internal representation data and the emotion expression data included in the second internal representation data as input data, and outputs data calculated by referring to the expression database. , is obtained as expression data.

The expression processing unit 126 selects, for example, expression data with the highest third degree of association with self-recognition data and emotional expression data, and also selects expression data with a degree of association equal to or greater than a preset threshold. may Also, the number of selected second internal representation data can be set arbitrarily.

<<generation unit 13>>
The generator 13 generates at least one piece of pseudo data based on the expression data acquired by the processor 12 . The generation unit 13 generates pseudo data including audio and images based on the expression data acquired by the expression processing unit 126, for example. By generating pseudo data, it is possible to output expressions of characters that are not stored in the storage unit 15 . The generator 13 may use a known technique when generating the pseudo data.

<<output unit 14>>
The output unit 14 outputs expression data. The output unit 14 may output the pseudo data generated by the generation unit 13, for example. The output unit 14 outputs expression data to the display unit 109 via the I/F 107, and also outputs expression data to the terminal 2 or the like via the I/F 105, for example.

<<storage unit 15>>
The storage unit 15 retrieves various data such as databases stored in the storage unit 104 as necessary. The storage unit 15 stores various data acquired or generated by each of the components 11 to 14 and 16 in the storage unit 104 as necessary.

<<DB generator 16>>
The DB generation unit 16 generates a database by machine learning using a plurality of learning data. For machine learning, for example, the above-described neural network or the like is used.

The DB generator 16 has, for example, a first internal representation database generator 161 , a second internal representation database generator 162 , and a representation database generator 163 .

The first internal representation database generation unit 161 performs machine learning using a plurality of first internal representation learning data, for example, a pair of user data and first internal representation data as first internal representation learning data. A first internal representation database is generated.

The second internal representation database generation unit 162 uses a pair of user data and second internal representation data as second internal representation learning data, for example, and performs machine learning using a plurality of second internal representation learning data. Generate a second internal representation database.

The representation database generation unit 163 generates a representation database by machine learning using a plurality of representation learning data, for example, using a pair of first internal representation data, second internal representation data, and representation data as learning data for representation. to generate

<Terminal 2>
The terminal 2 is owned by, for example, a user of a service using the content reproduction system 100 and is connected to the content reproduction device 1 via the communication network 4 . Terminal 2 may for example represent an electronic device that generates a database. As the terminal 2, for example, an electronic device such as a personal computer or a tablet terminal is used. The terminal 2 may have, for example, at least part of the functions of the content reproduction device 1 .

<Server 3>
Server 3 is connected to content reproduction device 1 via communication network 4 . The server 3 stores various past data and the like, and various data are transmitted from the content reproducing apparatus 1 as necessary. The server 3 may have, for example, at least part of the functions of the content reproduction device 1, and may perform at least part of the processing instead of the content reproduction device 1, for example. For example, the server 3 stores at least part of various data stored in the storage unit 104 of the content reproduction device 1, and may be used instead of the storage unit 104, for example.

<Communication network 4>
The communication network 4 is, for example, an Internet network or the like to which the content reproduction device 1 is connected via a communication circuit. The communication network 4 may be composed of a so-called optical fiber communication network. Moreover, the communication network 4 may be realized by a known communication technology such as a wireless communication network in addition to the wired communication network.

(Embodiment: learning method)
Next, an example of the learning method in the embodiment will be described. FIG. 14 is a flow chart showing an example of a learning method according to this embodiment.

The learning method includes an acquisition step S110, a first internal representation database generation step S120, a second internal representation database generation step S130, and a representation database generation step S140.

<Acquisition step S110>
An acquisition step S110 acquires user data, first internal representation data, second internal representation data, and representation data. The obtaining step S110 may obtain, as user data, for example, an interview video in which the user is asked a question about the user. Further, in the obtaining step S110, as user data, for example, a sound learning model is referred to, sound feature amount data for sound data included in the user data and visual learning model are referred to, and image feature data for image data included in the user data are referred to. Quantity data and text feature data for text data included in user data may be obtained by referring to a text learning model. In addition, the acquisition step S110 may acquire text feature amount data as user data by a known technique such as principal component analysis, morphological analysis, classification by random forest, or the like, without using a text learning model. Further, the acquisition step S110 may acquire speech feature amount data as user data by a known technique such as MFCC (Mel-Frequency Cepstrum Coefficient) without using a sound learning model. Also, in the acquisition step S110, text feature amount data may be acquired as user data by a known technique such as SIFT (Scale-Invariant Feature Transform) without using a text learning model.

In the obtaining step S110, any one of image data, voice data, and text data posted by the user on a social network service or the like may be obtained as the user data. At the acquisition step S110, for example, the acquisition unit 11 acquires each data described above. The acquisition unit 11 acquires, for example, user data and two or more types of data and expression data included in the internal representation data from the terminal 2 or the like, and may acquire from the storage unit 104 via the storage unit 15, for example. . As the user data, for example, only text data describing information about the user may be acquired. By acquiring , for example, it is possible to learn the user's visual expressions and auditory expressions, so that it is possible to perform more accurate learning.

<First Internal Representation Database Generation Step S120>
Next, a first internal representation database generation step S120 generates a first internal representation database through machine learning using a pair of user data and the first internal representation data as first internal representation learning data. For example, the first internal representation database generation unit 161 generates the first internal representation database by known machine learning. The first internal representation database generation unit 161 stores the generated first internal representation database in the storage unit 104 via the storage unit 15, for example. Note that the generated first internal representation database may be transmitted to the server 3 or another content reproduction device 1, for example. The first internal representation learning data may include a plurality of data (for example, about 1000) of a pair of user data and one or more types of data included in the internal representation data.

<Second Internal Representation Database Generation Step S130>
Next, a second internal representation database generating step S130 generates a second internal representation database by machine learning using the pair of user data and the second internal representation data as second internal representation learning data. For example, the second internal representation database generation unit 162 generates the second internal representation database by known machine learning. The second internal representation database generation unit 162 stores the generated second internal representation database in the storage unit 104 via the storage unit 15, for example. The generated second internal representation database may be transmitted to the server 3 or another content reproduction device 1, for example. The second internal representation learning data may include a plurality of data (for example, about 1000) of a pair of user data and one or more types of data included in the internal representation data. An internal representation showing the user's multifaceted emotions by independently acquiring internal representation data containing different types of data by referring to a first internal representation database and a second internal representation database. Data can be obtained. For example, using the first internal representation database, "laughter" is acquired as self-recognition data included in the internal representation data, and using the second internal representation database, "laughter" is acquired as emotional expression data included in the internal representation data. , and “anger”, it is possible to learn the internal representation data representing the multifaceted emotions of the user.

<Expression database generation step S140>
Next, in an expression database generation step S140, a set of first internal representation data, second internal representation data, and expression data are used as representation learning data, and machine learning is performed using a plurality of representation learning data. generates a representation database. For example, the expression database generation unit 163 generates an expression database by known machine learning. The representation database generation unit 163 stores the generated representation database in the storage unit 104 via the storage unit 15, for example. Note that the generated expression database may be transmitted to the server 3 or another content reproduction device 1, for example. The expression learning data may include a plurality of pairs (for example, about 1000) of pairs of first internal representation data, second internal representation data, and expression data. By using the first internal representation data and the second internal representation data as input data, it is possible to obtain representation data based on multifaceted emotions.

Also, the learning method described above is merely an example, and the timing of learning, the procedure of the steps of learning, and the like may be arbitrary. Also, one or more databases for outputting internal representation data may be generated by a different learning method in that different types of output data are used from the first internal representation database and the second internal representation database. good. This makes it possible to evaluate the user's multifaceted emotions from many angles.

(First Embodiment: Operation of Content Playback System)
Next, an example of the operation of the content reproduction system 100 according to this embodiment will be described. FIG. 15 is a flow chart showing an example of the operation of the content reproduction system 100 according to this embodiment.

<Acquisition Means S210>
Acquisition means S210 acquires stimulation data input by a user or the like. In acquisition means S210, for example, the acquisition unit 11 acquires stimulation data. The acquisition unit 11 may acquire stimulus data from, for example, the terminal 2 or the like, and may also acquire the stimulus data from the storage unit 104 via the storage unit 15, for example. Further, the acquisition means S210 may acquire only arbitrary voice data as stimulus data, for example, but may acquire a plurality of types of data linked to one data. For example, as stimulus data, image data such as moving images and audio data linked to the image data may be acquired.

<Feature amount processing means S220>
The feature amount processing means S220 refers to, for example, a sound learning model, a visual learning model, and a text learning model, and obtains text feature amount data, image feature amount data, and audio feature amount data corresponding to the stimulus data acquired by the acquisition means S210, respectively. get. The feature amount processing means S220 acquires text feature amount data for the acquired text data by referring to, for example, a text learning model. In addition, the feature amount processing means S220 refers to, for example, a sound learning model for audio feature amount data for audio data included in the stimulus data, and a visual learning model for image feature amount data for image data included in the stimulus data. , and text feature amount data corresponding to text data included in the stimulus data by referring to the text learning model. Also, the feature amount processing means S220 may acquire text feature amount data by a known technique such as principal component analysis, morphological analysis, classification by random forest, or the like, without using a text learning model. Also, the feature amount processing means S220 may acquire speech feature amount data by a known technique such as MFCC (Mel-Frequency Cepstrum Coefficient) without using a sound learning model. Also, the feature amount processing means S220 may acquire text feature amount data by a known technique such as SIFT (Scale-Invariant Feature Transform) without using a text learning model.

Further, the feature amount processing unit S220 may store the acquired voice data and feature point data in the storage unit 104 via the storage unit 15, for example. Each acquired data may be transmitted to the server 3 or another content reproduction device 1, for example. The data to be acquired may be pseudo-generated data. As for the data to be acquired, for example, a plurality of data may be acquired for one text data. In addition, by using a plurality of types of data linked to one data as input data, for example, it becomes possible to calculate a feature amount in a composite manner, and more accurate expression data can be acquired.

<Internal Representation Processing Means S230>
The internal representation processing means S230 refers to, for example, the first internal representation database and the second internal representation database, and the text data including the text feature amount data and the image data including the image feature amount data acquired by the feature amount processing means S220. , to acquire first internal representation data and second internal representation data corresponding to speech data including speech feature data. In the internal representation processing means S230, for example, the first internal representation processing unit 124 refers to the first internal representation database, acquires self-recognition data corresponding to the text feature amount data, and the second internal representation processing unit 125 The second internal representation database is referred to, and emotional expression data corresponding to the text feature amount data is acquired. The internal representation processing means S230 may store the acquired internal representation data in the storage unit 104 via the storage unit 15, for example. The acquired internal representation data may be transmitted to the server 3 or another content reproduction device 1, for example. A plurality of pieces of data may be acquired for one piece of input data. In addition, the internal representation processing means S230 outputs internal representation data by a learning method different from the first internal representation database and the second internal representation database in that different types of output data are used. One or more databases may be used to obtain internal representation data containing more types. As a result, the user's emotions can be judged from various angles.

<Expression processing means S240>
The representation processing means S240 refers to, for example, a representation database, receives the first internal representation data and the second internal representation data acquired by the internal representation processing means S230, and acquires representation data corresponding to the input. In the expression processing means S240, the expression processing unit 126 uses, for example, the self-recognition data included in the first internal representation data and the emotional expression data included in the second internal representation data as input data, and refers to the expression database. The calculated output data is obtained as expression data. For example, "laughter" is input as self-recognition data included in the first internal representation data, "anger" is input as emotional expression data included in the second internal representation data, and "self-recognition" is input as priority data. In this case, it is possible to acquire expression data based on multifaceted emotions, such as a person who is feeling angry inwardly, but is laughing while prioritizing the atmosphere of the place.

The expression processing means S240 may store the acquired expression data in the storage unit 104 via the storage unit 15, for example. Note that the acquired representation data may be transmitted to the server 3 or another content reproduction device 1, for example. The data to be acquired may be pseudo data. A plurality of pieces of data may be acquired for one piece of input data.

<Output Means S250>
In the output unit S250, for example, the output unit 14 outputs the expression data acquired by the expression processing unit S240 to the display unit 109, the terminal 2, or the like.

By performing each means described above, the operation of the content reproduction system 100 in this embodiment is completed.

While embodiments of the invention have been described, the embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

Reference Signs List 1 : Content reproduction device 2 : Terminal 3 : Server 4 : Communication network 10 : Case 11 : Acquisition unit 12 : Processing unit 13 : Generation unit 14 : Output unit 15 : Storage unit 16 : DB generation unit 100 : Content reproduction system 101 : CPU
102: ROM
103: RAM
104: Storage unit 105: I/F
106: I/F
107: I/F
108: Input unit 109: Display unit 110: Internal bus 121: Audio processing unit 122: Image processing unit 123: Text processing unit 124: First internal representation processing unit 125: Second internal representation processing unit 126: Expression processing unit 161: First internal representation database generation unit 162: Second internal representation database generation unit 163: Expression database generation unit S110: Acquisition step S120: First internal representation database generation step S130: Second internal representation database generation step S140 : Expression database generation step S210 : Acquisition means S220 : Feature amount processing means S230 : Internal representation processing means S240 : Expression processing means S250 : Output means

A learning method according to a first aspect of the invention is a learning method for generating a database used to generate expression data representing an expression of a character, the learning method including text data describing information about a user and an image of the user. an input data acquisition step of acquiring user data including at least one of image data and audio data relating to the user's voice; self-recognition data indicating the user's self-recognition; and the user's priority for events. including two or more types of data among priority order data indicating a ranking, emotional expression data indicating an emotional expression of the user's event, and causal relationship data indicating a presumed causal relationship of the user's event. an output data obtaining step for obtaining internal representation data representing a user's internal representation; and user data obtained by said input data obtaining step as first input data, and at least one type of data included in said internal representation data. Using the first internal representation data as first output data, the first input data and the first output data as a set of first internal representation learning data, and using a plurality of the first internal representation learning data A first internal representation database generation step for generating a first internal representation database by machine learning; Second internal representation data, which is data of a different type from the first output data and is one or more types of data included in the internal representation data, is used as second output data, and the second input data and the second output data are combined. a second internal representation database generating step of generating a second internal representation database by machine learning using a plurality of the second internal representation learning data as a set of second internal representation learning data; It is characterized by

A learning method according to a second invention is, in the first invention, the first internal representation data generated using the first internal representation database, and the second internal representation data generated using the second internal representation database. The method further causes the computer to execute an expression database generation step of generating an expression database for outputting expression data representing the expression of the character, using the expression data as input.

Claims (7)

A learning method for generating a database used to generate expression data indicating a character's expression,
an input data acquisition step of acquiring user data including at least one of text data describing information about a user, image data including an image of the user, and audio data relating to the voice of the user;
self-recognition data indicating the user's self-recognition; priority data indicating the priority of the user's event; emotional expression data indicating the user's emotional expression regarding the event; an output data acquisition step of acquiring internal representation data representing the user's internal representation, including two or more types of data from:
Using the user data acquired by the input data acquiring step as first input data, and the first internal representation data, which is one or more types of data contained in the internal representation data, as first output data, the first input data and the generating a first internal representation database by machine learning using a plurality of the first internal representation learning data, with the first output data as a set of first internal representation learning data; a step;
The user data acquired by the input data acquisition step is used as second input data, the data is of a different type from the first output data in the first internal representation database generation step, and is included in the internal representation data. A plurality of the second internal representation learning data, wherein the second internal representation data as data is used as second output data, the second input data and the second output data are used as a set of second internal representation learning data, and and a second internal representation database generating step of generating a second internal representation database by machine learning using . An expression indicating the expression of the character, with input of first internal representation data generated using the first internal representation database and second internal representation data generated using the second internal representation database. 2. The learning method according to claim 1, further comprising an expression database generating step of generating an expression database for outputting data. The input data obtaining step includes: the text data including data in text format regarding the content of the user's response to the question; and the image data including data in image format regarding the content of the user's response to the question. 3. The learning method according to claim 1 or 2, wherein the user data including at least one of the following: and audio format data regarding the content of the user's answer to the question. the user data has text feature amount data indicating features of the text data;
4. The method according to any one of claims 1 to 3, wherein said input data acquisition step includes a text feature amount data acquisition step of acquiring said text feature amount data extracted based on said acquired text data. learning method. The user data has image feature amount data indicating features of the image data,
5. The method according to any one of claims 1 to 4, wherein said input data acquisition step includes an image feature amount data acquisition step of acquiring said image feature amount data extracted based on said acquired image data. learning method. The user data has audio feature amount data indicating features of the audio data,
6. The method according to any one of claims 1 to 5, wherein said input data acquisition step includes a voice feature amount data acquisition step of acquiring said voice feature amount data extracted based on said acquired voice data. learning method. 3. A content reproduction device that refers to the first internal representation database, the second internal representation database, and the representation database generated by the learning method according to claim 2, and outputs representation data of the character. and
an acquisition unit that acquires stimulus data including any one or more of arbitrary text data, image data, and audio data;
a first internal representation processing unit that refers to the first internal representation database and acquires the first internal representation data corresponding to the stimulus data acquired by the acquisition unit;
a second internal representation processing unit that refers to the second internal representation database and acquires the second internal representation data corresponding to the stimulus data acquired by the acquisition unit;
Referring to the expression database, corresponding to first internal representation data generated using the first internal representation database and second internal representation data generated using the second internal representation database and an expression processing unit that outputs the expression data.

Images