JP7476091B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Technologies are provided that estimate emotions from a person's facial expressions, etc. For example, there is provided a technology that estimates emotions from a person's facial expressions using image data of the target person's face captured by an imaging device (e.g., Patent Document 1, etc.). There is also provided a technology that reads a person's emotions using the electromyogram of the person's face, which is a type of biosignal, rather than image data (e.g., Non-Patent Document 1, etc.). There is also provided a technology that estimates a person's emotions using data from non-biological sensors such as acceleration and gyroscopes in addition to image data and biosignals (e.g., Non-Patent Document 2, etc.).

JP 2020-149361 A

"A wearable interface for reading,facial expressions based on bioelectrical signals", Anna GRUEBLER, and Kenji SUZUKI, <Internet> https://www.keer.org/keer2010/Papers/2310.pdf (Retrieved November 19, 2020) “Recognition of human affection in Smartphone perspective based on accelerometer and user's sitting position”, Rasam Bin Hossain and Ben Taskar＜Internet＞https://www.researchgate.net/profile/Hasan_Mahmud3/publication/277004963_Recognition_of_human_affection_in_Smartphone_perspective_based_on_accelerometer_and_user%27s_sitting_position/links/5a9e2874aca272cd09c22cd3/Recognition-of-human-affection-in-Smartphone-perspective-based-on-accelerometer-and-users-sitting-position.pdf（Retrieved December 17, 2020）

However, for example, in Patent Document 1, images such as a facial image of the target person in a normal state are used, making it difficult to capture small changes that do not appear in the image. Also, for example, in Non-Patent Document 1, it is difficult to associate which emotion with which electromyogram state. Therefore, the above conventional technology has room for improvement in terms of estimating emotions.

The present application has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that enable appropriate estimation of a user's emotions.

The information processing device according to the present application is characterized by comprising: an acquisition unit that acquires learning data including label information indicating the psychological effect that content output by a terminal device worn by a user on the head has on the user, and input data that is the user's biosignal detected by the terminal device when the terminal device outputs the content; and a learning unit that uses the learning data to learn an estimation model that outputs information that estimates the emotion of a person corresponding to the input data in response to the input of the input data.

According to one aspect of the embodiment, it is possible to achieve the effect of enabling appropriate estimation of a user's emotions.

FIG. 1 is a diagram illustrating an example of processing by an information processing system according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of a learning data storage unit according to the embodiment. FIG. 4 is a diagram illustrating an example of a model information storage unit according to the embodiment. FIG. 5 is a diagram illustrating an example of a content information storage unit according to the embodiment. FIG. 6 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. FIG. 7 is a flowchart illustrating an example of processing by the information processing device according to the embodiment. FIG. 8 is a diagram illustrating an example of a hardware configuration.

Below, the information processing device, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. In addition, the same parts in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[1. Information Processing]
An example of information processing according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of processing by an information processing system according to the embodiment. In Fig. 1, an information processing device 100 learns an estimation model that estimates human emotions using label information indicating emotions evoked in a user by content output by a terminal device 10 worn by the user on the head and a biosignal of the user detected by the terminal device 10 when the content is output by the terminal device 10. Then, the information processing device 100 estimates the user's emotions using the estimation model.

In the example of FIG. 1, a head-mounted display (hereinafter simply referred to as a "head-mounted display") that is integrated with a headset and capable of displaying content and outputting audio is shown as an example of a terminal device 10 that a user wears on his/her head. Note that the terminal device 10 that a user wears on his/her head is not limited to a head-mounted display. For example, the terminal device 10 may be a head-mounted display alone, over-ear headphones, a headset-integrated VR (Virtual Reality) goggle, a smartphone-mounted VR goggle, a wearable device integrated with glasses, earphones, or the like. In addition, FIG. 1 shows the myoelectric potential of the user's face as an example of a user's biosignal, but the user's biosignal is not limited to the myoelectric potential of the face and may be the myoelectric potential of various physical parts of the head. In addition, the user's biosignal may be a biosignal of the user other than the myoelectric potential, which will be described later.

First, the configuration of the information processing system 1 will be described. As shown in FIG. 1, the information processing system 1 includes multiple terminal devices 10 and an information processing device 100. Each terminal device 10 and the information processing device 100 are connected to each other via a predetermined communication network (not shown) so as to be able to communicate with each other via a wired or wireless connection. Note that the information processing system 1 shown in FIG. 1 may include multiple information processing devices 100.

The information processing device 100 is a computer that learns an estimation model that estimates human emotions using learning data including label information (also simply called "label") indicating emotions associated with content and the user's biosignal at the time of outputting the content. The information processing device 100 also estimates the user's emotions using the estimation model that estimates human emotions. The information processing device 100 also provides content to the terminal device 10. Note that if the information processing system 1 includes a content providing device that provides content to the terminal device 10, the information processing device 100 does not need to provide content to the terminal device 10. In this case, the information processing device 100 may provide information indicating the estimated user's emotions to the content providing device.

The terminal device 10 is a device (computer) that is worn on the user's head and used by the user. The terminal device 10 outputs content. The terminal device 10 receives content from the information processing device 100 and outputs the received content. For example, the terminal device 10 displays the content and outputs audio. The terminal device 10 also accepts operations by the user.

The terminal device 10 also has a sensor (myogenic potential sensor) that detects myoelectric potential. In FIG. 1, the terminal device 10 has a myoelectric potential sensor that detects myoelectric potential of the face, and detects the myoelectric potential of the user's face as the user's biosignal. The terminal device 10 may detect and acquire a biosignal of the user other than myoelectric potential. For example, the terminal device 10 may have an electrooculography sensor that detects the user's electrooculography and detect the user's electrooculography. The terminal device 10 may also have a heart rate sensor or the like, not limited to a sensor that detects potential, but will be described later. For example, a user who uses the terminal device 10 may wear a wearable device that can communicate with the terminal device 10, thereby enabling the terminal device 10 to acquire the user's biosignal. For example, a user who uses the terminal device 10 may wear a wristband-type wearable device that can communicate with the terminal device 10, thereby enabling the terminal device 10 to acquire information about the user's own heart rate (pulse). Note that the above is just one example, and the terminal device 10 may have sensors that detect various information.

In addition, in the following, the terminal device 10 may be referred to as the user. In other words, in the following, the user may be read as the terminal device 10. In the example of FIG. 1, the terminal device 10 is a head-mounted display.

In the following, a user identified by user ID "U1" may be referred to as "user U1". In this way, when "user U* (* is an arbitrary numerical value)" is written below, it indicates that the user is identified by user ID "U*". For example, when "user U2" is written, the user is identified by user ID "U2". In addition, in the example shown in FIG. 1, the terminal device 10 will be described as terminal device 10-1 or 10-2 depending on the user who uses the terminal device 10. For example, the terminal device 10-1 is a terminal device 10 used by a user (user U1) identified by user ID "U1". In addition, for example, the terminal device 10-2 is a terminal device 10 used by a user (user U2) identified by user ID "U2". In addition, in the following, when the terminal devices 10-1 and 10-2 are described without any particular distinction, they will be described as terminal device 10.

An example of information processing will be described below with reference to FIG. 1. First, the learning process of the estimation model by the information processing device 100 will be described, and then the emotion estimation process using the estimation model by the information processing device 100 will be described.

First, the information processing device 100 provides content to the terminal device 10-1 used by the user U1 (step S11). The information processing device 100 determines the content to be provided to the user U1 from a list of content associated with a label indicating the psychological effect it has on the user. The information processing device 100 may randomly determine the content to be provided to the user U1 from the list of content, or may determine the content to be provided to the user U1 based on the user attributes such as the age, gender, and interests of the user U1. Note that the above is merely an example, and the information processing device 100 may determine the content to be provided to the user U1 by appropriately using various information.

For example, the information processing device 100 may determine the content to be provided to the user U1 according to the type of content that the terminal device 10-1 can play. In this case, if the terminal device 10-1 is capable of outputting video, the information processing device 100 may determine the content including only video as the content to be provided to the user U1. Furthermore, if the terminal device 10-1 is capable of outputting audio only, the information processing device 100 may determine the content including audio as the content to be provided to the user U1. Furthermore, if the terminal device 10-1 is capable of outputting video and audio, the information processing device 100 may determine the content including video and audio as the content to be provided to the user U1. For example, if the terminal device 10-1 is a device that can output only audio (such as over-ear headphones), the information processing device 100 determines the content including audio as the content to be provided to the user U1.

Then, the information processing device 100 transmits the determined content to the terminal device 10-1 used by the user U1. In FIG. 1, the information processing device 100 transmits content CT1 associated with a label LB1 indicating that the psychological effect on the user is an emotion of joy to the terminal device 10-1 used by the user U1.

The terminal device 10-1 that has received the content CT1 from the information processing device 100 outputs the content CT1 (step S12). For example, the terminal device 10-1 displays the content CT1 including video and audio, and performs audio output. If the terminal device 10-1 is a device that only performs audio output, such as over-ear headphones, the terminal device 10-1 may only perform audio output of the content CT1. The terminal device 10-1 outputs the content CT1 and detects the biosignal of the user U1, which is the myoelectric potential of the user U1's face when the content CT1 is output. As a result, the terminal device 10-1 collects the biosignal of the user U1, which is the myoelectric potential of the user U1's face when the content CT1 is output. Then, the terminal device 10-1 that has collected the biosignal of the user U1 transmits the collected biosignal of the user U1 (biological signal data DT1) to the information processing device 100 (step S13).

The information processing device 100 associates the biosignal data acquired from the terminal device 10 with a label indicating the content and the emotion evoked by the content, and adds it as learning data (step S14). The information processing device 100 acquires the biosignal (biosignal data DT1) of the user U1 from the terminal device 10-1, associates the acquired biosignal of the user U1 with the classification label of the content CT1, and adds it to the dataset DS1. For example, the information processing device 100 associates the biosignal data DT1 with the content CT1 and a label LB1 indicating the emotion of joy, and adds it to the dataset DS1 as learning data.

The information processing device 100 also provides content to other users, associates the biosignal data of the users at the time of outputting the provided content with the classification label of the provided content, and adds the data to the dataset DS1 as learning data. As a result, the information processing device 100 generates a dataset including multiple learning data as shown in the dataset DS1 in FIG. 1. Note that the information processing device 100 may obtain the dataset DS1 from an external device. That is, the learning data may be collected by an external device other than the information processing device 100, and the information processing device 100 may use the dataset obtained from the external device to learn the estimation model.

Then, the information processing device 100 uses the dataset DS1 to train an estimation model M1, which is an emotion estimation model that takes a user's biosignal as input and outputs information classifying the user's emotion (step S15). For example, the information processing device 100 trains an estimation model M1 that outputs a score for each emotion indicating the likelihood of the user's emotion in response to an input of the user's biosignal, which is the myoelectric potential of the user's face. In this case, the information processing device 100 estimates that the emotion output by the estimation model M1 with the highest score is the emotion of the user. For example, the information processing device 100 trains an estimation model M1 having a neural network network configuration. Note that the network configuration of the estimation model M1 may be any network configuration as long as it is capable of outputting information used for emotion estimation.

For example, when biosignal data DT1 is input, the information processing device 100 learns parameters such as weights of the estimation model M1 so that the score corresponding to the emotion "joy" corresponding to label LB1 is maximized. Also, for example, when biosignal data DT2 is input, the information processing device 100 learns parameters such as weights of the estimation model M1 so that the score corresponding to the emotion "anger" corresponding to label LB2 is maximized. The information processing device 100 generates model data MDT1 including the network configuration of the estimation model M1, the learned parameters, and the like. As a result, the information processing device 100 generates the estimation model M1, which is an emotion estimation model. Any method can be adopted for the learning process of the estimation model M1.

For example, the information processing device 100 performs a learning process using a method such as backpropagation. For example, the information processing device 100 adjusts the value of the weight (i.e., the connection coefficient) that is taken into account when a value is transmitted between nodes through the learning process. In this way, the information processing device 100 learns the estimation model M1 through a process such as backpropagation that corrects the parameters (connection coefficients) so as to reduce the error between the output in the estimation model M1 and the correct answer (correct answer data) corresponding to the input. For example, the information processing device 100 generates the estimation model M1 by performing a process such as backpropagation to minimize a predetermined loss function. This allows the information processing device 100 to perform a learning process that learns the parameters of the estimation model M1. The information processing device 100 stores the estimation model M1 generated through the learning process in the model information storage unit 122 (see FIG. 4).

Next, we will explain the emotion estimation process using the estimation model by the information processing device 100. Figure 1 shows a case where the estimation process is performed on a user U2 who uses the terminal device 10-2.

The terminal device 10-2 detects the biosignal of the user U2, which is the myoelectric potential of the user U2's face, and collects the biosignal of the user U2, which is the myoelectric potential of the user U2's face. The terminal device 10-2 that has collected the biosignal of the user U2 then transmits the collected biosignal of the user U2 to the information processing device 100 (step S21).

The information processing device 100 receives the biosignal of user U2 from the terminal device 10-2 and inputs the received biosignal of user U2 as input data IN to the estimation model M1 (step S22). The information processing device 100 acquires the estimation model M1 from the model information storage unit 122 and inputs the received biosignal of user U2 to the acquired estimation model M1 as input data IN. As a result, the estimation model M1 to which the biosignal of user U2 has been input outputs emotion estimation information of user U2 as output data OT (step S23). For example, the estimation model M1 outputs the score of each emotion corresponding to the biosignal of user U2. In FIG. 1, the estimation model M1 outputs the score of each emotion corresponding to the emotion "anger" corresponding to label LB2 in response to the input of the biosignal of user U2, with the score being the maximum.

Then, the information processing device 100 estimates the emotion of the user U2 based on the output data OT (step S24). In FIG. 1, the information processing device 100 estimates that the emotion of the user U2 is anger, as shown in the estimation result RS, because the score corresponding to the emotion "anger" corresponding to the label LB2 is the largest. Then, the information processing device 100 provides information to the user U2 according to the estimation result (step S25). For example, since the emotion of the user U2 is anger, the information processing device 100 transmits notification information urging the user U2 to calm down the emotion of anger. Furthermore, since the emotion of the user U2 is anger, the information processing device 100 may provide content for calming down the emotion of anger, which will be described later.

As described above, the information processing device 100 can generate a model that appropriately learns the correspondence between the user's emotions and the biosignals by learning an estimation model that estimates a person's emotions using a label indicating the psychological effect that the content has on the user and the user's biosignal at the time the content is output. Therefore, the information processing device 100 can appropriately estimate the user's emotions by estimating the user's emotions using the estimation model. By generating the estimation model as described above, the information processing system 1 can estimate the emotions of the user from the content such as the image displayed and the sound output even in a terminal device (head-mounted display, headphones, etc.) that does not acquire myoelectric potential or electrooculography. Furthermore, the information processing system 1 can estimate the emotions of the user using the estimation model, even for users who do not watch the content. That is, the information processing system 1 can estimate the emotions of the user for all users from whom data to be input to the estimation model can be acquired. Furthermore, the information processing device 1 may generate an estimation model that inputs various types of data, not limited to biosignals. For example, the information processing device 100 may generate an estimation model using sensor information detected by various sensors included in the sensor unit 16. In this case, the information processing device 100 generates an estimation model that receives as input sensor information detected by various sensors included in the sensor unit 16. For example, the information processing device 100 may generate an estimation model that receives as input other types of data in addition to biosignals. For example, the information processing device 100 may generate an estimation model that receives as input image data. Furthermore, the information processing device 100 may generate an estimation model that receives as input data from non-biometric sensors such as acceleration and gyro. In other words, the information processing device 100 may use various information, not limited to biosignals, as long as the information can be used to estimate the user's emotions.

[1-1. Biosignals]
In the above example, a case where facial myoelectric potential is used as a biosignal of the user has been described as an example, but the biosignal of the user is not limited to facial myoelectric potential. For example, the information processing device 100 may use the user's electrooculography as the biosignal of the user. In this case, the information processing device 100 may learn an estimation model including the user's electrooculography in input data. Then, the information processing device 100 may estimate the user's emotion by inputting the input data including the user's electrooculography to the estimation model.

The information processing device 100 may also use the user's heart rate as the user's biosignal. In this case, the information processing device 100 may learn an estimation model that includes the user's heart rate in the input data. Then, the information processing device 100 may estimate the user's emotions by inputting the input data that includes the user's heart rate into the estimation model.

The information processing device 100 may also use the user's brain waves as the user's biosignal. In this case, the information processing device 100 may learn an estimation model that includes the user's brain waves in the input data. Then, the information processing device 100 may estimate the user's emotions by inputting the input data that includes the user's brain waves into the estimation model.

The information processing device 100 may also use the user's pulse as the user's biosignal. In this case, the information processing device 100 may learn an estimation model that includes the user's pulse in the input data. Then, the information processing device 100 may estimate the user's emotions by inputting the input data that includes the user's pulse into the estimation model.

The information processing device 100 may also use the user's breathing as the user's biosignal. In this case, the information processing device 100 may learn an estimation model that includes the user's breathing in the input data. Then, the information processing device 100 may estimate the user's emotions by inputting the input data that includes the user's breathing into the estimation model.

The information processing device 100 may also use the user's sweating as the user's biosignal. In this case, the information processing device 100 may learn an estimation model that includes the user's sweating in the input data. Then, the information processing device 100 may estimate the user's emotions by inputting the input data that includes the user's sweating into the estimation model.

Note that the above is merely an example, and the information processing device 100 may use various types of biosignals of the user. Furthermore, the information processing device 100 may use a combination of the various types of biosignals described above. For example, the information processing device 100 may perform processing using an estimation model in which two types of biosignals, the myoelectric potential of the user's face and the user's breathing, are used as input data. For example, the information processing device 100 may learn an estimation model in which two types of biosignals, the myoelectric potential of the user's face and the user's breathing, are used as input data, and perform estimation processing using the estimation model.

[1-2. Emotions
In the example of FIG. 1, only four labels (emotions) "joy", "anger", "sadness", and "pleasure" are shown as examples of labels (emotions), but the label information is limited to these four. The classification of emotions is not limited to the above, and various classifications may be used depending on the purpose, etc. For example, labels indicating various emotions such as "neutral" (see FIG. 5) may be used. In addition, labels (emotions) can be more abstract, such as "negative emotions" that combine negative emotions such as anger and sadness, or "positive emotions" that combine positive emotions such as joy and pleasure. may be also possible.

2. Configuration of information processing device
Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 2, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

(Communication unit 110)
The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a predetermined communication network by wire or wirelessly, and transmits and receives information to and from the terminal device 10.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 2 , the storage unit 120 according to the embodiment has a learning data storage unit 121, a model information storage unit 122, and a content information storage unit 123.

(Learning Data Storage Unit 121)
The learning data storage unit 121 according to the embodiment stores various information related to data used for learning. The learning data storage unit 121 stores learning data (dataset) used for learning. FIG. 3 is a diagram illustrating an example of the learning data storage unit according to the embodiment of the present disclosure. For example, the learning data storage unit 121 stores various information related to various data such as learning data used for learning and evaluation data used for accuracy evaluation (measurement). FIG. 3 illustrates an example of the learning data storage unit 121 according to the embodiment. In the example of FIG. 3, the learning data storage unit 121 includes items such as "dataset ID", "data ID", "biological signal data", "content ID", and "classification label".

"Dataset ID" indicates identification information for identifying a dataset. "Data ID" indicates identification information for identifying each piece of learning data. Furthermore, "biological signal data" indicates data (biological signal data) identified by a data ID. "Biological signal data" indicates biological signal data (input data) used as input for a model. "Biological signal data" may be associated with information indicating which user's biological signal it is (e.g., a user ID).

"Content ID" indicates the content output when the corresponding biosignal data was detected. "Classification label" indicates label information (correct answer information) associated with the corresponding content. "Classification label" indicates the classification result of the user's emotion corresponding to the corresponding data (input data). In the example of Figure 3, "classification label" indicates a classification for estimating the user's emotion corresponding to the biosignal data that is the input data. "Classification label" indicates information (classification label) that indicates the user's emotion that is expected to be output by the model when the corresponding data (input data) is input to the model.

The example in Figure 3 shows that the dataset (dataset DS1) identified by the dataset ID "DS1" includes multiple data identified by data IDs "DID1", "DID2", "DID3", etc.

The biosignal data DT1 identified by the data ID "DID1" indicates the user's biosignal detected when the content (content CT1) identified by the content ID "CT1" is output. The biosignal data DT1 also indicates that the classification label is "LB1." Although illustrated abstractly in FIG. 3, the label LB1 is assumed to be a label (value) corresponding to the emotion of "joy."

The learning data storage unit 121 may store various information according to the purpose, not limited to the above. For example, the learning data storage unit 121 stores information indicating the type of teacher data, such as audio or image, in association with each piece of data. For example, the learning data storage unit 121 stores information indicating the type of data in association with each piece of data.

For example, the learning data storage unit 121 may store each piece of biosignal data in a manner that allows identification as to whether it is learning data or evaluation data. For example, the learning data storage unit 121 stores learning data and evaluation data in a manner that allows distinction between them. The learning data storage unit 121 may store information that identifies whether each piece of data is learning data or evaluation data. The information processing device 100 learns a model based on each piece of biosignal data used as learning data and the classification label. The information processing device 100 measures the accuracy of the model based on each piece of biosignal data used as evaluation data and the classification label. The information processing device 100 measures the accuracy of the model by collecting the results of a comparison between the output result (output information that estimates emotions) output by the model when evaluation data is input and the classification label.

(Model information storage unit 122)
The model information storage unit 122 according to the embodiment stores information related to a model. For example, the model information storage unit 122 stores information (model data) of a trained model (model) trained (generated) by a learning process. FIG. 4 is a diagram illustrating an example of a model information storage unit according to the first embodiment of the present disclosure. FIG. 4 illustrates an example of the model information storage unit 122 according to the first embodiment. In the example illustrated in FIG. 4, the model information storage unit 122 includes items such as "model ID", "purpose", and "model data".

"Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates the data of the model. Figure 4 etc. shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various information that constitutes the model is included, such as information on the model configuration (network configuration) and information on parameters. For example, "model data" includes information including the nodes in each layer of the network, the functions employed by each node, the connection relationships between the nodes, and the connection coefficients set for the connections between the nodes.

In the example shown in FIG. 4, the model identified by the model ID "M1" (estimation model M1) indicates that its use is "emotion estimation." For example, estimation model M1 indicates that it is an estimation model (emotion estimation model) that outputs information that estimates the user's emotion corresponding to input data when input data that is a user's biosignal is input. In addition, it indicates that the model data of estimation model M1 is model data MDT1.

The model information storage unit 122 may store various types of information depending on the purpose, not limited to the above.

(Content information storage unit 123)
The content information storage unit 123 according to the embodiment stores various information related to content (articles). Fig. 5 is a diagram showing an example of the content information storage unit according to the embodiment. The content information storage unit 123 shown in Fig. 5 has items such as "content ID", "content", and "label information". Note that in the example of Fig. 5, only five labels (emotions) of "joy", "anger", "sadness", "enjoyment", and "neutrality" are illustrated as label information, but the label information is not limited to the above five and may be labels indicating various emotions.

"Content ID" indicates identification information for identifying the content. "Content" may be video content such as a movie, or various types of content such as audio content such as music. Figure 5 shows an example in which conceptual information such as "Content A" is stored in "Content", but in reality, the content itself or a file path name indicating the storage location of the content is stored.

Furthermore, the "label information" indicates the psychological effect that the content has on the user. For example, the "label information" indicates the psychological state (emotion) that the user will be in when the content is output to the user. In other words, the "label information" indicates the emotion (correct answer information) that is evoked in the user by the output of the content.

In the example of FIG. 5, content A (content CT1) identified by content ID "CT1" indicates that label information LB1 is associated with it. For example, content CT1 indicates that it induces in the user the emotion of "joy" corresponding to label information LB1.

In addition, B content (content CT2) identified by the content ID "CT2" indicates that it is associated with label information LB2. For example, content CT2 indicates that it evokes in the user the emotion of "anger" corresponding to label information LB2.

The content information storage unit 123 may store various types of information depending on the purpose, not limited to the above.

(Control unit 130)
Returning to the explanation of Fig. 2, the control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 2, the control unit 130 has an acquisition unit 131, a learning unit 132, an estimation unit 133, and a provision unit 134, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 2, and may be other configurations as long as they perform the information processing described below. Also, the connection relationship between each processing unit of the control unit 130 is not limited to the connection relationship shown in FIG. 2, and may be other connection relationships.

(Acquisition unit 131)
The acquisition unit 131 acquires various types of information from the storage unit 120. The acquisition unit 131 receives a biosignal of a user from the terminal device 10 used by the user. The acquisition unit 131 acquires data to be used for learning from the learning data storage unit 121. The acquisition unit 131 acquires model information from the model information storage unit 122. The acquisition unit 131 acquires content to be provided to the user from the content information storage unit 123.

The acquisition unit 131 acquires learning data including label information indicating the psychological effect on the user of the content output by the terminal device 10 worn on the user's head, and input data which is the user's biosignal detected by the terminal device 10 when the content is output by the terminal device 10.

The acquisition unit 131 acquires learning data including input data that is the bioelectric potential of the user detected by the terminal device 10 when the terminal device 10 outputs content. The acquisition unit 131 acquires learning data including input data that includes the myoelectric potential of the user's head detected by the terminal device 10 when the terminal device 10 outputs content. The acquisition unit 131 acquires learning data including input data that includes the myoelectric potential of the user's face detected by the terminal device 10 when the terminal device 10 outputs content.

The acquisition unit 131 acquires learning data including input data including the user's electrooculography detected by the terminal device 10 when the terminal device 10 outputs content. The acquisition unit 131 acquires learning data including label information indicating the emotion evoked in the user by the content.

The acquisition unit 131 acquires learning data including label information of content output by the terminal device 10, which is a head-mounted display worn by a user on the head. The acquisition unit 131 acquires learning data including label information indicating the psychological effect that content including video has on the user.

The acquisition unit 131 acquires learning data including label information of content output by the terminal device 10, which is an over-ear headphone worn by a user on the head. The acquisition unit 131 acquires learning data including label information indicating the psychological effect that content including audio has on a user.

The acquisition unit 131 acquires an estimation model that is generated using learning data including label information indicating the psychological effect that content output by the terminal device 10 worn by the user on the head has on the user, and input data that is the user's biosignal detected by the terminal device 10 when the terminal device 10 outputs the content, and that outputs information that estimates the emotion of a person corresponding to the input data in response to the input of the input data. The acquisition unit 131 acquires the estimation model from the model information storage unit 122. The acquisition unit 131 acquires a target biosignal that is the biosignal of a target user wearing the terminal device 10.

(Learning Unit 132)
The learning unit 132 learns a model. The learning unit 132 learns various pieces of information based on information from an external information processing device and information stored in the storage unit 120. The learning unit 132 learns various pieces of information based on information stored in the learning data storage unit 121. The learning unit 132 stores the model generated by learning in the model information storage unit 122.

The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to input data in response to the input of the input data. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to a bioelectric potential. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to the myoelectric potential of the head. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to the myoelectric potential of the face.

The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates the emotion of a person corresponding to the electrooculography. The learning unit 132 uses the learning data to learn the estimation model. The learning unit 132 uses the learning data to learn an estimation model that estimates the emotion of a person wearing the terminal device 10. The learning unit 132 uses the learning data to learn an estimation model that estimates the emotion of a person wearing the terminal device 10.

The learning unit 132 learns the parameters of the model (network). The learning unit 132 learns parameters such as the connection coefficients (weights) between connected nodes. The learning unit 132 learns the model using various machine learning techniques. The learning unit 132 learns the model parameters by a learning process that uses data to be input to the model and classification labels (correct answer labels) that indicate the output when that data is input, that is, a supervised learning method. Note that the above is just one example, and the learning unit 132 may learn the model parameters by any learning process as long as it is capable of learning the model parameters.

Note that when the information processing device 100 acquires an estimation model learned by an external device from an external device and estimates human emotions using the acquired estimation model, the information processing device 100 does not need to include the learning unit 132.

(Estimation unit 133)
The estimation unit 133 executes an estimation process to estimate the emotion of a person. The estimation unit 133 estimates the emotion of a person by using the estimation model learned by the learning unit 132. The estimation unit 133 estimates the emotion of a target user by inputting a target biological signal to the estimation model.

The estimation unit 133 performs various processes using the estimated user emotion. The estimation unit 133 performs a decision process based on the estimated user emotion. The estimation unit 133 decides on content to be provided to the user based on the estimated user emotion. The estimation unit 133 decides to provide the user with content that corresponds to the estimated user emotion. The estimation unit 133 decides to provide the user with content that evokes an emotion opposite to the estimated user emotion. If the estimated user emotion is sadness, the estimation unit 133 decides to provide the user with content associated with the emotion of joy.

(Providing Unit 134)
The providing unit 134 provides information to the user. The providing unit 134 transmits information to the terminal device 10 via the communication unit 110. The providing unit 134 provides the user with content stored in the content information storage unit 123. The providing unit 134 transmits content associated with a label indicating the psychological effect that the content has on the user to the terminal device 10 of the user.

The providing unit 134 provides information according to the estimation result by the estimation unit 133. For example, the providing unit 134 provides content according to the decision by the estimation unit 133. The providing unit 134 provides the user with content corresponding to the user's emotion estimated by the estimation unit 133. The providing unit 134 transmits content that evokes an emotion opposite to the user's emotion estimated by the estimation unit 133 to the user's terminal device 10. When the user's emotion estimated by the estimation unit 133 is sadness, the providing unit 134 transmits content associated with the emotion of joy to the user's terminal device 10.

3. Configuration of terminal device
Next, the configuration of the terminal device 10 according to the embodiment will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the configuration of the terminal device 10 according to the embodiment. As shown in FIG. 6, the terminal device 10 has a communication unit 11, a voice output unit 12, a display unit 13, a storage unit 14, a control unit 15, and a sensor unit 16. Although not shown, the terminal device 10 may have an input unit to which various operations of the user are input. For example, the input unit has a microphone (voice input unit) that receives input from the user by voice. The input unit receives various operations by the user's speech. Also, for example, the input unit may receive operations by the user's gestures by a motion sensor that detects the user's gestures, etc. In this case, the sensor unit 16 has a motion sensor.

(Communication unit 11)
The communication unit 11 is realized by, for example, a NIC or a communication circuit. The communication unit 11 is connected to a predetermined communication network by wire or wirelessly, and transmits and receives information to and from an external information processing device. For example, the communication unit 11 is connected to a predetermined communication network by wire or wirelessly, and transmits and receives information to and from the information processing device 100.

(Audio output unit 12)
The audio output unit 12 is realized by a speaker that outputs audio, and is an output device for outputting various information as audio. The audio output unit 12 outputs content provided from the information processing device 100 as audio. For example, the audio output unit 12 outputs audio corresponding to information displayed on the display unit 13.

(Display unit 13)
The display unit 13 is a display screen of a tablet terminal or the like realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. For example, if the terminal device 10 is a head-mounted display, the display unit 13 is disposed in front of the user's eyes when the user wears the terminal device 10. Note that, if the terminal device 10 does not display information, such as over-ear headphones, the terminal device 10 does not need to have the display unit 13.

(Memory unit 14)
The storage unit 14 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores, for example, various information received from the information processing device 100. The storage unit 14 stores, for example, information related to an application (for example, an information output application) installed in the terminal device 10, such as a program.

(Control unit 15)
The control unit 15 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs stored in a storage device such as the storage unit 14 inside the terminal device 10 using the RAM as a working area. For example, the various programs include a program for an application that performs information processing (for example, an information output application). The control unit 15 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC or an FPGA.

As shown in FIG. 6, the control unit 15 has a collection unit 151, a transmission unit 152, a reception unit 153, and a processing unit 154, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in FIG. 6, and may be other configurations as long as they perform the information processing described below. Also, the connection relationship between each processing unit of the control unit 15 is not limited to the connection relationship shown in FIG. 6, and may be other connection relationships.

(Collection Unit 151)
The collection unit 151 collects various types of information. For example, the collection unit 151 collects various types of information from an external information processing device. For example, the collection unit 151 stores the collected various types of information in the storage unit 14. The collection unit 151 collects sensor information detected by the sensor unit 16.

The collection unit 151 collects the biosignals of the user detected by the sensor unit 16 when content is output by the terminal device 10. The collection unit 151 collects the bioelectric potential of the user detected by the sensor unit 16. The collection unit 151 collects the myoelectric potential of the user's head detected by the sensor unit 16. The collection unit 151 collects the myoelectric potential of the user's face detected by the sensor unit 16.

(Transmitting unit 152)
The transmitting unit 152 transmits various information to an external information processing device via the communication unit 11. The transmitting unit 152 transmits various information to the information processing device 100. The transmitting unit 152 transmits various information stored in the memory unit 14 to an external information processing device. The transmitting unit 152 transmits various information collected by the collecting unit 151 to the information processing device 100. The transmitting unit 152 transmits sensor information collected by the collecting unit 151 to the information processing device 100. The transmitting unit 152 transmits sensor information detected by the sensor unit 16 to the information processing device 100. The transmitting unit 152 transmits various biological signals of the user to the information processing device 100.

(Receiving unit 153)
The receiving unit 153 receives information from the information processing device 100 via the communication unit 11. The receiving unit 153 receives information provided by the information processing device 100. The receiving unit 153 receives content from the information processing device 100. The receiving unit 153 receives advertisements as notification information from the information processing device 100. The receiving unit 153 receives content to which label information indicating the psychological effect to be given to the user when the content is output is attached.

(Processing Unit 154)
The processing unit 154 executes various processes. The processing unit 154 displays various information via the display unit 13. For example, the processing unit 154 controls the display of the display unit 13. The processing unit 154 outputs various information as audio via the audio output unit 12. For example, the processing unit 154 controls the audio output of the audio output unit 12.

The processing unit 154 outputs information received by the receiving unit 153. The processing unit 154 outputs content provided by the information processing device 100. The processing unit 154 outputs the content received by the receiving unit 153 via the audio output unit 12 or the display unit 13. The processing unit 154 displays the content via the display unit 13. The processing unit 154 outputs the content as audio via the audio output unit 12.

Note that each of the processes by the control unit 15 described above may be realized by, for example, JavaScript (registered trademark). Furthermore, when the information processing and other processes by the control unit 15 described above are performed by a specified application, each unit of the control unit 15 may be realized by, for example, the specified application. For example, the information processing and other processes by the control unit 15 may be realized by control information received from an external information processing device. For example, when the display process described above is performed by a specified application (for example, an information output application), the control unit 15 may have, for example, an application control unit that controls the specified application or a dedicated application.

(Sensor unit 16)
The sensor unit 16 detects predetermined information. The sensor unit 16 has a sensor (biological signal sensor) for detecting a biosignal of the user. The sensor unit 16 has an electromyogram sensor for detecting the myoelectric potential of the user's face. The electromyogram sensor is disposed from the ear to the cheek of the user and detects the myoelectric potential of the user's face. The terminal device 10 may have a sensor that detects various biosignals of the user, not limited to the electromyogram sensor. The terminal device 10 may have an electrooculography sensor that detects the electrooculography of the user. For example, the terminal device 10 may have sensors that detect various biosignals, such as a heartbeat sensor that detects the user's heartbeat, an electroencephalogram sensor that detects the user's electroencephalogram, a pulse sensor (pulse wave sensor) that detects the user's pulse, a respiration sensor (breath sensor) that detects the user's respiration, and a sweat sensor that detects the user's sweat. The terminal device 10 is configured to be disposed at a position where each sensor can detect a corresponding biosignal when worn by the user.

The sensor unit 16 is not limited to the above-mentioned sensor for detecting the user's biosignal, and may have various sensors for detecting information used in information processing. For example, the sensor unit 16 has a sensor for detecting information used in processing related to the user's health. For example, the sensor unit 16 has a sensor for detecting various types of sensor information used as usage mode information indicating the usage mode of the terminal device 10 by the user.

For example, the sensor unit 16 has a motion sensor. For example, the sensor unit 16 may have a camera (image sensor) that detects an image. For example, the sensor unit 16 has an image sensor for capturing an image of the user.

For example, the sensor unit 16 has an acceleration sensor. For example, the sensor unit 16 has a sensor that detects acceleration information of the terminal device 10 when a user performs a predetermined operation. The sensor unit 16 has a sensor (position sensor) that detects the position of the terminal device 10. For example, the sensor unit 16 may have a GPS (Global Positioning System) sensor. Furthermore, when acquiring the position information of the terminal device 10 as sensor information, the sensor unit 16 may acquire the position information of a base station with which the terminal device 10 is communicating, or the position information of the terminal device 10 estimated using Wi-Fi (registered trademark) (Wireless Fidelity) radio waves.

The sensor unit 16 may include various sensors, not limited to those described above. For example, the sensor unit 16 may include a sensor that detects information outside the terminal device 10.

The sensors that detect the various types of information in the sensor unit 16 may be a common sensor, or each may be a different sensor.

4. Processing Flow
Next, a procedure of information processing by the information processing system 1 according to the embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of processing by the information processing device according to the embodiment.

As shown in FIG. 7, the information processing device 100 acquires learning data including label information indicating the psychological effect on the user of the content output by the terminal device 10 worn on the user's head, and input data which is the user's biosignal detected by the terminal device 10 when the content is output by the terminal device 10 (step S101).

The information processing device 100 uses the learning data to learn an estimation model that outputs information that estimates the emotion of a person corresponding to input data in response to the input of the input data (step S102). The information processing device 100 estimates the emotion of the person using the estimation model (step S103). For example, the information processing device 100 acquires a target biosignal that is a biosignal of a target user wearing the terminal device 10, and inputs the target biosignal into the estimation model to estimate the emotion of the target user.

5. Effects
As described above, the information processing device 100 according to the embodiment includes the acquisition unit 131 and the learning unit 132. The acquisition unit 131 acquires learning data including label information indicating a psychological effect on the user of content output by the terminal device 10 worn on the user's head, and input data which is a biosignal of the user detected by the terminal device 10 when the terminal device 10 outputs the content. The learning unit 132 uses the learning data to learn an estimation model that outputs information for estimating the emotion of a person corresponding to the input data in response to the input of the input data.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and bio-signals by learning an estimation model that estimates a person's emotions using a label indicating the psychological effect that the content has on the user and the user's bio-signals at the time the content is output. Therefore, the information processing device 100 can appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including input data that is the bioelectric potential of the user detected by the terminal device 10 when the terminal device 10 outputs content. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates the emotion of a person corresponding to the bioelectric potential.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and bioelectric potentials by learning an estimation model of human emotions using the user's bioelectric potentials as input. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including input data including the myoelectric potential of the user's head detected by the terminal device 10 when the terminal device 10 outputs content. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to the myoelectric potential of the head.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and the head's myoelectric potential by learning an estimation model of human emotions using the user's head's myoelectric potential as input. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including input data including the myoelectric potential of the user's face detected by the terminal device 10 when the terminal device 10 outputs content. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to the myoelectric potential of the face.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and the facial myoelectric potential by learning an estimation model of human emotions using the user's facial myoelectric potential as input. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including input data including the user's electrooculography detected by the terminal device 10 when the terminal device 10 outputs content. The learning unit 132 uses the learning data to learn an estimation model that outputs information that estimates a person's emotion corresponding to the electrooculography.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotion and the electrooculography by learning an estimation model of human emotion using the user's electrooculography as input. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotion.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including label information indicating the emotion evoked in the user by the content. The learning unit 132 uses the learning data to learn the estimation model.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and bio-signals by learning an estimation model that estimates human emotions using labels indicating the emotions evoked in the user by the content and the user's bio-signals at the time the content is output. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotions.

In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including label information of content output by the terminal device 10, which is a head-mounted display worn by a user on the head. The learning unit 132 uses the learning data to learn an estimation model that estimates the emotions of a person wearing the terminal device 10.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and bio-signals by learning an estimation model that estimates human emotions using the labels of content output to the user by the head-mounted display and the bio-signals of the user wearing the head-mounted display when the content is output. Therefore, the information processing device 100 can appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including label information indicating the psychological effect that content including video has on the user.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and biosignals by learning an estimation model that estimates a person's emotions using the content label and the user's biosignals at the time of video output. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including label information of content output by the terminal device 10, which is an over-ear headphone worn by a user on the head. The learning unit 132 uses the learning data to learn an estimation model that estimates the emotions of a person wearing the terminal device 10.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and biosignals by learning an estimation model that estimates human emotions using the labels of content output to the user through the over-ear headphones and the biosignals of the user wearing the over-ear headphones when the content is output. Therefore, the information processing device 100 can appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires learning data including label information indicating the psychological effect that content including audio has on the user.

As a result, the information processing device 100 according to the embodiment can generate a model that appropriately learns the correspondence between the user's emotions and the biosignals by learning an estimation model that estimates a person's emotions using the content labels and the biosignals when the user outputs voice. Therefore, the information processing device 100 can make it possible to appropriately estimate the user's emotions.

The information processing device 100 according to the embodiment also includes an estimation unit 133. The estimation unit 133 estimates human emotions using the estimation model learned by the learning unit 132.

This allows the information processing device 100 according to the embodiment to appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires an estimation model that is generated using learning data including label information indicating the psychological effect that content output by the terminal device 10 worn by the user on the head has on the user, and input data that is the user's biosignal detected by the terminal device 10 when the terminal device 10 outputs the content, and that outputs information that estimates the emotion of a person corresponding to the input data in response to the input of the input data. The estimation unit 133 estimates the emotion of the person using the estimation model.

This allows the information processing device 100 according to the embodiment to appropriately estimate the user's emotions.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a target biosignal, which is a biosignal of a target user wearing the terminal device 10. The estimation unit 133 estimates the emotion of the target user by inputting the target biosignal to an estimation model.

As a result, the information processing device 100 according to the embodiment can appropriately estimate the user's emotions for a target user wearing the terminal device 10.

6. Hardware Configuration
Moreover, the terminal device 10 and the information processing device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in Fig. 8, for example. The information processing device 100 will be described below as an example. Fig. 8 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected by a bus 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB memory or a Secure Digital (SD) memory card. The secondary storage device 1050 may be a cloud storage device (online storage device), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, projector, printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I/F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by a USB, etc.

In addition, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

The output device 1010 and the input device 1020 may be integrated together, such as a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated together as an input/output I/F.

The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

The network I/F 1080 receives data from other devices via the network N and sends it to the computing device 1030, and also transmits data generated by the computing device 1030 to other devices via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I/F 1060 and the input I/F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I/F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with other devices via the network I/F 1080 and use the functions and data of a program by calling them from other programs of the other devices.

[7. Other]
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, the information processing device 100 described above may be realized by multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing. Also, the information processing device 100 and the terminal device 10 may be integrated. In this case, for example, the terminal device 10 used by a user may have the function of the information processing device 100.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

In addition, the above-mentioned "section, module, unit" can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

REFERENCE SIGNS LIST 1 Information processing system 100 Information processing device 120 Storage unit 121 Learning data storage unit 122 Model information storage unit 123 Content information storage unit 130 Control unit 131 Acquisition unit 132 Learning unit 133 Estimation unit 134 Provision unit 10 Terminal device 11 Communication unit 12 Audio output unit 13 Display unit 14 Storage unit 15 Control unit 151 Collection unit 152 Transmission unit 153 Reception unit 154 Processing unit 16 Sensor unit

Claims (13)

An information processing device,
an acquisition unit that acquires from a storage unit learning data associated with a content, the learning data including label information that indicates a psychological effect of a content output by a terminal device worn on the head of a user, the label information being correct answer information that is associated in advance with the content, the label information being correct answer information that is associated with the content, the label information being correct answer information that is associated in advance with the content, the label information being correct answer information that is associated with the content , and a biological signal of the user including information indicating sweating of the user detected by a sensor unit of the terminal device when the content is output by the terminal device, the biological signal being associated with the content, the biological signal being correct answer information that is associated with the content, and the biological signal being associated with the content, the biological signal being correct answer information that is ... correct answer information that is associated with the content, the biological signal being correct answer information that is correct answer information that is associated with the content, the biological signal being correct answer information that is correct answer information that is correct answer information
a learning unit that uses the learning data to learn an estimation model that outputs a plurality of scores corresponding to each of a plurality of psychological effects, and that learns, by a learning process in which, when a biosignal of the user is input, a score corresponding to a psychological effect indicated by label information associated with the biosignal is maximized, to generate the plurality of scores corresponding to the biosignal used as an input, the plurality of scores being information necessary for estimating the emotion of a person who is the detection target of the biosignal used as an input;
An information processing device comprising: The biosignal of the user is
a bioelectric potential of the user detected by the terminal device when the terminal device outputs the content;
2. The information processing apparatus according to claim 1, The biosignal of the user is
The information processing apparatus according to claim 2 , wherein the input is a myoelectric potential of the head of the user detected by the terminal device when the content is output by the terminal device. The biosignal of the user is
The information processing apparatus according to claim 3 , wherein the input is a myoelectric potential of the face of the user detected by the terminal device when the content is output by the terminal device. The biosignal of the user is
5. The information processing device according to claim 2, wherein the electrooculography of the user is detected by the terminal device when the content is output by the terminal device. The acquisition unit is
acquiring the learning data including the label information indicating an emotion evoked in the user by the content;
The learning unit is
The information processing apparatus according to any one of claims 1 to 5, characterized in that the estimation model is trained using the learning data. The acquisition unit is
The information processing device according to any one of claims 1 to 6, characterized in that the learning data is acquired including the label information of the content output by the terminal device, which is a head-mounted display worn on the user's head. The acquisition unit is
The information processing apparatus according to claim 7 , further comprising: acquiring the learning data including the label information indicating a psychological effect that the content including a video has on the user. The acquisition unit is
The information processing device according to any one of claims 1 to 6, characterized in that the learning data is acquired including the label information of the content output by the terminal device, which is an over-ear headphone worn on the user's head. The acquisition unit is
10. The information processing apparatus according to claim 7, further comprising: acquiring the learning data including the label information indicating a psychological effect that the content including audio has on the user. an estimation unit that estimates human emotions using the estimation model trained by the learning unit;
11. The information processing apparatus according to claim 1, further comprising: 1. A computer-implemented information processing method, comprising:
an acquisition process for acquiring, from a storage unit, learning data in which label information indicating a psychological effect exerted on the user by content output by a terminal device worn on the user's head and determined by the computer to be provided to the user in accordance with the type of content that can be played back by the terminal device and the user attributes of the user, the label information being correct answer information previously associated with the content , a biological signal of the user including information indicating sweating of the user detected by a sensor unit of the terminal device when the content is output by the terminal device , and the content are associated with the label information;
a learning process for learning an estimation model that uses the learning data to output a plurality of scores corresponding to each of a plurality of psychological effects, the estimation model being used to generate the plurality of scores corresponding to the biosignal used as an input , the plurality of scores being information necessary for estimating the emotions of a person who is the detection target of the biosignal used as an input, by a learning process that learns so as to maximize a score corresponding to a psychological effect indicated by label information associated with the biosignal when the biosignal of the user is input;
13. An information processing method comprising: An information processing program executed by a computer,
an acquisition step of acquiring, from a storage unit, label information indicating a psychological effect exerted on the user by content output by a terminal device worn on the user's head, the content being determined by the computer to be provided to the user according to a type of content that can be played back by the terminal device and a user attribute of the user, the label information being correct answer information previously associated with the content , learning data associated with the content, and a biosignal of the user including information indicating sweating of the user detected by a sensor unit of the terminal device when the content is output by the terminal device ;
a learning procedure for learning an estimation model that uses the learning data to output a plurality of scores corresponding to each of a plurality of psychological effects, the estimation model being used to generate the plurality of scores corresponding to the biosignal used as an input , the plurality of scores being information necessary for estimating the emotions of a person who is the detection target of the biosignal used as an input, by a learning process that learns so as to maximize a score corresponding to a psychological effect indicated by label information associated with the biosignal when the biosignal of the user is input;
An information processing program characterized by causing a computer to execute the above.

Images