JP6166234B2 - Robot control apparatus, robot control method, and robot control program - Google Patents

Description

  The present invention relates to a technique for controlling a robot.

  When people watch videos together and when they watch them alone, even if they show the same comedy video, the co-viewers who are co-viewing are more likely to laugh and laugh than those who watch alone. It has been found that the fun with respect to itself is improved (for example, Non-Patent Document 1). Therefore, by co-viewing the video with the robot and making the robot perform a synchronous reaction such as laughter, joy, sadness, and anger with respect to the video content together, emotions such as laughter and joy than when watching the video alone It is possible to suppress emotions such as sadness and anger.

  Further, in communication research between a person and a CG person, it has been pointed out that an affinity motivation is given to a person by causing the CG person to perform a facial expression change that gives empathy. Affinity motivation is defined as a desire to approach, cooperate with, or reward the other person, and people are considered to have affinity motivation for others who have a similar attitude to them. Yes.

  For example, in Non-Patent Document 2, a robot provides information to a user when viewing a video. In Non-Patent Document 2, a user's evaluation of a viewing program is estimated as a profile from the user's viewing program log and the utterance being viewed. However, a technique is disclosed in which a robot recommends another television program using a profile if it seems to be bored.

  Also disclosed is a technology that acts as an intermediary for social media, in which robots use social media comments related to viewing programs as utterances, interact with users, and robots post comments on social media. (For example, Non-Patent Document 3).

  In addition, a technique is disclosed that promotes user laughing behavior by detecting changes in myoelectric potential of the chest in response to the laughing behavior of the person watching the video, and making the robot react and laugh (for example, non-patented) Reference 4).

Keiko Omori, 1 other person, “Effects of the existence of others on the expression of fun and expression of laughter”, 2011, Bulletin of Niai University, Faculty of Humanities, No. 10, pp.25-32 Yasushi Takama, 5 others, “Human Robot Communication based on Recommendation of Information when Watching TV”, 2007, National Congress of Population Intelligence Society 2007, 2D5-5 Tatsuhashi Takahashi and two others, “Social Media Mediation Robot for Increasing Speaking Opportunities of Elderly People”, October 2012, IEICE, IEICE Technical Report, vol.112, no.233, CNR2012-9, pp.21-26 Masaki Fukushima, 3 others, “Laughter Amplifier: Verification of Laughter Amplification Effect”, 2010, Journal of Human Interface Society, pp.199-207

  However, with the technique disclosed in Non-Patent Document 2, other programs are only introduced by user evaluation in units of programs, and emotional expressions at the time of viewing cannot be handled.

  Also in Non-Patent Document 3, only the utterance content of the robot is determined from social media, and does not deal with emotional expressions to be expressed. In Non-Patent Document 3, the speech operation of the robot is determined only from social media comment information. However, since the comment content on social media is not necessarily close to the emotion that the user receives from the video, the user receives from the emotion that the user received from the video and the utterance action of the robot determined from the social media comment information There is a problem that the reaction of the robot becomes unsympathetic to the user when the impressions are greatly different.

  Non-Patent Document 4 is a technique for promoting only the user's laughing behavior. “Laughter” is one element of emotion and can only handle a small part of the entire emotional expression such as excitement and sadness. In addition, since the input to the robot is only the user's laughter reaction, the robot can only perform reactive control on the user that responds to the detected content after detecting the user's reaction. Therefore, when there are few laughing reactions that cannot detect changes in myoelectric potential, it is impossible to perform proactive control for the user, such as working to laugh at the user from the robot. The conditions for improvement are limited.

  The present invention has been made in view of these problems, and provides a robot control device, a robot control method, and a robot control program that cause a robot to perform a synchronous reaction that is sympathetic to a video viewed by a user. For the purpose.

  A robot control apparatus according to the present invention is a robot control apparatus that causes a robot to perform an operation such as viewing a video together with a user, and a robot control apparatus that causes a robot to perform an operation such as viewing a video together with a user. Video impression information representing emotions aroused by a human when he / she sees the video and user emotion information representing the emotions of the user who viewed the video are input, and two types of emotions related to each other are input. In the case where n types (n is an integer of 1 or more) of emotion types are set in advance, the type of emotion is determined from the video impression information using a conversion rule prepared in advance. A value indicating the size is generated, and a value indicating the size of each type of emotion is generated from the user emotion information using the conversion rule. For each set, (1) the video A value obtained by multiplying a value for one emotion type of the set generated from the elephant information by a predetermined weight α and a value for the one emotion type of the set generated from the user emotion information. The sum of the values multiplied by the weight β is calculated as a value for the one emotion type of the set of the robot, and (2) for the other emotion type of the set generated from the video impression information The sum of the value obtained by multiplying the value by the weight α and the value for the other emotion type of the set generated from the user emotion information by the weight β is the other emotion of the set of the robot. An emotional state determination unit that calculates the value of the type of

  The robot control method of the present invention is a robot control method performed by a robot control device that causes a robot to perform an operation of viewing a video together with a user. When the human views the video, the robot is controlled. Video impression information representing emotions and user emotion information representing the emotions of the user who viewed the video, and n sets of two types of emotions related to each other (n is an integer of 1 or more) ) Emotion types are set in advance, a value indicating the size of each type of emotion is generated from the video impression information using a conversion rule prepared in advance, and from the user emotion information Then, using the conversion rule, a value indicating the magnitude of each type of emotion is generated, and for each set, (1) a value for one type of emotion of the set generated from the video impression information The sum of a value obtained by multiplying a predetermined weight α and a value obtained by multiplying a value for the one emotion type of the set generated from the user emotion information by a predetermined weight β is the value of the set of the robot. Calculated as a value for the one emotion type, and (2) generated from the value obtained by multiplying the value for the other emotion type in the set generated from the video impression information by the weight α and the user emotion information A sum of values obtained by multiplying the value for the other emotion type of the set by the weight β is calculated as a value for the other emotion type of the set of the robot.

  In the present invention, since the emotional state of the robot is determined using the video impression information and the user emotion information, it is possible to cause the robot to perform a synchronous reaction that makes it feel that empathy has been obtained.

1 is an overall configuration diagram including a robot control device 1 in the present embodiment. It is a figure which shows the structural example of the emotion word dictionary. It is a figure which shows the structural example of the emotion state conversion rule base. It is a figure which shows the structural example of the audio | voice expression database. It is a figure which shows the structural example of the body representation database. It is a figure which shows the structural example of the azimuth | direction information database 60 which the position information acquisition server 6 hold | maintains. It is a figure which shows the flow of the video relevant information and user information of this Embodiment. FIG. 7 is a diagram in which the flow of robot line-of-sight information is additionally written. It is a figure which shows the operation | movement flow of the video relevant information collection part. It is a figure which shows the operation | movement flow of the user information collection part. It is a figure which shows the operation | movement flow of the image | video impression estimation part. It is a figure which shows the operation | movement flow of the user emotion estimation part. It is a figure which shows the operation | movement flow of the emotional state determination part. It is a figure which shows the example which maps an emotion element on a two-dimensional coordinate. It is a figure which shows the operation | movement flow of the emotion expression production | generation part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an overall configuration diagram including a robot control device according to the present embodiment.

  The robot control device 1 according to the present embodiment is a device that acquires video, user moving image information, and user audio information, determines the emotional state of the robot 5 based on the information, and controls the robot 5. The video is acquired from the video display device 2 that can display and transmit the video. The user moving image information is acquired from the camera 3 that can capture the user's facial expression and transmit moving image information. The user voice information is acquired from the microphone 4 that can acquire the voice uttered by the user and transmit the voice information.

  The robot 5 is a robot capable of receiving voice data and a drive control command and driving the motor in accordance with the playback of the voice data and the drive control command to express the body. The position information acquisition server 6 is a server that can hold and transmit the azimuth and elevation angles of the user and the video display device 2 viewed from the robot 5, and includes an azimuth information database 60.

[Robot controller configuration]
First, the configuration of the robot control apparatus 1 in the present embodiment will be described.

  The robot control apparatus 1 includes a video related information collection unit 11, a user information collection unit 12, a user emotion estimation unit 13, a video impression estimation unit 14, an emotion word dictionary 15, an emotion state determination unit 16, an emotion expression generation unit 17, an emotion state. A conversion rule base 18, a speech expression database 19, and a body expression database 20 are provided. The robot control device 1 may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the robot control apparatus 1 and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. In addition, you may make it comprise each function structure part of the robot control apparatus 1 with a computer. In FIG. 1, the robot control device 1 and the robot 5 are shown separately, but the robot control device 1 may be incorporated in the robot 5.

  The video related information collection unit 11 receives video from the video display device 2 and transmits video related information regarding the video to the video impression estimation unit 14. Here, the video related information is information including video moving image information, audio information, and caption information.

  The user information collection unit 12 receives a user's facial expression photographed and transmitted by the camera 3 as user moving image information. In addition, the voice uttered by the user collected by the microphone 4 is received as user voice information. And the received user moving image information and user audio | voice information are transmitted to the user emotion estimation part 13 as user information.

  The video impression estimation unit 14 estimates video impression information, which is an emotion aroused when a person views the video related information, from the video related information transmitted from the video related information collection unit 11. The user emotion estimation unit 13 estimates user emotion information felt by the user from the user information transmitted from the user information collection unit 12.

  The emotional state determination unit 16 determines the robot emotional state, which is the emotional state of the robot 5, using two pieces of information: the video impression information estimated by the video impression estimation unit 14 and the user emotion information estimated by the user emotion estimation unit 13. To do. The emotion expression generation unit 17 receives the robot emotion state determined by the emotion state determination unit 16, and generates voice data and a drive control command to be transmitted to the robot 5 with reference to data held by the robot control device 1.

  Next, data held by the robot control apparatus 1 will be described.

  FIG. 2 shows a configuration example of the emotion word dictionary 15. The emotion word dictionary 15 stores words expressing emotions such as “beautiful” and “great” and the emotion strength of each word. Each word that expresses emotion is associated with the intensity of each emotion element that is broken down into emotion elements such as “excitement”, “joy”, “anger”, “sadness”, “…” Yes. For example, the strength is represented by a value of 0.0 to 1.0. The emotion word dictionary 15 is used when the video impression estimation unit 14 estimates video impression information.

  FIG. 3 shows a configuration example of the emotion state conversion rule base 18. The emotion state conversion rule base 18 stores an arousal value expression and a pleasant value expression corresponding to emotion elements in the emotion word dictionary 15. The arousal value for the emotion element “excitement” is given by, for example, arousal value = n * 0.15 + 0.35, and the pleasant value is given by, for example, pleasant value = n * 0.10 + 0.10. Here, n is a variable representing the strength of the emotion element. The emotional state conversion rule base 18 is used when the emotional state determination unit 16 determines the emotional state of the robot. The awakening value and the pleasant value will be described later in detail.

  FIG. 4 shows a configuration example of the speech expression database 19. The voice expression database 19 stores the arousal value, pleasant value, and voice file path associated with the voice name. For example, the arousal value for the voice name “Wow” is 0.85, and the pleasant value is 0.80. The audio file path represents the location of audio data such as “awesome”. For example, the voice data may be in the robot control apparatus 1 or in an external server. The voice expression database 19 is used when the emotion expression generation unit 17 generates voice data.

  FIG. 5 shows a configuration example of the body representation database 20. The body representation database 20 stores an action name, an arousal value, a pleasant value, a line-of-sight object, and a drive unit control function corresponding to the motion name. The drive unit control function is composed of an array of two values of a motor control location, a value, and a sequence movement interval. The motor control part and the value are the motor part of the robot 5 to be operated and the value (angle) of the operation amount. The sequence movement interval is a numerical value representing the time interval until the motor control location and value are changed to the next value. The body expression database 20 is used when the emotion expression generation unit 17 generates a drive control command.

[External data used by the robot controller]
Next, data held by an external server used by the robot control apparatus 1 will be described. FIG. 6 shows a configuration example of the azimuth information database 60 held by the position information acquisition server 6. The azimuth information database 60 stores an azimuth angle and an elevation angle corresponding to the line-of-sight object. The line-of-sight target is the user or the video display device 2 ahead of the line of sight of the robot 5. The azimuth angle and the elevation angle are the angles of the line of sight of the robot 5.

  The azimuth angle and elevation angle of each line-of-sight object are sequentially updated according to the movement of the user, the robot 5, and the video display device 2. The update may be set by the person in the position information acquisition server 6 when the positions of the video display device 2 and the user are determined. Alternatively, these positions may be automatically detected and sequentially updated.

  The azimuth angle indicates in which direction each object from the robot 5 is located when the north is 0 ° in the horizontal direction on the ground. The elevation angle indicates the angle at which each object from the robot 5 is at 0 ° in the direction perpendicular to the ground and 90 ° directly above the ground.

  The direction information database 60 is used when the emotion expression generation unit 17 generates a drive control command.

[Flow of video-related information and user information]
Before describing the present embodiment in detail, the basic concept of the present embodiment will be described with reference to FIG. 7 showing the flow of information of video-related information and user information, which are main information. The video-related information is information related to a viewing scene necessary for estimating a video impression, and is information including data such as audio of a viewing video, a moving image, and captions included in the viewing video. In addition, social comment information such as Twitter related to the viewing video may be included in the video related information.

  The user information is information including data such as the user's facial expression and the user's uttered voice during video viewing. In addition, you may include biometric information, such as a user's attitude | position and a user's heart rate, in user information.

  The video impression estimation unit 14 to which video related information is input estimates an emotion aroused when a general person views the video related information as video impression information. The user emotion estimation unit 13 to which user information is input estimates user emotion information felt by the user from the user information.

  The emotion state determination unit 16 receives the video impression information and the user emotion information as input, and generates an emotion expression that causes the robot 5 to perform a synchronous reaction that makes the user feel empathy. Emotional expressions to be performed by the robot 5 include physical expressions such as arm movements and head movements of the robot 5, voices uttered by the robot 5, facial expressions of the robot 5, and the like.

  The emotional state determination unit 16 determines the emotional state of the robot 5 using two pieces of information, so that the emotional reaction of the user is small or the robot 5 estimated from each of the video impression information and the user emotional information. Even if the emotional expression of the robot is greatly different, it is possible to cause the robot 5 to perform appropriate emotional expression.

  In addition to the image impression information and the user emotion information, emotion expression to be performed by the robot 5 may be generated using the robot line-of-sight information indicating the line-of-sight direction of the robot 5. FIG. 8 shows the information flow of the present embodiment to which robot line-of-sight information is added. By setting which of the video impression information and the user emotion information is weighted using the robot line-of-sight information, it becomes possible for the robot 5 to perform natural human-like empathy reaction.

  As described above, the present embodiment is an idea of determining an appropriate emotional state of the robot 5 using the video related information and the user information. Therefore, the robot control apparatus 1 according to the present embodiment can cause the robot 5 to perform a synchronous reaction that makes the user feel that the user has obtained sympathy by viewing the video together with the robot.

[Operation of robot controller]
Next, the operation of the robot control device 1 will be described. Below, operation | movement for every function structure part which comprises the robot control apparatus 1 is demonstrated in order.

[Video Related Information Collection Department]
FIG. 9 shows an operation flow of the video related information collection unit 11 and the operation will be described. The video related information collection unit 11 starts operation when receiving video from the video display device 2. First, the video-related information collection unit 11 determines whether the information received from the video display device 2 is a video (step S110). If it is not a video, the determination operation is repeated until the video is received (No in step S110).

  When the video is received, moving image information, audio information, and subtitle information of the video constituting the video are transmitted to the video impression estimation unit 14 as video related information (step S111). The video related information collection unit 11 ends the operation when the video related information is transmitted to the video impression estimation unit 14.

[User Information Collection Department]
FIG. 10 shows an operation flow of the user information collection unit 12 and its operation will be described. When the user information collection unit 12 receives any one of the user moving image information from the camera 3 and the user voice information from the microphone 4, the user information collecting unit 12 starts the operation. First, the user information collection unit 12 determines whether the information received from the camera 3 and the microphone 4 is user moving image information and user audio information (step S120). The determination operation is repeated until both the user moving image information and the user voice information can be received (No in step S120).

  When both the user moving image information and the user audio information can be received, the user moving image information and the user audio information are transmitted to the user emotion estimation unit 13 as user information (step S121). The user information collection unit 12 ends the operation when the user information is transmitted to the user emotion estimation unit 13.

[Image impression estimation part]
FIG. 11 shows an operation flow of the video impression estimation unit 14 and its operation will be described. The video impression estimation unit 14 first determines whether the information received from the video related information collection unit 11 is video related information (step S140).

  The video impression estimation part 14 will start operation | movement, if video related information is received (Yes of step S140). First, the video impression estimation unit 14 extracts voice moving image emotions from video moving image information and audio information in the received video related information (step S141). The voice moving image emotion includes a category of emotion elements and their strengths.

  For example, Reference 1 (Go Irie, Takashi Satou, Akira Kojima, Toshihiko Yamasaki and Kiyaharu Aizawa, “Affective Audio-Visual Words and Latent Topic Driving Model for Realizing Movie Affective Scene Classification”, IEEE Transactions on Multimedia, Vol.12, No.6, pp.523-534, 2010.). Reference 1 shows a technique for assigning an emotion label to a video section from audio information. That is, the video section is a section corresponding to the audio section. The voice section can be easily extracted by setting, for example, a section in which the amplitude of the user voice information is a predetermined value or more. The video section may be a section divided for each topic (Topics). The topic may be extracted by analyzing the result of speech recognition of the speech information, or may be extracted from the result of subtitle information described below after morphological analysis.

  The method described in Reference 1 estimates the most suitable emotion label among the eight types of labels representing emotion for each video section. Eight emotion labels are, for example, joy, acceptance, fear, surprise, sadness, disgust, anger, anticipation. It is a kind. This emotion label is a category of emotion elements of the voice moving image emotion.

  In the present embodiment, a plurality of scenes may be detected for a video section, a plurality of labels may be estimated, and the number of labels may be zero. When the estimation of emotion labels is completed for all scenes in the video section, the number of each emotion label is counted, and the number of each emotion label is defined as the emotion strength of the emotion element as a voice moving image emotion.

  Next, the video impression estimation unit 14 performs a morphological analysis process on the caption information of the video related information (step S142). The morphological analysis process is performed using a known method.

The phrase that matches each vocabulary as a result of the morphological analysis of the caption information is searched with reference to the emotion word dictionary 15. The method of extracting emotions from Japanese text using the emotion word dictionary 15 is described in, for example, Reference 2 (Hisaaki Sugawara, two others, “Emotion extraction from Japanese text using emotion word dictionary”, the 23 ^rd Annual Conference of the Japanese Society for Artificial Intelligence, 2009). If there is a matching phrase, the word is retained and the search is continued. For example, suppose that the vocabulary resulting from morphological analysis of subtitle information includes the words “great” and “beautiful”. In this case, the video impression estimation unit 14 holds “excitement: 0.6”, “joy: 0.2”, and “anger: 0.1” as emotion elements corresponding to “great” (see FIG. 2). Similarly, “excitement: 0.3” and “joy: 0.8” are also held as emotional elements corresponding to “beautiful”. This search operation is repeated until all vocabularies of the morphological analysis result are completed.

  Then, the video impression estimation unit 14 calculates the sum of the emotion intensities of the emotion elements of all the words held as matching words, and extracts the combination of the emotion elements and the sum of the emotion intensities as subtitle emotions (step S144). ). In the above example, “excitement: 0.9 (0.6 + 0.3)”, “joy: 1.0 (0.2 + 0.8)”, and “anger: 0.1 (0.1 + 0)” are extracted as subtitle emotions.

  Next, the video impression estimation unit 14 determines video impression information from the audio moving image emotion and the subtitle emotion (step S145). The determination of the video impression information is performed by adding the strengths of the emotional elements of the voice moving image emotion and the subtitle emotion, and setting the strength of the emotional element to 5 when the combined value is, for example, 5 or more of the maximum value. When calculation of the values of all the emotion elements is completed, a set of each emotion element and each calculated value is determined as video impression information. When the determined video impression information is transmitted to the emotional state determination unit 16, the video impression estimation unit 14 ends the operation (step S146).

[User Emotion Estimator]
FIG. 12 shows an operation flow of the user emotion estimation unit 13 and its operation will be described. The user emotion estimation part 13 determines whether the information received from the user information collection part 12 first is user information (step S130).

  The user emotion estimation part 13 will start operation | movement, if user information is received (Yes of step S130). The user emotion estimation part 13 extracts a user's facial expression emotion from the user moving image information in the received user information (step S131). For example, OKAO Vision, a product of OMRON Corporation, can be used to extract facial expressions.

OKAO Vision can measure seven emotion labels and their levels for the user's facial expressions in user video information. The seven emotion labels are included in the above eight types of emotion labels, and the emotion labels represent the category of emotion elements of facial expression emotion. Assuming that the maximum value representing the degree is E _max , the value obtained by adding 5 / E _max to the maximum value in the measurement section of the degree of each measured emotion label and the emotion element that matches each emotion label All pairs are determined as facial emotions.

  Next, the user emotion estimation unit 13 extracts a voice emotion from the user voice information in the received user information (step S132). As a voice emotion extraction method, for example, a method of giving an emotion label to a person's utterance described in Non-Patent Document 3 is used. The method of Non-Patent Document 3 estimates an emotion label that is most suitable among emotion labels representing five types of emotions in a speech section. The five types of emotion labels are joy, sadness and surprise. This emotion label is a category of emotion elements of voice emotion.

  In the present embodiment, a plurality of voice sections may be detected for user voice information, a plurality of emotion labels may be estimated, and the number of emotion labels may be zero. When estimation of emotion labels for the speech segment of the user speech information is finished, the number of emotion labels is counted for each speech segment, and the emotion strength of the emotion element that matches the number of each emotion label is defined as speech emotion. .

  And the user emotion estimation part 13 calculates the value which totaled the intensity | strength of each emotion element of the facial expression emotion extracted at step S131, and the audio | voice emotion extracted at step S132. For example, when the total value is 5 or more, which is the maximum value, the intensity of the emotion element is set to 5. When calculation of the values of all emotion elements is completed, a set of each emotion element and each calculated value is determined as user emotion information (step S133). When the determined user emotion information is transmitted to the emotion state determination unit 16, the user emotion estimation unit 13 ends the operation (step S134).

[Emotion state determination section]
FIG. 13 shows an operation flow of the emotion state determination unit 16 and its operation will be described. The emotional state determination unit 16 determines whether the information received from the video impression estimation unit 14 is video impression information (step S160). When it is video impression information (Yes in step S160), the emotion state determination unit 16 determines whether or not the information received from the user emotion estimation unit 13 is user emotion information (step S161). The order of determination may be user emotion information first.

  The emotion state determination unit 16 starts the operation when receiving both the video impression information and the user emotion information (Yes in step S161). The emotional state determination unit 16 determines the emotional state of the robot 5 using two pieces of information of video impression information and user emotion information.

  The reason for using two pieces of information in the emotional state determination unit 16 will be described. The reason for using the video impression information is that if the emotion of the robot 5 is determined only by the user emotion information, the robot 5 actively works on the user and makes the user laugh more when the emotional reaction of the user is small. This is because there arises a problem that proactive control for the user such as excitement cannot be performed. On the other hand, if it is attempted to determine the emotion of the robot 5 using only the video impression information, if the user's emotion and the impression estimated from the video impression information differ greatly, the reaction of the robot 5 cannot be sympathized with the user. This is because a problem occurs.

  For example, if the image has contents that provoke laughter but the user does not feel it interesting at all, the robot 5 continues to laugh regardless of the user's reaction. In order to avoid such a problem, in this embodiment, the emotional state of the robot 5 is determined using two pieces of information of video impression information and user emotion information.

  In order for the robot 5 to express human-like emotions that cause empathy for the user, it is necessary to determine an appropriate emotional state according to the video and the user situation and to express the emotional state to convey the emotional state. . Here, an appropriate emotional state means that a value representing the strength of each emotional element is appropriately determined for each emotional element that constitutes emotions such as joy, anger, sorrow, and comfort.

  For example, if the intensity of each emotional element is determined in the range of 0 to 1, the emotional state representing intense anger is joy = 0, anger = 1, sorrow = 0, comfort = 0. The emotional state representing small joy is determined as joy = 0.3, anger = 0, sorrow = 0, comfort = 0. If an appropriate emotional state can be determined, the robot 5 can perform emotional expression by facial expression, body movement, voice expression, etc. according to the emotional state.

  As a technique for mitigating or solving the above problem using two pieces of information, there is an intermediate value utilization technique, for example. Next, the intermediate value utilization method will be described.

[Method of using intermediate values]
In the intermediate value utilization technique, video impression information and user emotion information are mapped to the same kind of emotion state elements and value scales. Mapping can be realized by preparing a database describing mapping conversion correspondence. Mapping is performed as a set of elements of two types of emotional states related to each other. For example, when the emotional state elements are two sets of awakening-non-awakening and pleasant-uncomfortable, mapping conversion can be performed as shown in FIG. Here, the emotional state element is the type of emotion of the robot 5. Therefore, the emotion of the robot 5 when the elements of the emotional state are awakening-non-awakening and pleasant-uncomfortable are expressed by an arousal level (arousal value) and a comfort level (a pleasant value).

  The awakening-non-awaking and pleasant-uncomfortable group is based on Lassen's ring model. In addition to this example, the two axes representing emotional elements can be a combination of joy—grief, rage, and fear, and a longing—hate, vigilance—marvel.

  That is, the emotional state determination unit 16 arranges the image impression information and the user emotion information in the two-dimensional two-dimensional space with the elements of the emotional state for determining the emotional state of the robot 5 and the arranged video impression information and The value of the element of the emotion state between the user emotion information is determined as the emotion state of the robot 5.

  In this embodiment, an example in which two emotional state elements are used will be described. However, an axis representing an emotional element may be one set unit. That is, the intermediate value utilization method may be applied to one set (one axis) of awakening / non-wakening. Further, it may be expanded to four sets of joy-sadness, acceptance-disgust, fear-anger, surprise-expectation. In this way, the number of sets of two types of emotions related to each other can be expanded to n sets.

  In FIG. 14, input 1 and input 2 are two pieces of information. Input 1 shows an example in which, for example, the estimation result of the video impression information estimated by the video impression estimation unit 14 is represented by a single label representing the feeling of “anger”. Since the arousal value of “anger” is 0.85 and the pleasant value is −0.5, the estimation result α (−0.5, 0.85) is plotted on the coordinates on the two axes.

  Input 2 shows an example in which the estimation result of the user emotion information estimated by the user emotion estimation unit 13 is represented by a plurality of emotion elements and their degrees. Examples of estimation results of user emotion information are “joy (n = 5)” and “excitement (n = 2)”. Thus, when the estimation result is composed of a plurality of emotional elements and their levels, for each emotional element whose estimation result matches on the conversion formula rule, the value of that level is converted into an arousal value expression and a pleasant value expression. Find the assigned value. And the value which added each of the arousal value and pleasant value of the substitution result of each emotion element becomes an estimation result.

  In this example, the pleasant value = 2 * 0.05 + 0.10 + 5 * 0.08 + 0.35 = 0.95 and the arousal value = 2 * 0.10 + 0.35 + 5 * 0.05 + 0.25 = 1.05. In this embodiment, since a value of 1.0 or more is 1.0, the estimation result β (0.95, 1.0) is plotted on the two-axis coordinates.

  In the intermediate value utilization method, the emotional state of the robot 5 is determined as an intermediate value between the two pieces of information using an emotional state obtained by mapping two pieces of information of video impression information and user emotional information as shown in the following equation.

ここでRobotArousal_iとRobotPositive_iは時間ｉにおけるロボット５の感情状態の覚醒−非覚醒と快−不快要素の値を表す。時間ｉは、例えば上記の音声区間や映像区間に対応する時間である。式（１）は、感情状態の要素を覚醒−非覚醒と快−不快の２組とした場合の例である。

Here, RobotArousal _i and RobotPositive _i represent the values of the awakening-non-awaking and pleasant-unpleasant elements of the emotional state of the robot 5 at time i. The time i is, for example, a time corresponding to the above-described audio section or video section. Expression (1) is an example in the case where the elements of the emotional state are two sets of awakening-non-wakening and pleasant-uncomfortable.

  This emotional state element set may be n sets (n is an integer of 1 or more). For example, a combination of delight, grief, rage, and fear may be added to the set of elements of the emotional state to form four sets. The emotion of the robot 5 in that case is added with a level of joy-grief and a level of rage and fear. In this case, the expression (1) is expressed by four expressions.

MovieArousal _i and MoviePositive _i represent the values of awakening-non-awaking and pleasant-unpleasant elements of video impression information. UserArousal _i and UserPositive _i represent the values of arousal-non-awakening and pleasant-unpleasant elements of user emotion information. The val value is a value that defines how much the user emotion information or video impression information should be relied on, and is set to a value that takes a range of 0 ≦ val ≦ 1.

By using the intermediate value utilization method, the robot 5 proactively uses the estimated value of the video impression information even when the emotional state of the user is small, for example, when UserArousal _i and UserPositive _i are close to 0. It is possible to express emotions to the user. Moreover, even when the impression estimated from the user emotion information and the video impression information is largely different, it is possible to express the emotion in consideration of the user's reaction by appropriately setting the val value.

For example, when the val value is set to a large value, the image is a scene that arouses laughter, but in a scene where the user is not laughing at all, the robot 5 first expresses a reaction of laughing smallly, and the user responds to the subsequent reaction of the robot 5 When the robot 5 laughs, the robot 5 can control to strengthen the laughing reaction in accordance with changes in UserArousal _i and UserPositive _i .

  In short, the emotion state determination unit 16 inputs video impression information representing emotions aroused by a human when the human views the video, and user emotion information representing the emotion of the user who viewed the video. In the case where n types (n is an integer of 1 or more) of emotion types are set in advance with two types of emotions related to each other as a set, conversion rules prepared in advance are determined from the video impression information. A value indicating the magnitude of each type of emotion is generated, and from the user emotion information, a value indicating the size of each type of emotion is generated using the conversion rule. (1) Generated from the value obtained by multiplying a value for one emotion type of the set generated from the video impression information by a predetermined weight α (α = (1-val)) and the user emotion information About the type of emotion The sum of the values obtained by multiplying the value of the above by a predetermined weight β (β = val) is calculated as a value for the one emotion type of the set of the robot, and (2) generated from the video impression information A sum of a value obtained by multiplying the value of the other emotion type of the set by the weight α and a value obtained by multiplying the value of the other emotion type of the set generated from the user emotion information by the weight β. Calculated as a value for the other emotion type of the set of robots.

  Next, a method for further using the line-of-sight direction of the robot 5 as the intermediate value utilization method will be described.

[Method using robot's gaze direction]
Video-related information and user information used by the robot 5 to express emotions can be acquired regardless of which direction the robot 5 is facing. On the other hand, people often obtain surrounding information visually. When people interact with each other, communication is made on the premise that the other party is performing visual information processing.

  For this reason, if the robot 5 is input information in a direction that the robot 5 is not looking at and expresses emotions, the model assumed for the information processing system of the robot 5 and the person with whom the person interacts is inconsistent. It becomes an obstacle. Therefore, there is a method of reducing obstacles in empathy by making human-like and more empathetic reaction by setting which information of video related information or user information is weighted and using the robot's gaze direction. It is done.

An example of a method for realizing the method using the line-of-sight direction of the robot 5 can be realized by setting the val value, RobotArousal _i, and RobotPositive _i described in the intermediate value using method as shown in the following equation, for example.

ここでRobotView_iは時間ｉにおけるロボット５の視線方向を表す。Movie,User,Noneはそれぞれ映像表示デバイス２、ユーザ、その他の方向に向けたロボット５の視線方向を表すラベルである。また、val_see-movie,val_nosee-movie,val_nosee-user,val_nosee-userはロボット５の振る舞いを決める変数である。これらの変数は、0≦val_see-movie≦1,0≦val_nosee-movie≦1,0≦val_nosee-user≦1,0≦val_nosee-user≦1の範囲を取る。ロボット５に人らしい反応を行わせるためには、これらの変数の値を適切な値に設定すればよい。なお、ロボットの視線方向を用いる場合でも、上記と同様に感情状態の要素の組を増やすことができる。感情状態の要素を増やした場合の式（２）は、上記の式（１）と同様に感情状態の要素の組の数分増加することになる。

Here, RobotView _i represents the viewing direction of the robot 5 at time i. Movie, User, and None are labels indicating the line-of-sight direction of the robot 5 toward the video display device 2, the user, and other directions, respectively. In addition, val _see-movie , val _nosee-movie , val _nosee-user , and val _nosee-user are variables that determine the behavior of the robot 5. These variables take _{0 ≦ val see-movie ≦ 1,0} ≦ val nosee-movie ≦ 1,0 ≦ val nosee-user ≦ 1,0 ≦ val nosee-user ≦ 1 range. In order to cause the robot 5 to perform a human-like reaction, the values of these variables may be set to appropriate values. Even when the robot's line-of-sight direction is used, the set of emotional state elements can be increased as described above. Equation (2) when the number of emotional state elements is increased is increased by the number of sets of emotional state elements in the same manner as the above equation (1).

  When the emotional state determination unit 16 receives the video impression information and the user emotion information, the emotional state determination unit 16 first converts the video impression information into a pair of arousal value and pleasant value based on the above intermediate value utilization method (step S162). The conversion method searches for the emotion element item of the conversion rule stored in the emotion state conversion rule base 18 that matches each emotion element of the video impression information, and for the awakening value expression and the pleasant expression of the matching conversion rule. Then, the intensity value of the emotion element of the video impression information is substituted into the variable n of both conversion equations.

The search and substitution processes are repeated for all emotion elements whose image impression information intensity is not 0, and the sum of the results of substitution of all awakening expressions and pleasant expressions is used as an arousal value and a pleasant value. However, if the result of addition is −1 or less, −1 is set, and if the result of addition is 1 or more, 1 is set as a value of an arousal value or a pleasant value. A variable representing the arousal value obtained by converting the video impression information is MovieArousal _i , and a variable representing the pleasant value is MoviePositive _i .

  The emotion state determination unit 16 then converts the user emotion information into a pair of arousal value and pleasant value (step S163). The conversion method is the same as the video impression information.

  The process of converting user emotion information into a pair of arousal value and pleasant value will be described with a specific example. For example, it is assumed that the emotion element and the intensity n are “joy, n = 5” and “excitement, n = 4”. In that case, “n * 0.08 + 0.45” of “joy” and “n * 0.10 + 0.10” of “excitement” are used as the pleasant expression of the conversion rule stored in the emotion state conversion rule base 18 (see FIG. 3). . Therefore, the pleasant value is calculated as the pleasant value = 5 * 0.08 + 0.45 + 4 * 0.10 + 0.10 = 1.35.

The arousal value formula uses “n * 0.05 + 0.25” for “joy” and “n * 0.15 + 0.35” for “excitement”. Therefore, the arousal value is calculated as arousal value = 5 * 0.05 + 0.25 + 4 * 0.15 + 0.35 = 1.45. The result of addition is a pleasant value = 1.35 and an arousal value = 1.45, and since each is 1 or more, the pleasant value = 1.0 and the arousal value = 1.0. Therefore, the above variables UserArousal _i = 1.0 and UserPositive _i = 1.0.

Here wake values obtained from the image impression information (MovieArousal _i) 0, when the pleasure value (MoviePositive _i) assuming 0, arousal value of emotional state of the robot 5 (RobotArousal _i) and rice (RobotPositive _i) is It can be calculated by substituting the value of each variable into the above equation (1).

Assuming that val = 0.5 is a value that defines how much user emotion information or video impression information should be relied upon, RobotArousal _i of the robot 5 emotion state is RobotArousal _i = (1-0.5) * in this example. 0 + 0.5 + 1.0 = 0.5, RobotPositive _i is generated as RobotPositive _i = (1-0.5) * 0 + 0.5 + 1.0 = 0.5 (step S164).

The emotional state determination unit 16 ends the operation when it transmits the RobotArousal _i and RobotPositive _i representing the generated emotional state of the robot 5 to the emotional expression generation unit 17 (step S166). The process of step S165 illustrated in FIG. 11 is performed when robot line-of-sight information is received.

  When the robot line-of-sight information is input, the emotion state determination unit 16 increases the weight (1-val) of the video impression information if the robot line-of-sight information indicates the direction of the video, and the user if the robot's line-of-sight information indicates the direction of the user. The emotional state of the robot 5 is determined by increasing the weight (val) of the emotion information.

When the emotion state determination unit 16 receives the robot gaze information indicating the gaze direction of the robot 5 from the emotion expression generation unit 17, the emotion state determination unit 16 sets a gaze target variable (RobotView _i ). When the line-of-sight direction is the video display device 2, RobotView _i = Movie is set. When the line-of-sight direction is the user, RobotView _i = User is set. If the line-of-sight direction is neither the video display device 2 nor the user, RobotView _i = None is set.

When the line-of-sight target variable is set, the emotional state determination unit 16 generates RobotArousal _i and RobotPositive _i of the emotional state of the robot 5 using the above equation (2) (step S164). The arousal value and pleasant value obtained from the video impression information are MovieArousal _i and MoviePositive _i , respectively, the arousal value and pleasant value obtained from the user emotion information are UserArousal _i and UserPositive _i , respectively, and val _see-movie = 0.8, val _{nosee-movie = 0.2, val see} -user = 0.8, val as _nosee-user = 0.2 by using the above equation (2) determining the wakefulness value RobotArousal _i and rice RobotPositive _i emotional state of the robot 5.
Here, the _values of val _see-movie = 0.8, val _nosee-movie = 0.2, val _see-user = 0.8, val _nosee-user = 0.2 are examples.

When the gaze target variable is set in this way, the emotional state determination unit 16 determines the video impression that is predetermined for the case where the robot gaze information indicates the direction of the video when the robot gaze information recording direction is indicated. If the weight of information and the weight of user emotion information are set to weight α (val _see-movie ) and weight β (val _nosee-user ), respectively, and the robot gaze information indicates the direction of the user, the robot gaze information The weight of the video impression information and the weight of the user emotion information predetermined for the case of indicating the direction are set to the weight α (val _nosee-movie ) and the weight β (val _see-user ), respectively. The direction of the video impression information and the weight of the user emotion information determined in advance for the case where the robot line-of-sight information indicates other directions are weight α (val _nosee-movie ) and Set weight β (val _nosee-user ).

[Emotion expression generator]
FIG. 15 shows an operation flow of the emotion expression generation unit 17 and its operation will be described. When the emotion expression generation unit 17 receives the emotion state of the robot 5 from the emotion state determination unit 16, the emotion expression generation unit 17 starts the operation (Yes in step S170). When the emotion expression generation unit 17 starts to operate, the voice expression database 19 is referred to using the arousal value and the pleasant value of the emotion state of the robot 5, and the voice data emitted by the robot 5 is determined (step S171).

Here determined is to determine the audio data referring satisfies the following formula awake values of certain audio data O _n from the VoiceArousalo _n, all the audio data O _N when the pleasure value was VoicePositiveo _n ( (See FIG. 4). Although the voice expression database 19 may have the voice data itself, in the present embodiment, the voice expression database 19 stores a voice file path representing a place where the voice data is stored.

次に、感情表現生成部１７はロボット５の感情状態の覚醒値と快値を用いて身体表現データベース２０を参照し、ロボット５を駆動する駆動制御情報を決定する（ステップＳ１７２）。ここで決定とは、ある駆動制御情報K_nの覚醒値をMotionArousal_Kn、快値をMotionPositive_Knとしたときの全駆動制御情報K_Nの中から次式を満たす参照する駆動制御情報を決定することである（図５参照）。

Next, the emotional expression generating unit 17 refers to the body expression database 20 using the arousal value and the pleasant value of the emotional state of the robot 5, and determines drive control information for driving the robot 5 (step S172). Here decided is to determine certain MotionArousal _Kn arousal value of the drive control information K _n, drive control information for referencing satisfies the following expression from all drive control information K _N when the pleasure value was MotionPositive _Kn (See FIG. 5).

参照する駆動部制御情報が決定すると、感情表現生成部１７は決定した駆動部制御情報に含まれる視線対象の方位角と仰俯角を方位情報として参照する（ステップＳ１７３）。視線対象が例えば映像表示デバイス２であれば、駆動制御関数の頭部方位角dxと頭部仰俯角dyに、195.6°（図６参照）と-15.1°を代入する（ステップＳ１７４）。

When the driving unit control information to be referred to is determined, the emotion expression generating unit 17 refers to the azimuth angle and the elevation angle of the line-of-sight target included in the determined driving unit control information as the azimuth information (step S173). If the line-of-sight target is, for example, the video display device 2, 195.6 ° (see FIG. 6) and −15.1 ° are substituted into the head azimuth dx and head elevation angle dy of the drive control function (step S174).

  When the emotion state determination unit 16 uses the robot line-of-sight information, the emotion expression generation unit 17 transmits the line-of-sight target included in the drive unit control information to the emotion state determination unit 16 as the robot line-of-sight information (step S175). The drive control function and audio file path in which the angles are substituted for the head azimuth angle dx and the head elevation angle dy are transmitted to the robot 5 as drive control commands and audio data (step S176).

  The transmission of the drive control command and audio data is repeated every time set as the sequence movement interval of the drive control function. For example, when the drive unit control information in the first row of the body representation database 20 (FIG. 5) is referenced, the robot 5's gaze direction remains the video display device 2 and the right arm tilt angle of the robot 5 is 0 °. After the state of the left arm tilt angle of 0 ° continues for 20 seconds, the state changes to the state of the right arm tilt angle of 30 ° and the left arm tilt angle of 30 °.

  The emotion expression generation unit 17 generates the contents of the physical expression and the utterance expression of the robot 5 based on the emotional state of the robot 5. The physical expression is an operation of the robot 5 described in the operation name stored in the body expression database 20 (FIG. 5). The utterance expression is a voice such as “Wow” stored in the voice expression database 19.

  Note that the facial expression of the robot 5 may be included in the physical expression. In that case, the emotion expression generation unit 17 also generates the contents of the expression expression together with the physical expression and the utterance expression. The emotion expression generation unit 17 may dynamically generate the physical expression, the expression expression, and the utterance expression of the robot 5 based on the emotion state of the robot 5 or hold them in advance corresponding to the emotion state. A method using physical expression, facial expression and speech expression may be used. Also, the example of the emotional state elements for determining the emotional state of the robot 5 has been described as an example of two sets of awakening-non-wakening and pleasant-uncomfortable. The number may be n.

  As described above, according to the present embodiment, when the robot 5 views a video together with the user, the robot 5 performs more emotional expression so that the user has empathy for the robot 5 and views the video. It is possible to enjoy more richly.

1: Robot control device 11: Video related information collection unit 12: User information collection unit 13: User emotion estimation unit 14: Video impression estimation unit 15: Emotion word dictionary 16: Emotion state determination unit 17: Emotion expression generation unit 18: Emotion State conversion rule base 19: voice expression database 20: body expression database 2: video display device 3: camera 4: microphone 5: robot 6: position information acquisition server 60: direction information database

Claims (6)

A robot control device that causes a robot to perform an operation such as viewing a video with a user,
Video impression information representing emotions aroused by a human when he / she sees the video and user emotion information representing the emotions of the user who viewed the video are input, and two types of emotions related to each other are input. In the case where n types (n is an integer of 1 or more) of emotion types are set in advance, the type of emotion is determined from the video impression information using a conversion rule prepared in advance. A value indicating a size is generated, and a value indicating the size of each type of emotion is generated from the user emotion information using the conversion rule. For each set, (1) from the video impression information A weight obtained by multiplying a value obtained by multiplying a value for one emotion type of the set by a predetermined weight α and a value for the one emotion type of the set generated from the user emotion information. The sum of the values multiplied by β Calculated as a value for the one emotion type of the set of the robot, and (2) a value obtained by multiplying the value for the other emotion type of the set generated from the video impression information by the weight α and the user Emotion state determination unit that calculates a sum of values obtained by multiplying the value of the other emotion type of the set generated from emotion information by the weight β as a value of the other emotion type of the set of the robot A robot control device comprising: The robot control device according to claim 1,
The emotional state determination unit
The robot control apparatus according to claim 1, wherein the weight β is included in a range of 0 ≦ β ≦ 1, and the weight α is α = (1−β). The robot control device according to claim 1,
The robot gaze information indicating the robot gaze direction is input. If the robot gaze information indicates the direction of the video, α> β, and if the robot gaze information indicates the user direction, β> α. A robot controller characterized by being. The robot controller according to claim 3, wherein
The emotional state determination unit
When the robot line-of-sight information indicates the direction of the video, the weight α and the weight of user emotion information predetermined for the case where the robot line-of-sight information indicates the direction of the video are respectively set to the weight α and Set to the weight β,
When the robot line-of-sight information indicates the direction of the user, a weight of video impression information and a weight of user emotion information predetermined for the case where the robot line-of-sight information indicates the direction of the user are set as the weight α and Set to the weight β,
When the robot line-of-sight information indicates the other direction, the weight α and the weight are set as the weight of the video impression information and the weight of the user emotion information that are predetermined for the case where the robot line-of-sight information indicates the other direction, respectively. A robot controller characterized by being set to β. A robot control method performed by a robot control device that causes a robot to perform an operation such as viewing a video with a user,
Video impression information representing emotions aroused by a human when he / she sees the video and user emotion information representing the emotions of the user who viewed the video are input, and two types of emotions related to each other are input. In the case where n types (n is an integer of 1 or more) of emotion types are set in advance, the type of emotion is determined from the video impression information using a conversion rule prepared in advance. A value indicating a size is generated, and a value indicating the size of each type of emotion is generated from the user emotion information using the conversion rule. For each set, (1) from the video impression information A weight obtained by multiplying a value obtained by multiplying a value for one emotion type of the set by a predetermined weight α and a value for the one emotion type of the set generated from the user emotion information. The sum of the values multiplied by β Calculated as a value for the one emotion type of the set of the robot, and (2) a value obtained by multiplying the value for the other emotion type of the set generated from the video impression information by the weight α and the user Calculating a sum of values obtained by multiplying a value for the other emotion type of the set generated from the emotion information by the weight β as a value for the other emotion type of the set of the robot, Robot control method.   A robot control program for causing a computer to function as the robot control apparatus according to claim 1.

Images