JP2016012341A - Action control system, system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem of incapability of determining an adequate action to a state of a user.SOLUTION: An action control system includes: an action information storage part which stores determination information for determining an action on the basis of a state of a user; a user state recognition part which recognizes the state of the user; an action determination part which determines an action to be executed based on the state of the user recognized by the user state recognition part and determination information; a response recognition part which recognizes a response of the user to execution of the action determined by the action determination part; a transmission part which transmits the state of the user recognized by the user state recognition part, the action determined by the determination part, and the response of the user to the action recognized by the response recognition part, to an external server; a receiving part which receives an action appropriate to the state of the user recognized by the user state recognition part from a server when the action determination part is not able to determine the action to be executed based on the state of the user and the determination information; and an action information updating part which updates the determination information on the basis of the action received by the receiving part from the server.

Description

  The present invention relates to a behavior control system, a system, and a program.

There has been known a terminal that learns a conversation between a user and a call partner and accumulates a response of the call partner in response to the user's inquiry in a response table (see, for example, Patent Document 1).
[Prior art documents]
[Patent Literature]
[Patent Document 1] Japanese Patent Application Laid-Open No. 2011-253389

  Appropriate behavior may not be determined for the user's condition.

  In the first aspect of the present invention, an action information storage unit that stores determination information for determining an action based on a user state, a user state recognition unit that recognizes a user state, and a user state recognition unit include: Based on the recognized user state and the determination information, an action determination unit that determines an action to be executed, a reaction recognition unit that recognizes a user's reaction to the execution of the action determined by the action determination unit, and a user state The transmission unit that transmits the user status recognized by the recognition unit, the behavior determined by the behavior determination unit, and the user response recognized by the reaction recognition unit to an external server, and the behavior determination unit includes the user status and determination information If the action to be executed based on the condition cannot be determined, the receiver that receives the action appropriate for the user state recognized by the user state recognizer from the server, and the action that the receiver receives from the server Based on, behavior control system and a behavior information updating unit that updates the decision information is provided.

  Based on the state of the user recognized by the user state recognition unit, an emotion determination unit that determines an emotion value indicating the current emotion of the system, and an action determined by the behavior determination unit based on the emotion value determined by the emotion determination unit And an action control unit that determines the execution form.

  The transmission unit further transmits the user attribute to the server, and the reception unit recognizes the user state recognition unit when the behavior determination unit cannot determine the action to be executed based on the user state and the determination information. Actions appropriate for the user's condition and user attributes may be received from the server.

  The decision information includes information indicating a condition indicating the state of the user to be satisfied when performing the action, and information indicating the action associated with the condition, and the action control system determines the user state recognized by the user state recognition unit. And a condition determining unit that determines whether or not a condition included in the determination information is satisfied, and the receiving unit determines that any condition included in the determination information is not satisfied by the condition determining unit. When it is, you may receive the action suitable for the user's state which the user state recognition part recognized from the server.

  The determination information includes information indicating the appropriateness of each of the one or more actions, and the action determination unit is associated with the condition determined to be satisfied by the condition determination unit among the information included in the determination information. Among the one or more actions, an action having a degree of appropriateness included in the decision information higher than a predetermined value is determined as an action to be executed, and the receiving unit sets the condition determined to be satisfied by the condition determining unit. When the appropriateness of one or more associated actions is less than or equal to a predetermined value, an action suitable for the user state recognized by the user state recognition unit may be received from the server.

  The behavior information update unit may add the behavior received from the server by the reception unit to the determination information in association with the condition determined to be satisfied by the condition determination unit.

  The behavior information update unit is determined in association with the condition determined to be satisfied by the condition determination unit and the action determined by the behavior determination unit when the reaction of the user recognized by the reaction recognition unit is poor. You may reduce the appropriateness memorize | stored in.

  In the 2nd aspect of this invention, the program for functioning a computer as said behavior control system is provided.

  In the 3rd aspect of this invention, a system provided with said some action control system and a server is provided.

  In the fourth aspect of the present invention, the state of the user recognized in the first behavior control system, the behavior determined based on the state of the user in the first behavior control system, and the behavior are executed. A user response receiving unit that receives the user's response to the first behavior control system, and a user result receiving unit that receives the user response, the behavior and the result of the user's response And the user status recognized by the second behavior control system in association with the user status receiving unit that receives from the second behavior control system and the status that matches the user status received by the user status receiving unit. Based on the user's reaction stored in the action result storage unit, the action result description is associated with the state that matches the user state received by the user state reception unit. The behavior selection unit that selects a behavior suitable for the user's state received by the user status reception unit from the behaviors stored in the unit, and the behavior selected by the behavior selection unit are transmitted to the second behavior control system A server including a transmitting unit is provided.

  In the fifth aspect of the present invention, a program for causing a computer to function as the server is provided.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

An example of system 5 concerning this embodiment is shown roughly. The action which the robot 100 performs with respect to the user 10 is shown schematically. A scene in which the robot 100 executes data recording is schematically shown. 1 schematically shows a functional configuration of a robot 100. The function structure of the server 200 is shown schematically. The information perceived by the perception processing unit 416 is schematically shown. The information understood by the semantic understanding unit 418 is schematically shown. Information included in the reaction rule 421 is shown in a table format. The process based on a reaction rule is shown schematically. An example of the operation | movement flow by the robot 100 is shown schematically. An example of the reaction rule statistical information 522 stored in the server 200 is shown. The action performed in order to determine the label incidental to the information recorded on the server 200 is schematically shown. An example of the recording format switching rule 471 for switching the recording format is shown. An example of the operation | movement flow regarding switching of a recording format is shown roughly.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 schematically shows an example of a system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 200. The user 10 a, the user 10 b, the user 10 c, and the user 10 d are users of the robot 100. The user 11a, the user 11b, and the user 11c are users of the robot 101. The user 12 a and the user 12 b are users of the robot 102. In the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. In addition, the user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. In addition, the user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same function as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

  The robot 100 has a conversation with the user 10 and provides video to the user 10. At this time, the robot 100 cooperates with the server 200 or the like that can communicate via the communication network 20 and provides a conversation with the user 10 and a video to the user 10. For example, the robot 100 not only learns an appropriate conversation by itself, but also learns so that the conversation with the user 10 can be advanced more appropriately in cooperation with the server 200. In addition, the robot 100 causes the server 200 to record the captured video data of the user 10, requests the video data from the server 200 as necessary, and provides the user 10 with the video data.

  The robot 100 also has an emotion value that represents the type of emotion of the robot 100. For example, the robot 100 has “joy”, “anger”, “sorrow”, “easy”, “pleasant”, “unpleasant”, “relief”, “anxiety”, “sadness”, “excitement”, “worry”, It has emotion values that represent the emotional strengths of “Relief”, “Fulfillment”, “Nonsense” and “Normal”. For example, when talking with the user 10 in a state where the emotion value of excitement is large, the robot 100 emits a voice at a high speed. In this way, the robot 100 can express its own emotions with actions.

  The robot 100 has a function of recognizing the emotion of the user 10. The robot 100 recognizes the emotion of the user 10 by analyzing the face image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function. The robot 100 determines an action to be executed by the robot 100 based on the recognized emotion of the user 10 and the like. For example, when the robot 100 recognizes that the user 10 is likely to be lonely, the robot 100 performs an action such as calling the user 10.

  The robot 100 stores rules that define actions to be executed by the robot 100 based on the emotions of the user 10, self-emotions, and the like, and performs various actions according to the rules. An example of the action based on the rule will be schematically described with reference to FIGS.

  FIG. 2 schematically shows actions that the robot 100 performs on the user 10. The robot 100 has a reaction rule for determining an action based on the emotion of the user 10. According to the illustrated reaction rule, when the emotion of the robot 100 is “ordinary” and the condition of the user is “one person seems to be lonely”, the emotion of the robot 100 is “worried”. It is stipulated that the content of emotion change and the action of “calling out” can be executed. Here, the action is associated with a consideration point representing the strength of the consideration of the user 10.

  If the robot 100 recognizes that the current emotion of the robot 100 is “ordinary” and the user 10 is in a state of being lonely based on the reaction rule, the emotion of the robot 100 “anxiety”. Increase the value. In addition, the robot 100 selects the action of “calling a voice” defined in the reaction rule as an action to be performed on the user 10. For example, when the robot 100 selects the action of “calling a voice”, the robot 100 converts the word “What's wrong?” Indicating worried into a worried voice and outputs it.

  The robot 100 recognizes a change in the emotion of the user 10 after outputting the voice to the user 10. For example, when the user 10 replies “Thank you! All right!” After the robot 100 outputs the voice “What was it?”, The robot 100 analyzes the reply of the user 10 by voice analysis. Recognize that feelings improved. Since the emotion of the user 10 is improved by the action of “calling out”, the robot 100 determines that the action of “calling out” is a highly considerate action, and the action of “calling out” Increase compassion points. Thereby, the robot 100 can learn an action appropriate for the user 10.

  In addition, the robot 100 transmits user response information indicating that a positive response is obtained from the user 10 to the server 200 by this action. The user response information includes, for example, the condition “one person seems to be lonely”, the action “call out”, the user 10's reaction being positive, and the user's 10 attributes.

  The server 200 stores user reaction information received from the robot 100. The server 200 receives and stores user reaction information not only from the robot 100 but also from each of the robot 101 and the robot 102. Then, the server 200 analyzes user reaction information from the robot 100, the robot 101, and the robot 102, and generates a recommended reaction rule.

  The robot 100 receives the recommended reaction rule from the server 200 by inquiring the server 200 for the recommended reaction rule. The robot 100 incorporates the recommended reaction rule into the reaction rule stored in the robot 100. Thereby, the robot 100 can incorporate the reaction rule acquired by the robot 101, the robot 102, or the like into its own reaction rule.

  FIG. 3 schematically shows a scene in which the robot 100 executes switching of the recording format of data that the server 200 records. The robot 100 has a recording format switching rule for switching the recording format based on the emotion of the user 10. According to the recording format switching rule shown in the figure, the data recording format is set to “low compression” when the conditions “there are more than a certain number of families” and “a certain number of family smiles are detected” are satisfied. The action of “switch to form” is defined.

  The robot 100 continuously transmits high-compressed data such as the skeleton data of the user 10 to the server 200 and records it in the server 200 during a period when the conditions of the recording format switching rule are not satisfied. The skeleton data is an example of shape data that represents the shape of the user. When the robot 100 determines that the condition of the recording format switching rule is satisfied, the robot 100 starts transmission of full HD video data and audio data, and performs low compression including full HD video data and audio data in addition to the skeleton data. Data is recorded on the server 200.

  Then, when providing the user 10 with some fun video, the robot 100 requests the server 200 to transmit full HD video data and audio data, and provides the video data received from the server 200 to the user 10. To do. Thus, when the robot 100 encounters a scene that is strongly memorable, the robot 100 can accumulate high-quality video data of the scene in the server 200.

  FIG. 4 schematically shows a functional configuration of the robot 100. The robot 100 includes a sensor unit 400, a sensor module unit 410, a storage unit 420, a user state recognition unit 430, a condition determination unit 434, a user reaction recognition unit 432, an action determination unit 440, and an emotion determination unit 442. A rule update unit 422, a communication processing unit 480, a recording data generation unit 470, a switching control unit 477, a label generation unit 478, and a control object 452.

  The control object 452 includes a display device, a speaker, an eye LED, a motor that drives arms, hands, legs, and the like. The posture and gesture of the robot 100 are controlled by controlling motors such as arms, hands, and feet. A part of the emotion of the robot 100 can be expressed by controlling these motors. Also, a part of the emotion of the robot 100 can be expressed by controlling the light emission state of the LED of the eye part of the robot 100. Note that the posture, gesture, and facial expression of the robot 100 are examples of the attitude of the robot 100.

  The sensor unit 400 includes a microphone 401, a 3D depth sensor 402, a 2D camera 403, and a distance sensor 404. The microphone 401 continuously detects sound and outputs sound data. The microphone 401 may be provided on the head of the robot 100 and have a function of performing binaural recording. The 3D depth sensor 402 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by an infrared camera. The 2D camera 403 is an example of an image sensor. The 2D camera 403 shoots with visible light and generates visible light video information. The distance sensor detects the distance to the object by irradiating, for example, a laser or an ultrasonic wave. In addition, the sensor unit 400 may include a timepiece, a gyro sensor, a touch sensor, a motor feedback sensor, and the like.

  Note that, among the components of the robot 100 illustrated in FIG. 4, the components other than the control target 452 and the sensor unit 400 are examples of components included in the behavior control system included in the robot 100. The behavior control system of the robot 100 sets the control target 452 as a control target.

  The storage unit 420 includes a reaction rule 421, a perception confirmation rule 417, a meaning understanding rule 419, a recording format switching rule 471, and a person DB 424. At least a part of the storage unit 420 is implemented by a storage medium such as a memory. The person DB 424 stores the face image of the user 10, attribute information of the user 10, and the like. Among the components of the robot 100 shown in FIG. 4, the functions of the components other than the control object 452, the sensor unit 400, and the storage unit 420 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by a basic software (OS) and a program operating on the OS.

  The sensor module unit 410 includes a voice emotion recognition unit 411, an utterance understanding unit 412, a facial expression recognition unit 413, and a face recognition unit 414. Information detected by the sensor unit 400 is input to the sensor module unit 410. The sensor module unit 410 analyzes information detected by the sensor unit 400 and outputs an analysis result to the user state recognition unit 430.

  The voice emotion recognition unit 411 of the sensor module unit 410 analyzes the voice of the user 10 detected by the microphone 401 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 411 extracts a feature quantity such as a frequency component of voice and recognizes the emotion of the user 10 based on the extracted feature quantity. The utterance understanding unit 412 analyzes the voice of the user 10 detected by the microphone 401 and outputs character information representing the utterance content of the user 10.

  The facial expression recognition unit 413 recognizes the facial expression of the user 10 and the emotion of the user 10 from the image of the user 10 captured by the 2D camera 403. For example, the facial expression recognition unit 413 recognizes the facial expression and emotion of the user 10 based on the shape and positional relationship of the eyes and mouth.

  The face recognition unit 414 recognizes the face of the user 10. The face recognition unit 414 recognizes the user 10 by matching the face image stored in the person DB 424 with the face image of the user 10 taken by the 2D camera 403.

  The user state recognition unit 430 recognizes the state of the user 10. The user state recognition unit 430 includes a perception processing unit 416 and a semantic understanding unit 418. Information analyzed by the sensor module unit 410 is output to the perception processing unit 416. The perception processing unit 416 mainly performs processing related to perception using the analysis result of the sensor module unit 410 and the perception confirmation rule 417 of the storage unit 420. For example, the perception processing unit 416 generates perception information such as “a daddy is one person” and “a daddy has a 90% probability of not smiling”.

  The meaning understanding unit 418 performs processing for understanding the meaning of the perceptual information generated by the perception processing unit 416 using the meaning understanding rules 419 of the storage unit 420. For example, the semantic understanding unit 418 generates semantic information such as “One daddy seems lonely.” Note that perceptual information and semantic information are examples of information representing the state of the user 10. Here, the information representing the state of the user 10 may include information representing the emotion of the user 10.

  The semantic information generated by the semantic understanding unit 418 is output to the condition determination unit 434, the user reaction recognition unit 432, and the label generation unit 478.

  The condition determining unit 434 determines that the condition defined in the reaction rule 421 is satisfied based on the semantic information and the reaction rule 421 in the storage unit 420. The reaction rule 421 is an example of determination information for determining the action of the robot 100. As described above, the reaction rule 421 associates the plurality of actions and the consideration points of the plurality of actions. The reaction rule 421 includes information for determining an action based on an emotion value indicating an emotion in the robot 100 and the state of the user 10. As described above, the storage unit 420 functions as an example of an action information storage unit that stores information for determining an action based on the state of the user.

  The condition determination unit 434 determines that the condition stored by the reaction rule 421 is satisfied. The action determination unit 440 determines an action to be executed from among a plurality of actions stored in the reaction rule 421 in association with the condition determined to be satisfied by the condition determination unit 434. Specifically, the action determination unit 440 executes based on each consideration point of the plurality of actions stored in the reaction rule 421 in association with the condition determined to be satisfied by the condition determination unit 434. Decide what action to take. Thus, the action determination unit 440 determines an action to be executed based on the state of the user 10 recognized by the user state recognition unit 430 and the information of the reaction rule 421.

  The emotion determination unit 442 determines an emotion value indicating the current emotion of the robot 100 based on the semantic information generated by the meaning understanding unit 418. As described above, the emotion determination unit 442 determines an emotion value indicating the current emotion of the robot 100 based on the user state recognized by the user state recognition unit 430.

  The behavior control unit 450 controls the control object 452 based on the behavior determined by the behavior determination unit 440. For example, the behavior control unit 450 outputs a sound from a speaker included in the control target 452 when the behavior determination unit 440 determines to speak. At this time, the behavior control unit 450 determines the voice production speed based on the emotion value updated by the emotion determination unit 442. For example, the behavior control unit 450 determines a faster generation rate as the emotion value of excitement is larger. As described above, the behavior control unit 450 determines the execution form of the behavior determined by the behavior determination unit 440 based on the emotion value determined by the emotion determination unit 442.

  The user reaction recognition unit 432 recognizes a change in the emotion of the user 10 with respect to the execution of the action determined by the action determination unit 440. Note that the change in emotion of the user 10 is an example of the reaction of the user 10. That is, the user reaction recognition unit 432 recognizes the reaction of the user 10 to the execution of the action determined by the action determination unit 440. Note that the user reaction recognition unit 432 may recognize a change in emotion based on the voice or facial expression of the user 10. In addition, the user reaction recognition unit 432 may recognize a change in the emotion of the user 10 based on the detection of the impact by the touch sensor included in the sensor unit 400. For example, the user reaction recognition unit 432 may recognize that the emotion of the user 10 has deteriorated when an impact is detected by a touch sensor included in the sensor unit 400.

  Information indicating the reaction of the user 10 recognized by the user reaction recognition unit 432 is output to the emotion determination unit 442, the communication processing unit 480, and the label generation unit 478. As described above, the emotion determination unit 442 may increase the emotion value of sadness according to the reaction rule 421 when the user 10 is recognized to be lonely by the user state recognition unit 430. When the user reaction recognition unit 432 recognizes that the user 10 is smiling, the joy emotion value may be increased by the reaction rule 421. Thus, the emotion determination unit 442 updates the emotion value based on at least one of the user's state recognized by the user state recognition unit 430 and the user's reaction to the action determined by the behavior determination unit 440. .

  Also, the rule update unit 422 changes the consideration points stored by the reaction rule 421 in association with the action determined by the action determination unit 440 based on the change in the emotion of the user 10 recognized by the user reaction recognition unit 432. To do. Specifically, the rule update unit 422 associates the reaction rule 421 with the condition determined to be satisfied by the condition determination unit 434 based on the change in the emotion of the user 10 recognized by the user reaction recognition unit 432. The consideration point of the action determined by the action determination unit 440 is changed among the consideration points of the plurality of actions stored in the above. For example, when the user reaction recognition unit 432 recognizes that the emotion of the user 10 has improved, the rule update unit 422 determines that the condition and action determination unit 440 determined to be satisfied by the condition determination unit 434 The consideration point stored by the reaction rule 421 is increased in association with the determined action. On the other hand, the rule update unit 422 determines that the condition and action determination unit 440 determined to be satisfied by the condition determination unit 434 when the user reaction recognition unit 432 recognizes that the emotion of the user 10 has deteriorated. The consideration point stored in the reaction rule 421 in association with the determined action is reduced.

  The communication processing unit 480 is responsible for communication with the server 200. As described above, the communication processing unit 480 transmits user reaction information to the server 200. As described above, the communication processing unit 480 transmits the state of the user 10 recognized by the user state recognition unit 430, the action determined by the behavior determination unit 440, and the reaction of the user 10 recognized by the user reaction recognition unit 432 to the server 200. Send. Further, the communication processing unit 480 receives a recommended reaction rule from the server 200 when the behavior determination unit 440 cannot determine an action to be executed based on the state of the user 10 and the reaction rule 421. In this way, the communication processing unit 480 determines the state of the user 10 recognized by the user state recognition unit 430 when the behavior determination unit 440 cannot determine the action to be executed based on the state of the user 10 and the reaction rule 421. An action suitable for the server is received from the server 200.

  The rule update unit 422 updates the reaction rule 421 based on the recommended reaction rule received from the server 200 by the communication processing unit 480. As described above, the rule update unit 422 updates the reaction rule 421 based on the behavior received from the server 200 by the communication processing unit 480.

  In addition, the emotion determination part 442, when the action determination part 440 cannot determine the action with respect to the user based on the state of the user 10 recognized by the user state recognition part 430, the emotion value indicating the intensity of sadness You may change it to a value that represents the intensity of greater sadness.

  The recording data generation unit 470 generates data to be recorded on the server 200 based on the information detected by the sensor unit 400. The communication processing unit 480 transmits information generated by the recording data generation unit 470 to an external server and causes the server to record the information.

  For example, the recording data generation unit 470 continuously generates skeleton data for recording based on the skeleton information continuously detected by the 3D depth sensor 402. Further, the recording data generation unit 470 generates recording video data based on the video information captured by the 2D camera 403. For example, the recording data generation unit 470 generates full HD video data. The recording data generation unit 470 generates recording audio data based on the audio information continuously detected by the microphone 401.

  Here, the video data is, for example, full HD video data and has a larger amount of information than the skeleton data. Therefore, the skeleton data can be regarded as high-compression data, and the video data can be regarded as low-compression data. Thus, the skeleton data is an example of information in the first recording format, and the video data is an example of information in the second recording format that has a larger amount of information than the information in the first recording format. As described above, the recording data generation unit 470 continuously processes at least a part of the information detected by the sensor that continuously detects the user 10, so that the first recording format information or the first recording format information is obtained. The information of the second recording format having a larger amount of information is generated.

  Here, the perception processing unit 416 perceives the number of recognized families and the number of families from which smiles are detected based on at least part of the information detected by the sensor unit 400. As described above, the perception processing unit 416 perceives a laughing user based on at least one of the user 10 image and the user 10 voice detected by the sensor unit 400. Here, the smile is an example of information representing the strength of emotion of the user 10. Therefore, the number of families in which a smile is detected is an example of the total value of the emotional strengths of the plurality of users 10. Thus, the perception processing unit 416 detects the strength of the user's emotion based on at least a part of the information detected by the sensor unit 400.

  Based on the recording format switching rule 471, the perception processing unit 416 perceives whether or not a certain number of family members have been recognized and a certain number or more of family smiles have been detected. When the recording data generation unit 470 generates the skeleton data, the switching control unit 477 recognizes a certain number of families or more than a certain number of family smiles by the perception processing unit 416. In response to this, the recording data generating unit 470 starts generating video data. As described above, the switching control unit 477 detects that the number of the plurality of users 10 exceeds the predetermined number when the recording data generation unit 470 generates the information of the first recording format. When the total value of emotional intensity exceeds a predetermined threshold, the recording format of information to be generated by the recording data generating unit 470 is changed from the first recording format to the information amount of the first recording format. Switch to the second recording format with a large amount.

  The label generation unit 478 generates a label that is attached to the recording data and recorded. The label is an example of incidental information attached to the recording data. The label generation unit 478 generates a label including words such as “Yoshiko's birthday” and “Yoshiko's birthday party”. Other examples of labels include “celebration for higher education”, “Christmas”, “New Year”, and the like. The label generation unit 478 determines a situation such as a birthday or a birthday party based on the date / time information and the response content recognized by the user reaction recognition unit 432 in response to the question to the user 10. Then, a label is generated based on the determined situation. As described above, the label generation unit 478 generates a label including one or more words representing the current situation based on at least a part of the information detected by the sensor unit 400. The communication processing unit 480 associates the label generated by the label generation unit 478 with the recording data generated by the recording data generation unit 470, transmits the label to the server 200, and causes the server 200 to record it.

  FIG. 5 schematically shows a functional configuration of the server 200. The server 200 includes a storage unit 520, a reaction rule generation unit 530, a reaction rule search unit 540, a statistical information control unit 550, an access control unit 560, and a communication processing unit 580.

  The storage unit 520 stores reaction rules 521, reaction rule statistical information 522, and recorded data 561. The reaction rule 521 includes the same information as the reaction rule 421. The reaction rule statistical information 522 includes the attribute information of the user 10, the user 11, and the user 12, and information on the rate at which a positive reaction is obtained, in addition to the same information as the reaction rule 421. The recording data 561 is data for recording such as the above-described skeleton data, video data, and audio data.

  The communication processing unit 580 is responsible for communication between the robot 100, the robot 101, and the robot 102 and the server 200. The communication processing unit 580 receives user reaction information transmitted from the robot 100. As described above, the robot 101 and the robot 102 also have the same functions as the functions that the robot 100 has. Therefore, for example, the robot 101 transmits user response information including a response to an action performed on the user 11 to the server 200. The communication processing unit 580 receives the user reaction information transmitted from the robot 101 and the user reaction information transmitted from the robot 102. As described above, the communication processing unit 580 responds to the state of the user 11 recognized by the robot 101, the action determined based on the state of the user 11 in the robot 101, and the reaction of the user 11 to the execution of the action. , Received from the robot 101.

  The statistical information control unit 550 accumulates the user reaction information received by the communication processing unit 580 as the reaction rule statistical information 522 in the storage unit 520. As described above, the storage unit 520 stores the user state, action, and user reaction received by the communication processing unit 580 in association with each other. The user response information stored in the statistical information control unit 550 includes user response information regarding the user 10, user response information regarding the user 11, and user response information regarding the user 12.

  The reaction rule generation unit 530 generates a recommended reaction rule based on the user reaction information accumulated in the reaction rule statistical information 522 and stores it as the reaction rule 521. Based on the information included in the reaction rule statistical information 522, a reaction rule including an action for which a positive reaction is obtained is generated as a recommended reaction rule.

  Here, an operation when the robot 100 inquires the server 200 for a recommended reaction rule will be described. The robot 100 inquires of the server 200 about a recommended reaction rule by transmitting the state of the user 10 recognized by the robot 100 to the server 200. In this case, the reaction rule search unit 540 searches the reaction rule 521 and / or the reaction rule statistical information 522 for a reaction rule having a condition that is satisfied by the state of the user 10 received by the communication processing unit 580. The communication processing unit 580 transmits the reaction rule searched by the reaction rule search unit 540 to the robot 100 as a recommended reaction rule.

  As described above, the reaction rule search unit 540 is based on the reaction of the user 10 stored in the behavior result storage unit in association with the state that matches the state of the user 10 received by the communication processing unit 580. The behavior suitable for the state of the user 10 received by the communication processing unit 580 is selected from the behaviors stored in the reaction rule statistical information 522 in association with the state that matches the state of the user received by 580. The communication processing unit 580 transmits the action selected by the reaction rule search unit 540 to the robot 100.

  The access control unit 560 stores the recording data received by the communication processing unit 580 from the robot 100, the robot 101, and the robot 102 in the recording data 561 of the storage unit 520. When the communication processing unit 580 receives a recording data transmission request, the access control unit 560 reads the requested recording data from the recording data 561 and causes the communication processing unit 580 to transmit the recording data.

  Of the components of the robot 100 shown in FIG. 5, the functions of the components other than the storage unit 420 can be realized by the CPU operating based on the program. For example, the functions of these components can be implemented as the operation of the CPU by a basic software (OS) and a program operating on the OS.

  FIG. 6 schematically shows information perceived by the perception processing unit 416. When the human detection information 601 by the 3D depth sensor 402 and the face detection information 602 and user recognition information 603 by the face recognition unit 414 are input, the perception processing unit 416 receives the person detection information 601, the face detection information 602, and the user recognition information 603. Are combined to generate perceptual information 611, perceptual information 612, and perceptual information 613. Subsequently, when smile detection information 604 is input from the facial expression recognition unit 413, the smile detection information 604 is converted to generate perceptual information 614. Further, the perception processing unit 416 perceives “Dad or lonely?” Based on the high probability of not smiling as perceived by the perception information 614 (perception information 615).

  Here, in the perception confirmation rule 417, as a perception confirmation rule when perceiving “Is it lonely?”, A condition part of “one person is one” and “the probability of being a smile is 20% or less”, and “from microphone” A perception confirmation rule 630 having a conclusion part “confirm that there is no utterance for 30 seconds” is included. The perception processing unit 416 determines that the condition part of the perception confirmation rule 630 is satisfied from the perception information 613 and the smile detection information 604, and confirms the detection result by the microphone 401 based on the conclusion part of the perception confirmation rule 630. Run. Specifically, the perception processing unit 416 causes the speech understanding unit 412 to analyze the sound output from the microphone 401.

  Here, when the voice is not detected by the microphone 401 for 30 seconds, the speech understanding unit 412 outputs time-out information indicating that the speech has not been detected for 30 seconds. When the time-out information 605 from the utterance understanding unit 412 is input, the perception processing unit 416 converts the time-out information 605 into perception information 616 “Dad does not speak anything”.

  In this way, the perception processing unit 416 generates the perception history 610 including the perception information 611, the perception information 612, the perception information 613, the perception information 614, the perception information 615, and the perception information 616. The perception processing unit 416 outputs the generated perception information to the meaning understanding unit 418.

  FIG. 7 schematically shows information understood by the semantic understanding unit 418. The perception history 610 generated by the perception processing unit 416 is input to the meaning understanding unit 418. The semantic understanding unit 418 generates semantic information 711, semantic information 712, semantic information 713, semantic information 714, and semantic information 715 from the perceptual information 611, the perceptual information 612, the perceptual information 613, the perceptual information 614, and the perceptual information 615, respectively. .

  When the semantic information 714 is generated from the perceptual information 614, the semantic understanding unit 418 converts the numerical information “probability 90%” included in the perceptual information 614 into information “not so much”. Vague. As a result, the semantic understanding unit 418 generates semantic information 714 obtained by obscuring the perceptual information 614.

  Here, in the meaning understanding rule 419, as a meaning confirmation rule in the case of understanding “Is it lonely?”, A condition part “someone is alone”, “not so much smiling” and “does not speak anything”, and “ A semantic comprehension rule 730 having a conclusion that “someone seems to be lonely” is included. When the semantic understanding unit 418 determines that the semantic information 713, the semantic information 714, and the perceptual information 616 satisfy the condition part of the semantic understanding rule 730 based on the semantic understanding rule 730, “One dad seems to be lonely.” The semantic information 717 is generated. As described above, the semantic understanding unit 418 generates the semantic information 717 by integrating the semantic information 714 and the perceptual information 616 using the semantic understanding rule 730.

  In this way, the semantic understanding unit 418 generates the semantic history 710 including the semantic information 711, the semantic information 712, the semantic information 713, the semantic information 714, the semantic information 715, and the semantic information 717. The semantic information generated by the semantic understanding unit 418 is output to the condition determination unit 434, the user reaction recognition unit 432, and the label generation unit 478.

  As described with reference to FIGS. 6 and 7, the perceptual processing unit 416 and the semantic understanding unit 418 use a language in which the meaning of the perceptual result is patterned while losing the accuracy of the perceptual result. Memorize vaguely.

  FIG. 8 shows information included in the reaction rule 421 in a table format. The reaction rule 421 includes a condition part including conditions of “self-emotion” and “user state”, a conclusion part including “emotion change” and “behavior”, and “attachment points” including “compassion points” and “praised points”. Information ".

  The condition of “self emotion” is a condition related to the self emotion of the robot 100. “User state” is a condition relating to the state of the user 10. “Emotion change” indicates how the emotion of the robot 100 is changed. “Action” represents an action that the robot 100 can execute. The “compassion point” indicates the degree of compassion for the user 10. The “compassion point” is an example of appropriateness.

  As shown in the figure, the condition part of the reaction rule 421 includes a “self-emotion” condition of “normal” and a “user state” of “someone seems to be lonely”. The conclusion part corresponding to the condition part includes “emotion change” representing an emotion change “being worried”. In addition, as a conclusion part corresponding to this condition part, “behavior” indicating actions such as “call out”, “return only the anxious gesture” and “greet” are included. In addition, 30 points, 20 points, and 10 points correspond to the “consideration points” for these actions of “speaking”, “returning only worried gestures”, and “greeting” by “attached information”. It is attached. In addition, 50 points, 30 points, and 30 points are given as “praised points” for the actions of “speaking”, “returning only the gestures that are worried”, and “greeting” by the “attached information”, respectively. It is associated.

  Thus, the reaction rule 421 includes information indicating a condition indicating the state of the user to be satisfied and an action associated with the condition. Further, the reaction rule 421 includes information indicating a condition indicating the emotion of the robot 100 to be satisfied and an action associated with the condition. The reaction rule 421 includes emotion change information for changing the emotion value based on the emotion value of the robot 100 and the state of the user 10. Then, the emotion determination unit 442 is based on the emotion change information stored in the reaction rule 421 in association with the emotion value determined by the emotion determination unit 442 and the state of the user 10 recognized by the user state recognition unit 430. Update emotion values.

  FIG. 9 schematically shows processing based on the reaction rule 421. When the semantic determination unit 434 receives the semantic information 717 “Daddy seems to be lonely” from the meaning understanding unit 418, the condition of the user 10, “Daddy is alone, seems to be lonely” and the current emotion of the robot 100. A condition part that is satisfied by the “ordinary” feeling is searched from the reaction rule 421.

  The condition determination unit 434 determines that the state of the user 10 “Dad seems to be lonely” and the “normal” emotion satisfy the condition part of the reaction rule 810 in the reaction rule 421. As a result, the action determination unit 440 specifies “speak a voice”, “return only the gesture that seems to be worried”, and “greet” as the action candidates 910 defined in the reaction rule 810. Then, the action determining unit 440 selects an action to be actually executed from the identified action candidates 910. For example, the action determination unit 440 selects one action with a selection probability corresponding to the size of the consideration point. Here, it is assumed that the action determining unit 440 selects the action “call out”.

  In addition, the emotion determination unit 442 updates the emotion value based on the emotional change content “being worried” determined by the reaction rule 810. For example, the emotion determination unit 442 increases the emotion value of “anxiety” indicating the intensity of anxiety. As described above, the emotion determination unit 442 updates the emotion value based on the state of the user 10 recognized by the user state recognition unit 430 based on the reaction rule 421. For example, when the user state recognition unit 430 recognizes that the user 10 feels loneliness, the emotion determination unit 442 gives an emotion value representing the strength of worry to a greater worry strength. Change to the value you want to represent.

  Subsequently, the behavior control unit 450 causes the speech of the utterance content “What was it?” To be output from the speaker of the control target 452. At this time, the behavior control unit 450 outputs a voice having voice characteristics representing an emotion that is likely to be worried based on the current emotion value. For example, a voice with a relatively low speed is generated. Note that the behavior control unit 450 may determine the content of the utterance “What happened?” Based on the feeling “I am worried”.

  Subsequently, the user reaction recognition unit 432 waits for the reaction of the user 10 to be detected. For example, when the semantic information 940 “Dad is a little smile” is output from the meaning understanding unit 418 based on the voice “Thank you!” And the facial expression of the user 10 at that time, The user reaction recognizing unit 432 determines that the user 10 has changed from an emotion that seems to be lonely to an emotion that seems to be fun. Thereby, the rule update unit 422 determines that the emotion of the user 10 is improved by the action of “calling out”. Thereby, the consideration point of the action of “calling out” is increased from 30 points to 40 points. Also, the rule update unit 422 increases the number of points given up from 50 points to 60 points.

  In addition, the emotion determination unit 442 increases the emotion value of joy because the emotion of the user 10 is improved by the action executed by the robot 100. In addition, the behavior control unit 450 expresses that it has become a joyful emotion based on the increased emotion value of the joy. For example, the behavior control unit 450 causes a gesture representing joy to be performed.

  The communication processing unit 480 also includes information excluding “consideration points” and “praised points” in the information included in the reaction rule 810, information indicating that the reaction of the user 10 is positive, and “papa”. The user response information 960 including the attribute of the user 10 recognized as “” is transmitted to the server 200.

  As described above, the action determination unit 440 determines an action to be executed based on the emotion value determined by the emotion determination unit 442, the user state recognized by the user state recognition unit 430, and the reaction rule 421. Specifically, the action determination unit 440 satisfies the condition determination unit 434 among the plurality of actions stored by the reaction rule 421 in association with the condition determined to be satisfied by the condition determination unit 434. The action with the higher compassion point stored by the reaction rule 421 in association with the condition determined to have been determined is determined with a higher frequency as the action to be executed. Note that the action determination unit 440 may select the action having the maximum consideration point as the action to be actually executed. In this case, when there are a plurality of actions having the maximum consideration point, the action determination unit 440 may randomly select one action to be actually executed from the plurality of actions having the maximum consideration point. In this case, the action determination unit 440 may select one action to be actually executed with a selection probability corresponding to the point given up.

  FIG. 10 schematically shows an example of an operation flow related to an action of determining an action in the robot 100. The operation flow illustrated in FIG. 10 is started when, for example, semantic information including the emotion of the user 10 is input. Note that “S” in the operation flow represents a step to be executed.

  The condition determination unit 434 searches for a condition part satisfied by the emotion of the robot 100 and the input semantic information among the condition parts included in the reaction rule 421 (S1002). The condition determination unit 434 determines whether there is a reaction rule that satisfies the condition part (S1004). If it is determined that there is a reaction rule that satisfies the condition part, the action determination part 440 selects one action to be executed for the user 10 from the actions determined in the conclusion part of the retrieved reaction rule. (S1006). In addition, the emotion determination unit 442 updates the emotion value based on the emotion change determined in the conclusion part of the retrieved reaction rule (S1008).

  Subsequently, the behavior control unit 450 controls the control object 452 to execute the behavior selected in S1006 (S1010). At this time, the behavior control unit 450 controls the control object 452 based on the emotion value updated in S1008. For example, when expressing the feeling of “worry”, the behavior control unit 450 controls the control object 452 so that the posture of the robot 100 is likely to be worried.

  Subsequently, the user reaction recognition unit 432 monitors input of semantic information indicating the reaction of the user 10 (S1012). When the semantic information indicating the reaction of the user 10 is input, the user response recognition unit 432 determines whether the user's response is positive based on the input semantic information (S1016).

  If it is determined in S1016 that the reaction is positive, the emotion determination unit 442 increases the emotion value of “joy” (S1018). Also, the rule update unit 422 increases compassion points and praised points corresponding to the executed action. On the other hand, if a negative reaction is obtained in the determination of S1016, the emotion determination unit 442 increases the emotion value of “sadness” (S1022). In addition, the rule update unit 422 reduces compassion points and praised points corresponding to the executed action (S1024).

  As described above, the rule update unit 422 determines the condition and action determination unit 440 determined to be satisfied by the condition determination unit 434 when the user reaction recognized by the user reaction recognition unit 432 is not defective. The compassion point memorize | stored by the reaction rule 421 is increased in association with the performed action. On the other hand, when the user reaction recognized by the user reaction recognition unit 432 is bad, the rule update unit 422 determines that the condition determined by the condition determination unit 434 and the action determined by the action determination unit 440 are satisfied. Corresponding points stored by the reaction rule 421 are reduced in association with each other.

  In addition, after the behavior control unit 450 executes the behavior determined by the behavior determination unit 440 in the execution form determined according to the emotion, the emotion determination unit 442 is based on the user's reaction to the execution of the behavior. , Further change the emotion value. Specifically, the emotion determination unit 442 is delighted when the user's reaction to having performed the action determined by the behavior determination unit 440 on the user in the execution form determined by the behavior control unit 450 is not bad. The emotional value representing the strength of the emotion is changed to a value representing the strength of greater pleasure. In addition, the emotion determination unit 442 increases the sadness when the user's reaction to the user having performed the behavior determined by the behavior determination unit 440 in the execution mode determined by the behavior control unit 450 is poor. The emotional value representing the height is changed to a value representing the intensity of greater sadness.

  The behavior control unit 450 expresses the emotion of the robot 100 based on the emotion value determined in S1018 or S1022 (S1026). For example, when the emotion control value of “joy” is increased, the behavior control unit 450 controls the control object 452 to make the robot 100 perform a joyful gesture. In addition, when the emotion value of “sadness” is increased, the behavior control unit 450 controls the control object 452 so that the posture of the robot 100 becomes a prowled posture. Subsequently, the communication processing unit 480 transmits user reaction information to the server 200 (S1028), and ends the operation related to the determination of the operation of the robot 100.

  If it is determined in S1004 that there is no reaction rule that satisfies the condition part, the communication processing unit 480 inquires the server 200 about a recommended reaction rule. For example, information indicating the emotion of the robot 100, information indicating the state of the user represented by the input semantic information, and information indicating the attribute of the user 10 are transmitted to the server 200 (S1030). From the server 200, a reaction rule suitable for the information transmitted by the inquiry is transmitted to the robot 100 as a recommended reaction rule. When the communication processing unit 480 receives the recommended reaction rule from the server 200, the rule update unit 422 incorporates the recommended reaction rule into the reaction rule 421 (S1032), and proceeds to S1006. In this case, in S1006 and S1008, based on the reaction rule received from the server 200, action selection and emotion value update are performed. As described above, the communication processing unit 480 is suitable for the user state recognized by the user state recognition unit 430 when the condition determination unit 434 determines that none of the conditions included in the reaction rule 421 is satisfied. The received action is received from the server 200.

  According to this operation flow, when there is no reaction rule that satisfies the condition part, an inquiry about a recommended reaction rule is performed. Note that even when there is a reaction rule that satisfies the condition part, it may be determined that the action cannot be determined if the consideration point associated with any action is smaller than a predetermined value. In this case, the communication processing unit 480 may inquire a recommended reaction rule. In other words, the action determination unit 440 includes a consideration point included in the determination information among one or more actions associated with the condition determined to be satisfied by the condition determination unit 434 among the information included in the determination information. An action higher than a predetermined value is determined as an action to be executed. The communication processing unit 480 recognizes the user state when the consideration points of one or more actions associated with the condition determined to be satisfied by the condition determination unit 434 are all equal to or less than a predetermined value. An action suitable for the state of the user 10 recognized by the unit 430 may be received from the server 200. Then, the rule update unit 422 may add the action received from the server 200 by the communication processing unit 480 to the reaction rule in association with the condition determined to be satisfied by the condition determination unit 434.

  In addition, the inquiry of the recommended reaction rule may be periodically performed by batch processing or the like in addition to the case where the behavior determination unit 440 cannot determine the behavior. The inquiry of the recommended reaction rule may be performed when it is determined that the user 10 is sleeping or when the robot 100 transitions to an idle state or a sleep state. In addition, an inquiry about a recommended reaction rule may be made at any timing at which the robot 100 tries to act.

  Further, the reaction rule 421 may include a rule that defines an action when an action to be executed for the user 10 is not searched. For example, the reaction rule 421 includes a condition part that defines a condition that “an action to be performed on the user 10 has not been searched” and a conclusion that defines an action that “inquires the server 200 for a recommended reaction rule”. And a rule including a part. Thereby, the action determination unit 440 may determine to inquire the server 200 for a recommended reaction rule according to the reaction rule 421.

  FIG. 11 shows an example of the reaction rule statistical information 522 stored in the server 200. The reaction rule statistical information 522 includes “reaction rule”, “attribute”, “number of people”, and “reaction result”. The “reaction rule” includes a “condition part”, a “conclusion part”, and a “compassion point”. In the “condition part”, “conclusion part”, and “compassion point” of the reaction rule statistical information 522, information corresponding to the “condition part”, “conclusion part”, and “compassion point” of the reaction rule 421 is stored.

  In the “attribute” of the reaction rule statistical information 522, the attribute of the user 10 is stored. The attributes of the user 10 include age group, sex, role in the home, information on the area where the user lives, and the like. The “number of people” stores the number of people who contributed to the information accumulated in the reaction rule statistical information 522. The “reaction result” stores information indicating the number of times of positive reaction and the number of times of positive negative reaction.

  When the communication processing unit 580 receives the user reaction information from the robot 100, the statistical information control unit 550 associates the attribute that matches the attribute included in the user reaction information with the “condition” of the reaction rule included in the user reaction information. Part ”and“ conclusion part ”are stored in the reaction rule statistical information 522. Further, the statistical information control unit 550 increments the value of “positive count” or the value of “negative count” based on the response quality information included in the user response information. Further, the statistical information control unit 550 increments the value of “number of people” as necessary. For example, when the user response information received this time is user response information of a user who has reflected the user response information in the same record of the response rule statistical information 522 in the past, the value of “number of people” is not incremented.

  Further, the statistical information control unit 550 calculates a recommended compassion point and stores it in the “compassion point” of the “reaction rule”. For example, the statistical information control unit 550 calculates a larger value as the recommended compassion point as the value of “the number of positives” is larger. Specifically, the statistical information control unit 550 may calculate a larger value as a recommended compassion point as the ratio of the “positive count” value to the “negative count” value is larger.

  When the communication processing unit 580 receives an inquiry about a recommended reaction rule from the robot 100, the reaction rule search unit 540 searches for a recommended reaction rule from the reaction rules stored in the reaction rule 521. For example, the reaction rule search unit 540 searches the reaction rule 521 for a reaction rule having a “condition part” that matches the “condition part” included in the inquiry. The reaction rule retrieved by the reaction rule retrieval unit 540 is transmitted from the communication processing unit 580 to the robot 100 as a response to the inquiry.

  The reaction rule search unit 540 may search for a recommended reaction rule from the reaction rules stored in the reaction rule statistical information 522. Specifically, the reaction rule search unit 540 has a “condition part” that matches the “condition part” included in the inquiry among the reaction rules stored in the reaction rule statistical information 522, and the inquiry Reaction rules having “attributes” that match the “attributes” included may be searched. Thereby, the server 200 can provide the robot 100 with an action suitable for the attribute of the user 10.

  In the example of FIG. 12, in the reaction rule statistical information 522, reaction results are tabulated for each user attribute. As another example of the reaction rule statistical information, reaction results may be aggregated for each user.

  FIG. 12 schematically shows actions performed by the robot 100 to determine a label attached to the recording data. When it is determined that the date is the date of birth and the user's 10 attribute information, the behavior determination unit 440 asks the user 10 to confirm whether the date is the birthday. To decide. In this case, for example, the behavior control unit 450 causes the speaker of the control target 452 to output the voice of the utterance content “Mr. XX, today is a birthday. Happy birthday!”.

  Thereafter, the user reaction recognition unit 432 recognizes whether the reaction of the user 10 is positive or negative from the utterance content of the user 10. For example, when the user 10 replies “Thank you, I am happy!”, The user reaction recognition unit 432 recognizes that the reaction of the user 10 is positive. When the recognized name of the user 10 is “Yoshiko-chan”, the label generation unit 478 generates a label “Yoshiko-chan's birthday”.

  The label generation unit 478 generates a label including information indicating the name of the recognized user 10 and information “Yoshiko's birthday” as a label attached to the recording data. The communication processing unit 480 continuously transmits, to the server 200, recording data in which the label generated by the label generation unit 478 is added to the skeleton data based on the skeleton information acquired by the 3D depth sensor 402. In the server 200, the recording data received from the robot 100 is stored in the recording data 561 of the storage unit 520.

  The semantic understanding unit 418 calculates the number of people currently recognized based on the current skeleton data based on the skeleton information acquired from the 3D depth sensor 402. In addition, the skeleton data acquired in the same time zone in the past is acquired from the server 200, and the average value of the number of people recognized in the same room in the same time zone in the past is calculated. Here, when the semantic understanding unit 418 recognizes that the number of people currently recognized is more than a predetermined threshold value than the average number of people recognized in the same room in the same time period in the past, it is different from usual. Recognize the situation. The semantic understanding unit 418 compares the label attached to the skeleton data acquired at the same time in the past with the currently recognized user 10, and the user 10 who is not usually at home is currently recognized. If it is determined that it is, it is recognized that the situation is different from usual. When the behavior control unit 450 recognizes that the situation is different from usual, the behavior control unit 450 may control the sensor unit 400 or the sensor module unit 410 to perform more detailed detection operation. For example, the behavior control unit 450 may control the utterance understanding unit 412 to recognize the utterance content with higher accuracy. Further, the voice emotion recognition unit 411 may be controlled to recognize emotions with higher accuracy. Further, the facial expression recognition unit 413 may be controlled to recognize the facial expression with higher accuracy.

  When the meaning understanding unit 418 recognizes that the situation is different from the usual situation, the behavior determination unit 440 determines to ask the user 10 a question to confirm the situation. For example, in a case where the label “Yoshiko-chan's birthday” is determined, if it is determined that many people are recognized, the behavior control unit 450 determines “Yoshiko-chan ’s birthday party? ”Is output from the speaker of the control target 452.

  In this case, the user reaction recognition unit 432 recognizes whether the reaction of the user 10 is positive or negative from the utterance content of the user 10. For example, when the user 10 replies “Yes!”, The user reaction recognition unit 432 recognizes that the reaction of the user 10 is positive. In this case, the label generation unit 478 generates a label “Yoshiko's birthday party”. Note that the labels “Yoshiko-chan's birthday” and “Yoshiko-chan's birthday party” are examples of labels representing the current status of the user 10.

  As described above, the robot 100 interacts with the user 10 to confirm the current situation based on at least a part of the information detected by the sensor unit 400. And the label production | generation part 478 determines the present condition of the user 10 based on the interaction result with the user 10, Based on the determined information, 1 or more character information showing the present condition of the user 10 is obtained. Generate a label that contains it.

  FIG. 13 shows an example of the recording format switching rule 471. The recording format switching rule 471 includes a condition part that defines a condition that “a certain number of family members are in front” and “a certain number of family smiles are detected” and “switch to a low compression recording format”. And a rule 1300 having a conclusion part that defines the action. In addition, the recording format switching rule 471 includes a condition part that defines a condition that “the number of family members in front is less than a certain number” and “a certain number or more of family anger has been detected”; A rule having a conclusion part that defines an action of “switch to recording format”.

  Note that the value of the “certain number” may be larger as the number of members belonging to one family increases. For example, in the case of a family of four, the “certain number” value may be three. In the case of a family of five, the “certain number” value may be four. Further, the value of “a certain number” may be determined based on the history of the number of users 10 recognized by the meaning understanding unit 418. For example, even if a family of five people only recognizes four people by the semantic understanding unit 418, 3 may be determined as the value of “a certain number”.

  FIG. 14 schematically shows an example of an operation flow relating to switching of recording formats. This operation flow is started when the recording data generation unit 470 starts generating high-compression recording data.

  In S1402, the perception processing unit 416 determines the number of users 10 belonging to a specific family and the degree of smile exceeding a predetermined value based on information output from the sensor unit 400 and the sensor module unit 410. Start detecting the number of Here, the perception processing unit 416 counts the number of users 10 registered as a family in the person DB 424 among the users whose existence is recognized based on information output from the sensor unit 400 and the sensor module unit 410. In addition, the perception processing unit 416 counts the number of users 10 registered as a family in the person DB 424 among users whose smile level exceeds a predetermined value.

  In step S <b> 1404, the switching control unit 477 determines whether the condition part of the rule 1300 that determines to switch the recording format to the low compression format is satisfied. When it is determined that the condition part of the rule 1300 is satisfied, the action control unit 450 outputs the voice of the utterance content “I will take a picture because it looks fun!” From the speaker of the control target 452 (S1406). As a result, the user 10 is notified that the video is to be recorded.

  In step S <b> 1408, the switching control unit 477 determines whether there is a negative response from the user 10. For example, the switching control unit 477 determines that there is no negative reaction from the user 10 when a negative word / phrase is not recognized from the voice of the user 10. If it is determined that there is no negative response from the user 10, the switching control unit 477 controls the recording data generation unit 470 to start generating full HD video data and audio data in addition to the skeleton data ( S1410). In this way, the switching control unit 477 switches the recording format from the high compression format to the low compression format. If it is determined in S1408 that there is a negative response from the user 10, the process proceeds to S1404 after a predetermined wait time has elapsed (S1420).

  After switching to the low compression format in S1410, it is determined in S1412 whether or not the condition part of the rule 1310 that determines to switch to the high compression format is satisfied. When it is determined that the condition part of the rule 1310 is satisfied, the switching control unit 477 switches the recording format from the low compression format to the high compression format (S1414). If it is determined in S1412 that the condition part of the rule 1310 is not satisfied, the determination in S1412 is repeated until the condition part of the rule 1310 is satisfied.

  In addition to the determination based on the rule 1310 in S1412, the recording format is changed from the low compression format even when an utterance instructing to stop recording in the low compression format is recognized, such as “End recording”. You may switch to a high compression format. Further, after the recording format is switched to the low compression format, the recording format may be switched from the low compression format to the high compression format when a predetermined maximum recording time has elapsed.

  It should be noted that the switching control unit 477 may switch the recording format in accordance with not only the number of smiles but also the intenseness of the emotion of the user 10. For example, the switching control unit 477 may switch the recording format to the low compression format when an impact stronger than a predetermined value is detected by a touch sensor included in the sensor unit 400.

  The robot 100 is an example of an electronic device that includes a behavior control system. The application target of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. Further, the function of the server 200 may be implemented by one or more computers. At least some of the functions of the server 200 may be implemented by a virtual machine. Further, at least a part of the functions of the server 200 may be implemented in the cloud.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first,” “next,” etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

5 System 10, 11, 12 User 20 Communication network 100, 101, 102 Robot 200 Server 400 Sensor unit 401 Microphone 402 3D depth sensor 403 2D camera 404 Distance sensor 410 Sensor module unit 411 Speech emotion recognition unit 412 Speech understanding unit 413 Expression recognition Unit 414 face recognition unit 416 perception processing unit 417 perception confirmation rule 418 meaning understanding unit 419 meaning understanding rule 420 storage unit 421 reaction rule 422 rule update unit 424 person DB
430 User state recognition unit 432 User reaction recognition unit 434 Condition determination unit 440 Action determination unit 442 Emotion determination unit 450 Action control unit 452 Control object 470 Recording data generation unit 471 Recording format switching rule 477 Switching control unit 478 Label generation unit 480 Communication processing Unit 520 storage unit 521 reaction rule 522 reaction rule statistical information 530 reaction rule generation unit 540 reaction rule search unit 550 statistical information control unit 560 access control unit 561 record data 580 communication processing unit 601 person detection information 602 face detection information 603 user recognition information 604 Smile detection information 605 Time-out information 610 Perception history 611, 612, 613, 614, 615, 616 Perception information 630 Perception confirmation rule 710 Semantic history 711, 712, 713, 714, 715, 717 Semantic information 730 Understanding the rules 810 reaction Rule 910 candidate 940 semantic information 960 user response information 1300 and 1310 rules

Claims (11)

An action information storage unit for storing determination information for determining an action based on a user's state;
A user state recognition unit for recognizing the user state;
An action determination unit that determines an action to be executed based on the user state recognized by the user state recognition unit and the determination information;
A reaction recognition unit for recognizing the user's response to the execution of the action determined by the action determination unit;
A transmission unit that transmits the user status recognized by the user status recognition unit, the behavior determined by the behavior determination unit, and the user response recognized by the reaction recognition unit to an external server;
When the action determining unit fails to determine an action to be executed based on the user state and the determination information, the action suitable for the user state recognized by the user state recognizing unit is received from the server. A receiving unit to
A behavior control system comprising: a behavior information update unit that updates the determination information based on a behavior received by the reception unit from the server. An emotion determination unit that determines an emotion value indicating a current emotion of the system based on the state of the user recognized by the user state recognition unit;
The behavior control system according to claim 1, further comprising a behavior control unit that determines an execution form of the behavior determined by the behavior determination unit based on the emotion value determined by the emotion determination unit. The transmission unit further transmits the attribute of the user to the server,
The reception unit is configured to recognize the user status and the user attribute recognized by the user status recognition unit when the behavior determination unit cannot determine an action to be executed based on the user status and the determination information. The behavior control system according to claim 1, wherein a behavior suitable for the operation is received from the server. The decision information includes information indicating a condition indicating a state of a user to be satisfied when performing the action, and an action associated with the condition,
The behavior control system includes:
A condition determination unit that determines whether the condition included in the determination information is satisfied based on the state of the user recognized by the user state recognition unit;
When the condition determining unit determines that none of the conditions included in the determination information is satisfied, the receiving unit performs an action suitable for the user state recognized by the user state recognizing unit, The behavior control system according to claim 1, which is received from the server. The determination information includes information indicating the appropriateness of each of the one or more actions,
The degree of appropriateness included in the determination information among the one or more actions associated with the condition determined to be satisfied by the condition determination unit among the information included in the determination information. Determines an action that is higher than a predetermined value as an action to be executed,
The receiving unit, when the appropriateness of the one or more actions associated with the condition determined to be satisfied by the condition determining unit is less than or equal to the predetermined value, The behavior control system according to claim 4, wherein a behavior suitable for the state of the user recognized by the user state recognition unit is received from the server. The said action information update part is matched with the said condition judged that the said condition judgment part was satisfy | filled, The action which the said receiving part received from the said server is added to the said determination information. Behavior control system. The behavior information update unit is configured to determine the condition determined by the condition determination unit and the action determined by the behavior determination unit when the user's response recognized by the reaction recognition unit is poor. The behavior control system according to claim 5 or 6, wherein the degree of appropriateness stored in the determination information is reduced in association with the action information.   A program for causing a computer to function as the behavior control system according to any one of claims 1 to 7. A plurality of behavior control systems according to any one of claims 1 to 7,
A system comprising the server. The state of the user recognized in the first behavior control system, the behavior determined based on the state of the user in the first behavior control system, and the user's reaction to the execution of the behavior, A user response receiver that receives from the first behavior control system;
An action result storage unit that stores the state of the user, the action, and the reaction of the user received by the user reaction reception unit;
A user status receiving unit that receives the status of the user recognized in the second behavior control system from the second behavior control system;
The state of the user received by the user state reception unit based on the reaction of the user stored in the action result storage unit in association with a state that matches the state of the user received by the user state reception unit An action selection unit that selects an action suitable for the state of the user received by the user state reception unit from the behaviors stored in the action result storage unit in association with a state that matches
A server comprising: a transmission unit that transmits the behavior selected by the behavior selection unit to the second behavior control system.   The program for functioning a computer as a server of Claim 10.

Images