JP2012010856A - State recognition device, listening interaction continuing system, state recognition program and state recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a state recognition device, a listening interaction continuing system, a state recognition program and a state recognition method which are new.SOLUTION: A PC 10 included in the listening interaction continuing system 100 acquires user's action data from images picked up by an abdomen camera 14 of a robot 12 and a monitor camera 22 and voices collected by microphones 20. An individualized SVM in which the position of a boundary (ultraplane) is adjusted is constructed on the basis of an individual learning sample sampled from the action data of a certain user and a general learning sample for constructing SVM. When a recognition sample sampled from the certain user's action is inputted into the individualized SVM, the concentration state of a ceratin user is recognized. According to this device, the PC 10 can easily and correctly recognize the concentration state of the certain user.

Description

  The present invention relates to a state recognition device, a listening dialogue persistence system, a state recognition program, and a state recognition method, and more particularly, to a state recognition device, a listening dialogue persistence system, a state recognition program, and a state recognition method, for example.

The state monitoring device disclosed in Patent Document 1 uses a plurality of cameras and microphones to detect a change in the posture of a person and a face, and opening and closing of eyes and mouth. Further, the state monitoring device can determine, based on the detected information, a stop state in which a person is watching a television, a conversation state in which a person is speaking, a sleep state in which a person is sleeping, and the like. . Then, the state monitoring device controls the power supply of various electrical products set around the person or sends a message to the person according to the determined state.
JP-A-2005-199078 [A61B 5/11, A61B 5/107, G06T 1/00, G06T 7/20]

  In recent years, many recognition systems for recognizing a human state, such as the state monitoring device of Patent Document 1, have been developed and are beginning to be used in various fields.

  However, in a recognition system developed for recognizing various human states, a human state with extremely little change in posture and opening / closing of the mouth may not be recognized correctly. In this case, it may be possible to construct a recognition system specialized for that person, but in order to construct a personal recognition system, a large amount of learning data must be prepared. I can not say. Also, if a personal recognition system is constructed, other humans cannot use the personal recognition system at all, and the versatility of the recognition system is impaired.

  SUMMARY OF THE INVENTION Therefore, a main object of the present invention is to provide a novel state recognition device, a listening dialogue persistence system, a state recognition program, and a state recognition method.

  Another object of the present invention is to provide a state recognition device, a listening dialogue persistence system, a state recognition program, and a state recognition method that can easily and correctly recognize the state of a specific user.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  A first invention includes an acquisition unit that acquires a user's behavior and a recognition unit that has a recognition criterion and recognizes a user state from the user's behavior acquired by the acquisition unit. A state recognition apparatus, further comprising: a storage unit that stores a learning sample; and an adjustment unit that adjusts a recognition reference based on the plurality of learning samples.

  In the first invention, the state recognition device (10: reference numeral exemplifying a corresponding part in the embodiment, hereinafter the same) acquires a user's action from a camera (14, 22), a microphone (20), or the like. It has an acquisition means (26, S153, S155) and a recognition means (26, S195, S213) that has a recognition standard and recognizes the user state (concentrated state) from the user behavior acquired by the acquisition means. A memory | storage means (30) memorize | stores the some learning sample sampled from the action of the object user to recognize. The adjustment means (26, S171, S173) adjusts the recognition reference of the recognition means based on a plurality of learning samples sampled from the behavior of the user to be recognized.

  According to the first aspect, since the recognition standard is adjusted using the learning sample sampled from the behavior of a specific user, the user's state can be easily recognized correctly.

  A second invention is according to the first invention, wherein the recognition criterion is a boundary in the SVM, and the adjustment means includes a weight adjustment means for adjusting the weights of the plurality of learning samples, and the plurality of learning is performed by the weight adjustment means. When the weight of the sample is adjusted, the boundary position changes.

  In the second invention, the recognition criterion is a boundary (boundary line or hyperplane) in the SVM. The weight adjusting means (26, S171) adjusts the weight of each parameter value included in the learning sample. Then, when the weights of the plurality of learning samples are adjusted by the weight adjusting means and the SVM is reconstructed, the position of the boundary changes.

  According to the second invention, by adjusting the weight of each parameter value included in the learning sample, the boundary of the SVM can be changed to a position suitable for an individual's action.

  A third invention is dependent on the second invention, and further comprises provisional recognition means for temporarily recognizing a user state after the weight is adjusted by the weight adjustment means, and a recording means for recording the degree of recognition by the provisional recognition means. The weight adjusting means repeats the adjustment of the weights of the plurality of learning samples until the difference between the previous recognition degree recorded by the recording means and the current recognition degree becomes a predetermined value or less.

  In the third invention, the temporary recognition means (26, S175) temporarily recognizes the user state after the weights of the parameters included in the learning sample are adjusted by the weight adjustment means. The recording means (26, S177) records, for example, the number of recognition samples recognized as the active state as the recognition degree. Then, the weight adjustment unit repeats the adjustment of the weights of the plurality of learning samples until, for example, the difference between the previous recognition level and the current recognition level becomes a predetermined value or less.

  According to the third invention, the recognition accuracy of the user state is improved in proportion to the operating time of the state recognition device.

  4th invention is a state recognition apparatus which recognizes a user's user state, Comprising: The memory | storage means which memorize | stores a some learning sample, The acquisition means which acquires a user's action, An acquisition means based on a some learning sample A creation unit that creates a recognition sample with an adjusted weight from the user's behavior acquired by the step, and a recognition unit that has a recognition standard and recognizes the user state of the user from the recognition sample created by the creation unit It is a recognition device.

  In the fourth invention, the state recognition device (10) recognizes the user state of the user. The storage means (30) stores a plurality of learning samples sampled from the actions of a plurality of users. The acquisition means (26, S153, S155) acquires the user's action from the camera (14, 22), the microphone (20), or the like. The creating means (26, S313) creates a recognition sample in which the weight is adjusted from the user's action based on the plurality of learning samples sampled from the plurality of user's actions. The recognition means (26, S315) has a recognition standard, and recognizes the user state (concentrated state) of the user from the recognition sample created by the creation means.

  According to the fourth aspect of the present invention, it is only necessary to adjust the weight of the recognition sample, so that the state of a specific user can be easily and correctly recognized without changing the recognition standard.

  A fifth invention is according to any one of the first to fourth inventions, further comprising an elemental action determining means for determining presence / absence of a plurality of elemental actions of the user, wherein the user's action is a plurality of elemental actions It is characterized by including a complex action that combines the presence or absence of.

  In the fifth invention, the element behavior determining means (26, S13, S37, S45, S67, S75, S97, S105) determines the presence or absence of a plurality of elemental actions of the user based on the input of the camera or microphone. And a user's action contains the compound action which combined the existence of a plurality of elemental actions.

  A sixth invention is according to the fifth invention, and the plurality of element behaviors include a user's utterance, a user's gaze, a user's forward leaning posture, and a user's whisper.

  According to the fifth invention and the sixth invention, it is possible to obtain a complex action necessary for learning and recognition while reducing the types of element actions to be acquired. Therefore, it is possible to reduce the load on the state recognition device that records the user's elemental behavior.

  A seventh aspect of the present invention is a listening dialogue sustaining system having the state recognition device according to claim 6, further comprising partner utterance determination means for determining presence / absence of speech of the conversation partner, and a plurality of elemental actions in one speaker Is a listening dialogue sustaining system that further includes the utterance of the other dialogue partner.

  In the seventh invention, the listening dialogue sustaining system (100) has the state recognition device (10) according to claim 6 and is a system for sustaining the dialogue with the dialogue partner. Further, the other party utterance determination means (26, S123) of the listening dialogue sustaining system determines the presence or absence of the other party's utterance. The plurality of elemental actions in one speaker further includes the utterance of the other conversation partner.

  According to the seventh aspect, the recognition accuracy can be improved by recognizing the user's concentration state using the presence or absence of the other party's utterance.

  The eighth invention provides a processor (26) of a state recognition device comprising a storage means (30) for storing a plurality of learning samples, an acquisition means (26, S153, S155) for acquiring user actions, and a recognition standard. A recognition means (26, S213) for recognizing the user state from the user behavior acquired by the acquisition means, and an adjustment means (26, S171, S173) for adjusting the recognition reference based on a plurality of learning samples. It is a state recognition program that functions.

  In the eighth invention, similarly to the first invention, the recognition standard is adjusted using the learning sample sampled from the behavior of the specific user so that the user's state can be easily and correctly recognized. Become.

  According to a ninth aspect of the present invention, there is provided a processor (26) of the state recognition device (10) including a storage unit (30) for storing a plurality of learning samples, an acquisition unit (S153, S155) for acquiring a user action, a plurality of Based on the learning sample, a creation unit (S313) for creating a recognition sample in which the weight is adjusted from the user's action acquired by the acquisition unit, and the weight created by the creation unit having a recognition criterion is adjusted It is a state recognition program that functions as recognition means (S315) for recognizing the user state of a user from a recognition sample.

  In the ninth invention as well, as with the fourth invention, it is only necessary to adjust the weight of the recognition sample, so that the state of a specific user can be easily recognized correctly without changing the recognition standard.

  A tenth aspect of the present invention is an acquisition means (S153, S155) for acquiring a user's action, a recognition means (26, S213) having a recognition criterion and recognizing a user state from the user's action acquired by the acquisition means. In the state recognition method of the state recognition device (10), which includes a storage means (30) for storing the learning samples, the action of the user is acquired by the acquisition means (S153, S155), and recognition is performed based on the plurality of learning samples. The state recognition method is characterized in that the reference state is adjusted (S171, S173), and the user state is recognized from the user behavior acquired by the acquisition unit by the recognition unit whose recognition standard is adjusted.

  In the tenth invention, similarly to the first invention, the recognition standard is adjusted using the learning sample sampled from the behavior of a specific user so that the user's state can be easily and correctly recognized. Become.

  An eleventh aspect of the invention is a state recognition method of the state recognition device (10) that includes storage means (30) for storing a plurality of learning samples, acquires a user's action (S153, S155), and a plurality of learnings Based on the sample, a recognition sample in which the weight is adjusted from the user's action acquired by the acquisition unit is created (S313), and the user state of the user is recognized from the recognition sample having the recognition criterion and adjusted in the weight. (S315) is a state recognition method.

  In the eleventh invention, as in the fourth invention, it is only necessary to adjust the weights of the recognition samples, so that the state of a specific user can be easily and correctly recognized without changing the recognition criteria.

  According to the present invention, since the recognition standard is adjusted using a learning sample sampled from the behavior of a specific user, the user's state can be easily and correctly recognized.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an outline of a listening dialogue sustaining system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing an example of a positional relationship among the monitor camera, the monitor, the robot, and the user shown in FIG. 1 and an imaging range of the monitor camera and the abdominal camera. FIG. 3 is a block diagram showing an example of the electrical configuration of the PC shown in FIG. FIG. 4 is an illustrative view showing the appearance of the robot shown in FIG. 1 from the front. FIG. 5 is a block diagram showing an example of the electrical configuration of the robot shown in FIG. FIG. 6 is a block diagram showing an example of the electrical configuration of the server shown in FIG. FIG. 7 is an illustrative view showing one example of an action table stored in the memory shown in FIG. FIG. 8 is an illustrative view showing one example of image data taken by the monitor camera and the abdominal camera shown in FIG. FIG. 9 is an illustrative view showing one example of a time series graph based on an action table stored in the memory shown in FIG. FIG. 10 is an illustrative view showing one example of a composite action obtained from the time series data shown in FIG. FIG. 11 is an illustrative view showing one example of an action frequency of the composite action shown in FIG. FIG. 12 is an illustrative view showing one example of an SVM constructed by the PC shown in FIG. FIG. 13 is an illustrative view showing one example of the occurrence frequency of elemental behavior obtained from the behavior table stored in the memory shown in FIG. FIG. 14 is an illustrative view showing a list of behavior frequencies of complex behaviors calculated from the occurrence frequency of elemental behaviors shown in FIG. FIG. 15 is an illustrative view showing one example of numerical values used for personalizing the SVM shown in FIG. FIG. 16 is an illustrative view showing one example of a boundary line that is changed by personalizing the SVM shown in FIG. FIG. 17 is an illustrative view showing one example of a memory map of the memory of the PC shown in FIG. 18 is an illustrative view showing one example of a data storage area shown in FIG. FIG. 19 is an illustrative view showing one example of a configuration of the behavior determination program shown in FIG. FIG. 20 is a flowchart showing image / sound acquisition processing of the processor of the PC shown in FIG. FIG. 21 is a flowchart showing speech determination processing of the processor of the PC shown in FIG. FIG. 22 is a flowchart showing gaze direction determination processing of the processor of the PC shown in FIG. FIG. 23 is a flowchart showing the forward tilt posture determination process of the processor of the PC shown in FIG. FIG. 24 is a flowchart showing the whirling determination process of the PC processor shown in FIG. FIG. 25 is a flowchart showing the speech determination process of the other party of the PC processor shown in FIG. FIG. 26 is a flowchart showing synchronization processing of the processor of the PC shown in FIG. FIG. 27 is a flowchart showing sample accumulation processing of the processor of the PC shown in FIG. FIG. 28 is a flowchart showing the learning process of the processor of the PC shown in FIG. FIG. 29 is a flowchart showing the recognition processing of the processor of the PC shown in FIG. FIG. 30 is a flowchart showing centralized recognition processing of the processor of the PC shown in FIG. FIG. 31 is a flowchart showing robot control processing of the processor of the PC shown in FIG. FIG. 32 is an illustrative view showing one example when a recognition sample sampled from the behavior table stored in the memory shown in FIG. 3 is generalized. FIG. 33 is a flowchart showing the learning process of the second embodiment of the processor of the PC shown in FIG. FIG. 34 is a flowchart showing centralized recognition processing of the second embodiment of the processor of the PC shown in FIG. FIG. 35 is an illustrative view showing an outline of a listening dialogue sustaining system in another embodiment. FIG. 36 is an illustrative view showing a positional relationship between two users using the listening dialogue sustaining system of another embodiment shown in FIG.

<First embodiment>
Referring to FIG. 1, the listening dialogue sustaining system 100 according to this embodiment is used for a dialogue between a user B and a user B having a mild brain disorder such as a patient with dementia. Therefore, the listening dialogue sustaining system 100 includes a stuffed robot (hereinafter simply referred to as “robot”) 12a including a PC 10a and an abdominal camera 14a installed in the room 1 where the user A is located, a monitor 16a, a speaker 18a, and a microphone. 20a and monitor camera 22a, PC 10b installed in room 2 (remote location) where user B is located, robot 12b including abdominal camera 14b, monitor 16b, speaker 18b, microphone 20b and monitor camera 22b, and connected to network 200 Server 24. In the present specification, when the corresponding devices and persons in the room 1 and the room 2 are described without distinction, the alphabets attached to the reference numerals are omitted.

  The robot 12 performs a listening operation and a speech based on a control signal from the PC 10. The abdominal camera 14 provided on the abdomen of the robot 12 captures the user and outputs an image to the PC 10 via the robot 12. The PC 10 outputs a control signal to the robot 12 and receives an image captured by the abdominal camera 14 and the monitor camera 22 and a sound collected by the microphone 20. And PC10 recognizes a user's state (user state) by judging a user's action based on an inputted picture and sound. Further, the determined user behavior and user status are transmitted to the server 24 via the network 200. The PC 10 is sometimes called a state recognition device in order to recognize the user's state.

  The PC 10, the monitor 16, the speaker 18, the microphone 20, and the monitor camera 22 function as a video phone. For example, the PC 10a receives the image and sound of the user B transmitted from the PC 10b on the user B side. Therefore, the monitor 16a displays the image of the user B, and the speaker 18 outputs the user B's voice. Further, the microphone 20 collects the voice of the user A and outputs it to the PC 10, and the monitor camera 22 captures an image of the user A and outputs it to the PC 10. Then, the PC 10 transmits the image and sound of the user A to the PC 10b via the network 200. Therefore, the listening dialogue sustaining system 100 is sometimes called a videophone system.

  When the server 24 receives the data of the actions and states of the user A and the user B transmitted from the PC 10a and the PC 10b, the server 24 accumulates them in a database (DB). And when there exists a request | requirement which acquires the data of action and a state from PC10, data are transmitted to PC10 based on the request | requirement.

  In another embodiment, the robot 12 and the PC 10 may be wirelessly connected instead of wiredly connected. Further, the connection between the PC 10 and the server 24 with the network 200 may be a wired connection or a wireless connection.

  FIG. 2 shows an embodiment of the embodiment shown in FIG. 1 viewed from the side. As can be seen from FIG. 2, the monitor camera 22 is placed on the monitor 16, and the robot 12 and the monitor 16 are placed on the desk. The user is photographed by the abdominal camera 14 and the monitor camera 22 while facing the monitor 16 and the monitor camera 22 placed on the desk. Furthermore, since the robot 12 is disposed between the user and the monitor 16, the monitor camera 22 photographs the robot 12 and the user at the same time. As a result, the robot 12 performs a pseudo-listening operation (pseudo-listening operation) on the user A, or performs a pseudo-listening operation on the monitor 16a on which the user B is displayed.

  Note that the robot 12 does not have to be placed on a desk as long as it is photographed by the monitor camera 22 and can be photographed by the user.

  FIG. 3 is a block diagram showing the electrical configuration of the PC 10. The PC 10 has a processor 26 called a microcomputer or CPU. The processor 26 is connected to the memory 30, the audio input / output board 32, the I / O 34, and the communication LAN board 36 via the bus 28. The processor 26 has a built-in RTC (Real Time Clock) 26a for outputting date information.

  The memory 30 functioning as a storage means incorporates ROM, RAM, and HDD (not shown). The ROM is mainly expressed by a program for realizing a telephone function and flowcharts (FIGS. 20 to 31) described later. Program is stored. The RAM is mainly set with a buffer for temporarily storing images taken by the abdominal camera 14 and the monitor camera 22 and sounds collected by the microphone 20. The HDD mainly stores the result of determining the user's action, the result of recognizing the state, and the like.

  The audio data of the other user is given to the speaker 18 from the processor 26 via the audio input / output board 32, and in accordance with the audio data, audio corresponding to the data is output from the speaker 18. Then, the other user's voice collected by the microphone 20 is taken into the processor 26 via the voice input / output board 32.

  Each of the I / Os 34 is a digital port that can control input / output. From the output port, a control signal is output to the robot 12 and an image signal is output to the monitor 16. In addition, video signals are output from the robot 12 and the monitor camera 22 and applied to the input port.

  The communication LAN board 36 is configured by, for example, a DSP (Digital Signal Processor), and supplies transmission data given from the processor 26 to the wireless communication device 38. The wireless communication device 38 transmits the transmission data to an external computer (the server 24 and the partner PC 10) via the network 200. The communication LAN board 36 receives data via the wireless communication device 38 and gives the received data to the processor 26.

  For example, the transmission data may be a command, image data and audio data necessary for a videophone, a result of determining a user's action, and a result of recognizing the user's state. The received data may be image data and audio data of the other party obtained as a video phone, or a result of determining the action and state of the other user and recognizing the state.

  FIG. 4 illustrates the appearance of the robot 12. The robot 12 includes a head 42 and a body 44 that supports the head 42. The left arm 46L and the right arm 46R are provided on the left and right of the upper part of the body 44 (corresponding to a human shoulder), and the abdomen camera 14 is provided on the abdomen of the body 44. As the abdominal camera 14, a camera using a solid-state image sensor such as a CCD or a CMOS can be employed. The head 42 is provided with a mouth 48 on the front surface, and an eyeball 50 is provided above the mouth 48. An ear 52 is attached to the upper side surface of the head 42.

  The head 42 is supported by the body 44 so as to be able to turn and rise, and the eyeball 50 is also operatively held. Further, the body 44 can be tilted in the left-right direction around the waist. Further, the mouth 48 has a built-in speaker 72 (FIG. 5), and the ear 52 has a built-in microphone 74 (FIG. 5).

  If the microphones 74 are built in both ears 52, respectively, the microphones 74 function as stereo microphones, whereby the position of the sound input to the stereo microphones can be specified as necessary. Further, the appearance of the robot 12 is not limited to a bear, but may be other animals or a humanoid.

  FIG. 5 is a block diagram showing the electrical configuration of the robot 12. Similar to the PC 14, the robot 12 includes a processor 54. The processor 26 is connected to the memory 58, the motor control board 60, the audio input / output board 70, the sensor input / output board 76, and the I / O 78 via a bus 56 that is an example of a communication path.

  The memory 58 incorporates a ROM and a RAM (not shown). The ROM mainly includes a listening operation by the robot 12, a program for making a speech, and audio data output from the speaker 72 when the speech is made. Stored in advance. The RAM is used as a temporary storage memory and a working memory.

  The motor control board 60 is composed of, for example, a DSP, and controls each axis motor of each arm and head of the robot 12 shown in FIG. That is, the motor control board 60 receives the control data from the processor 54 and controls the three angles for controlling the X, Y, and Z axes so that the right arm 46R (FIG. 4) can be moved back and forth and left and right. The rotation angle of the motor 62R (collectively shown as “right arm motor” in FIG. 5) 62R is adjusted. Further, the motor control board 60 adjusts the rotation angle of three motors 62L of the left arm 46L (collectively shown as “left arm motor” in FIG. 5) 62L. The motor control board 60 also adjusts the rotation angle of three motors 64 (collectively shown as “head motors” in FIG. 5) that control the turning angle and the elevation angle of the head 42. The motor control board 60 also controls an eyeball motor 66 that moves the eyeball 50 and a waist motor 68 that tilts the body 44.

  Note that the motors described above are stepping motors or pulse motors in order to simplify the control, but may be direct current motors.

  The synthesized voice data is given to the speaker 72 from the processor 54 via the voice input / output board 70, and the voice or voice according to the data is outputted from the speaker 72 accordingly. Then, the sound collected by the microphone 74 is taken into the processor 54 via the sound input / output board 70.

  Similar to the motor control board 60, the sensor input / output board 76 is constituted by a DSP, takes in a signal from the abdominal camera 14, and gives it to the processor 54. The video signal from the abdominal camera 14 is input to the processor 54 after being subjected to predetermined processing by the sensor input / output board 76 as necessary.

  Similar to the I / O 32 of the PC 10, the I / O 78 is a digital port that can control input / output, and a video signal is output from the output port and applied to the PC 10. On the other hand, a control signal is output from the PC 10 and applied to the input port.

  FIG. 6 is a block diagram showing the electrical configuration of the server 24. Similarly to the processors 26 and 54, the server 24 incorporates a processor 80. The processor 80 is connected to the memory 84, the first user information DB 86, the second user information DB 88, and the communication LAN board 90 via the bus 82.

  The memory 84 incorporates a ROM and a RAM (not shown), and mainly stores a program for performing data communication between the server 24 and the PCs 10a and 10b. The RAM is used as a temporary storage memory and a working memory.

  The first user information DB 86 is a database for accumulating user A's behavior data and state data transmitted from the PC 10a. The second user information DB 88 is a database for accumulating user B's behavior data and state data transmitted from the PC 10b. The first user information DB 86 and the second user information DB 88 are configured from storage media such as HDDs and SSDs.

  Similar to the communication LAN board 38 of the PC 10, the communication LAN board 90 is composed of, for example, a DSP, and provides transmission data given from the processor 80 to the wireless communication device 92. The wireless communication device 92 transmits the transmission data to an external computer (PC 10a, 10b) via the network 200. The communication LAN board 90 receives data via the wireless communication device 92 and provides the received data to the processor 80.

  For example, the received data is the action data of the user A transmitted from the PC 10a, and the processor 80 stores the action data of the user A in the first user information DB 86. Further, when the action data acquisition request of the user A is given to the processor 80 from the PC 10b as received data, the processor 80 gives the action data of the user A to the communication LAN board 90 as transmission data.

  FIG. 7 shows an action table stored in the memory 30 of the PC 10. The behavior table is a table in which the result of determining the user's behavior is used as behavior data, and the behavior data is recorded every predetermined time (for example, 1 second).

  Referring to FIG. 7, the action table is composed of columns of “time”, “speech”, “gaze direction”, “forward tilt”, “whit”, and “speech of the other party” from the left side. Each action data is composed of a determination result recorded in each column in synchronization with the “time” column.

  The numerical value recorded in the “time” column is the date and time information output by the RTC 26a. For example, “10:00:30” represents “10:00:30”.

  In the “utterance” column, a determination result indicating whether or not the user is speaking is recorded. For example, if “present” is recorded in the “utterance” column, it indicates that the user is speaking, and if “not present” is recorded, it indicates that the user is not speaking. The presence / absence of the utterance is determined from the sound level of the sound data collected by the microphone 20. For example, if the sound level of the sound data is greater than or equal to a predetermined value, it is determined as “present”, and if it is less than the determined value, it is determined as “not present”.

  In the “Gaze direction” column, an object being watched by the user is recorded. For example, if “monitor” is recorded, it indicates that the user is gazing at the monitor 16, and if “robot” is recorded, it indicates that the user is gazing at the robot 12, "Indicates that the user is gazing at something other than the robot 12 or the monitor 16. Then, “monitor”, “robot”, and “other” in the “gazing direction” column are determined by recognizing the face of the user taken by the abdominal camera 14 or the monitor camera 22.

  FIG. 8A shows a success sequence of face recognition results by the monitor camera 22, FIG. 8B shows a success example of face recognition results by the abdominal camera 14, and FIG. Even the camera of shows the state that face recognition failed.

  Referring to FIG. 8A, the left side is an image by the abdominal camera 14, the right side is an image by the monitor camera 22, and both images are images taken at the same time. At this time, the user is watching the monitor camera 22. Therefore, in the image by the monitor camera 22, the face of the user is reflected in the front, and thus the face recognition is successful. On the other hand, in the image by the abdominal camera 14, the user's face is tilted and the face recognition fails. Therefore, the gaze direction of the user is determined as “monitor”.

  Next, referring to FIG. 8B, similar to FIG. 8A, the images are taken at the same time. At this time, since the user is gazing at the abdominal camera 14, face recognition is successful in the image by the abdominal camera 14. On the other hand, the face recognition fails because the user's face is seen in the image by the monitor camera 22. Therefore, the user's gaze direction is determined as “robot”.

  Then, referring to FIG. 8C, the images are taken at the same time as in FIGS. 8A and 8B. At this time, since the user is crawling, face recognition has failed in both the abdominal camera 14 and the monitor camera 22. Therefore, the user's gaze direction is determined as “other”.

  Returning to FIG. 7, the direction in which the user is taking the forward leaning posture is recorded in the “forward leaning posture” column. Also, in the column of “forward tilt posture”, “monitor”, “robot”, and “others” are recorded, similarly to the column of “gaze direction”. Then, “monitor”, “robot”, and “other” in the column of “forward tilt posture” are determined based on the recognition result of the user's face area photographed by the abdominal camera 14 or the monitor camera 22.

  For example, in the image shown in FIG. 8A, since the face area photographed by the monitor camera 22 is equal to or greater than the threshold value, the forward tilt posture is determined as “monitor”. Further, in the image shown in FIG. 8B, the face region photographed by the abdominal camera 14 is equal to or greater than the threshold value, and therefore the forward tilt posture is determined as “robot”. In the image shown in FIG. 8C, since face recognition has failed in either the abdominal camera 14 or the monitor camera 22, the face area is less than the threshold value in either camera image. Therefore, the forward leaning posture is determined as “other”.

  Returning to FIG. 7, the presence / absence of whispering by the user is recorded in the “whispering” column. For example, if “present” is recorded in the “whispering” column, it indicates that the user has asked, and if “not present” is recorded, it indicates that the user has not spoken. The presence / absence of the user's whispering is determined based on the change in the face position in the images taken by the abdominal camera 14 and the monitor camera 22. For example, the position of the face (the center of gravity of the face area) is recognized and recorded at regular intervals. Then, when “whispering” is determined, if the distance between the previous face position and the current face position is more than a certain distance, it is determined as “present”. Also, if the distance between the previous face position and the current face position is less than a certain distance, it is determined as “none”. Note that the user's whispering may be detected from a change in acceleration by attaching a head acceleration sensor to the user.

  In the “other party's utterance” column, whether the other party's user is speaking is recorded. That is, if “present” is recorded in the column of “the other party's utterance”, it indicates that the other party's user is speaking. Further, if “None” is recorded in the “other party utterance” column, it indicates that the other party user is not speaking. The presence / absence of “the other party's utterance” is recorded based on the action data of the other party's user recorded in the server 24. For example, if the behavior table shown in FIG. 7 corresponds to the user A, the behavior data corresponding to the user B is read from the server 24 and the presence / absence of “the other party's utterance” is recorded.

  In addition, although the Haar-like feature value is used in the face recognition process, the face area recognition process, and the face position recognition process described above, other feature values may be used. Further, the behavior table shown in FIG. 7 is stored not only in the memory 30 of the PC 30 but also in the first user information DB 86 and the second user information DB 88 of the server 24.

  Here, in the PC 10, the user state is recognized every predetermined time (for example, 10 seconds) from the action data recorded at regular intervals. The user state includes “utterance in dialogue”, “object of interest”, and “concentration in dialogue”. These user states are recognized by an utterance recognition process, an interest object recognition process, and a centralized recognition process.

  The utterance recognition process is a process for recognizing whether the user is in the “Talk” state where the user is speaking and is speaking, or the “Listen” state where the user is listening to the other user's story. In the speech recognition process, the speech state (talk / listen) is recognized based on the user's speech time.

  Next, interest recognition is a process for recognizing whether the user's interest is “robot”, “monitor”, or “other”. Therefore, in interest recognition, an object (robot, monitor, etc.) that the user is interested in is recognized based on the user's gaze direction among the user's behavior data for a predetermined time.

  The centralized recognition processing is an “active: active” state in which the user actively participates in the dialogue and concentrates on the dialogue, or “passive” in which the user is inactive in the dialogue and does not concentrate on the dialogue. : Passive "state. In the centralized recognition process of this embodiment, the centralized state (active / passive) is recognized by the SVM.

  SVM is a kind of recognition technique (recognition model), and is constructed by learning a plurality of learning samples labeled in advance. A plurality of learning samples in this embodiment are created in advance by the system administrator and stored in the memory 30. When creating a learning sample, first, action data for a predetermined time is cut out from the action table. Next, feature quantities are extracted from the extracted behavior data and manually or passively labeled. Furthermore, a learning sample is completed by associating a label with the extracted feature quantity. And SVM is constructed | assembled by learning the learning sample produced in this way.

  The learning sample will be described with reference to FIGS. FIG. 9 is an illustrative view showing a behavior table by a time series graph. Referring to FIG. 9, in this time series graph, the horizontal axis indicates a change in time, and the vertical axis indicates an action. The behavior on the vertical axis corresponds to “utterance”, “gaze direction”, “forward leaning posture”, “whit”, and “partner's speech” in the column of the behavior table. Furthermore, in the time-series graph, each action is expressed in two states, “Wake up” or “Not up”. For example, in the case of “utterance”, the user is expressed in two states of “speaking” / “not speaking”. In addition, the “gaze direction” is divided into “monitor”, “robot”, and “other”, and is expressed in two states of “gazing” / “not watching”. Then, “speech”, “gaze (monitor)”, “gaze (robot)”, “gaze (other)”, “posture (monitor)”, “posture (robot)”, “posture (other) shown in FIG. ) ”,“ Whispering ”, and“ the other party's utterance ”are elemental actions constituting the extracted feature quantity.

  For example, referring to FIGS. 10A and 10B, when attention is paid to the three element actions of “utterance”, “gaze (monitor)”, and “whit” in FIG. 9, “utterance only”, “gaze” Eight combinations (compound behavior) of “only”, “whispering”, “speech and gaze”, “speech and whispering”, “gaze and whisper”, “speech and gaze and whisper” and “none” are obtained. In the present embodiment, the frequency of occurrence of these complex actions is the feature amount of the learning sample. Further, the frequency of occurrence of the composite action is calculated based on the sampling window described below.

  FIG. 11 (A) is an illustrative view showing labeled behavior data, FIG. 11 (B) is an illustrative view showing sampled behavior data, and FIG. 11 (C) is a frequency at which composite behavior occurs. FIG. Referring to FIG. 11A, labels “active” and “passive” are attached to the time series graph shown in FIG. 10A by the system administrator. Then, learning samples are sampled from the action data labeled “active” and “passive” based on the window width of the sampling window.

  For example, a first learning sample sampled from behavior data labeled “active” and a second learning sampled behavior data labeled “passive” by a sampling window having a window width of 10 seconds. A sample can be represented as shown in FIG. Moreover, when the frequency (occurrence frequency) at which each composite action occurs is calculated from each of the first learning sample and the second learning sample, it can be represented by the graph of FIG.

  Referring to FIGS. 11B and 11C, if the time when “gaze only” occurs in the first learning sample is 2.5 seconds, the occurrence frequency of “gaze only” is “0.25 (= 2.5 / 10) ". If the time when “speech and gaze” occurs is 5.8 seconds, the occurrence frequency of “speech and gaze” is “0.58”. Further, if the time when “speech and gaze and whisper” occurs is 1.7 seconds, the occurrence frequency of “speech and gaze” becomes “0.17”. If the occurrence time of “utterance only”, “whispering only”, “utterance and whispering”, “gaze and whispering”, and “none” is 0 seconds, the frequency of occurrence of these complex actions is “0”. It becomes.

Here, M types of combined action, pat the combined action, d the frequency of the combined action, time t _pat the combined action _occurs, when the window width represented as t _s, a feature vector indicating the feature quantity ( Action frequency) D can be expressed as: If the type of elemental action is m, the number of complex actions is “M = 2 ^m ”.

[Equation 1]
D = [d ₁ , d ₂ ,..., D _M ]
However, d _pat = t _pat / t _s (pat = 1,..., M)
Further, when a plurality of learning samples are acquired from K (k = 1,... K) users, N _k feature vectors acquired based on the action data of user k are D ^(k) _1. ,... D ^(k) _Nk . At this time, the action frequency D ^{￣ (k)} related to the user k is expressed as shown in Expression 2, and the action frequency D ^の of all learning samples learned by the SVM is expressed as shown in Expression 3.

  However, when “Nk” is a subscript, for the sake of expression, “k” is not used as a subscript of “N” except for mathematical expressions, and is described as a subscript. In addition to the formula, the superscript bar is described as a superscript.

In addition, since the occurrence frequency of the composite action of each action is different for each user, each occurrence frequency is normalized. When normalizing the occurrence frequency, the ratio of the occurrence frequency d ^構成 constituting the action frequency D ^の of all the learning samples to the occurrence frequency d ^{￣ (k)} constituting the action frequency D ^{￣ (k)} of the user k. And the product of each occurrence frequency d ^{） (k)} _i of the user k. Then, the action frequency D ^{^ (k)} of the user k composed of each normalized occurrence frequency d ^{￣ (k)} _i can be obtained according to the equation shown in Equation 4. Note that the normalized learning sample is referred to as a general learning sample.

  However, although the subscript “^” is added above “D” in the mathematical expression, it is described as a superscript “D” for other than the mathematical expression for convenience of expression.

And the general learning sample of normalized action frequency D ^{^ (k)} is learned by SVM. Further, the learned SVM sets a boundary line (hyperplane) that separates a plurality of learned general learning samples. Further, the general SVM obtained by learning all the general learning samples can be expressed as shown in FIG.

  Referring to FIG. 12, the general learning samples learned by SVM are arranged on a plane. Of the general learning samples on the plane, learning samples labeled active are indicated by circles, and learning samples labeled passive are indicated by triangles. Further, the boundary line is set at a position for separating the general learning sample with “active” from the general learning sample with “passive”. Then, when recognition processing is performed and a recognition sample is input to this SVM, the concentration state is recognized based on the boundary line. That is, when the recognition sample is arranged on the left side of the boundary line, it is recognized as a passive state, and when it is arranged on the right side of the boundary line, it is recognized as an active state.

  Here, when a user's concentration state is recognized by a general SVM, the occurrence frequency of a specific elemental action may be low depending on the user, so that even if the user is in an active state, the user is recognized as a passive state. There is. Therefore, in the first example, the behavior data of the recognition target user is used to personalize the general SVM to recognize the user state.

  As a method for personalizing a general SVM, first, an individual learning sample is sampled from the behavior data of a user to be recognized, and the occurrence frequency of elemental behavior is obtained. Next, the occurrence frequency of the complex action is estimated from the occurrence frequency of each element action, and the occurrence frequency of the complex action is also estimated from each element action in the general learning sample. Furthermore, the general learning sample is personalized based on the occurrence frequency of the complex behavior estimated from the individual learning sample and the occurrence frequency of the complex behavior estimated from the general learning sample. The general SVM can be personalized by reconstructing the SVM learned from the personalized general learning sample. Hereinafter, these processes will be specifically described.

  First, a personal learning sample is sampled from the action data of the user who recognizes the state. Next, the frequency of occurrence of each elemental behavior is calculated from the individual learning sample.

  For example, referring to FIGS. 13A and 13B, if the window width is 10 seconds and the total time recorded as “speaking” in the personal learning sample is 7.5 seconds, The action frequency of “utterance” is “0.75 (= 7.5 / 10)”. Similarly, if the total time recorded as “gazing at the monitor” is 3.3 seconds, the action frequency of “gazing” is “0.33”, and the total time when “growing” is recorded If it is 0.17 seconds, the action frequency of “whit” is “0.17 (= 1.7 / 10)”.

Here, when the element action is represented as Pri and the time (total recorded time) when the element action occurs is represented as t _pri , the occurrence frequency p of the element action can be expressed as shown in Equation 5.

[Equation 5]
p _{= t} pri / _{t s}
Then, focusing on the fact that the composite action is a combination of element actions, the action frequency of the recognition target user is estimated from the expected value of each element action. Here, assuming that the action frequency estimated from the individual learning sample is Qp, the estimated action frequency Qp is calculated based on the mathematical formula according to Equation 6.

  For example, referring to FIG. 14, the action frequency of “utterance, gaze, and whisper” is the expected value of each element action, that is, “0.042 (= 0.75 × 0.33 × 0.17)”. . In addition, the action frequency of “utterance only” is the action frequency of “utterance” and an expected value at which “gaze” and “whit” do not occur, that is, “0.417 (= 0.75 × 0.67 × 0.83) " And the estimated action frequency Qp is comprised from the expected value (behavior frequency) calculated for every composite action.

  Further, assuming that the behavior frequency estimated from the general learning sample is Qn, the estimated behavior frequency Qn is calculated based on an equation according to Equation 7.

And the action frequency Dp ^{^} of the personalized general learning sample is calculated based on the mathematical formula according to Equation 8.

FIG. 15A is a table showing a plurality of expected values (occurrence frequencies) constituting the estimated action frequency Qn estimated from the general learning sample, and FIG. 15B is an estimated action estimated from the individual learning sample. It is a table which shows a plurality of expected values (occurrence frequency) which constitute frequency Qp. FIG. 15C is a table showing a plurality of occurrence frequencies constituting the behavior frequency D ^{^} of the general learning sample. FIG. 15D is a table showing a plurality of occurrence frequencies constituting the action frequency Dp ^{^} of the personalized general learning sample.

For example, referring to FIG. 15 (A) to FIG. 15 (D), focusing on “utterance only” of the complex action, the occurrence frequency “0.500” of “utterance only” of the action frequency Dp ^{^} is the action frequency. The product of the ratio of the occurrence frequency “0.417” of the action frequency Qp to the occurrence frequency “0.208” of the Qn and the occurrence frequency “0.250” of the action frequency of the general learning sample (0.417 / 0.208) X 0.250). In the table of FIG. 15D, “gaze only”, “whisker only”, “speech and gaze”, “speech and whisper”, “gaze and whisper”, “speech and gaze and whisper” and “none” The occurrence frequency of “is also the result calculated in the same manner as the occurrence frequency of“ utterance only ”described above. And the action frequency D ^{^} p of the personalized general learning sample is constituted by the plurality of occurrence frequencies shown in FIG.

  Furthermore, when all the general learning samples are personalized, the SVM is reconstructed. In other words, a personalized SVM is newly constructed by learning a personalized general learning sample. In the centralized recognition process, since the user's concentration state is recognized by the personalized SVM, the user's concentration state is correctly recognized.

  For example, referring to FIG. 16A, in the case of a user with a low occurrence frequency of “gaze”, a recognition sample sampled from the user is passive even if the recognition sample is recognized by a general SVM even if it is concentrated on conversation. It may be misrecognized as a (non-concentrated) state. However, when the weight of the general learning sample is adjusted using the personal learning sample sampled from the user behavior data, the boundary line of the personalized SVM is set at a different position from the boundary line of the general SVM. The Therefore, the recognition sample of the user is recognized as an active state.

In this way, by adjusting the weight of the occurrence frequency constituting the action frequency D ^{^} p of the general learning sample, the boundary line of the SVM can be changed to a position suitable for the individual action.

  In the above description, only three elemental actions have been described for the sake of simplicity, but in practice, learning and recognition are performed using nine elemental actions.

  In the present embodiment, personal learning samples are accumulated every predetermined time, and the personalized SVM is sequentially updated with the accumulated personal learning samples, thereby improving the recognition accuracy. Specifically, first, every time the personalized SVM is updated, all accumulated personal learning samples are recognized. Next, the recognition result of the concentrated state is labeled on the individual learning sample, and the number of the individual learning samples recognized as the active state is recorded as the recognition degree of the personalized SVM. Each time the personalized SVM is updated, the update of the personalized SVM is repeated until the difference between the previous time and the current level of recognition is equal to or less than a threshold value (predetermined value).

  Therefore, the recognition accuracy of the concentration state is improved in proportion to the time for which the user uses the listening dialog sustaining system 100. Further, by using the stored personalized learning samples for the recognition of the concentration state, it is possible to improve the recognition accuracy of the concentration state while recognizing the user's concentration state. Furthermore, if the recognition accuracy of the personalized SVM becomes a certain accuracy, the processor 26 can end the learning process.

  In the listening dialogue sustaining system 100, the robot 12 operates so that the dialogue between the two people is continued based on the recognized user status and the status data of the other user. Then, the robot 12 performs three types of operations, “pseudo-listening operation”, “speech control operation”, and “attention attracting operation”, for the dialogue between the user A and the user B, and continues the dialogue.

  For example, the pseudo-listening operation is an operation that behaves as if the user A and the user B are actively listening to one of the utterances. Furthermore, when one user is speaking unilaterally, the utterance control action is to suppress the utterance by looking at the user or to talk to the user in order to balance the two utterances. It is an action that promotes speech. The attention inducing and attracting actions are actions to attract the user's attention by talking to the user when the user is not concentrated on the dialogue.

  FIG. 17 is an illustrative view showing one example of a memory map 300 of the memory 30 in the PC 10 shown in FIG. As shown in FIG. 17, the memory 30 includes a program storage area 302 and a data storage area 304. The program storage area 302 includes a data communication program 312, an action determination program 314, a sample accumulation program 316, a learning program 318, a recognition program 320, a centralized recognition program 322, and a robot control program 324 as programs for operating the PC 10. Remembered.

  The data communication program 312 is a program for performing data communication with the server 24. The behavior determination program 314 is a program for determining a user's behavior. The sample accumulation program 316 is a program for accumulating individual learning samples from user behavior every predetermined time. The stored personal learning sample may be read as a recognition sample.

  The learning program 318 is a program for constructing a personalized SVM. The recognition program 320 is a program for reading a recognition sample and recognizing the concentration state, the utterance state, and the object of interest from the recognition sample. The concentrated recognition program 322 is a program for recognizing the user's concentrated state by the personalized SVM constructed by the learning program. The robot control program 324 is a program for determining the operation of the robot 12.

  Although not shown, the program for operating the PC 10 includes a program for realizing a videophone function.

  Referring to FIG. 18, data storage area 304 includes time buffer 330, monitor camera buffer 332, abdominal camera buffer 334, audio buffer 336, determination result buffer 338, face position buffer 340, data communication buffer 342, and SVM. A recognition buffer 344 is provided. The data storage area 304 stores general learning data 346, action table data 348, personal learning data 350, and state data 352, and a concentration flag 354 and a predetermined time counter 356 are further provided.

  The time buffer 330 is a buffer in which date / time information output from the RTC 26a is temporarily stored. The monitor camera buffer 332 is a buffer in which an image taken by the monitor camera 22 is temporarily stored. The abdominal camera buffer 334 is a buffer in which an image taken by the abdominal camera 14 is temporarily stored. The audio buffer 336 is a buffer in which audio collected by the microphone 20 is temporarily stored.

  The determination result buffer 338 determines the utterance determination for determining the presence or absence of the user's utterance, the gaze direction determination for determining the user's gaze direction, the forward tilt posture determination for determining the direction of the user's forward tilt posture, and the presence / absence of the user's whisper. It is a buffer for temporarily storing the determination results of the whisper determination and the other party's speech determination for determining the presence or absence of the other user's speech. The face position buffer 340 is a buffer for storing the position of the user's face in the photographed user image. Further, based on the position data stored in the face position buffer 340, the user's whisper is determined.

  The data communication buffer 342 is a buffer that temporarily stores the other party's action data, state data, and the like obtained by data communication with the server 24. The SVM recognition buffer 344 is a buffer that temporarily stores the recognition degrees of the general SVM and the personalized SVM.

  The general learning data 346 is data composed of a plurality of general learning samples. The behavior table data 348 is the behavior table shown in FIG. 7, and the latest behavior data is added every certain time. The personal learning data 350 is data composed of a plurality of personal learning samples, and the latest personal learning sample is added every predetermined time. The state data 352 is data indicating the result of recognizing the user's concentration state, speech state, and object of interest.

  The concentration flag 354 is a flag indicating the recognition result of the concentration state. For example, the concentration flag 354 is composed of a 1-bit register. When the concentration flag 354 is turned on (established), a data value “1” is set in the register. On the other hand, when the concentration flag 354 is turned off (not established), a data value “0” is set in the register. The concentration flag 354 is turned on when recognized as an active state and turned off when recognized as a passive state.

  The predetermined time counter 356 is a counter for measuring a predetermined time, and starts counting when initialized. The predetermined time counter 356 is also called a predetermined time counter, and the predetermined time counter 356 is initialized when processing for measuring time by a predetermined time timer is executed. For example, the predetermined time counter 356 is initialized when the PC 10 is turned on, and is reset every time a personal learning sample is sampled.

  Although not shown, the data storage area 304 is provided with a buffer for temporarily storing various calculation results and other counters and flags necessary for the operation of the PC 10.

  FIG. 19 is an illustrative view showing a program corresponding to a subroutine of the action determination program 314. Referring to FIG. 19, the situation recognition program 314 includes an image / sound acquisition program 314a, an utterance determination program 314b, a gaze direction determination program 314c, a forward tilt posture determination program 314d, a whisper determination program 314e, an opponent utterance determination program 314f, and It consists of a synchronization program 314g.

  The image / sound acquisition program 314a is a program for taking in an image captured by the monitor camera 22 and the abdomen camera 14 and the sound collected by the microphone 20 into a buffer. The utterance determination program 314b is a program for determining whether or not the user is speaking. The gaze direction determination program 314c is a program for determining the direction in which the user is gazing. The forward lean posture determination program 314d is a program for determining the direction in which the user is taking the forward lean posture. The whisper determination program 314e is a program for determining whether or not the user has whispered. The partner's utterance determination program 314f is a program for determining whether or not the partner user is speaking. The synchronization program 314g is a program for synchronizing the speech determination result, the gaze direction determination result, the forward tilt posture determination result, the whirl determination result, and the partner user's speech determination result into action data.

  Hereinafter, a flowchart of the first embodiment executed by the PC 10 will be described. Also, the flowcharts of FIGS. 20 to 26 show the processing of each program constituting the action determination program 314. Further, the flowchart of FIG. 27 shows processing by the sample accumulation program 316, the flowchart of FIG. 28 shows processing by the learning program, the flowchart of FIG. 29 shows processing by the recognition program 320, and the flowchart of FIG. The process by the centralized recognition program is shown, and the flowchart of FIG. 31 shows the process by the robot control program.

  FIG. 20 is a flowchart showing the processing of the image / sound acquisition program 314a. For example, when the power of the PC 10 is turned on by the user, the processor 26 of the PC 10 acquires image data from the abdominal camera 14 in step S1. That is, image data is acquired from the video signal input from the robot 12 and temporarily stored in the abdominal camera buffer 334. Subsequently, the processor 26 acquires image data from the monitor camera 22 in step S3. That is, image data is acquired from the video signal input from the monitor camera 22 and temporarily stored in the monitor camera buffer 332. Subsequently, in step S5, the processor 26 acquires audio data and returns to step S1. That is, the processor 26 extracts audio data from the audio collected by the microphone 20 and temporarily stores it in the audio buffer 336.

  FIG. 21 is a flowchart showing the processing of the speech determination program 314b. In step S11, the processor 26 determines whether audio data has been acquired. That is, the processor 26 determines whether or not new audio data is stored in the audio buffer 336. If “NO” in the step S11, that is, if the audio data is not acquired, the processor 26 repeatedly executes the process of the step S11. On the other hand, if “YES” in the step S11, that is, if audio data is acquired, the processor 26 determines whether or not the sound volume is equal to or higher than a threshold value in a step S13. That is, the processor 26 determines whether or not the sound level of the sound data stored in the sound buffer 336 is equal to or higher than the threshold value.

  If “YES” in the step S13, that is, if the sound level is equal to or higher than the threshold value, the processor 26 determines that “there is an utterance” in a step S15. On the other hand, if “NO” in the step S13, that is, if the sound level is lower than the threshold value, the processor 26 determines that “no utterance” in a step S17.

  Subsequently, the processor 26 acquires the current time in step S19. That is, the processor 26 acquires date / time information stored in the time buffer 330. Subsequently, in step S21, the processor 26 associates the current time with the utterance determination result. That is, the processor 26 associates the current time with each other in order to synchronize a plurality of determination results. The utterance determination result associated with the current time is temporarily stored in the determination result buffer 338.

  In the other determination processes shown in FIG. 22 to FIG. 25, there is a process for acquiring date / time information as in step S19 and associating the date / time information as in step S21. Detailed description is omitted in other flowcharts.

  FIG. 22 is a flowchart showing the processing of the gaze direction determination program 314c. In step S31, the processor 26 determines whether an image has been acquired. That is, the processor 26 determines whether new image data is stored in the monitor camera buffer 332 and the abdominal camera buffer 334. If “NO” in the step S31, that is, if the image data is not acquired, the processor 26 repeatedly executes the process of the step S31. On the other hand, if “YES” in the step S31, that is, if image data is acquired, the processor 26 acquires an image of the monitor camera 22 in a step S33. That is, the processor 26 acquires image data from the monitor camera buffer 332.

  Subsequently, the processor 26 executes face recognition processing in step S35. That is, the processor 26 executes a predetermined face recognition process on the image data stored in the monitor camera buffer 332. Subsequently, the processor 26 determines whether or not the face has been recognized in step S37. That is, the processor 26 determines whether or not the process of step 35 is successful. If “YES” in the step S37, that is, if the face is successfully recognized in the image taken by the monitor camera 22, the processor 26 sets the gaze direction to “monitor” in a step S39.

  If “NO” in the step S37, that is, if the face recognition has failed in the image taken by the monitor camera 22, the processor 26 acquires an image of the abdominal camera in a step S41. That is, the processor 26 acquires image data from the abdominal camera buffer 334. Subsequently, the processor 26 performs face recognition processing on the image data stored in the abdominal camera buffer 334 in step S43, as in step S35.

  Subsequently, the processor 26 determines whether or not a face is recognized in step S45. That is, the processor 26 determines whether or not the face recognition is successful in the process of step S43. If “YES” in the step S45, that is, if the face has been successfully recognized in the image taken by the abdominal camera 14, the processor 26 sets the gaze direction to “robot” in a step S47.

  On the other hand, if “NO” in the step S45, that is, if the user's face recognition has failed in both the monitor camera 22 and the abdominal camera 14, the processor 26 sets the gaze direction to “others” in the step S49. To "".

  Subsequently, the processor 26 acquires the current time in step S51, and associates the current time with the utterance determination result in step S53. And the processor 26 returns to step S31, after the process of step S53 is complete | finished. In step S53, the determination result of the gaze direction associated with the current time is temporarily stored in the determination result buffer 338.

  In another embodiment, the gaze direction may be determined using only one image of the monitor camera 22 or the abdominal camera 14. Also, in the other determination processes shown in FIG. 23 and FIG. 24, the same processes as in steps S31, S33, and S41 exist, but the process contents are all the same, and thus detailed description is omitted in the other flowcharts. .

  FIG. 23 is a flowchart showing the processing of the forward tilt posture determination program 314d. In step S61, the processor 26 determines whether an image has been acquired. If “NO” in the step S61, that is, if an image is not acquired, the process of the step S61 is repeatedly executed. On the other hand, if “YES” in the step S61, that is, if an image is acquired, the processor 26 acquires an image of the monitor camera 22 in a step S63.

  Subsequently, the processor 26 executes face area recognition processing in step S65. That is, the processor 26 recognizes the face area from the image data stored in the monitor camera buffer 332. Subsequently, in step S67, the processor 26 determines whether or not the face area is equal to or greater than a threshold value (for example, a value indicating 50% of the area of the image data). That is, the processor 26 determines whether or not the area of the recognized face area is equal to or greater than a threshold value. If “YES” in the step S67, for example, as shown in FIG. 8A, if the user's face is photographed, the processor 26 sets the forward tilt posture to “monitor” in a step S69.

  On the other hand, if “NO” in the step S67, if the area of the face region is less than the threshold value or the recognition of the face region fails, the processor 26 acquires an image of the abdominal camera 14 in a step S71. Subsequently, the processor 26 executes face area recognition processing in step S73. That is, the processor 26 recognizes the face area from the image data stored in the abdominal robot buffer 334. Subsequently, in step S75, the processor 26 determines whether or not the face area is greater than or equal to a threshold value. That is, the processor 26 determines whether or not the user's face is greater than or equal to the threshold in the image data captured by the abdominal camera 14. If “YES” in the step S75, for example, as shown in FIG. 8B, if the user's face is photographed, the processor 26 sets the forward tilt posture to “robot” in a step S77.

  On the other hand, if “NO” in the step S75, for example, as shown in FIG. 8C, if the user's face is photographed, the processor 26 sets the forward leaning posture to “other” in a step S79. To do.

  Subsequently, the processor 26 acquires the current time in step S81, and associates the current time with the utterance determination result in step S83. Then, when the process of step S83 ends, the processor 26 returns to step S61. In step S83, the gaze direction determination result associated with the current time is temporarily stored in the determination result buffer 338.

  In another embodiment, the forward tilt posture determination may be performed using only one image of the monitor camera 22 or the abdominal camera 14.

  FIG. 24 is a flowchart showing the processing of the whirl determination program 314e. The processor 26 determines whether an image has been acquired in step S91. If “NO” in the step S91, that is, if an image is not acquired, the processor 26 repeatedly executes the process of the step S91. On the other hand, if “YES” in the step S91, that is, if an image is acquired, the processor 26 acquires an image of the monitor camera 22 in a step S91.

  Subsequently, the processor 26 executes face position recognition processing in step S95. That is, the processor 26 recognizes the position of the center of gravity of the face area in the image data stored in the monitor camera buffer 332. Subsequently, in step S97, the processor 26 determines whether or not the face position has changed. That is, the processor 26 acquires the previous face position corresponding to the monitor camera 22 from the face position buffer 340. Further, the processor 26 compares the acquired previous face position with the current face position recognized in step S95, and determines whether or not the user's face position has changed. If “YES” in the step S97, that is, if the user's face position changes in the image taken by the monitor camera 22, the processor 26 determines that the whisper is “present” in a step S99.

  If “NO” in the step S97, that is, if the user's face position is not changed in the image of the monitor camera 22, the processor 26 acquires the image of the abdominal camera 14 in a step S101. Subsequently, in step S103, the processor 26 executes face position recognition processing. That is, the processor 26 recognizes the gravity center position of the face area in the image data stored in the abdominal camera buffer 334. Subsequently, in step S105, the processor 26 determines whether or not the face position has changed. That is, the processor 26 acquires the previous face position corresponding to the abdominal camera 14 from the face position buffer 340, and determines whether or not the user's face position has changed as in step S97. If “YES” in the step S105, that is, if the user's face position has changed in the image of the abdominal camera 14, the processor 26 determines that the whisper is “present” in a step S99. On the other hand, if “NO” in the step S105, that is, if a change in the user's face position is not detected even in the image of the abdominal camera 14, the processor 26 determines that the user's whisper is “none” in a step S107.

  Subsequently, the processor 26 stores the current face position in step S109. That is, the processor 26 stores the face position of the user recognized in steps S95 and S103 in the face position buffer 340.

  Subsequently, the processor 26 acquires the current time in step S111, and associates the current time with the utterance determination result in step S113. And the processor 26 returns to step S91, after the process of step S113 is complete | finished. Further, in step S113, the determination result of the handing associated with the current time is temporarily stored in the determination result buffer 338.

  Note that the processor 26 that executes the processes of steps S13, S37, S45, S67, S75, S97, and S105 functions as an element behavior determination unit.

  FIG. 25 is a flowchart showing the processing of the partner's speech determination program 314f. In step S121, the processor 26 acquires the other party's action data. That is, the processor 26 establishes data communication with the server 24 and stores the other party's action data in the data communication buffer 342. Subsequently, the processor 26 determines whether or not the other party has spoken in step S123. That is, the processor 26 determines whether or not “present” is recorded in the utterance column in the behavior data of the other party stored in the data communication buffer 342. If “YES” in the step S123, that is, if the utterance is recorded as “present” in the behavior data of the other party, the processor 26 determines in the step S125 that the other party has the utterance. On the other hand, if “NO” in the step S123, that is, if the utterance is recorded as “none” in the behavior data of the other party, the processor 26 determines that the other party's utterance is “none” in a step S127. Note that the processor 26 that executes the process of step S123 functions as a partner utterance determination unit.

  Subsequently, the processor 26 acquires the current time in step S129, and associates the current time with the utterance determination result in step S131. And the processor 26 returns to step S121, after the process of step S131 is complete | finished. In step S131, the determination result of the other party's utterance associated with the current time is temporarily stored in the determination result buffer 338.

  FIG. 26 is a flowchart showing the processing of the synchronization program 314g. The processor 26 determines whether or not each determination is completed in step S141. For example, the processor 26 determines whether or not the determination result buffer 338 stores the determination results of the utterance, the gaze direction, the forward leaning posture, the whispering, and the opponent's utterance. If “NO” in the step S141, that is, if each determination is not completed, the processor 26 repeatedly executes the process of the step S143. On the other hand, if “YES” in the step S143, that is, if each determination is completed, the processor 26 synchronizes each determination result and the face recognition result based on the time in a step S143. That is, the processor 26 synchronizes based on the time associated with each determination result.

  Subsequently, in step S145, the processor 26 sets each synchronized determination result as behavior data and records it in the behavior table. That is, the processor 26 records each determination result in a new row in the behavior table shown in FIG. Subsequently, the processor 26 transmits the current behavior data to the server 24 in step S147. Then, the processor 26 returns to step S141 when the process of step S147 is completed. That is, the processor 26 transmits the behavior data corresponding to the newly added row to the server 24 in the behavior table.

  As described above, the actions of the user are determined and transmitted to the server 24 by executing the processes of FIGS.

  FIG. 27 is a flowchart showing the processing of the sample accumulation program 316. The processor 26 determines whether or not the timer has expired for a predetermined time in step S151. That is, the processor 26 determines whether or not a predetermined time has elapsed since the previous personal learning sample was accumulated based on the value of the predetermined time counter 356. Subsequently, the processor 26 acquires action data for a predetermined time in step S153. That is, the processor 26 reads behavior data for a predetermined time from the behavior table data 348 based on the time column of the behavior table.

  Subsequently, in step S155, the processor 26 calculates the action frequency of the composite action. For example, the processor 26 calculates the occurrence frequency of the element behavior from the read behavior data. Then, the processor 26 calculates an expected value of the calculated occurrence frequency as the action frequency of the composite action. Subsequently, in step S157, the processor 26 adds (accumulates) the calculated action frequency to the personal learning data 350 as a personal learning sample. For example, the processor 26 stores the behavior frequency calculated as shown in FIG. 14 in the RAM 30 as a personal learning sample constituting the personal learning data 350. Subsequently, in step S159, the processor 26 resets the timer for a predetermined time, and returns to step S151. That is, the processor 26 initializes the predetermined time counter 354.

  FIG. 28 is a flowchart showing the processing of the learning program 318. The processor 26 acquires the general learning data 346 in step S161. That is, the processor 26 reads the general learning data 346 from the RAM 30. Subsequently, the processor 26 constructs a general SVM in step S163. That is, the processor 26 constructs a general SVM as shown in FIG. 16A from a plurality of general learning samples constituting the read general learning data 346. Subsequently, in step S167, the processor 26 recognizes all personal learning samples constituting the personal learning data 346 by the general SVM. That is, in order to improve the recognition accuracy of the personalized SVM, the personal learning sample is first recognized by the general SVM. Subsequently, in step S169, the processor 26 records the recognition degree of the general SVM. That is, the processor 26 stores, in the SVM recognition buffer 344, the number of personal learning samples recognized as the active state among the plurality of personal learning samples constituting the personal learning data 350 as the recognition degree of the general SVM.

  Subsequently, the processor 26 adjusts the weight of the general learning sample using the personal learning data 350 in step S171. That is, the processor 26 calculates the expected value of the composite action from the individual learning sample and the general learning sample based on the mathematical formulas 6 and 7, and configures the general learning data 346 based on the mathematical formula 8 Adjust the weight of the general learning sample. Subsequently, the processor 26 constructs a personalized SVM at step S173. That is, the processor 26 constructs a personalized SVM as shown in FIG. 16B from the general learning sample (personalized general learning sample) whose weight is adjusted. Note that the processor 26 that executes the processes of steps S171 and S173 functions as an adjusting unit. In addition, the processor 26 that executes the process of step S171 functions as a weight adjusting unit.

  Subsequently, in step S175, the processor 26 recognizes all personal learning samples constituting the personal learning data 350 by the personalized SVM. That is, the processor 26 recognizes the personal learning sample by the personalized SVM in order to improve the recognition accuracy of the personalized SVM. Subsequently, in step S177, the processor 26 stores the recognition degree of the personalized SVM. That is, the processor 26 stores, in the SVM recognition buffer 344, the number of personal learning samples recognized as an active state among the plurality of personal learning samples constituting the personal learning data 346 as the recognition degree of the personalized SVM. The processor 26 that executes the process of step S175 functions as a temporary recognition unit, and the processor 26 that executes the process of step S177 functions as a recording unit.

  Subsequently, in step S179, the processor 26 determines whether or not the difference between the previous recognition level and the current recognition level is equal to or less than a threshold value. For example, the processor 26 calculates the difference between the recognition degree of the general SVM and the recognition degree of the personalized SVM stored in the SVM recognition buffer 344, and whether the difference is 10% or less of the total number of personal learning samples. Judge whether or not. When the processes in steps S171 to S181 are performed for the second time or later, the determination in step S179 is determined from the previous recognition level and the current recognition level by the personalized SVM. If “YES” in the step S179, that is, if the difference in recognition degree is equal to or less than the threshold value, the processor 26 ends the learning process.

  If “NO” in the step S179, that is, if the difference in recognition degree is larger than the threshold value, the processor 26 determines whether or not an individual learning sample is added in a step S181. That is, the processor 26 determines whether or not a new personal learning sample has been added to the personal learning data 350. If “NO” in the step S181, for example, if the predetermined time has not elapsed and the personal learning sample has not been added, the processor 26 repeatedly executes the process of the step S181. On the other hand, if “YES” in the step S181, that is, if a personal learning sample is added, the processor 26 returns to the step S171. In step S171, the weight of the general learning sample is adjusted using the individual learning data 350 including the added individual learning sample.

  In other embodiments, steps S163 to S169 and steps S175 to S181 may be omitted, and a personalized SVM may be constructed using only already accumulated personal learning samples. For example, if the user to be recognized is a user who has provided a personal learning sample used to create a general learning sample, many personal learning samples have already been accumulated, so the process is simplified as described above. It becomes.

  FIG. 29 is a flowchart showing the processing of the recognition program 320. For example, when the power of the PC 10 is turned on by the user, the processor 26 of the PC 10 determines whether or not a personal learning sample has been added in step S191 as in step S181. If “NO” in the step S191, that is, if a personal learning sample is not added, the processor 26 repeats the process of the step S191. On the other hand, if “YES” in the step S191, that is, if a personal learning sample is added, the processor 26 acquires the added personal learning sample as a recognition sample in a step S193. That is, the processor 26 reads the personal learning sample added last from the personal learning data 350 as a recognition sample in order to recognize the concentration state.

  Subsequently, the processor 26 executes centralized recognition processing in step S195. Further, since the process of step S195 will be described later with reference to the flowchart shown in FIG. 30, detailed description thereof is omitted here. Subsequently, the processor 26 executes an utterance recognition process in step S197, and executes an interest object recognition process in step S199. As described above, the processor 26 recognizes a talk state in which the user is speaking as a speaker or a listen state in which the user is listening to the other party's story by the speech recognition process. Further, the processor 26 recognizes the user's interest (robot, monitor, etc.) by the interest recognition process.

  Subsequently, the processor 26 stores each recognition result as the state data 352 in step S201. For example, if the result of centralized recognition is active, the result of speech recognition is talk, and the result of interest recognition is “robot”, the processor 26 indicates that the state data 352 includes “active talk robot”. Is stored in the memory 30. Subsequently, in step S83, the processor 26 transmits the status data 352 to the server 24, and returns to step S191.

  FIG. 30 is a flowchart showing the processing of the centralized recognition program 322. When the process of step S195 is executed, the processor 26 calculates the action frequency of the composite action from the acquired recognition sample in step S211. That is, as shown in FIG. 14, the processor 26 calculates the action frequency of the composite action from the occurrence frequency of the element action in the recognition sample. Subsequently, in step S213, the processor 26 recognizes the user's concentration state. That is, the processor 26 inputs the action frequency calculated in step S211 to the personalized SVM and recognizes the user's concentration state. The processor 26 that executes the process of step S213 functions as a recognition unit.

  Subsequently, the processor 26 determines whether or not the recognition result is in an active state in step S215. If “YES” in the step S215, that is, if the recognition result by the personalized SVM is in the active state, the processor 26 sets the active state in step S217, and ends the centralized recognition process. That is, the processor 26 sets the concentration flag 354 to ON. On the other hand, if “NO” in the step S215, that is, if the recognition result by the personalized SVM is in the passive state, the processor 26 sets the passive state in step S219 and ends the centralized process. That is, the processor 26 sets the concentration flag 354 to OFF.

  Then, when the centralized recognition process is completed, the processor 26 returns to the recognition process and executes the process of step S197.

  FIG. 31 is a flowchart showing the processing of the robot control program 324. For example, when the telephone call by the videophone function is started, the processor 26 of the PC 10 determines whether or not the end operation is performed in step S231. For example, the processor 26 determines whether or not an operation for ending a call using the videophone function has been performed. If “YES” in the step S231, that is, if an end operation is performed, the processor 26 ends the robot control processing. On the other hand, if “NO” in the step S231, that is, if the end operation is not performed, the processor 26 refers to the state data 352 in a step S233.

  Subsequently, in step S235, the processor 26 determines whether or not the monitor state is set. That is, the processor 26 determines whether or not “monitor” indicating that the user's interest is the monitor 16 is included in the state data 352. If “NO” in the step S235, that is, if the user's interest is not the monitor 16, the processor 26 proceeds to a step S245. On the other hand, if “YES” in the step S235, that is, if the user's interest is the monitor 16, the processor 26 determines whether or not it is active in a step S237. That is, the processor 26 determines whether or not “active” indicating that the user is in the active state is included in the state data 352. If “NO” in the step S237, that is, if the user is in a passive state, the processor 26 proceeds to a step S247.

  On the other hand, if “YES” in the step S237, that is, if the user is in an active state, the processor 26 determines whether or not it is a talk in a step S239. That is, the processor 26 determines whether or not the status data 352 includes “talk” indicating that the user is in the talk state. If “YES” in the step S239, that is, if the user is in the talk state, the state data 352 is “active talk monitor”, and therefore the processor 26 executes the active talk process in the step S171, and the step S231. Return to. When this active talk process is executed, the robot 12 performs a pseudo-listening operation and an utterance control operation.

  If “NO” in the step S239, that is, if the user is in a listening state, the state data 352 is “active listening monitor”, and therefore the processor 26 executes an active listening process in a step S243, and the step S161. Return to. When this active listening process is executed, the robot 12 performs a pseudo-listening operation and an operation for prompting the user to speak.

  On the other hand, if the user's interest is other than the monitor 16, the processor 26 determines whether or not the robot is in a step S145. That is, the processor 26 determines whether or not the status data 352 includes “robot” indicating that the user is interested in the robot 12. If “YES” in the step S245, that is, if the user's interest is the robot 12, the processor 26 executes an inactive process in a step S247, and the process returns to the step S231. When this inactive process is executed, the robot 12 performs an operation that attracts the user's attention and an operation that prompts the user to speak.

  On the other hand, if “NO” in the step S245, that is, if the user is not interested in the robot 12 or the monitor 16, the other process is executed in a step S249, and the process returns to the step S231. When the other process is executed, the robot 12 performs an action to attract attention and an action to talk to the user.

  As described above, in the first embodiment, the user's concentration state is correctly recognized because the SVM is personalized based on the user's behavior.

<Second embodiment>
In the second embodiment, the concentration state of the recognition target user is correctly recognized by generalizing the recognition sample, not personalizing the SVM. In the second embodiment, the recognition sample is generalized based on the mathematical formula shown in Equation 4 that is used to normalize (generalize) the occurrence frequency of the user.

  Note that the listening dialogue sustaining system 100 of the second embodiment is the same as that of the first embodiment, and therefore, redundant description such as an electrical configuration of the PC 10 and the robot 12 and a memory map of the PC 10 is omitted.

  For example, referring to FIG. 32 (A), as in FIG. 16 (A), even if a user with low occurrence frequency of “gaze” concentrates on the conversation, the recognition sample is placed on the passive side. Therefore, it is misrecognized as a passive state. Therefore, when the user's recognition sample is generalized based on the mathematical formula shown in Equation 4, the generalized recognition sample is arranged on the right side (active side) of the boundary line, so that the user is recognized as being in an active state. Become so.

  In this way, since it is only necessary to adjust the weight of the recognition sample, the concentration state of a specific user can be easily and correctly recognized without changing the recognition reference.

  Hereinafter, a flowchart of the present invention executed by the PC 10 of the second embodiment will be described. However, since the flowcharts shown in FIGS. 20 to 27, 29, and 31 are the same as those in the first embodiment, detailed description thereof is omitted. 33 shows the processing by the learning program 318 of the second embodiment, and the flowchart of FIG. 34 shows the processing by the centralized recognition program 322 of the second embodiment.

  FIG. 33 is a flowchart showing the processing of the learning program 318 of the second embodiment. The processor 26 acquires general learning data in step S301, and constructs a general SVM in step S303. Then, when the process of constructing the general SVM ends, the processor 26 ends the learning process. In other words, the learning process of the second embodiment is a process that only constructs a general SVM.

  FIG. 34 is a flowchart showing the processing of the centralized recognition program 322 of the second embodiment. In step S <b> 311, the processor 26 calculates the action frequency of the composite action from the recognition sample acquired by the central processing. Subsequently, in step S313, the processor 26 creates a recognition sample with the weight adjusted. That is, the processor 26 adjusts the weight of the recognition sample based on the behavior frequency calculated in step S311 and the mathematical formula shown in Equation 4. The recognition sample whose weight has been adjusted is referred to as a “generalized recognition sample”. Subsequently, in step S315, the processor 26 recognizes the user's concentration state based on the recognition sample whose weight is adjusted. That is, the processor 26 recognizes the user's concentration state by recognizing the generalized recognition sample by the general SVM constructed by the learning process. Note that the processor 26 that executes the process of step S315 functions as a creation unit, and the processor 26 that executes the process of step S317 functions as a recognition unit.

  Subsequently, in step S317, the processor 26 determines whether or not the recognition result is in an active state. That is, the processor 26 determines whether the recognition result by the general SVM is in the active state. If “YES” in the step S317, that is, if the recognition result is the active state, the processor 26 sets the active state in a step S319. That is, in step S319, the concentration flag 354 is set to ON. On the other hand, if “NO” in the step S317, that is, if the recognition result is the passive state, the processor 26 sets the passive state in a step S321. That is, the concentration flag 354 is set off.

  In step S323, the processor 26 labels the recognition sample with the recognition result and adds it to the individual learning sample, and ends the intensive recognition process. When the processor 26 ends the intensive recognition process, the processor 26 returns to the recognition process that is a higher-level routine.

  Here, since the labeled personal learning sample is added to the personal learning data 350 in step S232, in the second embodiment, the general learning sample is recalculated using the labeled personal learning sample. Can do. That is, since the number of general learning samples learned by the general SVM can be increased, the recognition accuracy can be improved.

  According to these embodiments, the PC 10 included in the listening dialogue sustaining system 100 uses the image taken by the abdomen camera 14 and the monitor camera 22 of the robot 12 and the voice collected by the microphone 20 to obtain user action data. To get. Further, in the first embodiment, the personalized SVM in which the position of the boundary line (hyperplane) is adjusted based on the personal learning sample sampled from the action data of the specific user and the general learning sample for constructing the SVM. Is built. Further, in the second embodiment, the recognition sample is generalized based on the individual learning sample sampled from the action data of a specific user and the general learning sample for constructing the general SVM.

  And in each Example, a personal learning sample is sampled from the action of a specific user, and a specific user's state is recognized using the personal learning sample and the existing general learning sample. Therefore, the PC 10 can easily and correctly recognize the concentration state of a specific user.

  Further, in these embodiments, it is possible to obtain complex actions necessary for learning and recognition while reducing the types of element actions to be acquired. Therefore, it is possible to reduce the load on the PC 10 that records the user's elementary actions. Furthermore, the recognition accuracy can be improved by recognizing the user's concentration state using the presence or absence of the other party's utterance.

  As shown in FIG. 1, the listening dialogue sustaining system 100 is not necessarily premised on remote dialogue, and can also be used when estimating the state of a person in the same space (room). Referring to FIG. 35, when two users are in the same room, PC 10 is directly connected to server 24 without network 200. Referring to FIG. 36, when two people sit in the same room facing each other, the monitor 16 displays nothing because each other can directly confirm the other's face. In another embodiment, the concentration state of the user A and the user B may be recognized by one PC 10 by the PC 10 fulfilling the function of the server 24. That is, the action data of each user is stored in one PC 10. However, the robot 12, the speaker 18, the microphone 20, and the monitor camera 22 are each installed near each user.

  In another embodiment, not only the concentration state but also the utterance state and the object of interest may be recognized by the SVM.

  In this embodiment, the SVM is used as a method for recognizing the concentration state. However, in other embodiments, other recognition methods such as a nearest neighbor method may be used.

  Further, the composite action may include a control result of the robot 12, a gesture of the user's hand, and the like. When the control result of the robot 12 is used, the operation history data of the robot 12 is referred to. The user's hand gesture is recognized using a process for detecting the user's hand.

  In addition to the abdominal camera 14 and the monitor camera 22, a third camera that captures the face of the user at a position other than the monitor 16 and the robot 12 may be installed.

  Further, the state of the user may be recognized by the server 24 instead of the PC 10. In this case, the image captured by the abdominal camera 14 and the monitor camera 22 and the sound collected by the microphone 20 are directly transmitted to the server 24.

  Further, the user's image and sound may be transmitted / received to / from the PC 10 via the telephone network or the like without using the PC 10 and the network 200. Further, the PC 10, the monitor 16, the speaker 18, the microphone 20, and the monitor camera 22 may be incorporated in the same casing. Further, in this case, a videophone call is started with the robot 12 and the videophone connected.

  The data of various programs stored in the memory 30 may be stored in the HDD of the data distribution server and distributed to the PC 10 via communication. Furthermore, the storage medium may be sold or distributed in a state where the data of these programs is stored in a storage medium such as an optical disk.

  The specific numerical values such as the window width time, the fixed time, and the predetermined time threshold listed in this specification are merely examples, and can be appropriately changed according to the needs of the product specifications and the like. .

10a, 10b ... PC
12a, 12b ... Robot 14a, 14b ... Abdominal camera 16a, 16b ... Monitor 20a, 20b ... Microphone 22a, 22b ... Monitor camera 24 ... Server 26 ... Processor 30 ... Memory 36 ... Communication LAN board 38 ... Wireless communication device 100 ... Listen dialog Sustainable system 200 ... network

Claims (11)

In a state recognition apparatus comprising: an acquisition unit that acquires a user's behavior; and a recognition unit that has a recognition criterion and recognizes a user state from the user's behavior acquired by the acquisition unit.
A state recognition apparatus, further comprising: a storage unit that stores a plurality of learning samples; and an adjustment unit that adjusts the recognition reference based on the plurality of learning samples. The recognition criterion is a boundary in SVM;
The adjusting means includes weight adjusting means for adjusting the weights of the plurality of learning samples,
The state recognition apparatus according to claim 1, wherein the position of the boundary changes when weights of the plurality of learning samples are adjusted by the weight adjustment unit. A provisional recognition means for temporarily recognizing a user state after the weight is adjusted by the weight adjustment means; and a recording means for recording the degree of recognition by the provisional recognition means.
The weight adjustment means repeats the adjustment of the weights of the plurality of learning samples until the difference between the previous recognition degree recorded by the recording means and the current recognition degree is a predetermined value or less. The state recognition apparatus according to claim 2. A state recognition device for recognizing a user state of a user,
Storage means for storing a plurality of learning samples;
Acquisition means for acquiring the user's behavior;
Based on the plurality of learning samples, from the user's action acquired by the acquisition unit, a creation unit that creates a recognition sample in which a weight is adjusted, and a recognition sample that has a recognition reference and is created by the creation unit A state recognition device comprising recognition means for recognizing the user state of the user. Further comprising an elemental action determination means for determining the presence or absence of a plurality of elemental actions of the user,
The state recognition apparatus according to claim 1, wherein the user's action includes a composite action that combines the presence or absence of the plurality of element actions.   The state recognition apparatus according to claim 5, wherein the plurality of elemental actions include the user's utterance, the user's gaze, the user's forward leaning posture, and the user's whisper. A listening dialogue sustaining system comprising the state recognition device according to claim 6,
It further comprises a partner utterance judging means for judging the presence or absence of the utterance of the conversation partner,
The listening dialogue sustaining system, wherein the plurality of elemental actions in one speaker further include the speech of the other conversation partner. A processor of a state recognition device comprising storage means for storing a plurality of learning samples;
Acquisition means for acquiring user behavior,
A state recognition program having a recognition standard and functioning as a recognition unit for recognizing a user state from a user action acquired by the acquisition unit, and an adjustment unit for adjusting the recognition standard based on the plurality of learning samples . A processor of a state recognition device comprising storage means for storing a plurality of learning samples;
Acquisition means for acquiring user behavior,
Based on the plurality of learning samples, a creation unit that creates a recognition sample in which a weight is adjusted from the user's action obtained by the obtaining unit, and a recognition criterion, and the weight created by the creation unit is A state recognition program that functions as a recognition unit that recognizes the user state of the user from the adjusted recognition sample. An acquisition unit for acquiring user behavior, a recognition unit having a recognition standard, a recognition unit for recognizing a user state from a user behavior acquired by the acquisition unit, and a storage unit for storing a plurality of learning samples In the state recognition method,
Acquiring the user's behavior by the acquisition means;
Adjusting the recognition standard based on the plurality of learning samples, and recognizing a user state from a user action acquired by the acquisition unit by the recognition unit in which the recognition standard is adjusted, State recognition method. A state recognition method for a state recognition device, comprising storage means for storing a plurality of learning samples,
Get user behavior,
Based on the plurality of learning samples, a recognition sample with a weight adjusted from the user's behavior acquired by the acquisition unit is created, and the recognition sample having a recognition criterion and having the weight adjusted is used to determine the user's behavior. A state recognition method for recognizing a user state.

Images