JP2021135363A - Control system, control device, control method, and computer program - Google Patents

Abstract

To provide a control system capable of further clearly indicating a person whom an interaction device is speaking to.SOLUTION: A control system 1 includes an acquisition unit that acquires, for each person, information about people who face a prescribed interaction device for speaking to a person, an operation deciding unit 166 that decides, among the people, a person toward whom the interaction device is to be directed, on the basis of the acquired information and information about the history of interactions between the persons and the interaction device stored in a prescribed storage device, a speech deciding unit 165 that decides a speech of the interaction device on the basis of a voice given by the person, and a device control unit 167 that controls operation of the interaction device such that the interaction device is directed to the direction, and controls a loudspeaker 40 disposed near the interaction device so as to output the speech.SELECTED DRAWING: Figure 1

Description

The present invention relates to control systems, control devices, control methods and computer programs.

In recent years, research has been conducted on technologies for dialogue devices such as robots and dialogue agents to interact with humans. Such technology is utilized in fields where dialogue with people is required, such as customer service. The dialogue device may have a dialogue with a plurality of people at the same time.

Japanese Unexamined Patent Publication No. 2005-246502

However, when the dialogue device interacts with a plurality of people at the same time, there is a problem that it becomes difficult to understand which person is talking to. In view of the above circumstances, it is an object of the present invention to provide a technique capable of showing which person the dialogue device is talking to in an easy-to-understand manner.

One aspect of the present invention is an acquisition unit that acquires information about a plurality of people facing a predetermined dialogue device that speaks to a person for each person, the acquired information, and the person stored in the predetermined storage device. Based on the history information of the dialogue between the user and the dialogue device, an action determining unit that determines which of the plurality of people the dialogue device should be directed to, and a voice spoken by the person. Controls the speech determination unit that determines the speech content of the dialogue device and the speaker provided in the vicinity of the dialogue device so as to control the operation of the dialogue device so as to face the direction and output the speech content. It is a control system including a device control unit.

One aspect of the present invention is the control system, and whether or not the operation determining unit directs the dialogue device toward the person, which is calculated based on the acquired information and the history information. It is determined that the dialogue device is directed to the person with the highest priority, and the operation content is determined.

One aspect of the present invention is the control system, wherein the utterance content score obtained by quantifying the utterance content based on the history information and the utterance content and the history information indicate the above-mentioned dialogue device. One or more of the utterance history score obtained by quantifying the utterance history of a person and the person state score indicated by the history information and quantifying the state of the person is calculated, and the calculated score is obtained. A priority calculation unit for calculating the priority is further provided, and the operation determination unit determines the direction in which the dialogue device is directed and the operation content based on the priority calculated by the priority calculation unit. decide.

One aspect of the present invention is the control system, wherein the operation determination unit is based on an estimator generated by machine learning teacher data in which the history information and the priority are associated with each other. It is determined which of the plurality of people the dialogue device is directed toward.

One aspect of the present invention is an acquisition unit that acquires information about a plurality of people facing a predetermined dialogue device that speaks to a person for each person, the acquired information, and the person stored in the predetermined storage device. Based on the history information of the dialogue between the user and the dialogue device, an action determining unit that determines which of the plurality of people the dialogue device should be directed to, and a voice spoken by the person. Controls the speech determination unit that determines the speech content of the dialogue device and the speaker provided in the vicinity of the dialogue device so as to control the operation of the dialogue device so as to face the direction and output the speech content. It is a control device including a device control unit for the operation.

One aspect of the present invention is an acquisition step in which a control device acquires information about a plurality of people facing a predetermined dialogue device that speaks to a person for each person, and a control device obtains the acquired information and a predetermined information. Based on the history information of the dialogue between the person and the dialogue device stored in the storage device, an operation determination step for determining which of the plurality of people the dialogue device is directed to, and a control device. However, the utterance determination step of determining the utterance content of the dialogue device based on the voice uttered by the person, and the control device controls the operation of the dialogue device so as to face the direction, and the utterance content is determined. It is a control method having a device control step for controlling a speaker provided in the vicinity of the dialogue device so as to output.

One aspect of the present invention is a computer program for operating a computer as the control system described above.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to show which person the dialogue device is talking to in a more understandable manner.

It is a system configuration diagram which shows the system configuration of the control system 1. It is a figure which shows one specific example of a dialogue device 50. It is a figure which shows a specific example of a person information table. It is a figure which shows a specific example of the utterance content score table. It is a figure which shows a specific example of the utterance history score table. It is a figure which shows a specific example of a human condition score table. It is a flowchart which shows a specific example of the process concerning the generation of person information. It is a flowchart which shows a specific example of the process which determines the operation of a dialogue device 50. It is a flowchart which shows a specific example of the flow of the utterance history score calculation process. It is a flowchart which shows a specific example of the flow of the human condition score calculation process. It is a flowchart which shows a specific example of the flow of the utterance content score calculation process. It is a figure which shows a specific example of the operation of the dialogue device 50. It is a figure which shows a specific example of the case where the control system 1 is provided with a predetermined device such as a display device or an actuator instead of a robot.

FIG. 1 is a system configuration diagram showing a system configuration of the control system 1. The control system 1 includes a sensor 10, a camera 20, a microphone 30, a speaker 40, a dialogue device 50, and a control device 100. In the control system 1, the control device 100 controls the operation of the dialogue device 50 based on the information input to the control device 100. In the control system 1, the control device 100 outputs sound from the speaker 40 based on the information input to the control device 100.

The sensor 10 detects a person who has visited in the vicinity of the dialogue device 50. When the sensor 10 detects a person, the sensor 10 outputs information about the detected person to the control device 100. The detected information may differ depending on the type of the sensor 10. For example, when the sensor 10 detects a person, the sensor 10 may output a signal indicating that the person has visited to the control device 100. For example, as the sensor 10, a sensor that detects the position of a person, such as a depth sensor, may be used. In this case, the sensor 10 may output coordinate information indicating the position of the detected person to the control device 100. The sensor 10 is installed in the vicinity of the dialogue device 50. A plurality of sensors 10 may be installed. In this case, the sensor 10 may be installed in a different direction so that it can detect people in various directions. The sensor 10 is a specific example of the acquisition unit.

The camera 20 captures a moving image in the vicinity of the dialogue device 50. The camera 20 captures, for example, a person facing the dialogue device 50. The camera 20 may be an interface for connecting an imaging device such as a camera to the control device 100. In this case, the camera 20 generates a video signal from the moving image captured by the imaging device and inputs it to the control device 100. The camera 20 may be provided at any position as long as it can image a person facing the dialogue device 50. The camera 20 is a specific example of the acquisition unit.

The microphone 30 collects sound in the vicinity of the dialogue device 50. The microphone 30 is installed in the vicinity of the dialogue device 50. The microphone 30 picks up the voice spoken by, for example, a person facing the dialogue device 50. The microphone 30 collects, for example, the voice spoken to the dialogue device 50. The microphone 30 generates an audio signal based on the picked-up sound. The microphone 30 outputs the generated audio signal to the control device 100. A plurality of microphones 30 may be installed. In this case, the microphone 30 may be installed in a different direction so that sound in various directions can be picked up. The microphone 30 may be an interface for connecting a sound collecting device such as an external microphone to the control device 100. In this case, the microphone 30 generates an audio signal from the audio input by the sound collecting device and outputs the audio signal to the control device 100. The microphone 30 is a specific example of the acquisition unit.

The speaker 40 outputs the audio signal output from the control device 100 as audio. The speaker 40 is installed in the vicinity of the dialogue device 50. With this configuration, the dialogue device 50 can appear to be talking to a person facing the dialogue device 50. A plurality of speakers 40 may be provided. By providing a plurality of speakers 40, it is possible to have a dialogue with a plurality of people at the same time. The speaker 40 may be, for example, a directional speaker.

The dialogue device 50 is a device that talks to a person facing the own device and has a dialogue. The dialogue device 50 performs a predetermined operation based on the control of the control device 100. The dialogue device 50 may be a device such as a robot. Hereinafter, in the present embodiment, the dialogue device 50 will be described as a robot. The robot may operate by operating a drive mechanism provided at each joint of the neck, shoulder or arm, for example. The robot may be, for example, an animal type that walks by operating a drive mechanism provided at each joint such as a shoulder or a leg. The robot may be a robot such as a bipedal walking robot that walks independently by operating a drive mechanism provided at each joint such as a shoulder or a leg. The robot may be a mobile robot capable of moving on wheels or tracks. The robot is installed on a plate-shaped table such as a table or a reception table.

FIG. 2 is a diagram showing a specific example of the dialogue device 50. The dialogue device 50 includes a head 510 and a body portion. The body portion is composed of the body portion of the dialogue device 50, and the left arm portion 520, the right arm portion 530, and the leg portion 540 connected to the body portion. The head 510 includes a microphone 511, a camera 512, a first light emitting unit 513, and a second light emitting unit 514. The head 510 is driven in three axial directions, up, down, left, right, front and back, based on the control of the device control unit 167 of the control device 100. The dialogue device 50 controls the direction of the head 510 in a predetermined direction, for example, based on the control of the device control unit 167. The front portion of the head 510 will be described as the face of the dialogue device 50. The front surface of the dialogue device 50 is a surface of the head 510 provided with a microphone 511, a camera 512, a first light emitting unit 513, and a second light emitting unit 514.

The microphone 511 picks up the sound in the vicinity of the dialogue device 50. The microphone 511, for example, picks up the voice spoken by the person facing the dialogue device 50. The microphone 511 generates an audio signal based on the picked-up audio. The microphone 511 may record the generated audio signal in the control device 100, or may record it in an external device via a network.

The camera 512 captures a moving image or a still image in the direction in which the front portion of the dialogue device 50 provided with the camera 512 is facing. For example, the camera 512 captures a moving image or a still image in the vicinity of a person facing the dialogue device 50 and a person facing the dialogue device 50. The camera 512 may record the captured moving image or still image in the control device 100, or may record it in an external device via a network. The camera 512 may capture a moving image or a still image under the control of the device control unit 167.

The first light emitting unit 513 is a light emitting member such as a full-color LED (Light Emitting Diode). The first light emitting unit 513 may emit light in a predetermined color under the control of the device control unit 167. For example, a case where the device control unit 167 controls the first light emitting unit 513 to emit green light will be described. In this case, the first light emitting unit 513 emits green light. The first light emitting unit 513 corresponds to the right eye and the left eye of the dialogue device 50. The first light emitting unit 513 may simultaneously emit light of the same color to the left and right eyes, may emit light of different colors at the same time for the left and right eyes, may emit light of different colors alternately, or may alternately emit light of different colors. May emit light in the same color.

The second light emitting unit 514 is a light emitting member such as a full-color LED. The second light emitting unit 514 may emit light in a predetermined color under the control of the device control unit 167. For example, a case where the device control unit 167 controls the second light emitting unit 514 to blink red during the audio output of the speaker 40 will be described. In this case, the second light emitting unit 514 blinks red during the audio output of the speaker 40. The second light emitting unit 514 corresponds to the mouth of the dialogue device 50. With this configuration, the second light emitting unit 514 can be made to blink when the speaker 40 is outputting sound. By emitting light from the second light emitting unit 514 in this way, the dialogue device 50 can appear to be interacting with a person facing the dialogue device 50.

The left arm portion 520 includes a left shoulder joint portion 521 and a left arm joint portion 522. The left shoulder joint portion 521 is a drive mechanism that drives the left arm portion 520 up, down, left and right. The left shoulder joint portion 521 is rotationally driven under the control of the control device 100. For example, a case where the device control unit 167 controls to move the left arm portion 520 upward will be described. In this case, the left shoulder joint portion 521 is rotationally driven so as to move the left arm portion 520 upward. The right arm portion 530 includes a right shoulder joint portion 531 and a right arm joint portion 532. The right arm portion 530 drives the right shoulder joint portion 531 and the right arm joint portion 532 under the control of the device control unit 167, similarly to the left arm portion 520. The leg portion 540 changes the direction of the dialogue device 50 or moves the dialogue device 50 back and forth and left and right based on the control of the device control unit 167.

Returning to FIG. 1, the description of the control system 1 will be continued. The control device 100 is configured by using an information processing device such as a personal computer, a tablet computer, or a server. The control device 100 is equipped with a control function for operating the dialogue device 50. The control function may be implemented in the control device 100 by hardware, or may be implemented by installing software. The control device 100 includes a communication unit 101, a human information storage unit 102, a score map storage unit 103, an utterance content storage unit 104, an operation content storage unit 105, and a control unit 106.

The communication unit 101 is a communication device such as a network interface. The communication unit 101 is communicably connected to the network by a predetermined protocol. The communication unit 101 communicates data with other devices via the network under the control of the control unit 106.

The human information storage unit 102 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The human information storage unit 102 stores the human information table. The person information table stores the history of dialogue between the person and the dialogue device 50 for each person. The person indicates a person detected by the sensor 10. The person information table will be described later. The human information storage unit 102 is a specific example of a storage device. The person information table is a specific example of history information. The history information indicates the history of the dialogue between the person and the dialogue device 50 stored in the predetermined storage device.

FIG. 3 is a diagram showing a specific example of the human information table. The person information table has a person information record. The person information record has each value of a person identifier, a facial expression, the number of utterances of a person, the line of sight, the staying time, the number of times the dialogue device utters, the number of times the face is turned, the utterance time, the language, and the priority. A person information record is generated for each person. The person identifier is information that can uniquely identify the person information record. The person identifier is associated with a person who visits the vicinity of the dialogue device 50. The person identifier may be, for example, information that is a combination of numbers and characters. The person identifier may be any information as long as it does not overlap with other person identifiers. The person identifier is determined by the person identification unit 161 when the person information record is generated. Facial expressions indicate information about a person's facial expressions. The facial expression is estimated based on a moving image of a person captured by the camera 20. The facial expression is estimated by the feature estimation unit 162. The facial expression may be any information as long as it is information on a person's facial expression such as a smiling face, a sad face, an angry face, or an expressionless face. The number of human utterances indicates the number of utterances from a person to the dialogue device 50. The number of human utterances is counted by 1 each time the voice of the utterance content spoken by a person is picked up by the microphone 30. The number of human utterances is counted, for example, by the voice processing unit 164. The line of sight indicates information about a person's line of sight. The line of sight is estimated based on a moving image of a person captured by the camera 20. The line of sight is estimated by the feature estimation unit 162. The line of sight may be indicated by, for example, any of the three "dialogue device", "non-dialogue device", or "closed eyes". The “dialogue device” indicates that the line of sight of a person is directed toward the dialogue device 50. “Other than the dialogue device” indicates that the line of sight of the person is directed to other than the dialogue device. "Closed eyes" indicates that the human eye is closed. The staying time indicates the time during which a person stays in the vicinity of the dialogue device 50. The staying time is measured after being detected by the sensor 10. The number of utterances of the dialogue device indicates the number of utterances from the dialogue device 50 to a person. The number of times the dialogue device utters is counted by 1 each time the speaker 40 speaks to a person. The number of times the face is turned indicates the number of times the dialogue device 50 turns the face to a person. The number of times the face is turned is counted by 1 each time the dialogue device 50 is determined to turn the face to a person. The utterance time indicates the cumulative time during which the dialogue device 50 speaks to a person. The utterance time is measured each time the dialogue device 50 speaks to a person. The language indicates the language of the utterance output to the person. The language may be specified by a person. The priority is an index that determines whether or not to point the dialogue device 50 at a person. The priority is calculated for each person at a predetermined timing. The higher the priority, the easier it is for the dialogue device 50 to turn its face or body toward a person.

In the example shown in FIG. 3, the person information record at the top of the person information table has a person identifier value of "User001", a facial expression value of "smile", a person utterance count value of "2", and a line-of-sight value. Is "dialogue device", the value of staying time is "18 seconds", the value of the number of times the dialogue device is spoken is "3", the value of the number of times the face is turned is "2", the value of the speech time is "25 seconds", and the language The value of is "Japanese" and the priority value is "15". Therefore, the record at the top of the person information table is a record in which information about the person identified by the person identifier "User001" is registered. According to this record, the person is "smiling" and speaks twice to the dialogue device 50. Also, people are looking at the dialogue device. The person is in the vicinity of the dialogue device 50 for 18 seconds. Further, the dialogue device 50 speaks to a person three times, and the utterance time is 25 seconds. In addition, the dialogue device 50 turns the face toward the person twice. The dialogue device 50 speaks in Japanese when speaking to a person. The priority of a person is 15. The person information table shown in FIG. 3 is only a specific example. Therefore, the person information table may be configured in a manner different from that shown in FIG. For example, the person information table may have a column of time that was faced by the dialogue device 50. Moreover, the person information record may be generated for each group instead of an individual. A group is a group of people composed of a plurality of people such as family members and friends.

The score map storage unit 103 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The score map storage unit 103 stores various types of information used for calculating the priority of a person. For example, the score map storage unit 103 stores the utterance content score table, the utterance history score table, and the human state score table. Each score table will be described later.

The utterance content storage unit 104 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The utterance content storage unit 104 stores the utterance content uttered by the dialogue device 50. The utterance content storage unit 104 stores a predetermined character string in association with the utterance content. The predetermined character string is, for example, a character string indicating the content uttered by a person. Further, the utterance content storage unit 104 may store the utterance content in association with the fact that the sensor 10 has detected that a new person has visited instead of the predetermined character string. The utterance content is recorded in advance in the utterance content storage unit 104. The utterance content storage unit 104 may store the utterance content expressed in a language other than Japanese.

The operation content storage unit 105 is configured by using a storage device such as a magnetic hard disk device or a semiconductor storage device. The operation content storage unit 105 stores the content of the operation (hereinafter, referred to as “operation content”) that the dialogue device 50 behaves with respect to a person. The operation content may be, for example, an operation of turning a face toward a predetermined person. The operation content may be, for example, an operation of turning the body toward a predetermined person. The operation content may be, for example, an operation of raising the left arm with respect to a predetermined person. The operation content may be any nonverbal operation possible by the dialogue device 50. The predetermined person is, for example, a person determined as an object of action. The operation content storage unit 105 stores the operation content and the utterance content in association with each other. Further, the operation content storage unit 105 may store the possible values of each column of the human information record and the operation content in association with each other. With this configuration, the control device 100 can operate the dialogue device 50 based on the state of the person and the utterance content of the dialogue device 50. The operation content is recorded in advance in the operation content storage unit 105.

Returning to FIG. 1, the description of the control system 1 will be continued. The control unit 106 controls the operation of each unit of the control device 100. The control unit 106 is configured by using a processor such as a CPU (Central Processing Unit) and a RAM (Random Access Memory). When the processor executes a specific program, the control unit 106 includes a person identification unit 161, a feature estimation unit 162, a priority calculation unit 163, a voice processing unit 164, an utterance determination unit 165, an operation determination unit 166, and a device control unit. Functions as 167.

The person identification unit 161 identifies a person facing the dialogue device 50. Specifically, the person identification unit 161 acquires information about a person detected by the sensor 10 from the sensor 10. The person identification unit 161 identifies whether or not the person is a new visitor in the vicinity of the dialogue device 50 based on the acquired information of the person. When the person detected by the sensor 10 is a new visitor to the dialogue device 50, the person identification unit 161 generates human information about the detected person. The person identification unit 161 creates a person information record including the generated person information. The person identification unit 161 registers the newly detected person information by newly adding the generated person information record to the person information table. The person identification unit 161 may use a known means such as face recognition to identify whether or not the person is a new visitor to the dialogue device 50. When the person identification unit 161 cannot acquire the information about the person indicated by the person information record from the sensor 10, the person identification unit 161 may delete the person information record from the person information table. With such a configuration, the control device 100 can delete the person information record about the person who has left from the vicinity of the dialogue device 50.

The feature estimation unit 162 estimates the characteristics of the person detected by the sensor 10. The feature estimation unit 162 records the estimated feature in the human information record. Specifically, the feature estimation unit 162 estimates the characteristics of the captured person based on the moving image of the person captured by the camera 20. For example, the feature estimation unit 162 estimates the facial expression of an imaged person. The feature estimation unit 162 estimates whether a person has a smiling face, a sad face, an angry face, or an expressionless face by analyzing the captured moving image of the person. The feature estimation unit 162 records the estimated facial expression in the facial expression column of the human information record. In addition, the feature estimation unit 162 estimates the line of sight of the photographed person. The feature estimation unit 162 estimates whether the line of sight of the person is "dialogue device 50", "other than the dialogue device 50", or "closed eyes" by analyzing the captured moving image of the person. .. The feature estimation unit 162 records the estimated line of sight in the line of sight column of the human information record. The feature estimation unit 162 may estimate the facial expression or the column by using known means.

The priority calculation unit 163 calculates the priority for each person based on the information acquired by the sensor 10, the camera 20, and the microphone 30 and the score map storage unit 103. The specific method for calculating the priority will be described later. The priority calculation unit 163 records the calculated priority in the person information table.

The voice processing unit 164 executes the voice recognition process. The voice recognition process is a process of generating a character string based on a voice signal. The voice processing unit 164 generates a character string based on the voice signal output by the microphone 30 by executing the voice recognition process. The voice processing unit 164 outputs the generated character string to the utterance determination unit 165. The voice processing unit 164 may generate a character string by using a known method. A case where there are a plurality of people detected by the sensor 10 will be described. In this case, the voice processing unit 164 associates a person with a voice signal related to a voice uttered by the person based on a known sound source separation process.

The utterance determination unit 165 determines the utterance content of the dialogue device 50. Specifically, the utterance determination unit 165 determines the utterance content based on the character string generated by the voice processing unit 164. The utterance determination unit 165 acquires the utterance content associated with the generated character string from the utterance content storage unit 104. The utterance determination unit 165 determines the acquired utterance content as the utterance content of the dialogue device 50.

The operation determination unit 166 determines the operation content of the dialogue device 50. Specifically, the motion determination unit 166 determines the direction in which the dialogue device 50 faces. The direction in which the dialogue device 50 turns its face is the direction in which any one of the detected persons is present. For example, the motion determination unit 166 determines as the direction in which the face of the dialogue device 50 is directed, the direction of the person having the highest priority among the priorities calculated for each person as the direction in which the face of the dialogue device 50 is directed. In this case, the motion determination unit 166 estimates the coordinates indicating the position of the person based on the information acquired by the sensor 10 and the camera 20. The motion determining unit 166 determines the angles of the head 510 and the legs 540 so that the face of the dialogue device 50 faces in the direction of the estimated coordinates.

In addition, the operation determination unit 166 determines the operation content other than the direction in which the dialogue device 50 faces the face. For example, the operation determination unit 166 may determine the operation content associated with the utterance content of the dialogue device 50. The operation determination unit 166 may determine the operation content based on the information recorded in the human information record. The operation determination unit 166 acquires the determined operation content from the operation content storage unit 105.

The device control unit 167 controls the speaker 40 and the dialogue device 50 based on the determined operation content and utterance content. For example, the device control unit 167 generates an audio signal based on the determined utterance content. The device control unit 167 outputs an audio signal to the speaker 40 to output audio from the speaker 40. When a plurality of speakers are installed, the device control unit 167 may output audio signals to a plurality of people at the same time. At this time, the device control unit 167 may output a different audio signal for each speaker 40 depending on the position where a person is present. Further, the device control unit 167 controls the dialogue device 50 based on the determined operation content.

FIG. 4 is a diagram showing a specific example of the utterance content score table. The utterance content score table is a table in which the utterance content of the dialogue device 50 and the score are associated with each other. A case where the dialogue device 50 makes a predetermined utterance to a person will be described. The priority calculation unit 163 uses the score associated with the utterance content made by the dialogue device 50 to calculate the priority. The utterance content score table has an utterance content score record. The utterance content score record has each value of the utterance content and the score. The utterance content table has an utterance content record for each utterance content. The utterance content indicates the content uttered from the dialogue device 50 to a person. The utterance content indicates the same content as the utterance content stored in the utterance content storage unit 104. The score is information that quantifies which of a plurality of people should be turned to. The score is used to calculate the priority.

In the example shown in FIG. 4, the utterance content score record at the top of the utterance content score table has a utterance content value of "greeting (good morning, hello)" and a score value of "5". Therefore, according to the record at the top of the utterance content score table, when the dialogue device 50 gives a greeting such as good morning or hello to a person, the priority is calculated using a score of 5 for the person. The utterance content score table shown in FIG. 4 is only a specific example. Therefore, the utterance content score table may be configured in a mode different from that shown in FIG. For example, the utterance content score table may have a column of utterance content expressed in another language.

FIG. 5 is a diagram showing a specific example of the utterance history score table. The utterance history score table is a table in which information on utterances and actions from the dialogue device 50 to a person is associated with a score. The utterance history score table has a utterance history score record. The utterance history score record has each value of history content, number of times / time, and score. The history content indicates the content related to utterances and actions from the dialogue device 50 to a person. The history content includes, for example, the number of times the face is turned, the number of times the dialogue device utters, and the utterance time. The history content indicates the content related to the utterance and operation of the dialogue device 50 in the column of the human status record. The number of times / time indicates the number of times the utterance or action shown in the history content has been performed. The number of times / time indicates the time during which the utterance or action shown in the history content was performed. The history content and the number of times / time are associated with the score.

In the example shown in FIG. 5, in the utterance content score record at the top of the utterance history score table, the value of the history content is "the number of times the face is turned", the value of the number of times / time is "5 times or more", and the score value. Is "1". Therefore, according to the record at the top of the utterance history score table, when the dialogue device 50 faces a person five times or more, the priority is calculated using a score of 1 for the person. The utterance history score table shown in FIG. 5 is only a specific example. Therefore, the utterance history score table may be configured in a mode different from that shown in FIG.

FIG. 6 is a diagram showing a specific example of the human condition score table. The human condition score table is a table in which information about a person's condition and a score are associated with each other. The human status score table has a human status score record. The human condition score record has each value of the person's condition, details and score. The state of a person indicates the content related to the state of the person. A person's condition has, for example, a facial expression, the number of utterances, a line of sight, and a staying time. The human state indicates the contents related to the human state in the column of the human state record. The details represent the specific content of a person's condition. For example, in FIG. 6, a “smile” and a “sad face” are shown as specific contents of the human condition “facial expression”. The score indicates the score given to the person's condition. The score is set for each specific content of a person's facial expression, the number of utterances, a person's line of sight, and a person's staying time.

In the example shown in FIG. 6, in the human condition score record at the top of the human condition score table, the human condition value is "facial expression", the detailed value is "smile", and the score value is "3". Therefore, according to the record at the top of the human state score table, when the person smiles at the dialogue device 50, the priority is calculated using the score of 3 for the person. The human condition score table shown in FIG. 6 is only a specific example. Therefore, the human condition score table may be configured in a manner different from that shown in FIG.

FIG. 7 is a flowchart showing a specific example of the process related to the generation of human information. The process related to the generation of human information is executed each time a person is detected by, for example, the sensor 10. The person identification unit 161 of the control device 100 identifies the person detected by the sensor 10 (step S101). Specifically, the person identification unit 161 acquires information about a person detected by the sensor 10 from the sensor 10. The person identification unit 161 identifies whether or not the person is a new visitor in the vicinity of the dialogue device 50 based on the acquired information about the person.

When the detected person is a new visitor to the dialogue device 50 (step S102: YES), the person identification unit 161 generates human information about the detected person (step S103). The person identification unit 161 records the generated person information as a person information record in the person information table. If the detected person is not a new visitor to the dialogue device 50 (step S102: NO), the process transitions to step S104.

The feature estimation unit 162 of the control device 100 estimates the features of the detected person (step S104). Specifically, the feature estimation unit 162 estimates the facial expression of the captured person based on the moving image of the person captured by the camera 20. The feature estimation unit 162 estimates the line of sight of the captured person based on the moving image of the person captured by the camera 20. The feature estimation unit 162 records the estimated facial expression and line of sight in the human information record (step S105).

FIG. 8 is a flowchart showing a specific example of the process of determining the operation of the dialogue device 50. The process of determining the operation of the dialogue device 50 (hereinafter, referred to as “operation determination process”) is executed at the timing of calculating the priority, for example. The timing for calculating the priority is, for example, the timing when a new person is detected, the timing when the utterance by the dialogue device 50 ends, a predetermined interval (for example, once every 5 seconds), and the like. In this case, when it is time to calculate the priority, the control device 100 executes the operation determination process. First, the priority calculation unit 163 of the control device 100 acquires the person information table from the person information storage unit 102 (step S201). Next, the priority calculation unit 163 determines a person (hereinafter, referred to as “target person”) to be the target of priority calculation based on the person information table (step S202). Specifically, the priority calculation unit 163 determines any one of the person information records held in the person information table. The priority calculation unit 163 determines the person indicated by the determined person information record as the target person. The priority calculation unit 163 may determine the target person in order from the top of the person information table, for example. The priority calculation unit 163 may determine the target person by any means as long as it determines the person whose priority has not been calculated in the current operation determination process. The priority calculation unit 163 acquires the determined person information record of the target person (step S203).

Next, the priority calculation unit 163 calculates the utterance history score (step S204). The priority calculation unit 163 calculates the utterance history score of the target person by executing the utterance history score calculation process. The utterance history score calculation process will be described later. Next, the priority calculation unit 163 calculates the human condition score (step S205). The priority calculation unit 163 calculates the human condition score of the target person by executing the human condition score calculation process. The human condition score calculation process will be described later. Next, the priority calculation unit 163 calculates the utterance content score (step S206). The priority calculation unit 163 calculates the utterance content score of the target person by executing the utterance content score calculation process. The utterance content score calculation process will be described later. When calculating the utterance content score, the utterance content output from the speaker 40 is determined. The priority calculation unit 163 calculates the priority of the target person based on the calculated utterance history score, human condition score, and utterance content score (step S207). For example, the priority calculation unit 163 may calculate the priority by adding the utterance history score, the human condition score, and the utterance content score as in the following mathematical formula (1). For example, the priority calculation unit 163 may calculate the priority by weighting each score of the utterance history score, the human state score, and the utterance content score as in the following mathematical formula (2). Any value may be specified for the weight.

The priority calculation unit 163 determines whether or not the priority of all persons has been calculated (step S208). Specifically, the priority calculation unit 163 determines that the priority of all the persons has been calculated when there is no person for whom the priority has not been calculated in the operation determination process this time. If there is a person whose priority has not been calculated in the operation determination process this time, the priority calculation unit 163 determines that the priority of all the people has not been calculated.

If the priority of all persons has not been calculated (step S208: NO), the process proceeds to step S202. The priority calculation unit 163 determines a person to be the target of the priority calculation. When the priority of all people is calculated (step S208: YES), the operation determination unit 166 determines the person to perform the operation (step S209). Specifically, the motion determination unit 166 determines the person with the highest priority among the calculated priorities as the person to whom the dialogue device 50 faces. If there are a plurality of people with the highest priority, the motion determination unit 166 determines one of the plurality of people. In this case, the motion determination unit 166 may determine a person by any means. For example, the motion determination unit 166 may determine the person who has turned his / her face the least number of times based on the person information record. For example, the motion determination unit 166 may determine the person who has been in the posture of the dialogue device 50 for the shortest time.

The operation determination unit 166 determines the operation content of the dialogue device 50 to be performed on the determined person (step S210). For example, the motion determination unit 166 determines the direction in which the face of the dialogue device 50 is directed. The motion determination unit 166 estimates the coordinates indicating the position of the person based on the information acquired by the sensor 10 and the camera 20. The motion determining unit 166 determines the angles of the head 510 and the legs 540 so that the face of the dialogue device 50 faces in the direction of the estimated coordinates. Further, the operation determination unit 166 acquires the operation content associated with the utterance content for the determined person from the operation content storage unit 105. The operation determination unit 166 determines the acquired operation content as the operation content of the dialogue device 50. The device control unit 167 controls the speaker 40 or the dialogue device 50 based on the determined operation content and utterance content (step S211).

FIG. 9 is a flowchart showing a specific example of the flow of the utterance history score calculation process. The priority calculation unit 163 acquires the score of the number of times the face is turned from the utterance history score table (step S241). Specifically, the priority calculation unit 163 acquires the number of times the face is turned from the person information record acquired in step S203. The priority calculation unit 163 acquires the utterance history score table from the score map storage unit 103. The priority calculation unit 163 acquires a score associated with the number of times the face is turned from the utterance history score table. For example, when the number of times the person turns the face is 3, the priority calculation unit 163 acquires 3 as the score of the number of times the face is turned from the utterance history score table.

The priority calculation unit 163 acquires the score of the number of utterances of the dialogue device from the utterance history score table (step S242). Specifically, the priority calculation unit 163 acquires the number of speeches of the dialogue device from the person information record acquired in step S203. The priority calculation unit 163 acquires the utterance history score table from the score map storage unit 103. The priority calculation unit 163 acquires a score associated with the number of utterances of the dialogue device from the utterance history score table. For example, when the number of utterances of the dialogue device held by the human information record is 0, the priority calculation unit 163 acquires 5 as the score of the number of utterances of the dialogue device from the utterance history score table.

The priority calculation unit 163 acquires the utterance time score from the utterance history score table (step S243). Specifically, the priority calculation unit 163 acquires the utterance time from the person information record acquired in step S203. The priority calculation unit 163 acquires the utterance history score table from the score map storage unit 103. The priority calculation unit 163 acquires the score associated with the utterance time from the utterance history score table. For example, when the utterance time of the person information record is 25 seconds, the priority calculation unit 163 acquires 1 as the utterance time score from the utterance history score table.

The priority calculation unit 163 calculates the utterance history score (step S244). Specifically, the priority calculation unit 163 may calculate the utterance history score by performing a predetermined calculation on each score acquired in steps S241 to S243. For example, the priority calculation unit 163 may calculate the utterance history score by adding each score. For example, the priority calculation unit 163 may calculate the utterance history score by giving a predetermined weight to each score. Any value may be specified for the weight.

FIG. 10 is a flowchart showing a specific example of the flow of the human condition score calculation process.
The priority calculation unit 163 acquires the facial expression score from the human condition score table (step S251). Specifically, the priority calculation unit 163 acquires a facial expression from the person information record acquired in step S203. The priority calculation unit 163 acquires a person status score table from the score map storage unit 103. The priority calculation unit 163 acquires the score associated with the facial expression from the human state score table. For example, when the facial expression of the person information record is a smile, the priority calculation unit 163 acquires 3 as the score of the person's condition from the person information score table.

The priority calculation unit 163 acquires the score of the number of human utterances from the human status score table (step S252). Specifically, the priority calculation unit 163 acquires the number of utterances of a person from the person information record acquired in step S203. The priority calculation unit 163 acquires a person status score table from the score map storage unit 103. The priority calculation unit 163 acquires a score associated with the number of human utterances from the human state score table. For example, when the number of human utterances held by the human information record is 0, the priority calculation unit 163 acquires 5 as the score of the number of human utterances from the human status score table.

The priority calculation unit 163 acquires the line-of-sight score from the human condition score table (step S253). Specifically, the priority calculation unit 163 acquires the line-of-sight value from the person state record acquired in step S203. The priority calculation unit 163 acquires a person status score table from the score map storage unit 103. The priority calculation unit 163 acquires the score associated with the line of sight from the human state score table. For example, when the line of sight of the person information record is directed to a position other than the dialogue device, the priority calculation unit 163 acquires 5 as the line of sight score from the person state score table.

The priority calculation unit 163 acquires the score of the staying time from the human condition score table (step S254). Specifically, the priority calculation unit 163 acquires the value of the staying time from the person status record acquired in step S203. The priority calculation unit 163 acquires a person status score table from the score map storage unit 103. The priority calculation unit 163 acquires the score associated with the staying time from the human condition score table. For example, when the staying time of the person information record is 12 seconds, the priority calculation unit 163 acquires 5 as the line-of-sight score from the person status score table.

The priority calculation unit 163 calculates the human condition score (step S244). Specifically, the priority calculation unit 163 may calculate the human condition score by performing a predetermined calculation on each score acquired in steps S251 to S254. For example, the priority calculation unit 163 may calculate the human condition score by adding each score. For example, the priority calculation unit 163 may calculate the human condition score by giving a predetermined weight to each score. Any value may be specified for the weight.

FIG. 11 is a flowchart showing a specific example of the flow of the utterance content score calculation process. The voice processing unit 164 of the control device 100 determines whether or not a voice signal has been acquired from the microphone 30 (step S261). When the voice signal is acquired (step S261: YES), the voice processing unit 164 executes the voice recognition process. The voice processing unit 164 generates a character string based on the voice signal by voice recognition processing. When the voice processing unit 164 acquires the voice signals of a plurality of people, the voice processing unit 164 generates a character string for each person. In this case, the voice processing unit 164 associates the person with the voice signal based on a known sound source separation process. If the audio signal has not been acquired (step S261: NO), the process proceeds to step S263.

The utterance determination unit 165 determines the utterance content of the dialogue device 50 (step S263). Specifically, the utterance determination unit 165 determines the utterance content based on the generated character string. The utterance determination unit 165 acquires the utterance content associated with the generated character string from the utterance content storage unit 104. The utterance determination unit 165 determines the acquired utterance content as the utterance content of the dialogue device 50.

The priority calculation unit 163 calculates the utterance content score (step S264). Specifically, the priority calculation unit 163 acquires the utterance content score table from the score map storage unit 103. The priority calculation unit 163 acquires the score associated with the utterance content from the utterance content score table. For example, when the utterance content is "good morning", the priority calculation unit 163 acquires 5 as the utterance content score from the utterance content score table. The priority calculation unit 163 may calculate the utterance content score by giving a predetermined weight to the utterance content score. Any value may be specified for the weight.

In the control system 1 configured in this way, the operation determination unit 166 of the control device 100 determines which of the plurality of people detected by the sensor 10 the face of the dialogue device 50 is directed. Specifically, the motion determination unit 166 determines in which direction the dialogue device 50 should be directed based on the history of dialogue between the person and the dialogue device 50. The device control unit 167 controls the dialogue device 50 so as to face the determined person. Therefore, it is possible to show which person the dialogue device 50 is talking to in an easy-to-understand manner.

FIG. 12 is a diagram showing a specific example of the operation of the dialogue device 50. FIG. 12 shows a region 200, a region 200a and a region 200b. Regions 200-200b show changes over time. The region 200, the region 200a, and the region 200b all include a person 60, a person 60a, and a person 60b. In FIG. 12, the dialogue device 50 is talking to the person 60, the person 60a, and the person 60b at the same time. In FIG. 12, the time elapses in the order of region 200, region 200a, and region 200b. The priority calculation unit 163 repeatedly calculates the priority according to the passage of time.

In region 200, the priority of person 60 is 15. In region 200, the priority of person 60a is 10. In region 200, person 60b has a priority of 5. Therefore, the dialogue device 50 turns its face toward the person 60 and speaks. In region 200a, the priority of person 60 is 10. In the region 200a, the priority of the person 60a is 5. In region 200a, person 60b has a priority of 15. Therefore, the dialogue device 50 turns its face toward the person 60b and speaks. In region 200b, person 60 has a priority of 10. In region 200b, person 60a has a priority of 15. In region 200b, person 60b has a priority of 10. Therefore, the dialogue device 50 turns its face toward the person 60a and speaks. In this way, the priority calculation unit 163 periodically calculates the priority. The dialogue device 50 takes turns to speak to a plurality of people without being biased toward a specific person. Therefore, a person can recognize that he / she is being spoken to himself / herself.

In the above-described embodiment, the priority calculation unit 163 is configured to calculate the priority based on the utterance history score, the human condition score, and the utterance content score, but is not limited thereto. For example, the priority calculation unit 163 may be configured to calculate the priority based on any one or more of the utterance history score, the human condition score, and the utterance content score. In this case, the priority calculation unit 163 may be configured not to calculate the unused score.

In the above-described embodiment, the control system 1 has described, but is not limited to, the dialogue device 50 as being a robot. The dialogue device 50 may be composed of a display device such as a display or a drive device such as an actuator instead of the robot. FIG. 13 is a diagram showing a specific example of a case where the control system 1 is provided with a predetermined device such as a display device or an actuator instead of the robot. FIG. 13A is a diagram showing a specific example in the case where the control system 1 is provided with a display device instead of the robot. According to FIG. 13A, the control system 1 includes a dialogue device 50a (display device) instead of the dialogue device 50. The dialogue device 50a is a display device such as a touch panel or a display. The dialogue device 50a displays the human image 51. The human image 51 is an image showing a human figure. The figure of a person may be a figure representing the whole body or a part of a figure such as a bust-up. As the human image 51, a live-action image may be used, or CG (Computer Graphics) may be used. The human image 51 operates based on the operation content determined by the device control unit 167. The image displayed on the dialogue device 50a may be a plurality of human images, or may be an image of an animal, an object, a plant, or the like other than a human.

FIG. 13B is a diagram showing a specific example in the case where the control system 1 includes an actuator instead of the robot. According to FIG. 13B, the control system 1 includes a dialogue device 50b (actuator) instead of the dialogue device 50. The dialogue device 50b includes a motor, a transmission gear, and the like inside. An object 52 can be placed on the dialogue device 50b. The object 52 is an object such as a book, food, or a tool. The dialogue device 50b operates based on the operation content determined by the control device 100. Specifically, the dialogue device 50b performs a predetermined operation such as rotational movement, vibration up and down, and the like. The object 52 mounted on the dialogue device 50b rotates and moves or vibrates up and down according to the operation of the dialogue device 50b. In this way, the control system 1 can make the object 52 behave like a living thing by outputting sound from the speaker 40 while operating the dialogue device 50b. Further, even when the object 52 talks to a person, the person can know which person the object 52 is talking to.

In the above-described embodiment, the control system 1 has been described as controlling one dialogue device 50, but the control system 1 is not limited to one. For example, the control system 1 may be configured to control two or more dialogue devices 50. In this case, the operation determination unit 166 may determine the target person by the number of dialogue devices 50 in descending order of priority.

In the above-described embodiment, the control system 1 determines in which direction the dialogue device 50 is directed by calculating the priority. However, the control system 1 may be configured to determine which person the face of the dialogue device 50 should face based on the history of dialogue between the person and the dialogue device 50 and a predetermined estimator. The dialogue history may be any information as long as it is information indicating the history between the person and the dialogue device 50, such as a human information record. The estimator is a learning model generated by machine learning a plurality of learning data. The estimator has a function of estimating the priority based on the history of dialogue between the person and the dialogue device 50. In this case, the priority calculation unit 163 estimates the priority based on the person information record stored in the person information storage unit 102 and the estimator. The estimator may be generated by, for example, the priority calculation unit 163, or may be stored in advance by the control device 100. The learning data is, for example, data in which the history of dialogue between a person and the dialogue device 50 and the priority are associated with each other. The training data is used to generate the estimator. The machine learning may be any known machine learning such as SGD (Stochastic Gradient Descent), random forest, linear regression, decision tree or CNN (Convolutional Neural Network).

The control device 100 may be implemented by using a plurality of information processing devices that are communicably connected via a network. In this case, each functional unit included in the control device 100 may be distributed and mounted in a plurality of information processing devices. For example, even if the person identification unit 161 and the feature estimation unit 162, the priority calculation unit 163, the voice processing unit 164, the utterance determination unit 165, the operation determination unit 166, and the device control unit 167 are mounted on different information processing devices. good.

The control device 100 in the above-described embodiment may be realized by a computer. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. It may be realized by using a programmable logic device such as FPGA (Field Programmable Gate Array).

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

1 ... control system, 10 ... sensor, 20 ... camera, 30 ... microphone, 40 ... speaker, 50 ... dialogue device, 100 ... control device, 101 ... communication unit, 102 ... human information storage unit, 103 ... score map storage unit, 104 ... utterance content storage unit, 105 ... operation content storage unit, 106 ... control unit, 161 ... person identification unit, 162 ... feature estimation unit, 163 ... priority calculation unit, 164 ... voice processing unit, 165 ... utterance determination unit, 166 ... Operation determination unit, 167 ... Equipment control unit

Claims (7)

An acquisition unit that acquires information about a plurality of people facing a predetermined dialogue device that talks to a person for each person, and an acquisition unit.
Which of the plurality of people the dialogue device should be directed to, based on the acquired information and the history information of the dialogue between the person and the dialogue device stored in the predetermined storage device. The action decision unit to decide and
An utterance determination unit that determines the utterance content of the dialogue device based on the voice uttered by the person.
A device control unit that controls the operation of the dialogue device so as to face the direction and controls a speaker provided in the vicinity of the dialogue device so as to output the utterance content.
A control system. The operation determining unit calculates the dialogue device based on the acquired information and the history information, and determines whether or not to direct the dialogue device in the direction of the person. The dialogue device has the highest priority in the direction of the person. Decide to point and decide what to do,
The control system according to claim 1. An utterance content score in which the utterance content is quantified based on the history information and the utterance content, an utterance history score in which the history information indicates and quantifies the utterance history of the dialogue device to the person, and the above. A priority calculation unit that calculates one or more of the person status score, which is indicated by the history information and quantifies the status of the person, and calculates the priority based on the calculated score. Further prepare
The operation determination unit determines the direction in which the dialogue device is directed and the operation content based on the priority calculated by the priority calculation unit.
The control system according to claim 2. The operation determining unit moves the dialogue device in the direction of any of the plurality of people based on the estimator generated by machine learning the teacher data in which the history information and the priority are associated with each other. Decide whether to turn
The control system according to claim 2. An acquisition unit that acquires information about a plurality of people facing a predetermined dialogue device that talks to a person for each person, and an acquisition unit.
Which of the plurality of people the dialogue device should be directed to, based on the acquired information and the history information of the dialogue between the person and the dialogue device stored in the predetermined storage device. The action decision unit to decide and
An utterance determination unit that determines the utterance content of the dialogue device based on the voice uttered by the person.
A device control unit that controls the operation of the dialogue device so as to face the direction and controls a speaker provided in the vicinity of the dialogue device so as to output the utterance content.
A control device. An acquisition step in which the control device acquires information about a plurality of people facing a predetermined dialogue device that speaks to a person for each person.
Based on the acquired information and the history information of the dialogue between the person and the dialogue device stored in the predetermined storage device, the control device uses the dialogue device of any of the plurality of people. The action decision step to decide whether to turn and
An utterance determination step in which the control device determines the utterance content of the dialogue device based on the voice uttered by the person.
A device control step in which the control device controls the operation of the dialogue device so as to face the direction and controls a speaker provided in the vicinity of the dialogue device so as to output the utterance content.
Control method to have. A computer program for operating a computer as the control system according to any one of claims 1 to 4.

Images