JP7032284B2 - A device, program and method for estimating the activation timing based on the image of the user's face. - Google Patents

Description

The present invention relates to a technique of a dialogue device that realizes a natural dialogue with a user.

The dialogue device interacts with the user via the interface of the smartphone or tablet terminal. The user's utterance voice is converted into text, and the utterance meaning is estimated from the context structure. Then, it responds to the user based on the dialogue scenario corresponding to the utterance meaning. For example, there are dialogue systems such as "Siri®" and "Shabette Concierge®".

In recent years, smart speakers such as "Google Home (registered trademark)" and "Amazon Echo (registered trademark)" and "SOTA (registered trademark)" and "Unibo (registered trademark)" have been used as devices for interacting with users. Robots have been used (hereinafter referred to as "robots"). The user needs to speak an active command ("OK, XXX", etc.) before interacting with these robots. The robot that detects this command activates the voice recognition function and operates to recognize the subsequent user's spoken voice.

Conventionally, there is a technique of a guidance robot that calls out in consideration of the user's intention (see, for example, Patent Document 1). According to this technology, the direction the user is looking at is determined for each frame image that is continuous in time series, the amount of change in direction representing the change in that direction with the passage of time is calculated, and the amount of change in direction is calculated based on the amount of change in direction. Decide whether to call out. Specifically, it is possible to reach out to users who are having trouble with something in the exhibition hall or in the store.

Further, there is also a technique of a customer purchase intention prediction device that displays an optimum advertisement from the aspect of the customer in the store (see, for example, Patent Document 2). According to this technology, changes in the customer's face orientation are tracked over a certain period of time, and the product whose customer's face orientation is stopped with respect to the product is recommended to the customer for the longest product attention time. can do.

JP-A-2017-159396 Japanese Unexamined Patent Publication No. 2016-076109

"Head Pose Optimization using OpenCV and Dlib", [online], [Searched October 4, 2018], Internet <URL: https://www.learnopencv.com/head-pose-estimation-using-opencv-and -dlib /> "Short-time Fourier transform", [online], [Search on October 4, 2018], Internet <URL: https://www.ieice.org/jpn/event/FIT/pdf/d/2014/H- 039.pdf ＞ "Wavelet transform", [online], [Search on October 4, 2018], Internet <URL: http://www.cqpub.co.jp/hanbai/books/30/30961/30961_9syo.pdf>

It can be tedious for the user to say the active command to the robot every time.
For example, in a house or a store where there is a lot of ambient noise such as TV or conversation, the robot may not be able to recognize the active command spoken by the user.
On the contrary, even when the user does not intentionally speak to the robot, the robot may malfunction in response to ambient noise.

Further, as in Patent Documents 1 and 2, it is difficult to accurately optimize the timing at which the robot speaks only by the amount of change in the orientation of the user's face.
On the other hand, the inventors of the present application thought that there might be some characteristic change in the image of the user's face at the timing when the user wants to talk to the robot. I thought that if we could learn and find this characteristic change from the experience value, we would be able to identify the optimal activation timing.

Therefore, an object of the present invention is to provide a device, a program, and a method for estimating with high accuracy the timing of talking to a user or invoking an action from an image of a user's face.

According to the present invention, in a dialogue device that interacts with a user,
A face area detection means that inputs a time-series image of the user's face taken by the camera and detects the face area from each image, and
Face parameter extraction means that extracts each parameter of the face from the face area reflected in the image,
A feature amount extraction means for extracting a time-frequency feature amount from the time-series change of the face parameter, and a feature amount extraction means.
It was learned in advance from the teacher data in which the time-frequency feature amount and the activation availability (positive example / negative example) were associated with each different time span in the time-series image, and the time-frequency feature for each time span was learned at the time of estimation. A machine learning engine that inputs an amount and estimates whether or not the current timing is the activation timing for the user based on the time span that maximizes the estimation accuracy .
It is characterized by having an activation means that is activated for the user when it is determined to be true by the machine learning engine.

According to another embodiment of the dialogue device of the present invention.
It is also preferable that the face parameter extraction means includes Euler angles for the face, the center position of the face, and / or the size of the face as each parameter of the face.

According to another embodiment of the dialogue device of the present invention.
It also has a voice recognition means to extract text from the user's spoken voice.
When it is determined to be false by the machine learning engine, it is also preferable to reduce the error in speech recognition by increasing the threshold value of the speech recognition probability in the speech recognition means.

According to another embodiment of the dialogue device of the present invention.
It is also preferable that the triggering means utters an initial text based on a dialogue scenario as a triggering to the user.

According to another embodiment of the dialogue device of the present invention.
If the dialogue device is a movable robot,
It is also preferable that the activation means operates with the initial behavior based on the action scenario as the activation for the user.

According to another embodiment of the dialogue device of the present invention.
When the dialogue with the user is not established after the dialogue scenario trigger means utters the initial text.
It is also preferable that the machine learning engine collects teacher data as inoperable (negative example) for the time-frequency features up to that time.

According to another embodiment of the dialogue device of the present invention.
When spoken by the user during the interruption of the dialogue scenario
It is also preferable that the machine learning engine collects teacher data as being operable (normal example) for the time-frequency features up to that time.

According to another embodiment of the dialogue device of the present invention.
The feature amount extraction means extracts the time-frequency feature amount by short-time Fourier transform or wavelet transform for the time series of each parameter of the face.
It is also preferable that the machine learning engine functions the computer as if it were an LSTM (Long Short-Term Memory).

According to the present invention, in a program for operating a computer mounted on a device that interacts with a user.
A face area detection means that inputs a time-series image of the user's face taken by the camera and detects the face area from each image, and
Face parameter extraction means that extracts each parameter of the face from the face area reflected in the image,
A feature amount extraction means for extracting a time-frequency feature amount from the time-series change of the face parameter, and a feature amount extraction means.
It was learned in advance from the teacher data in which the time-frequency feature amount and the activation availability (positive example / negative example) were associated with each different time span in the time-series image, and the time-frequency feature for each time span was learned at the time of estimation. A machine learning engine that inputs an amount and estimates whether or not the current timing is the activation timing for the user based on the time span that maximizes the estimation accuracy .
When it is determined to be true by the machine learning engine, it is characterized in that the computer functions as an activation means to be activated for the user.

According to the present invention, in a method of interacting with a device that interacts with a user,
The device is
It has a machine learning engine that has been learned in advance using teacher data that associates time-frequency features with activation availability (positive and negative examples) for each different time span in a time-series image .
The first step of inputting a time-series image in which the user's face taken by the camera is reflected and detecting the face area from each image, and
The second step of extracting each parameter of the face from the face area reflected in the image,
The third step of extracting the time-frequency feature amount from the time-series change of the face parameter, and
A fourth method of estimating whether or not the current time is the activation timing for the user based on the time span that maximizes the estimation accuracy from the time frequency feature amount for each time span at the time of estimation using the machine learning engine. Steps and
If it is determined to be true by the 4th step, the 5th step to be activated for the user
It is characterized by executing.

According to the dialogue device, the program, and the method of the present invention, it is possible to estimate the timing of talking to the user and the activation timing of the operation with high accuracy from the image of the user's face.

It is a functional block diagram of the dialogue apparatus in this invention. It is a functional block diagram of a server in a dialogue system. It is explanatory drawing which shows the processing flow of each functional component part in the estimation stage. It is an external view which a robot of a dialogue device is taking a picture of a user's face. It is explanatory drawing which shows the process of a face area detection part and face parameter extraction part. It is explanatory drawing which shows the process of a feature amount extraction part. It is explanatory drawing which shows the processing flow of each functional component part in an initial stage. It is explanatory drawing which shows the process flow of each functional component part to be trained.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional configuration diagram of the dialogue device in the present invention.

According to FIG. 1, the dialogue device 1 is a robot (including a smart speaker) that interacts with a user. The dialogue device 1 is equipped with a microphone, a speaker, and a camera as input / output devices of the user interface. The camera captures an image of the user's face. The microphone acquires the user's spoken voice. The speaker speaks to the user by voice.
The user can realize a natural dialogue with the dialogue device 1 of the robot as a character.

FIG. 2 is a functional configuration diagram of a server in a dialogue system.

According to FIG. 2, it is exactly the same as the functional configuration of the present invention in FIG. 1, but is configured by a server. A smartphone or tablet terminal owned by the user serves as a user interface in the dialogue system.

The dialogue device 1 of the present invention can actively talk to or act on the user (for example, the robot raises a hand or starts walking) at the best timing for the user.
According to FIG. 1, the dialogue device 1 has a dialogue with a face area detection unit 11, a face parameter extraction unit 12, a feature amount extraction unit 13, a machine learning engine 14, an activation unit 15, and a voice recognition unit 101. It has an execution unit 102 and a voice conversion unit 103. These functional components can be realized by executing a program that makes a computer mounted on the dialogue device function. Further, the flow of processing of these functional components can be understood as a method of dialogue of the device.

The voice recognition unit 101, the dialogue execution unit 102, and the voice conversion unit 103 are functional components of a general dialogue device.
The voice recognition unit 101 inputs the user's uttered voice from the microphone, converts the uttered voice into text, and outputs the text to the dialogue execution unit 102.
The dialogue execution unit 102 searches for the text that becomes the next dialogue scenario with respect to the text input from the voice recognition unit 101. The text is output to the voice conversion unit 103. The dialogue scenario is a scenario in which the following dialogue text is associated with the user's utterance text, and a dialogue node composed of a question and an answer is configured in a tree shape.
The voice conversion unit 103 inputs a dialogue sentence to the user from the dialogue execution unit 102, converts the dialogue sentence into a voice signal, and outputs the voice signal to the speaker.

According to the present invention, the dialogue device 1 is divided into a <estimation stage> and an <initial stage> using the machine learning engine 14.

<Estimation stage>
FIG. 3 is an explanatory diagram showing a processing flow of each functional component in the estimation stage.

[Face area detection unit 11]
The face area detection unit 11 inputs a time-series image (video) in which the user's face captured by the camera is reflected, and detects the face area from each image.

FIG. 4 is an external view in which the robot of the dialogue device photographs the user's face.
FIG. 5 is an explanatory diagram showing the processing of the face area detection unit and the face parameter extraction unit.

The face area detection unit 11 identifies the face itself by extracting conspicuous features of the face from each image frame in the time series taken by the camera. As features, for example, the relative positions and sizes of facial parts, and the shapes of eyes, nose, cheekbones, and chin are used. The image part that matches the template created from the features of the face image is searched for as the face area. There are various existing methods as the face recognition algorithm.
According to FIG. 5, time-series images of the user's face are arranged side by side, and the face area detection unit 11 detects the face area from each image.

[Face parameter extraction unit 12]
The face parameter extraction unit 12 extracts each parameter of the face from the face region reflected in the image. Each face parameter includes the following.
Euler angles for the face Center position of the face Face size

For the detection of face parameters, for example, a head pose estimation method can be used (see, for example, Non-Patent Document 1).
In order to determine the orientation of the face, it can be implemented as image recognition using OpenCV (image processing), Dlib (machine learning), and deep learning classification model of the open source library.
The center position and size of the face can be derived as the position and size of the face region with respect to the entire angle of view.

The face parameters change in time series, for example, as follows, depending on the user's actions.
(Sit down and look at your cellphone)->
The face orientation changes from the front to the bottom, and the center position of the face changes from the top to the bottom.
(Stand up and go out)->
The face orientation changes from front to back, and the center position of the face changes from bottom to top.
(Take medicine)->
The face orientation changes from bottom to top and then down again.
(Approaching the robot while looking around)->
The orientation of the face changes from side to side, and the size of the face changes greatly.
(Check the surrounding area while looking at the guide map)->
The face orientation changes from bottom to left and right, and then down again.
(See the robot)->
The face orientation changes up, down, left and right, and the face position changes up, down, left and right.

The extracted time-series face parameters are output to the feature amount extraction unit 13.

[Feature amount extraction unit 13]
The feature amount extraction unit 13 extracts a "time-frequency feature amount" from the time-series change of the face parameter. That is, the feature quantities related to time and frequency are extracted at the same time.

FIG. 6 is an explanatory diagram showing the processing of the feature amount extraction unit.

The feature amount extraction unit 13 extracts the time-frequency feature amount from the time series of each parameter of the face by, for example, "short-time Fourier transform" or "wavelet transform".

The short-time Fourier transform (STFT) is a method of cutting out time at regular intervals and performing a Fourier transform one after another (see, for example, Non-Patent Document 2). In this way, the frequency and phase (change) of the parameter that changes with time are analyzed.

When the short-time Fourier transform is used, each parameter of the face is converted into frequency in the first column and amplitude in the second column as shown in Table 1 below. On the other hand, a short-time Fourier transform is applied.

The wavelet transformation is a method of changing the time width for analysis according to the frequency (see, for example, Non-Patent Document 3). It leaves information in the time domain that is lost when the frequency characteristics are obtained by the Fourier transform. In the wavelet transform, a waveform in a given wide frequency domain can be expressed by enlarging / reducing a small wave (wavelet), moving it in parallel, and adding them together.

When wavelet transform is used, as shown in Table 2 below for each parameter of the face, the first column is the frequency (converted from the output Scale of the wavelet transform), the second column is the start time to the end time, and the third column is the amplitude. Convert to. On the other hand, by applying the wavelet transform and acquiring the frequency component that fluctuates with time, the time-frequency feature amount can be derived in more detail than the short-time Fourier transform.

The time-frequency feature amount based on each parameter of the face can improve the completeness of the user's behavior pattern recognition and the robustness in which the activation timing is less susceptible to disturbance.

[Machine learning engine 14]
The machine learning engine 14 has been learned in advance using teacher data in which the time-frequency feature amount and the activation possibility (positive example / negative example) are associated with each other. "Activation" means talking to the user or operating the robot to receive the user's attention. That is, the interrelationship between the time-frequency feature amount based on the time-series change of each parameter of the face and the activation timing OK or NG is constructed as a learning model.

The machine learning engine 14 is preferably LSTM (Long Short-Term Memory), for example. LSTM is a kind of RNN (Recurrent Neural Network) that can learn long-term dependencies. The RNN is a chain that repeats the modules of the neural network.

Then, the machine learning engine 14 inputs the time-frequency feature amount output from the feature amount extraction unit 13, and estimates whether or not the current time is the activation timing for the user.
According to FIG. 3, when the activation timing is OK, the machine learning engine 14 outputs to that effect to the activation unit 15.
Further, when the activation timing is NG, the machine learning engine 14 instructs the voice recognition unit 101 to raise the threshold value of the voice recognition probability. The activation timing NG means that the user does not pay attention to the dialogue device 1, so that the voice recognition probability is increased so that the voice is not recognized from the ambient noise. This makes it possible to reduce the misrecognition of the user's utterance.

[Activator 15]
When the machine learning engine 14 determines that the activation unit 15 is true (activation timing is OK), the activation unit 15 is activated for the user. Here, "invocation" refers to, for example, the following aspects.
(1) Speak an initial text based on a dialogue scenario as an activation for the user.
(2) When the dialogue device is an operable robot, it operates in the initial behavior based on the behavior scenario as an activation for the user.

Next, the learning process in the machine learning engine 14 will be described.
The machine learning engine 14 executes a learning process in an initial stage of constructing a learning model based on pre-stored teacher data, and a learning process in an estimation stage of constructing a learning model while collecting teacher data.

<Learning process at the initial stage of machine learning engine 14>
FIG. 7 is an explanatory diagram showing a processing flow of each functional component in the initial stage.

According to FIG. 7, as the teacher data, the image of the user's face and whether or not it can be activated (positive example / negative example) are associated with each other. The teacher data is, for example, recording the model facial movements of a plurality of subjects in front of the camera of the dialogue device 1. From the video of the subject's facial movement, whether or not to activate the conversation or movement is associated.

The image of the user's face is processed by the face area detection unit 11, the face parameter extraction unit 12, and the feature amount extraction unit 13 described above, and the time-frequency feature amount is obtained. The time-frequency feature amount is associated with the activation availability (positive example / negative example) and input to the machine learning engine 14. As a result, the machine learning engine 14 builds a learning model.

<Learning process at the estimation stage of the machine learning engine 14>
It becomes difficult for the machine learning engine 14 to cover all learning patterns at an initial stage. Therefore, even in the estimation stage (operation stage), positive and negative teacher data are collected based on the positive reaction or the negative reaction of the user.
The machine learning engine 14 collects positive teacher data and negative teacher data while executing the estimation stage.

FIG. 8 is an explanatory diagram showing a processing flow of each functional component to be learned.

(When collecting positive teacher data)
When the user speaks during the interruption of the dialogue scenario, the machine learning engine 14 collects teacher data as activating (normal example) for the time-frequency features up to that time.
If the user's reaction is positive (speaks to the robot by himself) even though it is determined that the dialogue device 1 should not talk or operate (negative example), up to this point immediately before. It is determined that the time-frequency feature amount can be activated.

(When collecting negative teacher data)
When the dialogue with the user is not established after talking or operating from the activation unit 15 at the activation timing, the machine learning engine 14 collects teacher data as it cannot be activated for the time frequency feature amount up to that time. ..
If the user's reaction is negative (ignored) even though it is determined that the dialogue device 1 may speak or operate (normal example), the time-frequency feature amount up to this point immediately before this. Determines that it could not be activated.

<Setting different time spans>
As another embodiment, it is also preferable that the teacher data of the machine learning engine associates the time-frequency feature amount with the activation availability for each different time span.
For example, a plurality of fine particle size and coarse particle size time spans are set, and the fixed number of frames in the last N seconds is set as a default value.
(Fine grain timespan) Latest 1 second / 10 frames-> Derivation of time frequency features (Coarse grain timespan) Latest 5 seconds / 10 frames-> Derivation of time frequency features What is timespan? The time interval (sampling interval) of the image used to derive the quantity.

The feature amount extraction unit 13 described above outputs a different time frequency feature amount for each time span even if the same image in which the user's face is reflected is input. Then, the time-frequency feature amount is input to the machine learning engine 14 for each time span. As a result, a different learning model will be constructed for each time span.

The machine learning engine 14 may evaluate the estimation accuracy for each time span for the estimation stage. It is also possible to use the time span that maximizes the accuracy. The estimation accuracy here may be calculated as a concordance rate by collating the estimation result for each time span with the positive reaction or the negative reaction of the user.

As described above in detail, according to the dialogue device, the program, and the method of the present invention, it is possible to estimate the timing of talking to the user and the activation timing of the operation with high accuracy from the image of the user's face. That is, it is possible to realize a robot or a smart speaker that can read human air by improving convenience and intelligence from the user's point of view.

Various modifications, modifications and omissions of the above-mentioned various embodiments of the present invention within the scope of the technical idea and viewpoint of the present invention can be easily carried out by those skilled in the art. The above explanation is just an example and does not attempt to limit anything. The present invention is limited only to the scope of claims and their equivalents.

1 Dialogue device 11 Face area detection unit 12 Face parameter extraction unit 13 Feature quantity extraction unit 14 Machine learning engine 15 Activation unit 101 Voice recognition unit 102 Dialogue execution unit 103 Voice conversion unit

Claims (10)

In a dialogue device that interacts with the user
A face area detection means that inputs a time-series image of the user's face taken by the camera and detects the face area from each image, and
Face parameter extraction means that extracts each parameter of the face from the face area reflected in the image,
A feature amount extraction means for extracting a time-frequency feature amount from the time-series change of the face parameter, and a feature amount extraction means.
It was learned in advance from the teacher data in which the time-frequency feature amount and the activation availability (positive example / negative example) were associated with each different time span in the time-series image, and the time-frequency feature for each time span was learned at the time of estimation. A machine learning engine that inputs an amount and estimates whether or not the current timing is the activation timing for the user based on the time span that maximizes the estimation accuracy .
A dialogue device comprising an activation means that is activated for a user when it is determined to be true by the machine learning engine. The dialogue device according to claim 1, wherein the face parameter extracting means includes Euler angles facing the face, a center position of the face, and / or the size of the face as each parameter of the face. It also has a voice recognition means to extract text from the user's spoken voice.
The dialogue device according to claim 1 or 2, wherein when it is determined to be false by the machine learning engine, the error in voice recognition is reduced by increasing the threshold value of the voice recognition probability in the voice recognition means. The dialogue device according to any one of claims 1 to 3, wherein the activation means utters an initial text based on a dialogue scenario as an activation to a user. If the dialogue device is a movable robot,
The dialogue device according to any one of claims 1 to 3, wherein the activation means operates with an initial behavior based on an action scenario as an activation for a user. When the dialogue with the user is not established after the dialogue scenario invoking means utters the initial text.
The dialogue device according to claim 4, wherein the machine learning engine collects teacher data as inoperable (negative example) with respect to the time-frequency feature amount up to that time. When spoken by the user during the interruption of the dialogue scenario
The dialogue device according to claim 4 or 6, wherein the machine learning engine collects teacher data as being operable (normal example) with respect to the time-frequency feature amount up to that time. The feature amount extraction means extracts the time-frequency feature amount by short-time Fourier transform or wavelet transform for the time series of each parameter of the face.
The program according to any one of claims 1 to 7 , wherein the machine learning engine operates a computer so as to be an LSTM (Long Short-Term Memory). In a program that activates a computer installed in a device that interacts with a user
A face area detection means that inputs a time-series image of the user's face taken by the camera and detects the face area from each image, and
Face parameter extraction means that extracts each parameter of the face from the face area reflected in the image,
A feature amount extraction means for extracting a time-frequency feature amount from the time-series change of the face parameter, and a feature amount extraction means.
It was learned in advance from the teacher data in which the time-frequency feature amount and the activation availability (positive example / negative example) were associated with each different time span in the time-series image, and the time-frequency feature for each time span was learned at the time of estimation. A machine learning engine that inputs an amount and estimates whether or not the current timing is the activation timing for the user based on the time span that maximizes the estimation accuracy .
A program characterized in that a computer functions as an invoking means to be activated for a user when it is determined to be true by the machine learning engine. In the method of interacting with the device that interacts with the user
The device is
It has a machine learning engine that has been learned in advance using teacher data that associates time-frequency features with activation availability (positive and negative examples) for each different time span in a time-series image .
The first step of inputting a time-series image in which the user's face taken by the camera is reflected and detecting the face area from each image, and
The second step of extracting each parameter of the face from the face area reflected in the image,
The third step of extracting the time-frequency feature amount from the time-series change of the face parameter, and
A fourth method of estimating whether or not the current time is the activation timing for the user based on the time span that maximizes the estimation accuracy from the time frequency feature amount for each time span at the time of estimation using the machine learning engine. Steps and
If it is determined to be true by the 4th step, the 5th step to be activated for the user
A method of interacting with a device, characterized by performing.

Images