JP2021086354A - Information processing system, information processing method, and program - Google Patents

Abstract

To cause a speaker to actively output sounds to a user so that a conference or the like smoothly progresses.SOLUTION: An information processing system includes an acquisition unit configured to acquire a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker, an emotion analysis unit configured to analyze an emotion of the user based on the face image, a line-of-sight determination unit configured to determine whether the user is looking at the speaker device based on the face image, a time calculation unit configured to calculate the duration time of the user watching the speaker device based on a determination result by the line-of-sight determination unit, a response content identification unit configured to identify a predetermined response content based on the user's emotion and the duration time, and a transmission unit configured to transmit sound information indicating the predetermined response content to the speaker device in order to cause the speaker device to output sounds in accordance with the predetermined response content identified by the response content identification unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to information processing systems, information processing methods, and programs.

A determination device for determining whether or not to output audio to a speaker based on image information detected by an image sensor is disclosed (Patent Document 1).

Japanese Unexamined Patent Publication No. 2019-35897

Patent Document 1 discloses a determination device that determines the timing of audio output in a speaker provided in a house in which a user resides based on image information. Further, the determination device described in Patent Document 1 causes the speaker to output voice at the timing when the voice information is interrupted. According to the determination device of Patent Document 1, the speaker can output audio according to the situation in the house. However, in the determination device described in Patent Document 1, since it is not possible for the speaker to actively output voice from the speaker in consideration of the user's emotions, it is not possible to smoothly perform a conversation between the speaker and the user. There was the problem that it was enough.

An object of the present invention has been made in view of the above problems, and an object of the present invention is to provide a system for actively outputting voice from a speaker in consideration of user's emotions.

The information processing system according to one aspect of the present invention includes an acquisition unit that acquires a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker, and the user's face image based on the face image. Based on the emotion analysis unit that analyzes emotions, the line-of-sight determination unit that determines whether or not the user is looking at the speaker device based on the face image, and the determination result in the line-of-sight determination unit, the user A time calculation unit that calculates the duration of viewing the speaker device, a response content specifying unit that specifies a predetermined response content based on the user's emotion and the duration, and the response content specifying unit. The speaker device is provided with a transmission unit that transmits voice information indicating the predetermined response content to the speaker device in order to output the sound to the speaker device in accordance with the predetermined response content specified in the above.

In the information treatment method according to one aspect of the present invention, a computer acquires a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker, and the computer obtains a face image showing the user's face based on the face image. The user is looking at the speaker device based on the analysis of the user's emotion, determining whether or not the user is looking at the speaker device based on the face image, and based on the determination result. Specifying a predetermined response content based on the calculation of the duration, the emotion of the user, and the duration, and the voice to the speaker device according to the specified predetermined response content. In order to output, the voice information indicating the predetermined response content is transmitted to the speaker device, and so on.

A program according to one aspect of the present invention causes a computer to acquire a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker, and based on the face image, the user's To analyze emotions, to determine whether or not the user is viewing the speaker device based on the face image, and to determine the duration of the user viewing the speaker device based on the determination result. Is calculated, the predetermined response content is specified based on the user's emotion and the duration, and the speaker device is made to output audio according to the specified predetermined response content. Therefore, the speaker device is made to transmit the voice information indicating the predetermined response content, and is executed.

According to the present invention, it is possible to encourage the user to speak by actively outputting voice to the user based on the user's emotion and line of sight.

It is a figure which shows an example of the structure of a voice notification system. It is a figure which shows the outline of the processing in a voice notification system. It is a figure which shows an example of the functional structure of the response server apparatus. It is a figure which shows an example of the response information table. It is a figure which shows an example of the functional structure of a speaker apparatus. It is a flow chart which shows an example of the processing of a response server apparatus. It is a figure which shows an example of the hardware configuration of a computer. It is a figure which shows an example of the functional structure in another embodiment of a speaker apparatus. It is a figure which shows an example of a speaker response information table.

Hereinafter, the voice notification system 1 according to the embodiment of the present invention will be described in detail with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention of excluding the application of various modifications and techniques not specified below. That is, the present invention can be implemented in various ways without departing from the spirit of the present invention, or by combining each embodiment. Further, in the description of the following drawings, the same or similar parts are represented by the same or similar reference numerals.
== Composition ==

FIG. 1 is a diagram showing an example of the configuration of the voice notification system 1. As shown in FIG. 1, the voice notification system 1 includes, for example, a response server device 10 and a speaker device 20. The functions of the response server device 10 and the speaker device 20 may be realized by one system. Further, the functions of the response server device 10 and the speaker device 20 may be realized by a plurality of other devices. Hereinafter, each component of the voice notification system 1 will be described.

The response server device 10 acquires a face image showing the user's face from the speaker device 20, and transmits predetermined response information to the speaker device 20 based on the face image, so that the speaker device 20 actively responds to the user. It is a device that makes the speaker speak. The response server device 10 is composed of an information processing device such as a server computer, and is connected to the speaker device 20 via a network 300. The transmission and reception of various data between the response server device 10 and the speaker device 20 will be described later.

The speaker device 20 has an image acquisition device 21 that acquires a face image, and is a device that transmits the face image acquired by the image acquisition device 21 to the response server device 10. Then, the speaker device 20 speaks based on the response information acquired from the response server device 10. The speaker device 20 is a so-called smart speaker.

Here, in order to help understanding the following description, an example of the hardware configuration of the speaker device 20 will be described. The speaker device 20 is for communicating with, for example, a microphone (not shown) that detects voice and converts it into an electric signal, a speaker that outputs voice information acquired from the response server device 10 (not shown), and an external device. A communication module (not shown), a display unit 22 that visually displays an icon that imitates the status and expression of the speaker device 20, operation buttons (not shown) for giving various operation instructions, and a control unit that controls each component. (Not shown). There are various types of speaker devices 20, for example, those having a plurality of microphones and speakers, those having microphones arranged at equal intervals on the outer peripheral portion of the upper surface, and those having equal intervals on the outer peripheral portion of the side surface. There are some that have speakers arranged in the room, and the specifications are not limited.
== Overview of voice notification system 1 ==

FIG. 2 is a diagram showing an outline of processing in the voice notification system 1. An outline of the operation of the voice notification system 1 will be described with reference to FIG.

First, in S1, the image acquisition device 21 of the speaker device 20 acquires the user's face image.

Next, in S2, the speaker device 20 transmits the acquired face image to the response server device 10. Then, the response server device 10 analyzes the user's facial expression and line of sight based on the face image. The response server device 10 identifies the user's emotions and the duration during which the user is viewing the speaker device 20 based on the analyzed facial expression and line of sight of the user.

Next, in S3, the response server device 10 identifies the response information based on the specified emotion and duration, and transmits the response information to the speaker device 20. As a result, the speaker device 20 can actively speak appropriately to the user based on the response information.

Next, in S4, the speaker device 20 transmits a voice corresponding to the response information to the user.
== Configuration of voice notification system 1 ==

Hereinafter, the functions provided by the response server device 10 and the speaker device 20 will be described.
<< Response server device 10 >>

The functional configuration of the response server device 10 will be described with reference to FIG. FIG. 3 is a diagram showing an example of the functional configuration of the response server device 10. As shown in FIG. 3, the response server device 10 includes a storage unit 11, an acquisition unit 12, an emotion analysis unit 13, a line-of-sight determination unit 14, a time calculation unit 15, a timing identification unit 16, a response content identification unit 17, and a facial expression icon identification unit. 18. It has the function of the transmission unit 19.

The storage unit 11 has, for example, a response information table 11a.

The response information table 11a is, for example, a table that stores response information indicating the content of the response to the user. As shown in FIG. 4, the data structure of the response information table 11a is based on data such as emotions, duration, timing, user attributes, response contents, and facial expression icons, using an appropriate item such as a response content ID as a primary key. It is a collection of records consisting of. Here, the emotion is a user's emotion estimated based on, for example, a face image. The duration is, for example, the time during which the user is continuously watching the speaker device 20. The timing is, for example, the timing at which the speaker device 20 speaks. The user attribute is, for example, the gender or job title of the user. The response content is, for example, the content output from the speaker device 20 as voice. The facial expression icon is, for example, an icon that imitates a person's facial expression displayed on the display unit of the speaker device 20. The contents of the response information table 11a are appropriately updated by, for example, the administrator of the response server device 10. The response information table 11a shows an example, and the content thereof is not limited.

Further, the storage unit 11 may store, for example, the trained models in the emotion analysis unit 13 and the time calculation unit 15, which will be described later. Further, the storage unit 11 may store training data for generating a trained model. The learning data is, for example, a face image used in the emotion analysis in the emotion analysis unit 13, and a user's emotion and line of sight that are associated with the image and serve as teacher data. These will be described later.

The acquisition unit 12 acquires the face image transmitted from the speaker device 20.

The emotion analysis unit 13 inputs the feature amount in the face image into the trained model, analyzes the user's emotion, and outputs it. Specifically, the sentiment analysis unit 13 extracts a region of interest such as an eye region, a mouth region, a nose region, or a cheek region, and extracts feature points from the attention region. The sentiment analysis unit 13 specifies the feature amount from the distance between the extracted feature points. Then, by inputting the feature amount into the trained model, the user's emotion is output. Here, the trained model is, for example, a convolutional neural network, and is trained using a set of a feature amount of a region of interest and an emotion (teacher data) corresponding to the feature amount as training data. As a result, the sentiment analysis unit 13 analyzes the facial expression that appears due to the contraction of the facial muscles that occurs in response to the deformation of facial elements such as the user's eyes, mouth, nose, and cheeks from the facial image, and the emotion expressed by the facial expression. Can be identified.

The emotion may be, for example, a so-called 27 types of basic emotions, or an emotion in which the basic emotions are combined. In the following, for convenience of explanation, the emotions specified by the sentiment analysis unit 13 will be limited to “fun”, “anger”, and “sorrow” as an example.

The line-of-sight determination unit 14 inputs a feature amount related to the user's eyes from the face image into the trained model, analyzes the user's line of sight, and determines whether or not the user is looking at the speaker device 20. Specifically, the line-of-sight determination unit 14 extracts, for example, a partial image including at least one eye of the face image, and extracts the feature amount indicated by the partial image. The line-of-sight determination unit 14 inputs the feature amount to the trained model and determines whether or not the user is viewing the speaker device 20. Here, the trained model is, for example, a convolutional neural network, and is trained using a set of a feature amount shown by the partial image and a line of sight (teacher data) corresponding to the feature amount as training data. is there.

It should be noted that the above-mentioned techniques for extracting features in the sentiment analysis unit 13 and the line-of-sight determination unit 14 and the trained model using the same are merely examples and are not limited to them, and are well known in place of them. It does not prevent you from using other technologies.

The time calculation unit 15 calculates the duration while the line-of-sight determination unit 14 determines that the user is looking at the speaker device 20. Specifically, the time calculation unit 15 acquires the determination result from the line-of-sight determination unit 14 at predetermined time intervals, and from the time when it is determined that the user is looking at the speaker device 20, the user uses the speaker device 20. The time until the time when it is determined not to see is calculated as the duration.

The timing specifying unit 16 outputs a predetermined response content to the speaker device 20 by voice based on at least one of, for example, the emotion analyzed by the emotion analysis unit 13 and the duration calculated by the time calculation unit 15. Specify the timing to make it. As a result, the speaker device 20 can output voice at an appropriate timing according to the user's emotions.

Further, the timing specifying unit 16 specifies the first timing for the first emotion of the user analyzed by the emotion analysis unit 13, and unlike the first emotion, the second emotion for the second emotion analyzed by the emotion analysis unit 13. The timing is specified to be longer than the first timing. Specifically, as shown in FIG. 4, for example, when the duration is "momentary" and the second emotion is "fun", the first timing of "immediately after" is specified, and the second emotion is "sorrow". In the case of "", the second timing of "around 3 seconds", which is longer than the first timing (immediately after) of "fun", is specified. When the second emotion is "anger", the second timing of "around 6 seconds", which is longer than the first timing (immediately after) of "fun", is specified. In this way, when the user has positive emotions such as "fun", the speaker device 20 outputs the response content by voice as soon as possible, and as the negative emotions such as "sorrow" and "anger" become stronger. , The speaker device 20 delays the audio output of the response content. As a result, the speaker device 20 can output voice at an appropriate timing according to the user's emotions, so that communication between the user and the speaker device 20 can be promoted.

The timing specifying unit 16 refers to the response information table 11a whose timing has been determined in advance, and is associated with the emotion analyzed by the emotion analysis unit 13 and the duration calculated by the time calculation unit 15. The timing and the second timing may be specified.

The response content specifying unit 17 refers to the response information table 11a and specifies the response content to be output to the speaker device 20 by voice, at least based on emotions and duration. As a result, the speaker device 20 can output voice content appropriate for the user's emotions, so that communication between the user and the speaker device 20 can be promoted. Further, in a situation where a plurality of people gather, such as a meeting or a party, it is possible to output an accurate response content according to the situation by voice, and it is possible to smoothly proceed with the meeting or the party.

The facial expression icon specifying unit 18 identifies the facial expression icon corresponding to the response content with reference to the response information table 11a. By displaying the facial expression icon specified by the facial expression icon specifying unit 18 on the display unit 22, it is possible to create an atmosphere that is close to the heart of the user who is looking at the speaker device 20, so that communication between the user and the speaker device 20 can be promoted. ..

It should be noted that the functions of the timing specifying unit 16, the response content specifying unit 17, and the facial expression icon specifying unit 18 described above are not prevented from being realized by using a well-known learned model. That is, in this case, each functional unit does not need to refer to the response information table 11a.

The transmission unit 19 transmits the response information including the information regarding the timing, the response content, and the facial expression icon to the speaker device 20.
<< Speaker device 20 >>

Next, the functional configuration of the speaker device 20 will be described. The speaker device 20 has the functions of the transmission / reception unit 20a and the display control unit 20b.

The transmission / reception unit 20a controls the transmission / reception of data in the speaker device 20. For example, the transmission / reception unit 20a transmits the face image acquired from the image acquisition device 21 to an external device such as the response server device 10. Further, the transmission / reception unit 20a receives the response information from an external device such as the response server device 10.

The display control unit 20b controls the display of the display unit 22 (see FIG. 1) included in the speaker device 20 or connected to the speaker device 20. The display control unit 20b causes the display unit 22 to display the facial expression icon included in the response information.
=== Processing procedure ===

FIG. 6 is a flow chart showing an example of processing of the response server device 10. An example of the processing executed by the response server device 10 will be described with reference to FIG.

First, in S100, the response server device 10 acquires a face image from the speaker device 20 (or the image acquisition device 21). Next, in S101, the response server device 10 inputs a face image into the trained model to identify the user's emotions. Next, in S102, the response server device 10 inputs a partial image including, for example, the user's eyes, among the facial images into the trained model, determines whether or not the user is looking at the speaker device 20, and is looking at it. When it is determined, the duration from that point to the point when it is determined that the user is not watching is calculated. Next, in S103, the response server device 10 specifies the timing of voice output from the speaker device 20 based on the emotion and the duration. Next, in S104, the response server device 10 identifies the response content based on at least emotions and duration. Next, in S105, the response server device 10 transmits the response information to the speaker device 20. As a result, the speaker device 20 can output an accurate response content to the user by voice at an appropriate timing.
== Hardware configuration of voice notification system 1 ==

An example of the hardware configuration when the response server device 10 and the speaker device 20 are realized by the computer 100 will be described with reference to FIG. 7. The function of each device can be realized by dividing it into a plurality of devices. Further, some hardware configurations in the speaker device 20 are as described above. The functions of the response server device 10 and the speaker device 20 are realized by the processor 101 reading and executing the computer program stored in the storage device 103 and controlling each configuration of the response server device 10 and the speaker device 20. Will be done.

FIG. 7 is a diagram showing an example of the hardware configuration of the computer. As shown in FIG. 7, the computer 100 includes a processor 101, a memory 102, a storage device 103, an input I / F unit 104, a data I / F unit 105, a communication I / F unit 106, and a display device. Includes 107.

The processor 101 is a control unit that controls various processes in the computer 100 by executing a program stored in the memory 102.

The memory 102 is, for example, a storage medium such as a RAM (Random Access Memory). The memory 102 temporarily stores the program code of the program executed by the processor 101 and the data required when the program is executed.

The storage device 103 is a non-volatile storage medium such as a hard disk drive (HDD) or a flash memory. The storage device 103 stores an operating system and various programs for realizing each of the above configurations.

The input I / F unit 104 is a device for receiving input from the user. Specific examples of the input I / F unit 104 include a keyboard, a mouse, a touch panel, various sensors, a wearable device, and the like. The input I / F unit 104 may be connected to the computer 100 via an interface such as USB (Universal Serial Bus).

The data I / F unit 105 is a device for inputting data from the outside of the computer 100. Specific examples of the data I / F unit 105 include a drive device for reading data stored in various storage media. It is also conceivable that the data I / F unit 105 is provided outside the computer 100. In that case, the data I / F unit 105 is connected to the computer 100 via an interface such as USB.

The communication I / F unit 106 is a device for performing data communication via the Internet N by wire or wirelessly with an external device of the computer 100. It is also conceivable that the communication I / F unit 106 is provided outside the computer 100. In that case, the communication I / F unit 106 is connected to the computer 100 via an interface such as USB.

The display device 107 (display unit 22) is a device for displaying various types of information. Specific examples of the display device 107 include a liquid crystal display, an organic EL (Electro-Lumisensence) display, and a display of a wearable device. The display device 107 may be provided outside the computer 100. In that case, the display device 107 is connected to the computer 100 via, for example, a display cable or the like. Further, when the touch panel is adopted as the input I / F unit 104, the display device 107 can be integrally configured with the input I / F unit 104.
=== Other embodiments ===
<< Response server device 210 (not shown) >>

In the above, the response server device 10 has been described so that the emotion analysis unit 13 analyzes the emotion in order to specify the response information, but instead, the emotion analysis unit 213 (not shown) analyzes the user's facial expression. You may. In this case, for example, the facial expressions of "laughing", "angry", or "sad" are analyzed, and the response content is specified based on these facial expressions and the duration. You may. That is, in the response server device 210 (not shown), the response information may be specified by associating the behavior of the user expressing emotions with the emotions of the user.
<< Speaker device 220 >>

In the above, the speaker device 20 transmits the user's face image to the response server device 10. After that, the response information is received from the response server device 10 to output voice to the user. That is, the speaker device 20 has been described so as to output voice to the user without analyzing and processing the face image, but the present invention is not limited to this.

For example, the speaker device 220 may respond to the user by identifying the user's emotion from the user's face image. FIG. 8 is a diagram showing the configuration of the speaker device 220 in another embodiment. As shown in FIG. 8, the speaker device 220 includes, for example, a storage unit 221, an acquisition unit 222, a speaker emotion analysis unit 223, a voice analysis unit 224, a timing identification unit 225, a speaker response content identification unit 226, and a display control unit 227. The output unit 228 is included. In the following, for convenience of explanation, the same ones as the speaker device 20 will be omitted, and different ones will be mainly described.

The storage unit 221 has, for example, a speaker response information table 221a.

The speaker response information table 221a is, for example, a table in which response contents to the user are stored. FIG. 9 is a diagram showing a speaker response information table 221a. As shown in FIG. 9, the data structure of the speaker response information table 221a is based on data such as emotions, voice volume, timing, response contents, and facial expression icons, using an appropriate item such as a response content ID as a primary key. It is a collection of records consisting of. Here, the emotion, timing, response content, and facial expression icon are the same as those in the response information table 11a described above, and thus description thereof will be omitted. The loudness of the voice is the loudness of the user's voice specified from the information about the user's voice acquired by the microphone of the speaker device 220. By referring to the speaker response information table 221a, the speaker device 220 can promptly make the user speak an aizuchi or the like, so that communication with the user can be promoted. It should be noted that it does not prevent the "duration" from being stored in place of the loudness of the voice or together with the loudness of the voice. The speaker response information table 221a shows an example, and the content thereof is not particularly limited.

Further, the storage unit 221 may store, for example, the trained model in the speaker emotion analysis unit 223, which will be described later. Further, the storage unit 221 may store the training data for generating the trained model.

Since the speaker emotion analysis unit 223 is the same as the emotion analysis unit 13 of the response server device 10, the description thereof will be omitted.

The voice analysis unit 224 analyzes the loudness of the user's voice based on the voice information indicating the user's voice acquired by the microphone of the speaker device 220. For example, the voice analysis unit 224 evaluates the voice information showing a loudness (voice volume) equal to or more than a predetermined threshold value as "large", and the voice information showing a loudness less than a predetermined threshold value is evaluated as "small". .. As a result, as will be described later, it is possible to specify an appropriate timing for outputting voice from the speaker device 220 according to, for example, the degree of anger of the user, so that communication with the user can be smoothly executed.

The voice analysis unit 224 is not limited to analyzing "large" and "small" according to the loudness (voice volume) of the voice. For example, the voice analysis unit 224 analyzes words contained in the voice and "large" and "small" emotions. May be distinguished. Specifically, for example, after converting voice information into text and extracting morphemes from the text, if a predetermined sentence contains a predetermined emotion term, an emotion vector is calculated based on the emotion term. .. Then, the typical emotional strengths "large" and "small" indicated by the emotion vector are specified. It is possible to use a well-known technique for identifying the "large" and "small" emotions by analyzing the words contained in the voice.

The timing identification unit causes the speaker device 220 to output a predetermined response content by voice based on at least one of the emotion analyzed by the speaker emotion analysis unit 223 and the voice information analyzed by the voice analysis unit 224. To identify. Specifically, as shown in FIG. 9, the timing for emotions is specified for the user analyzed by the speaker emotion analysis unit 223. For example, when the emotion is "fun", the timing of "immediately after" is specified, and when the emotion is "sorrow", the timing of "immediately after" is specified. Furthermore, when the emotion is "anger", the timing of "immediately after" is specified when the user's voice is "small", and the timing of "waiting" (not responding) when the user's voice is "loud". Identify. In this way, by specifying a timing simpler than the timing specified in the response server device 10, it is possible to realize a primary response to the user with hardware resources having lower performance than the response server device 10, so that the user and the speaker can be realized. Communication with the device 220 can be promoted.

Here, the “wait” shown in FIG. 9 means, for example, a process of waiting for response information to be transmitted from the response server device 210. For example, when it is determined that the user has a strong feeling of "anger", if the speaker device 220 immediately responds with an aizuchi or the like, it is presumed that the user has a feeling in the negative direction, so that the process of waiting is performed. Is.

The timing specifying unit 225 refers to the speaker response information table 221a whose timing has been determined in advance, and refers to the timing associated with the emotion analyzed by the speaker sentiment analysis unit 223 and the voice information analyzed by the voice analysis unit 224. May be specified.

The speaker response content specifying unit 226 refers to the speaker response information table 221a and specifies the response content to be output by the speaker device 220 based on at least emotion and voice information. The speaker response content specifying unit 226 specifies a response content such as "aizuchi", which is a simpler response content than the response server device 210. As a result, a primary response to the user can be realized with hardware resources having lower performance than the response server device 210.

The speaker response content specifying unit 226 refers to the speaker response information table 221a in which the response content is determined in advance, and refers to the emotions analyzed by the speaker sentiment analysis unit 223, the voice information analyzed by the voice analysis unit 224, and the timing. The response content associated with the timing specified by the specific unit 225 may be specified.

In this way, the voice notification system 200 first immediately issues a response such as an aizuchi to the user in the speaker device 220, and then outputs an appropriate message via the response server device 10 according to the user's emotions. Since it can be processed, it is possible to actively communicate appropriately with the user.

The output unit 228 outputs a voice signal indicating the response content to the speaker in order to cause the speaker to output voice according to the response content specified by the speaker response content specifying unit 226.

Further, the speaker device 220 may include a position information generation unit (not shown). The position information generation unit generates, for example, position information indicating the positions of each of a plurality of users. The position information generation unit may specify the position where the user exists based on, for example, a face image, or acquires the position information of the mobile terminal device from a beacon wirelessly connected to the mobile terminal device possessed by the user. You may specify the position where the user exists. The method of identifying the position where the user exists is not limited.

In this case, the facial expression icon display unit of the speaker device 220 has, for example, a facial expression icon in a predetermined area of the display device facing the position where a plurality of users exist, based on the position information generated by the position information generation unit. May be displayed. As a result, the speaker device 220 can produce an atmosphere that is close to the user's mind at the same time for a plurality of users, so that communication between the plurality of users and the speaker device 220 can be promoted. It is desirable that the display device is provided with a predetermined region on the circumference of the speaker device 220 over 360 degrees. Further, it is desirable that the image acquisition device 21 is provided in the speaker device 220 as a camera capable of photographing 360 degrees. As a result, the facial expression icon can be reliably displayed so as to correspond to the position where the user exists.
=== Summary ===

The voice notification system 1 according to the present embodiment is based on the acquisition unit 12 that acquires a face image showing the user's face from the image acquisition device 21 connected to the speaker device 20 having a speaker, and the user's face image. Based on the sentiment analysis unit 13 that analyzes emotions, the line-of-sight determination unit 14 that determines whether or not the user is looking at the speaker device 20 based on the face image, and the determination result in the line-of-sight determination unit 14, the user uses the speaker. The time calculation unit 15 that calculates the duration of viewing the device 20, the response content specifying unit 17 that specifies a predetermined response content based on the user's emotion and the duration, and the response content specifying unit 17 The speaker device 20 includes a transmission unit 19 that transmits voice information indicating the predetermined response content to the speaker device 20 so that the speaker device 20 can output the sound according to the specified predetermined response content. According to the present embodiment, it is possible to encourage the user to speak by actively outputting voice to the user based on the user's emotion and line of sight.

The voice notification system 1 according to the present embodiment further includes a timing specifying unit 16 that specifies a timing for causing the speaker device 20 to output a predetermined response content by voice based on the user's emotions analyzed by the sentiment analysis unit 13. The transmission unit 19 transmits voice information to the speaker device 20 so that the speaker device 20 can output voice at the timing specified by the timing specifying unit 16. According to the present embodiment, since the speaker device 20 can output voice at an appropriate timing according to the emotion of the user, communication between the user and the speaker device 20 can be promoted.

The timing specifying unit 16 of the voice notification system 1 according to the present embodiment specifies the first timing for the user's first emotion analyzed by the emotion analysis unit 13, and is analyzed by the emotion analysis unit 13 unlike the first emotion. The second timing for the second emotion to be performed is specified to be longer than the first timing. According to the present embodiment, since the speaker device 20 can output voice at a more appropriate timing according to the emotion of the user, communication between the user and the speaker device 20 can be promoted.

The voice notification system 1 according to the present embodiment corresponds to the response content by referring to the response information table 11a in order to display the facial expression icon (icon) imitating the facial expression on the display unit 22 in the speaker device 20 having the display unit. A facial expression icon specifying unit 18 for specifying the facial expression icon (icon) to be used is further provided, and the transmitting unit 19 transmits the facial expression icon (icon) specified by the facial expression icon specifying unit 18 to the speaker device 20. According to the present embodiment, it is possible to create an atmosphere that is close to the heart of the user who is looking at the speaker device 20, so that communication between the user and the speaker device 20 can be promoted.

The voice notification system 1 according to the present embodiment further includes a speaker device 220, and the speaker device 220 has a speaker sentiment analysis unit 223 that analyzes the user's emotions based on a face image, and is analyzed by the user sentiment analysis unit. Based on the sentiment of the user, the speaker response content specifying unit 226 that specifies a predetermined response content and the speaker respond content specifying unit 226 are specified to output audio to the speaker. It is provided with an output unit 228 that outputs an audio signal indicating the response content of the above to the speaker. According to this embodiment, the system design can be efficiently carried out by configuring the system as one system including the response server device 210 and the speaker device 220.

The speaker device 220 of the voice notification system 1 according to the present embodiment has a cylindrical shape, and is analyzed by at least one of the emotion analysis unit 13 and the speaker emotion analysis unit 223 on the side peripheral portion at the upper end portion of the cylindrical shape. It has a display unit 22 for displaying a facial expression icon. According to the present embodiment, the facial expression icon can be reliably displayed so as to correspond to the position where the user exists, so that the communication between the user and the speaker device 220 can be promoted more reliably.

It should be noted that the above-described embodiment is for facilitating the understanding of the present invention, and is not for limiting and interpreting the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention also includes an equivalent thereof.

1,200 ... Voice notification system, 10 ... Response server device, 12 ... Acquisition unit, 13 ... Sentiment analysis unit, 14 ... Line-of-sight determination unit, 15 ... Time calculation unit, 16 ... Timing identification unit, 17 ... Response content identification unit, 18 ... Facial expression icon identification unit, 19 ... Transmission unit, 20, 220 ... Speaker device, 21 ... Image acquisition device, 223 ... Speaker sentiment analysis unit, 226 ... Speaker response content identification unit, 228 ... Output unit

Claims (9)

An acquisition unit that acquires a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker.
An emotion analysis unit that analyzes the user's emotions based on the face image,
A line-of-sight determination unit that determines whether or not the user is looking at the speaker device based on the face image.
A time calculation unit that calculates the duration of the user watching the speaker device based on the determination result in the line-of-sight determination unit.
A response content specifying unit that specifies a predetermined response content based on the user's emotion and the duration.
A transmission unit that transmits voice information indicating the predetermined response content to the speaker device in order to cause the speaker device to output voice in accordance with the predetermined response content specified by the response content specifying unit.
Information processing system equipped with. A timing specifying unit for specifying a timing for outputting the predetermined response content to the speaker device by voice based on the emotion of the user analyzed by the emotion analysis unit is further provided.
The information processing system according to claim 1, wherein the transmission unit transmits the voice information to the speaker device so that the speaker device can output voice at the timing specified by the timing specifying unit. The timing specifying unit specifies the first timing for the first emotion of the user analyzed by the sentiment analysis unit, and unlike the first emotion, the second sentiment for the second emotion analyzed by the sentiment analysis unit. The information processing system according to claim 2, wherein the timing is specified to be longer than the first timing. In the speaker device having a display unit, in order to display an icon imitating a human facial expression on the display unit, a facial expression icon specifying unit that specifies the icon corresponding to the response content is further provided with reference to predetermined information. Prepare,
The information processing system according to any one of claims 1 to 3, wherein the transmitting unit transmits the icon specified by the facial expression icon specifying unit to the speaker device. The acquisition unit acquires the face image including the position information of each of the plurality of users, and obtains the face image.
The transmitting unit causes each of the plurality of users to display the icon specified by the facial expression icon specifying unit in a predetermined area of the display unit corresponding to the position information of each of the plurality of users. The information processing system according to claim 4, which is transmitted to the speaker device. Further equipped with the speaker device
The speaker device is
It has a speaker sentiment analysis unit that analyzes the user's emotions based on the face image.
A speaker response content specifying unit that specifies a predetermined response content based on the user's emotion analyzed by the speaker emotion analysis unit, and a speaker response content specifying unit.
An output unit that outputs a voice signal indicating the predetermined response content to the speaker in order to output the voice to the speaker in accordance with the predetermined response content specified by the speaker response content specifying unit.
The information processing system according to any one of claims 1 to 5. The speaker device has a cylindrical shape, and displays an icon imitating a human facial expression analyzed by at least one of the sentiment analysis unit and the speaker sentiment analysis unit on a side peripheral portion at an upper end portion of the cylindrical shape. The information processing system according to claim 6, which has a display unit. The computer
Acquiring a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker,
Analyzing the user's emotions based on the face image,
Determining whether or not the user is looking at the speaker device based on the face image.
Based on the determination result, the duration of viewing the speaker device by the user is calculated.
Identifying a predetermined response content based on the user's emotion and the duration.
To transmit the voice information indicating the predetermined response content to the speaker device in order to output the voice to the speaker device according to the specified predetermined response content.
Information processing method to execute. On the computer
Acquiring a face image showing a user's face from an image acquisition device connected to a speaker device having a speaker,
To analyze the user's emotions based on the face image,
To determine whether or not the user is looking at the speaker device based on the face image.
To calculate the duration of viewing the speaker device by the user based on the determination result.
To specify a predetermined response content based on the user's emotion and the duration.
In order to have the speaker device output voice according to the specified predetermined response content, the speaker device is to transmit voice information indicating the predetermined response content.
A program that executes.

Images