JP2021111239A - Providing system, providing method, providing device, and computer program - Google Patents

Abstract

To provide a providing system for assisting smooth communication among users.SOLUTION: A providing system comprises: a video obtainment section for obtaining a video of a second user which is obtainable by photographing the second user listening to voice of a first user representing a speaker; a determination section for, based on the obtained video of the second user, determining at least one of an emotion or a concentration level of the second user; and a providing section for providing a determination result by the determination section to the first user.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to a providing system, a providing method, a providing device, and a computer program.

Conventionally, an electronic conference system has been proposed in which users communicate with each other via a network (see, for example, Patent Document 1).

JP-A-2018-139652

In an electronic conference system as described in Patent Document 1, compared to a normal face-to-face conference, the faces of the dialogue partners appear smaller on the screen, and it is difficult for the eyes of each other to meet each other. It is difficult to communicate with each other. Therefore, there is a problem that the productivity of the conference is lowered.

In addition, even if you have an opinion against the speaker, it may be difficult to give an opinion to your superiors, and it may not be a constructive discussion.

This is thought to be due to the fact that it is difficult for the listener's emotions to be conveyed to the speaker in conversations over the network.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide a providing system, a providing method, a providing device, and a computer program that support smooth communication between users.

The providing system according to one aspect of the present disclosure includes an image acquisition unit that acquires an image of the second user, which is obtained by photographing a second user who listens to the voice of the first user who is a speaker. A determination unit that determines at least one of the emotion and concentration of the second user based on the video of the second user, and a providing unit that provides the determination result by the determination unit to the first user. ..

The providing method according to another aspect of the present disclosure includes a step of acquiring the video of the second user, which is obtained by photographing the second user who views the voice and video of the first user who is the speaker, and the acquisition. It includes a step of determining at least one of the emotion and concentration of the second user based on the video of the second user, and a step of providing the determination result in the determination step to the first user. ..

The providing device according to another aspect of the present disclosure includes a video acquisition unit that acquires the video of the second user, which is obtained by photographing the second user who views the voice and video of the first user who is the speaker. A determination unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user, and a providing unit that provides the determination result by the determination unit to the first user. And.

The computer program according to another aspect of the present disclosure is a video for acquiring the video of the second user, which is obtained by photographing the computer with a second user who watches the voice and video of the first user who is a speaker. The acquisition unit, a determination unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user, and the determination result by the determination unit are provided to the first user. To function as a provider.

Needless to say, computer programs can be distributed via computer-readable non-temporary recording media such as CD-ROMs (Compact Disc-Read Only Memory) and communication networks such as the Internet. Further, the present disclosure can be realized as a semiconductor integrated circuit that realizes a part or all of the providing device, or can be realized as a providing system including the providing device.

According to the present disclosure, it is possible to support smooth communication between users.

FIG. 1 is a diagram showing a schematic configuration of a provision system according to the first embodiment of the present disclosure. FIG. 2 is a block diagram showing a functional configuration of the first device according to the first embodiment of the present disclosure. FIG. 3 is a diagram showing an example of an image displayed on the display. FIG. 4 is a block diagram showing a functional configuration of the second device according to the first embodiment of the present disclosure. FIG. 5 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the first user from the first device to the second device by the providing system according to the first embodiment of the present disclosure. FIG. 6 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the second user from the second device to the first device by the providing system according to the first embodiment of the present disclosure. FIG. 7 is a block diagram showing a functional configuration of the first device according to the second embodiment of the present disclosure. FIG. 8 is a block diagram showing a functional configuration of the second device according to the second embodiment of the present disclosure. FIG. 9 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the first user from the first device to the second device by the providing system according to the second embodiment of the present disclosure. FIG. 10 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the second user from the second device to the first device by the providing system according to the second embodiment of the present disclosure. FIG. 11 is a block diagram showing a functional configuration of the first device according to the third embodiment of the present disclosure. FIG. 12 is a block diagram showing a functional configuration of the second device according to the third embodiment of the present disclosure. FIG. 13 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the first user from the first device to the second device by the providing system according to the third embodiment of the present disclosure. FIG. 14 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the second user from the second device to the first device by the providing system according to the third embodiment of the present disclosure.

[Summary of Embodiments of the present disclosure]
First, an outline of the embodiments of the present disclosure will be listed and described.
(1) The providing system according to the embodiment of the present disclosure is an image acquisition unit that acquires an image of the second user, which is obtained by photographing a second user who listens to the voice of the first user who is a speaker. A determination unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user, and a determination result by the determination unit are provided to the first user. It has a part.

According to this configuration, the determination result of at least one of the emotion and the concentration of the second user who is the listener of the utterance content of the first user is provided to the first user. Therefore, the first user can know what kind of emotion the second user has with respect to the content of his / her utterance, or whether the second user concentrates on listening to the story. On the other hand, the first user asks a question to the second user who has a negative feeling about the utterance content, or changes the topic when the second user is not concentrated. It can be performed. This makes it possible to support smooth communication between users.

(2) Preferably, the providing system further includes an audio acquisition unit that acquires the audio of the second user, and the determination unit is based on the acquired video and audio of the second user. 2 Judge at least one of the user's emotions and concentration.

According to this configuration, at least one of the emotion and the degree of concentration of the second user can be determined in consideration of the voice of the second user. Therefore, the emotion or concentration of the second user can be determined with higher accuracy than in the case of determining at least one of the emotion and the concentration of the second user using only the video of the second user.

(3) More preferably, the provided system includes a first device and a second device connected to each other via a network, and the first device acquires audio and video of the first user. An acquisition unit, a first transmission unit that transmits the acquired audio and video of the first user to the second device, and a first reception unit that receives the audio and video of the second user from the second device. The second device includes a first output unit as the providing unit that outputs the received audio and video of the second user and the determination result by the determination unit. A second acquisition unit as the audio acquisition unit and the video acquisition unit that acquires audio and video, a second transmission unit that transmits the acquired audio and video of the second user to the first device, and the above. It includes a second receiving unit that receives the audio and video of the first user from the first device, and a second output unit that outputs the received audio and video of the first user.

According to this configuration, it is possible to have a dialogue between the first user and the second user over the network and provide the first user with the determination result of at least one of the emotion and the concentration of the second user. Therefore, for example, in an electronic conference system in which the first user is the facilitator of the conference, the first user grasps the emotion or concentration of the second user and asks the second user for opinions as appropriate. You can proceed. This makes the discussion constructive and enables highly productive meetings. As in the case of the second user, the determination unit determines at least one of the emotion and concentration of the first user from the audio and video of the first user, and the second output unit of the second device is the first. The judgment result of at least one of the user's emotion and concentration may be output. As a result, the first user and the second user can mutually grasp the emotions or the degree of concentration of the other party.

(4) Further, the determination unit is provided in the first device, and at least one of the emotion and concentration of the second user is based on the audio and video of the second user received by the first receiving unit. May be judged.

According to this configuration, the first device can determine at least one of the emotion and concentration of the second user based on the audio and video of the second user transmitted from the second device. Therefore, the first device can accurately synchronize the audio and video of the second user with the emotion or concentration of the second user. As a result, the audio and video of the second user can be accurately associated with the emotion or concentration of the second user and provided to the first user.

(5) Further, the provided system includes a first device and a second device connected to each other via a network, and the first device includes a first acquisition unit for acquiring the voice of the first user. The first transmitting unit that transmits the acquired voice of the first user to the second device, the first receiving unit that receives the acquired voice of the second user from the second device, and the second that has been received. The second device acquires the voice and video of the second user, including the first output unit as the providing unit, which outputs the voice of the user and the judgment result by the determination unit. A second acquisition unit as a unit and the video acquisition unit, a second transmission unit that transmits the acquired voice of the second user to the first device, and the voice of the first user from the first device. The second receiving unit for receiving and the second output unit for outputting the received voice of the first user are included, and the determining unit is provided in the second device and acquired by the second acquisition unit. Based on the audio and video of the second user, at least one of the emotion and the degree of concentration of the second user is determined, and the second transmission unit further transmits the determination result by the determination unit and receives the first reception. The unit may further receive the determination result by the determination unit, and the first output unit may output the determination result by the determination unit received by the first receiving unit.

According to this configuration, the second device determines at least one of the emotions and the concentration of the second user together with the voice of the second user without transmitting the image of the second user from the second device to the first device. It can be transmitted to the first device. Therefore, it is possible to transmit the determination result of the emotion or the degree of concentration of the second user to the first device while reducing the transmission data from the second device to the first device. Further, it is not necessary to transmit the video from the second device to the first device. Therefore, for example, the first output unit can output the avatar of the second user having a facial expression based on the emotion of the second user to the display device instead of the video of the second user. Thereby, the privacy of the second user can also be protected.

(6) Further, the second device may further include a speech promotion unit that prompts the second user to speak based on the determination result of at least one of the emotion and the concentration of the second user. ..

According to this configuration, for example, when the second user has a negative feeling toward the utterance content of the first user, or when the second user is not concentrated, the second user can be prompted to speak. can. This makes the discussion meaningful and supports smooth communication between users.

(7) Further, the providing system further includes a calculation unit that calculates the degree of contribution of the second user in the dialogue between the first user and the second user based on the judgment result by the judgment unit. May be good.

According to this configuration, the degree of contribution in the dialogue of the second user can be calculated based on the judgment result of at least one of the emotion and the degree of concentration of the second user. For example, it is possible to calculate the contribution of the second user who was concentrating on the dialogue high, or to calculate the contribution of the second user who has low feelings of contempt and disgust and high feelings of joy and surprise. Is.

(8) Further, the judgment unit further corrects the judgment result of at least one of the emotion and concentration of the second user based on the history of the judgment result of at least one of the emotion and concentration of the second user. You may.

According to this configuration, when the judgment result of emotion or concentration is expressed by a score, each score of the second user in which the change in emotional ups and downs and concentration is relatively small, and the change in emotional ups and downs and concentration are Each score of the relatively large second user can be normalized or standardized. This makes it possible to accurately compare emotions or concentration levels among the second users.

(9) The providing method according to another embodiment of the present disclosure acquires the video of the second user obtained by photographing the second user who views the voice and video of the first user who is the speaker. The first user is provided with a step of determining at least one of the emotion and concentration of the second user based on the step and the acquired video of the second user, and a determination result in the determination step. Including steps.

This configuration includes steps corresponding to the characteristic processing units included in the above-mentioned providing system. Therefore, according to this configuration, the same operation and effect as the above-mentioned providing system can be obtained.

(10) The providing device according to another embodiment of the present disclosure acquires the video of the second user obtained by photographing the second user who views the voice and video of the first user who is the speaker. Based on the video acquisition unit and the acquired video of the second user, a determination unit that determines at least one of the emotion and concentration of the second user, and a determination result by the determination unit are transmitted to the first user. It has a providing unit to provide.

According to this configuration, the judgment result of at least one of the emotion and the concentration of the second user who is the listener is provided to the first user in the utterance content of the first user. Therefore, the first user can know what kind of emotion the second user has with respect to the content of his / her utterance, or whether the second user concentrates on listening to the story. On the other hand, the second user asks a question to the second user who has a negative feeling about the utterance content, or changes the topic when the second user is not concentrated. It can be performed. This makes it possible to support smooth communication between users.

(11) The computer program according to another embodiment of the present disclosure is obtained by photographing a second user who watches the voice and video of the first user who is a speaker, and the video of the second user. A determination unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user, and a determination result by the determination unit. 1 Make it function as a provider to provide to users.

According to this configuration, the computer can function as the above-mentioned providing device. Therefore, the same operation and effect as the above-mentioned providing device can be obtained.

[Details of Embodiments of the present disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be noted that all of the embodiments described below show a specific example of the present disclosure. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, step order, and the like shown in the following embodiments are examples, and do not limit the present disclosure. Further, among the components in the following embodiments, the components not described in the independent claims are components that can be arbitrarily added. Further, each figure is a schematic view and is not necessarily exactly illustrated.

Further, the same components are designated by the same reference numerals. Since their functions and names are the same, their description will be omitted as appropriate.

<Embodiment 1>
[Overall configuration of the provided system]
FIG. 1 is a diagram showing a schematic configuration of a provision system according to the first embodiment of the present disclosure. The providing system 1 includes a first device 2, a second device 4, and an emotion / concentration database (hereinafter, referred to as “emotion / concentration DB”) 5 connected to each other via a network 3.

The first device 2 acquires, for example, video data (hereinafter, referred to as “video”) and audio data (hereinafter, referred to as “audio”) of one or a plurality of first users in the first base. At least one of the emotion and concentration of the first user is determined for each first user based on the video and audio of the first user. The first device 2 provides the second device 4 by transmitting the video and audio of the first user and at least one of the emotion and concentration of the first user to the second device 4. The first device 2 is installed, for example, at a first base, which is a business establishment of a company.

The second device 4 receives the data transmitted by the first device 2 from the first device 2. The second device 4 displays the received video of the first user on the display, and outputs the received voice of the first user from the speaker. In addition, the second device 4 displays at least one of the received emotions and concentration of the first user on the display. The display and the speaker may be built in the second device 4, or may be connected by wire or wirelessly.

The second device 4 acquires, for example, the video and audio of one or a plurality of second users in the second base, and based on the acquired video and audio of the second user, for each second user, the second user Judge at least one of emotions and concentration. The second device 4 provides the first device 2 by transmitting the video and audio of the second user and at least one of the emotion and concentration of the second user to the first device 2. The second device 4 is installed, for example, at a second base, which is another business establishment of the above-mentioned company.

The first device 2 receives the data transmitted by the second device 4 from the second device 4. The first device 2 displays the received video of the second user on the display, and outputs the received voice of the second user from the speaker. In addition, the first device 2 displays at least one of the received emotions and concentration of the second user on the display. The display and the speaker may be built in the first device 2 or may be externally connected.

The emotion / concentration DB 5 stores the history of the determination result of the emotion or the degree of concentration for each user determined by each of the first device 2 and the second device 4.

[Structure of First Device 2]
FIG. 2 is a block diagram showing a functional configuration of the first device 2 according to the first embodiment of the present disclosure.

The first device 2 includes a video acquisition unit 21, a video coding unit 22, a video analysis unit 23, an audio acquisition unit 24, an audio coding unit 25, an audio analysis unit 26, a multiplexing unit 27, and the like. The emotion / concentration determination unit 28, the first transmission unit 29, the first reception unit 30, the separation unit 31, the video decoding unit 32, the audio decoding unit 33, and the emotion / concentration processing unit 34. , Display / output unit 35.

The first device 2 is realized by a general computer equipped with a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), a communication interface, an input / output interface, and the like. be able to. For example, by expanding the computer program recorded in the HDD on the RAM and executing it on the CPU, each processing unit 21 to 35 is functionally realized. However, a part or all of the processing units 21 to 35 may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array).

The image acquisition unit 21 functions as a first acquisition unit, and acquires images taken by the camera from a camera built in the first device 2 or connected by wire or wirelessly. The video acquisition unit 21 outputs the acquired video to the video coding unit 22 and the video analysis unit 23.

The video coding unit 22 receives the video from the video acquisition unit 21 and encodes the video according to a predetermined coding method. For example, when the video is a 4K video or an 8K video, the video coding unit 22 has H. Video is encoded according to 265 / HEVC (High Efficiency Video Coding). The video coding unit 22 outputs the coded video to the multiplexing unit 27.

The video analysis unit 23 receives the video from the video acquisition unit 21 and identifies the user (hereinafter, referred to as “first user”) shown in the video and the position of the first user in the video. For example, the image analysis unit 23 identifies the first user in the image by using the face recognition algorithm. However, the information for identifying the first user (for example, the user name) and the position of the first user in the video may be specified by the operator of the first device 2 by external input.

The video analysis unit 23 determines the emotion and concentration of the first user by analyzing the video. That is, the video analysis unit 23 calculates an emotion score that quantifies the degree of emotion for each type of emotion based on the input video. The types of emotions analyzed by the image analysis unit 23 are, for example, anger, contempt, disgust, surprise, fear, joy, sadness, surprise, happiness, and discomfort. The image analysis unit 23 calculates the emotion score of the first user by inputting an image into the classifier using a classifier provided for each type of emotion.

As the discriminator, for example, a multi-layer neural network that receives an image as an input and outputs an emotion score can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's video and emotion score as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

Further, the video analysis unit 23 calculates the degree of concentration in which the degree of concentration of the user is quantified based on the input video. That is, the video analysis unit 23 calculates the degree of concentration of the first user by inputting the video to the classifier.

As the discriminator, for example, a multi-layer neural network that receives an image as an input and outputs a degree of concentration can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's image and concentration as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

When a plurality of first users are included in the video, the video analysis unit 23 calculates the emotion score and the degree of concentration for each first user.

The video analysis unit 23 determines the emotion / concentration determination unit for the information for identifying the first user, the position in the video of the first user, and the calculated emotion score and concentration for each type of emotion of the first user. Output to 28.

The voice acquisition unit 24 functions as a first acquisition unit, and acquires the voice of the first user from a microphone built in the first device 2 or connected by wire or wirelessly. The voice acquisition unit 24 outputs the acquired voice of the first user to the voice coding unit 25 and the voice analysis unit 26.

The voice coding unit 25 receives the voice from the voice acquisition unit 24 and encodes the voice according to a predetermined coding method. For example, the voice coding unit 25 encodes voice according to MPEG-4 AAC. The voice coding unit 25 outputs the coded voice to the multiplexing unit 27.

The voice analysis unit 26 receives a voice from the voice acquisition unit 24 and identifies the first user who is emitting the voice. The first user may be specified, for example, by identifying the speaker based on the voice data registered in advance. The voice analysis unit 26 analyzes the voiceprint of the speaker from the voice, and identifies the speaker by using an identification method such as a hidden Markov model, a neural network, or a decision tree. However, when the first user speaks, the information for identifying the first user may be externally input by the first user or the operator of the first device 2.

The voice analysis unit 26 determines the emotion and concentration of the first user by analyzing the voice. That is, the voice analysis unit 26 calculates the emotion score for each type of emotion based on the input voice. The type of emotion to be analyzed by the voice analysis unit 26 is the same as the type of emotion to be analyzed by the video analysis unit 23. The voice analysis unit 26 calculates an emotion score by inputting voice into the classifier using a classifier provided for each type of emotion.

As the discriminator, for example, a multi-layer neural network that receives voice as an input and outputs an emotion score can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's voice and emotion score as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

In addition, the voice analysis unit 26 calculates the degree of concentration of the first user based on the input voice.

As the discriminator, for example, a multi-layer neural network that receives voice as an input and outputs a degree of concentration can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's voice and concentration as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

When a plurality of first users are included in the voice, the voice analysis unit 26 calculates the emotion score and the degree of concentration for each first user.

The voice analysis unit 26 outputs the information for identifying the first user, the calculated emotion score for each type of emotion of the first user, and the degree of concentration to the emotion / concentration determination unit 28.

The multiplexing unit 27 receives the encoded video and the encoded audio from the video coding unit 22 and the audio coding unit 25, respectively, and multiplexes the encoded video and the encoded audio to obtain the multiplexed data. To generate. For example, the multiplexing unit 27 performs multiplexing according to MPEG-H MMT (MPEG Media Transport). The multiplexing unit 27 outputs the generated multiplexed data to the first transmission unit 29.

The emotion / concentration determination unit 28 includes information for identifying the first user from the video analysis unit 23, a position in the video of the first user, an emotion score for each type of emotion of the first user, and the emotion score of the first user. Receive a degree of concentration. Further, the emotion / concentration determination unit 28 receives information for identifying the first user from the voice analysis unit 26, an emotion score for each type of emotion of the first user, and a degree of concentration of the first user.

The emotion / concentration determination unit 28 determines the emotion of the first user based on the emotion score for each type of emotion of the first user received from the video analysis unit 23 and the voice analysis unit 26. For example, the emotion / concentration determination unit 28 has an emotion score corresponding to the type of the first user received from the video analysis unit 23 and the type of the first user received from the voice analysis unit 26 for each type of emotion. The emotion score of the type is calculated by simply adding or weighting the emotion score corresponding to. The weight of the weighting addition may be set in advance or may be changed according to the two emotion scores.

The method of calculating the emotion score is not limited to this. For example, the emotion / concentration determination unit 28 may calculate the emotion score of the first user by using the discriminator provided for each type of emotion. Specifically, the emotion / concentration determination unit 28 receives the emotion score corresponding to the type of the first user received from the video analysis unit 23 and the voice analysis unit 26 to the discriminator of each emotion type. By inputting the emotion score corresponding to the type of one user, the emotion score of the first user for the type of emotion is calculated.

As the discriminator, for example, a multi-layer neural network that receives an emotion score calculated based on video and an emotion score calculated based on audio as input and outputs an emotion score can be used. In this multi-layer neural network, for example, the emotion score calculated based on the video and the emotion score calculated based on the voice and the emotion score determined by the designer of the neural network are used as teacher data, and the parameters of the multi-layer neural network are deep-layered. It is constructed by performing machine learning such as learning. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

Further, the emotion / concentration determination unit 28 determines the concentration level of the first user based on the concentration level of each type of emotion of the first user received from the video analysis unit 23 and the voice analysis unit 26. For example, the emotion / concentration determination unit 28 simply adds or weights the concentration of the first user received from the video analysis unit 23 and the concentration of the first user received from the voice analysis unit 26. , Calculate the concentration of the first user. The weight of the weighting addition may be set in advance or may be changed according to the two degrees of concentration.

The method of calculating the degree of concentration is not limited to this. For example, the emotion / concentration determination unit 28 may calculate the concentration level of the first user by using the discriminator. Specifically, the emotion / concentration determination unit 28 inputs the concentration level of the first user received from the video analysis unit 23 and the concentration level of the first user received from the voice analysis unit 26 into the classifier. To calculate the concentration of the first user.

As the discriminator, for example, a multi-layer neural network that receives the concentration level calculated based on the video and the concentration level calculated based on the audio as inputs and outputs the concentration level can be used. In this multi-layer neural network, for example, the concentration calculated based on the video and the concentration calculated based on the sound and the concentration determined by the designer of the neural network are used as training data, and the parameters of the multi-layer neural network are deep-layered. It is constructed by performing machine learning such as learning. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

When there are a plurality of first users, the emotion / concentration determination unit 28 calculates the emotion score and the degree of concentration for each first user.

The emotion / concentration determination unit 28 associates the calculated emotion score of the first user for each type of emotion with the degree of concentration of the first user in association with the identifier of the first user and the calculated time, and the emotion / concentration DB5. Write to. It should be noted that the identifier of the first user includes information for identifying the first user (for example, a user name) and position information in the video of the first user.

The emotion / concentration determination unit 28 may correct the calculated emotion score of the first user based on the history of the emotion score of the first user. For example, the emotion / concentration determination unit 28 reads the emotion score of the first user for a certain period in the past from the emotion / concentration DB 5 for each type of emotion, and based on the read emotion score, the standard deviation of the emotion score and the standard deviation of the emotion score. Calculate the average. The emotion / concentration determination unit 28 standardizes the calculated emotion score of the first user for each type of emotion using the standard deviation and average of the calculated emotion score according to the following equation 1. Thereby, the emotion score can be standardized among the first users.

Standardized emotion score = (calculated emotion score-average emotion score)
/ Standard deviation of emotion score ... (Equation 1)

Further, the emotion / concentration determination unit 28 may normalize the emotion score instead of standardizing the emotion score. For example, the emotion / concentration determination unit 28 reads the emotion score of the first user for a certain period in the past from the emotion / concentration DB5 for each type of emotion, and based on the read emotion score, the maximum value of the emotion score and Calculate the minimum value. The emotion / concentration determination unit 28 normalizes the calculated emotion score of the first user for each type of emotion using the calculated maximum and minimum values of the emotion score according to the following equation 2. This makes it possible to normalize the emotional score among the first users.

Normalized emotion score = (emotion score-minimum emotion score)
/ (Maximum emotional score-Minimum emotional score) ... (Equation 2)

In addition, the emotion / concentration determination unit 28 may correct the calculated concentration level of the first user based on the history of the concentration level of the first user. For example, the emotion / concentration determination unit 28 reads the concentration of the first user for a certain period in the past from the emotion / concentration DB 5, and calculates the standard deviation and the average of the concentration based on the read concentration. The emotion / concentration determination unit 28 standardizes the calculated concentration of the first user according to the following equation 3 using the standard deviation and the average of the calculated concentration. As a result, the degree of concentration can be standardized among the first users.

Standardized concentration = (calculated concentration-average concentration)
/ Standard deviation of concentration ratio ... (Equation 3)

Further, the emotion / concentration determination unit 28 may normalize the concentration level instead of standardizing the concentration level. For example, the emotion / concentration determination unit 28 reads the concentration of the first user for a certain period in the past from the emotion / concentration DB 5, and calculates the maximum value and the minimum value of the concentration based on the read concentration. The emotion / concentration determination unit 28 normalizes the calculated concentration of the first user for each type of emotion using the calculated maximum and minimum values of the concentration according to the following equation 4. As a result, the degree of concentration can be normalized among the first users.

Normalized concentration = (concentration-minimum concentration)
/ (Maximum concentration-Minimum concentration) ... (Equation 4)

The emotion / concentration determination unit 28 sends the calculated emotion score of the first user for each type of emotion and the degree of concentration of the first user to the first transmission unit 29 together with the identifier of the first user and the calculated time. Output.

The first transmission unit 29 receives the multiplexed data from the multiplexing unit 27, and the emotion / concentration determination unit 28 adds the identifier of the first user and the calculated time to the emotion score of the first user for each type of emotion. , Receives the concentration of the first user. The first transmission unit 29 transmits these received data to the second device 4.

The first receiving unit 30 includes multiplexed data in which encoded video and encoded audio are multiplexed from the second device 4, emotion scores of the second user for each type of emotion, and concentration of the second user. And receive. An identifier of the second user and a calculation time of the emotion score and the degree of concentration of the second user are added to these data. The first receiving unit 30 outputs a set of these data received from the second device 4 to the separating unit 31. It is assumed that the identifier of the second user includes information for identifying the second user (for example, a user name) and position information in the video of the second user.

The separation unit 31 receives the data set from the first receiving unit 30 and separates the data set. That is, the separation unit 31 separates the multiplexed data included in the data set into the encoded video and the encoded audio, and separates the separated encoded video and the encoded audio into the video decoding unit 32 and the audio decoding. Output to each unit 33. Further, the separation unit 31 separates the emotion score and the degree of concentration for each type of emotion to which the second user's identifier and the calculated time are added from the data set, and separates these separated data into the emotion / concentration processing unit 34. Output to.

The video decoding unit 32 receives the encoded video from the separation unit 31 and decodes the video according to a predetermined decoding method. The decoding method is a method corresponding to the video coding method in the second device 4. For example, when the video is a 4K video or an 8K video, the second device 4 is H.I. When the video is encoded according to 265 / HEVC, the video decoding unit 32 uses H.I. The encoded video is decoded according to 265 / HEVC. The video decoding unit 32 outputs the decoded video to the emotion / concentration processing unit 34 and the display / output unit 35.

The voice decoding unit 33 receives the encoded voice from the separation unit 31 and decodes the voice according to a predetermined decoding method. The decoding method is a method corresponding to the voice coding method in the second device 4. For example, when the second device 4 encodes the voice according to the MPEG-4 AAC, the voice decoding unit 33 decodes the voice according to the MPEG-4 AAC. The voice decoding unit 33 outputs the decoded voice to the display / output unit 35.

The emotion / concentration processing unit 34 receives the identifier and measurement time of the second user from the separation unit 31 and the emotion score and concentration level for each type of emotion of the second user. Further, the emotion / concentration processing unit 34 receives an image from the image decoding unit 32.

The emotion / concentration processing unit 34 creates display data for display on the display from these data. For example, the emotion / concentration processing unit 34 creates display data of a display icon corresponding to an emotion by performing threshold processing of an emotion score for each type of emotion with a predetermined threshold value. For example, the emotion / concentration processing unit 34 displays an icon corresponding to the happy emotion in the vicinity of the position in the video of the second user for the second user having an emotion score of 80 or more for the happy emotion. Create display data for display. In addition, the emotion / concentration processing unit 34 displays an icon corresponding to the unpleasant emotion in the vicinity of the position in the video of the second user for the second user having an emotion score of 80 or more for the unpleasant emotion. Create display data for display.

Further, the emotion / concentration processing unit 34 cuts out the image of the second user from the image, and displays the measurement time of the emotion and concentration of the second user, the detected emotion and concentration next to the cut out image, for example. Create display data to do so.

The emotion / concentration processing unit 34 outputs the created display data to the display / output unit 35.

The display / output unit 35 functions as a providing unit and a first output unit, receives audio from the audio decoding unit 33, and outputs the audio from the speaker.

Further, the display / output unit 35 receives an image from the image decoding unit 32, receives display data from the emotion / concentration processing unit 34, superimposes the display data on the image, and displays the superimposed image on the display. Let me.

FIG. 3 is a diagram showing an example of an image displayed on the display.

The video includes a video display area 60 and an emotion history notification area 61. The video received from the video decoding unit 32 is displayed in the video display area 60. Here, the second users, users 71A to 71C, are displayed. Further, in the vicinity of the users 71A to 71C, icons 72A to 72C shown in the display data received from the emotion / concentration processing unit 34 are displayed, respectively. The icons 72A and 72C are icons corresponding to happy emotions, and the icons 72B are icons corresponding to unpleasant emotions. That is, it is shown that the emotion scores of the happy emotions of the users 71A and 71C are 80 or more, and the emotion scores of the unpleasant emotions of the icon 72B are 80 or more.

In the emotion history notification area 61, the images of the users 71A to 71C cut out from the images are displayed. Next to that, the judgment results 73A to 73C are displayed for each second user. The determination results 73A to 73C indicate the emotions detected from the users 71A to 71C, the measurement time of the emotions, and the degree of concentration, respectively. For example, the judgment result 73A indicates that the emotion score for the happy emotion of the user 71A became 80 or more at 14:10:25, and the concentration ratio at that time was 80%. Further, the judgment result 73B indicates that the emotion score of the user 71B regarding the unpleasant emotion became 80 or more at 14:08:10, and the concentration ratio at that time was 60%. Further, the judgment result 73C indicates that the emotion score for the happy emotion of the user 71C became 80 or more at 14:07:50, and the concentration ratio at that time was 90%. The determination results 73A to 73C may indicate the degree of concentration of the current time instead of the degree of concentration of the measurement time.

[Structure of the second device 4]
FIG. 4 is a block diagram showing a functional configuration of the second device 4 according to the first embodiment of the present disclosure. The configuration of the second device 4 is paired with the configuration of the first device 2.

The second device 4 includes a video acquisition unit 41, a video coding unit 42, a video analysis unit 43, an audio acquisition unit 44, an audio coding unit 45, an audio analysis unit 46, a multiplexing unit 47, and the like. The emotion / concentration determination unit 48, the second transmission unit 49, the second reception unit 50, the separation unit 51, the video decoding unit 52, the audio decoding unit 53, and the emotion / concentration processing unit 54. , Display / output unit 55.

The second device 4 can be realized by a general computer including a CPU, ROM, RAM, HDD, communication interface, input / output interface, and the like. For example, by expanding the computer program recorded in the HDD on the RAM and executing it on the CPU, each processing unit 21 to 35 is functionally realized. However, a part or all of the processing units 41 to 55 may be realized by hardware such as LSI, ASIC, and FPGA.

The image acquisition unit 41 functions as a second acquisition unit, and acquires images taken by the camera from a camera built in the second device 4 or connected by wire or wirelessly. The video acquisition unit 41 outputs the acquired video to the video coding unit 42 and the video analysis unit 43.

The video coding unit 42 receives the video from the video acquisition unit 41 and encodes the video according to a predetermined coding method. For example, when the video is a 4K video or an 8K video, the video coding unit 42 may use the H. Video is encoded according to 265 / HEVC (High Efficiency Video Coding). The video coding unit 42 outputs the coded video to the multiplexing unit 47.

The image analysis unit 43 receives an image from the image acquisition unit 41, and identifies a user (hereinafter, referred to as “second user”) shown in the image and a position in the image of the second user. For example, the image analysis unit 43 uses a face recognition algorithm to identify a second user in the image. However, the information for identifying the second user (for example, the user name) and the position of the second user in the video may be specified by the operator of the second device 4 by external input.

The image analysis unit 43 functions as a determination unit, and determines the emotion and concentration of the second user by analyzing the image. That is, the video analysis unit 43 calculates an emotion score that quantifies the degree of emotion for each type of emotion based on the input video. The types of emotions analyzed by the image analysis unit 43 are, for example, anger, contempt, disgust, surprise, fear, joy, sadness, surprise, happiness, and discomfort. The image analysis unit 43 calculates the emotion score of the second user by inputting an image into the classifier using the classifiers provided for each type of emotion.

As the discriminator, for example, a multi-layer neural network that receives an image as an input and outputs an emotion score can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's video and emotion score as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

Further, the video analysis unit 43 calculates the degree of concentration in which the degree of concentration of the user is quantified based on the input video. That is, the video analysis unit 43 calculates the degree of concentration of the second user by inputting the video to the classifier.

As the discriminator, for example, a multi-layer neural network that receives an image as an input and outputs a degree of concentration can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's image and concentration as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

When a plurality of second users are included in the video, the video analysis unit 43 calculates the emotion score and the degree of concentration for each second user.

The video analysis unit 43 determines the emotion / concentration determination unit for the information for identifying the second user, the position in the video of the second user, and the calculated emotion score and concentration for each type of emotion of the second user. Output to 48.

The voice acquisition unit 44 functions as a second acquisition unit, and acquires the voice of the second user from a microphone built in the second device 4 or connected by wire or wirelessly. The voice acquisition unit 44 outputs the acquired voice of the second user to the voice coding unit 45 and the voice analysis unit 46.

The voice coding unit 45 receives the voice from the voice acquisition unit 44 and encodes the voice according to a predetermined coding method. For example, the voice coding unit 45 encodes voice according to MPEG-4 AAC. The voice coding unit 45 outputs the coded voice to the multiplexing unit 47.

The voice analysis unit 46 receives a voice from the voice acquisition unit 44 and identifies a second user who is emitting the voice. The second user may be specified, for example, by identifying the speaker based on the voice data registered in advance. The voice analysis unit 46 analyzes the voiceprint of the speaker from the voice, and identifies the speaker by using an identification method such as a hidden Markov model, a neural network, or a decision tree. However, when the second user speaks, the information for identifying the second user may be externally input by the second user or the operator of the second device 4.

The voice analysis unit 46 functions as a determination unit, and determines the emotion and concentration of the second user by analyzing the voice. That is, the voice analysis unit 46 calculates the emotion score for each type of emotion based on the input voice. The type of emotion to be analyzed by the voice analysis unit 46 is the same as the type of emotion to be analyzed by the video analysis unit 43. The voice analysis unit 46 calculates an emotion score by inputting voice into the classifier using a classifier provided for each type of emotion.

As the discriminator, for example, a multi-layer neural network that receives voice as an input and outputs an emotion score can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's voice and emotion score as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

In addition, the voice analysis unit 46 calculates the degree of concentration of the second user based on the input voice.

As the discriminator, for example, a multi-layer neural network that receives voice as an input and outputs a degree of concentration can be used. This multi-layer neural network is constructed, for example, by performing machine learning such as deep learning on the parameters of the multi-layer neural network using the user's voice and concentration as teacher data. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

When a plurality of second users are included in the voice, the voice analysis unit 46 calculates the emotion score and the degree of concentration for each second user.

The voice analysis unit 46 outputs the information for identifying the second user, the calculated emotion score for each type of emotion of the second user, and the degree of concentration to the emotion / concentration determination unit 48.

The multiplexing unit 47 receives the encoded video and the encoded audio from the video coding unit 42 and the audio coding unit 45, respectively, and multiplexes the encoded video and the encoded audio to obtain the multiplexed data. To generate. For example, the multiplexing unit 47 performs multiplexing according to MPEG-H MMT. The multiplexing unit 47 outputs the generated multiplexing data to the second transmission unit 49.

The emotion / concentration determination unit 48 includes information for identifying the second user from the video analysis unit 43, a position in the video of the second user, an emotion score for each type of emotion of the second user, and the second user's emotion score. Receive a degree of concentration. Further, the emotion / concentration determination unit 48 receives information for identifying the second user from the voice analysis unit 46, an emotion score for each type of emotion of the second user, and a degree of concentration of the second user.

The emotion / concentration judgment unit 48 functions as a judgment unit, and determines the emotion of the second user based on the emotion score for each type of emotion of the second user received from the video analysis unit 43 and the voice analysis unit 46. .. For example, the emotion / concentration determination unit 48 has an emotion score corresponding to the type of the second user received from the video analysis unit 43 and the type of the second user received from the voice analysis unit 46 for each type of emotion. The emotion score of the type is calculated by simply adding or weighting the emotion score corresponding to. The weight of the weighting addition may be set in advance or may be changed according to the two emotion scores.

The method of calculating the emotion score is not limited to this. For example, the emotion / concentration determination unit 48 may calculate the emotion score of the second user by using the discriminator provided for each type of emotion. Specifically, the emotion / concentration determination unit 48 receives the emotion score corresponding to the type of the second user received from the video analysis unit 43 in the discriminator of each emotion type, and the voice analysis unit 46. By inputting the emotion score corresponding to the type of the two users, the emotion score of the second user for the type of emotion is calculated.

As the discriminator, for example, a multi-layer neural network that receives an emotion score calculated based on video and an emotion score calculated based on audio as input and outputs an emotion score can be used. In this multi-layer neural network, for example, the emotion score calculated based on the video and the emotion score calculated based on the voice and the emotion score determined by the designer of the neural network are used as teacher data, and the parameters of the multi-layer neural network are deep-layered. It is constructed by performing machine learning such as learning. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

Further, the emotion / concentration determination unit 48 determines the concentration level of the second user based on the concentration level of each type of emotion of the second user received from the video analysis unit 43 and the voice analysis unit 46. For example, the emotion / concentration determination unit 48 simply adds or weights the concentration of the second user received from the video analysis unit 43 and the concentration of the second user received from the voice analysis unit 46. , Calculate the concentration of the second user. The weight of the weighting addition may be set in advance or may be changed according to the two degrees of concentration.

The method of calculating the degree of concentration is not limited to this. For example, the emotion / concentration determination unit 48 may calculate the concentration level of the second user by using the discriminator. Specifically, the emotion / concentration determination unit 48 inputs the concentration level of the second user received from the video analysis unit 43 and the concentration level of the second user received from the voice analysis unit 46 into the classifier. To calculate the concentration of the second user.

As the discriminator, for example, a multi-layer neural network that receives the concentration level calculated based on the video and the concentration level calculated based on the audio as inputs and outputs the concentration level can be used. In this multi-layer neural network, for example, the concentration calculated based on the video and the concentration calculated based on the sound and the concentration determined by the designer of the neural network are used as training data, and the parameters of the multi-layer neural network are deep-layered. It is constructed by performing machine learning such as learning. The classifier is not limited to the multi-layer neural network, and other classifiers such as a linear regression model, a logistic regression model, a support vector machine, a random forest, AdaBoost, a naive bays, and a k-nearest neighbor method are used. be able to.

When there are a plurality of second users, the emotion / concentration determination unit 48 calculates the emotion score and the degree of concentration for each second user.

The emotion / concentration determination unit 48 associates the calculated emotion score of the second user for each type of emotion with the degree of concentration of the second user with the identifier of the second user and the calculated time, and the emotion / concentration DB5. Write to. It should be noted that the identifier of the second user includes information for identifying the second user (for example, a user name) and position information in the video of the second user.

The emotion / concentration determination unit 48 may correct the calculated emotion score of the second user based on the history of the emotion score of the second user. For example, the emotion / concentration determination unit 48 reads the emotion score of the second user for a certain period in the past from the emotion / concentration DB5 for each type of emotion, and based on the read emotion score, the standard deviation of the emotion score and the standard deviation of the emotion score. Calculate the average. The emotion / concentration determination unit 48 standardizes the calculated emotion score of the second user for each type of emotion using the calculated standard deviation and variance of the emotion score according to the above equation 1. Thereby, the emotion score can be standardized among the second users.

Further, the emotion / concentration determination unit 48 may normalize the emotion score instead of standardizing the emotion score. For example, the emotion / concentration determination unit 48 reads the emotion score of the second user for a certain period in the past from the emotion / concentration DB5 for each type of emotion, and based on the read emotion score, the maximum value of the emotion score and Calculate the minimum value. The emotion / concentration determination unit 48 normalizes the calculated emotion score of the second user for each type of emotion using the calculated maximum and minimum values of the emotion score according to the above equation 2. This makes it possible to normalize the emotional score among the second users.

In addition, the emotion / concentration determination unit 48 may correct the calculated concentration level of the second user based on the history of the concentration level of the second user. For example, the emotion / concentration determination unit 48 reads out the concentration of the second user in the past fixed period from the emotion / concentration DB 5, and calculates the standard deviation and the average of the concentration based on the read concentration. The emotion / concentration determination unit 48 standardizes the calculated concentration of the second user according to the above equation 3 by using the standard deviation and the average of the calculated concentration. As a result, the degree of concentration can be standardized among the second users.

Further, the emotion / concentration determination unit 48 may normalize the concentration level instead of standardizing the concentration level. For example, the emotion / concentration determination unit 48 reads the concentration of the second user for a certain period in the past from the emotion / concentration DB 5, and calculates the maximum value and the minimum value of the concentration based on the read concentration. The emotion / concentration determination unit 48 normalizes the calculated concentration of the second user for each type of emotion using the calculated maximum and minimum values of the concentration according to the above equation 4. As a result, the degree of concentration can be normalized among the second users.

The emotion / concentration determination unit 48 sends the calculated emotion score of the second user for each type of emotion and the degree of concentration of the second user to the second transmission unit 49 together with the identifier of the second user and the calculated time. Output.

The second transmission unit 49 receives the multiplexed data from the multiplexing unit 47, and the emotion / concentration determination unit 48 adds the second user's identifier and the calculated time to the emotion score of the second user for each type of emotion. , Receives the concentration of the second user. The second transmission unit 49 transmits these received data to the first device 2.

The second receiving unit 50 includes multiplexed data in which encoded video and encoded audio are multiplexed from the first device 2, an emotion score of the first user for each type of emotion, and a degree of concentration of the first user. And receive. An identifier of the first user and a calculation time of the emotion score and the degree of concentration of the first user are added to these data. The second receiving unit 50 outputs a set of these data received from the first device 2 to the separating unit 51. It is assumed that the identifier of the first user includes information for identifying the first user (for example, a user name) and position information in the video of the first user.

The separation unit 51 receives the data set from the second receiving unit 50 and separates the data set. That is, the separation unit 51 separates the multiplexed data included in the data set into the encoded video and the encoded audio, and separates the separated encoded video and the encoded audio into the video decoding unit 52 and the audio decoding. Output to unit 53, respectively. Further, the separation unit 51 separates the emotion score and the degree of concentration for each type of emotion to which the identifier of the first user and the calculated time are added from the data set, and separates these separated data into the emotion / concentration processing unit 54. Output to.

The video decoding unit 52 receives the encoded video from the separation unit 51 and decodes the video according to a predetermined decoding method. The decoding method is a method corresponding to the video coding method in the first device 2. For example, when the video is a 4K video or an 8K video, the first device 2 is H.I. When the video is encoded according to 265 / HEVC, the video decoding unit 52 uses H. The encoded video is decoded according to 265 / HEVC. The video decoding unit 52 outputs the decoded video to the emotion / concentration processing unit 54 and the display / output unit 55.

The voice decoding unit 53 receives the encoded voice from the separation unit 51 and decodes the voice according to a predetermined decoding method. The decoding method is a method corresponding to the voice coding method in the first device 2. For example, when the first device 2 encodes the voice according to the MPEG-4 AAC, the voice decoding unit 53 decodes the voice according to the MPEG-4 AAC. The voice decoding unit 53 outputs the decoded voice to the display / output unit 55.

The emotion / concentration processing unit 54 receives the identifier and measurement time of the first user from the separation unit 51, and the emotion score and concentration level for each type of emotion of the first user. Further, the emotion / concentration processing unit 54 receives an image from the image decoding unit 52.

The emotion / concentration processing unit 54 creates display data for display on the display from these data. For example, the emotion / concentration processing unit 54 creates display data of a display icon corresponding to an emotion by performing threshold processing of an emotion score for each type of emotion with a predetermined threshold value. For example, the emotion / concentration processing unit 54 displays an icon corresponding to the happy emotion in the vicinity of the position in the image of the first user for the first user having an emotion score of 80 or more for the happy emotion. Create display data for display. Further, the emotion / concentration processing unit 54 displays an icon corresponding to the unpleasant emotion in the vicinity of the position in the image of the first user for the first user having an emotion score of 80 or more for the unpleasant emotion. Create display data for display.

Further, the emotion / concentration processing unit 54 cuts out the image of the first user from the image, and displays the measurement time of the emotion and concentration of the first user, the detected emotion and concentration next to the cut out image, for example. Create display data to do so.

The emotion / concentration processing unit 54 outputs the created display data to the display / output unit 55.

The display / output unit 55 functions as a second output unit, receives audio from the audio decoding unit 53, and outputs the audio from the speaker.

Further, the display / output unit 55 receives an image from the image decoding unit 52, receives display data from the emotion / concentration processing unit 54, superimposes the display data on the image, and displays the superimposed image on the display. Let me.

[Processing flow of providing system 1]
FIG. 5 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the first user from the first device 2 to the second device 4 by the providing system 1 according to the first embodiment of the present disclosure.

The image acquisition unit 21 of the first device 2 acquires an image from the camera (S1).

The voice acquisition unit 24 of the first device 2 acquires voice from the microphone (S2).

The video analysis unit 23 of the first device 2 identifies the first user from the video by analyzing the video acquired in step S1, and the emotion score for each position of the first user and the type of emotion of the first user. And the degree of concentration is determined (S3).

The voice analysis unit 26 of the first device 2 identifies the first user from the voice by analyzing the voice acquired in step S2, and determines the emotion score and the degree of concentration for each type of emotion of the first user. (S4).

The emotion / concentration determination unit 28 of the first device 2 determines the emotion score and concentration level for each type of emotion of the first user determined in step S3, and each type of emotion of the first user determined in step S4. Based on the emotional score and the degree of concentration of the first user, the emotional score and the degree of concentration for each type of emotion of the first user are determined (S5).

The video coding unit 22 of the first device 2 encodes the video acquired in step S1 (S6).

The voice coding unit 25 of the first device 2 encodes the voice acquired in step S2 (S7).

The multiplexing unit 27 of the first device 2 multiplexes the video encoded in step S6 and the audio encoded in step S7 to generate multiplexed data (S8).

The first transmission unit 29 of the first device 2 sets the multiplexing data generated in step S8, the emotion score and the degree of concentration for each type of emotion of the first user determined in step S5, and the identifier of the first user. The data set to which the calculated time is added is transmitted to the second device 4. The second receiving unit 50 of the second device 4 receives the data set (S9).

The second receiving unit 50 of the second device 4 adds the encoded video, the encoded audio, the identifier of the first user, and the calculated time to the data set received in step S9 for each emotion type. Separated into score and concentration (S10).

The video decoding unit 52 of the second device 4 decodes the coded video separated in step S10 (S11).

The voice decoding unit 53 of the second device 4 decodes the coded voice separated in step S10 (S12).

The emotion / concentration processing unit 54 of the second device 4 adds an emotion score for each type of emotion to which the video decoded in step S11, the identifier of the first user separated in step S10, and the calculated time are added. Based on the concentration level, display data for displaying the emotion and concentration level of the first user on the display is created (S13).

The display / output unit 55 of the second device 4 superimposes the display data created in step S13 on the video decoded in step S11, and displays the superposed video on the display (S14).

The display / output unit 55 of the second device 4 outputs the sound decoded in step S12 from the speaker (S15).

By executing the process shown in FIG. 5, the emotion and concentration of the first user are provided to the second user.

FIG. 6 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the second user from the second device 4 to the first device 2 by the providing system 1 according to the first embodiment of the present disclosure.

The image acquisition unit 41 of the second device 4 acquires an image from the camera (S21).

The voice acquisition unit 44 of the second device 4 acquires voice from the microphone (S22).

The video analysis unit 43 of the second device 4 identifies the second user from the video by analyzing the video acquired in step S21, and the emotion score for each position of the second user and the type of emotion of the second user. And the degree of concentration is determined (S23).

The voice analysis unit 46 of the second device 4 identifies the second user from the voice by analyzing the voice acquired in step S22, and determines the emotion score and the degree of concentration for each type of emotion of the second user. (S24).

The emotion / concentration determination unit 48 of the second device 4 has an emotion score and concentration for each type of emotion of the second user determined in step S23, and each type of emotion of the second user determined in step S24. Based on the emotional score and the degree of concentration of the second user, the emotional score and the degree of concentration for each type of emotion of the second user are determined (S25).

The video coding unit 42 of the second device 4 encodes the video acquired in step S21 (S26).

The voice coding unit 45 of the second device 4 encodes the voice acquired in step S22 (S27).

The multiplexing unit 47 of the second device 4 multiplexes the video encoded in step S26 and the audio encoded in step S27 to generate multiplexed data (S28).

The second transmission unit 49 of the second device 4 sets the multiplexing data generated in step S28, the emotion score and the degree of concentration of each type of emotion of the second user determined in step S25, and the identifier of the second user and the degree of concentration. The data set to which the calculated time is added is transmitted to the first device 2. The first receiving unit 30 of the first device 2 receives the data set (S29).

The first receiving unit 30 of the first device 2 adds the encoded video, the encoded audio, the identifier of the second user, and the calculated time to the data set received in step S29 for each emotion type. Separated into score and concentration (S30).

The video decoding unit 32 of the first device 2 decodes the coded video separated in step S30 (S31).

The voice decoding unit 33 of the first device 2 decodes the coded voice separated in step S30 (S32).

The emotion / concentration processing unit 34 of the first device 2 has an emotion score for each type of emotion to which the video decoded in step S31, the identifier of the second user separated in step S30, and the calculated time are added. Based on the concentration level, display data for displaying the emotion and concentration level of the second user on the display is created (S33).

The display / output unit 35 of the first device 2 superimposes the display data created in step S33 on the video decoded in step S31, and displays the superposed video on the display (S34).

The display / output unit 35 of the first device 2 outputs the sound decoded in step S32 from the speaker (S35).

By executing the process shown in FIG. 6, the emotion and concentration of the second user are provided to the first user.

[Effects of Embodiment 1 and the like]
According to the first embodiment, the determination result of at least one of the emotion and the concentration of the second user who is the listener of the utterance content of the first user is provided to the first user. Therefore, the first user can know what kind of emotion the second user has with respect to the content of his / her utterance, or whether the second user concentrates on listening to the story. On the other hand, the first user asks a question to the second user who has a negative feeling about the utterance content, or changes the topic when the second user is not concentrated. It can be performed. This makes it possible to support smooth communication between users.

Similarly, the judgment result of at least one of the emotion and the concentration of the first user who is the listener of the utterance content of the second user is provided to the second user. As a result, the second user can take the same measures as the first user.

In addition, at least one of the emotions and the degree of concentration of the second user is determined in consideration of the voice of the second user. Therefore, the emotion or concentration of the second user can be determined with higher accuracy than in the case of determining at least one of the emotion and the concentration of the second user using only the video of the second user. The same applies to the judgment of the emotion and concentration of the first user.

Further, it is possible to have a dialogue between the first user and the second user over the network 3 and provide the first user with a judgment result of at least one of the emotion and the concentration of the second user. Therefore, for example, in an electronic conference system in which the first user is the facilitator of the conference, the first user grasps the emotion or concentration of the second user and asks the second user for opinions as appropriate. You can proceed. This makes the discussion constructive and enables highly productive meetings. As in the case of the second user, the first device 2 determines at least one of the emotion and concentration of the first user from the audio and video of the first user, and the second device 4 determines the emotion and concentration of the first user. The result of judgment of at least one of emotion and concentration is displayed on the display. As a result, the first user and the second user can mutually grasp the emotions or the degree of concentration of the other party.

Further, the first device 2 determines the emotion and concentration of the first user, and the second device 4 determines the emotion and concentration of the second user. Therefore, the first device 2 may not transmit the video to the second device 4, and the second device 4 may not transmit the video to the first device 2. As a result, the determination result of the emotion or concentration of the first user can be transmitted to the second device 4 while reducing the transmission data from the first device 2 to the second device 4. Further, the determination result of the emotion or the degree of concentration of the second user can be transmitted to the first device 2 while reducing the transmission data from the second device 4 to the first device 2.

The emotion / concentration processing unit 34 of the first device 2 may create display data for displaying the avatar of the second user having a facial expression based on the emotion of the second user. Similarly, the emotion / concentration processing unit 54 of the second device 4 may create display data for displaying the avatar of the first user based on the emotion of the first user. As a result, the avatar can be displayed instead of the user's image, so that the privacy of the second user and the first user can be protected.

Further, the emotion / concentration determination unit 28 of the first device 2 and the emotion / concentration determination unit 48 of the second device 4 standardize the emotion score and concentration based on the emotion score and concentration of the past user. be able to. That is, it is possible to standardize each score of a user having a relatively small change in emotional ups and downs and concentration, and each score of a user having a relatively large change in emotional ups and downs and concentration. This makes it possible to accurately compare emotions or concentration levels among users.

<Embodiment 2>
In the first embodiment, the first device 2 determines the emotion and concentration of the first user, and the second device 4 determines the emotion and concentration of the second user. In the second embodiment, an example will be described in which the first device 2 determines the emotion and concentration of the second user, and the second device 4 determines the emotion and concentration of the first user.

The configuration of the provision system 1 according to the second embodiment is the same as that of the first embodiment.

[Structure of First Device 2]
FIG. 7 is a block diagram showing a functional configuration of the first device 2 according to the second embodiment of the present disclosure.

The first device 2 includes a video acquisition unit 21, a video coding unit 22, an audio acquisition unit 24, an audio coding unit 25, a multiplexing unit 27, a first transmission unit 29, and a first reception unit 30. A separation unit 31, a video decoding unit 32, an audio decoding unit 33, a video analysis unit 23, an audio analysis unit 26, an emotion / concentration determination unit 28, and a display / output unit 35. ..

The processing of the video acquisition unit 21, the video coding unit 22, the audio acquisition unit 24, the audio coding unit 25, and the multiplexing unit 27 is the same as that of the first embodiment.

The first transmission unit 29 receives the multiplexing data from the multiplexing unit 27 and transmits the multiplexing data to the second device 4.

The first receiving unit 30 receives the multiplexed data in which the encoded video and the encoded audio are multiplexed from the second device 4. The first receiving unit 30 outputs the received multiplexed data to the separating unit 31.

The separation unit 31 receives the multiplexed data from the first receiving unit 30 and separates the multiplexed data into a coded video and a coded audio. The separation unit 31 outputs the separated encoded video and encoded audio to the video decoding unit 32 and the audio decoding unit 33, respectively.

The processing of the video decoding unit 32 and the audio decoding unit 33 is the same as that of the first embodiment. The video decoding unit 32 functions as a video acquisition unit and outputs the decoded video to the video analysis unit 23 and the emotion / concentration determination unit 28, and the audio decoding unit 33 functions as an audio acquisition unit and decodes the video. The converted voice is output to the voice analysis unit 26.

The video analysis unit 23 receives the video from the video decoding unit 32 and identifies the second user and the position in the video of the second user in the video. In addition, the video analysis unit 23 functions as a determination unit, and determines the emotion and concentration of the second user by analyzing the video. The video analysis unit 23 determines the emotion / concentration determination unit for the information for identifying the second user, the position in the video of the second user, and the calculated emotion score and concentration for each type of emotion of the second user. Output to 28. The processing of the video analysis unit 23 is the same as that of the video analysis unit 23 of the first embodiment except that the user to be processed is the second user.

The voice analysis unit 26 receives a voice from the voice decoding unit 33 and identifies a second user who is emitting the voice. In addition, the voice analysis unit 26 functions as a determination unit, and determines the emotion and concentration of the second user by analyzing the voice. The voice analysis unit 26 outputs the information for identifying the second user, the calculated emotion score for each type of emotion of the second user, and the degree of concentration to the emotion / concentration determination unit 28. The processing of the voice analysis unit 26 is the same as that of the voice analysis unit 26 of the first embodiment except that the user to be processed is the second user.

The emotion / concentration determination unit 28 includes information for identifying the second user from the video analysis unit 23, a position in the video of the second user, an emotion score for each type of emotion of the second user, and the second user's emotion score. Receive a degree of concentration. Further, the emotion / concentration determination unit 28 receives information for identifying the second user from the voice analysis unit 26, an emotion score for each type of emotion of the second user, and a degree of concentration of the second user.

The emotion / concentration judgment unit 28 functions as a judgment unit, and determines the emotion of the second user based on the emotion score for each type of emotion of the second user received from the video analysis unit 23 and the voice analysis unit 26. .. For example, the emotion / concentration determination unit 28 has an emotion score corresponding to the type of the second user received from the video analysis unit 23 and the type of the second user received from the voice analysis unit 26 for each type of emotion. The emotion score of the type is calculated by simply adding or weighting the emotion score corresponding to. The weight of the weighting addition may be set in advance or may be changed according to the two emotion scores.

Further, the emotion / concentration determination unit 28 determines the concentration level of the second user based on the concentration level of each type of emotion of the second user received from the video analysis unit 23 and the voice analysis unit 26. For example, the emotion / concentration determination unit 28 simply adds or weights the concentration of the second user received from the video analysis unit 23 and the concentration of the second user received from the voice analysis unit 26. , Calculate the concentration of the second user. The weight of the weighting addition may be set in advance or may be changed according to the two degrees of concentration.

When there are a plurality of second users, the emotion / concentration determination unit 28 calculates the emotion score and the degree of concentration for each of the second users.

In addition, the emotion / concentration determination unit 28 creates display data for display on the display. For example, the emotion / concentration determination unit 28 creates display data of a display icon corresponding to an emotion by performing threshold processing of an emotion score for each type of emotion with a predetermined threshold value. For example, the emotion / concentration determination unit 28 displays an icon corresponding to the happy emotion in the vicinity of the position in the video of the second user for the second user having an emotion score of 80 or more for the happy emotion. Create display data for display. In addition, the emotion / concentration determination unit 28 displays an icon corresponding to the unpleasant emotion in the vicinity of the position in the video of the second user for the second user having an emotion score of 80 or more for the unpleasant emotion. Create display data for display.

Further, the emotion / concentration determination unit 28 cuts out the image of the second user from the image, and displays the measurement time of the emotion and concentration of the second user, the detected emotion and concentration next to the cut out image, for example. Create display data to do so.

The emotion / concentration determination unit 28 outputs the created display data to the display / output unit 35.

The processing of the display / output unit 35 is the same as that of the display / output unit 35 of the first embodiment.

[Structure of the second device 4]
FIG. 8 is a block diagram showing a functional configuration of the second device 4 according to the second embodiment of the present disclosure.

The second device 4 includes a video acquisition unit 41, a video coding unit 42, an audio acquisition unit 44, an audio coding unit 45, a multiplexing unit 47, a second transmission unit 49, and a second reception unit 50. The separation unit 51, the video decoding unit 52, the audio decoding unit 53, the video analysis unit 43, the audio analysis unit 46, the emotion / concentration determination unit 48, and the display / output unit 55 are provided. ..

The processing of the video acquisition unit 41, the video coding unit 42, the audio acquisition unit 44, the audio coding unit 45, and the multiplexing unit 47 is the same as that of the first embodiment.

The second transmission unit 49 receives the multiplexing data from the multiplexing unit 47 and transmits the multiplexing data to the first device 2.

The second receiving unit 50 receives the multiplexed data in which the encoded video and the encoded audio are multiplexed from the first device 2. The second receiving unit 50 outputs the received multiplexed data to the separating unit 51.

The separation unit 51 receives the multiplexed data from the second receiving unit 50, and separates the multiplexed data into a coded video and a coded audio. The separation unit 51 outputs the separated encoded video and encoded audio to the video decoding unit 52 and the audio decoding unit 53, respectively.

The processing of the video decoding unit 52 and the audio decoding unit 53 is the same as that of the first embodiment. The video decoding unit 52 outputs the decoded video to the video analysis unit 43 and the emotion / concentration determination unit 48, and the audio decoding unit 53 outputs the decoded audio to the audio analysis unit 46.

The video analysis unit 43 receives the video from the video decoding unit 52 and identifies the first user and the position in the video of the first user in the video. In addition, the video analysis unit 43 determines the emotion and concentration of the first user by analyzing the video. The video analysis unit 43 determines the emotion / concentration determination unit for the information for identifying the first user, the position in the video of the first user, and the calculated emotion score and concentration for each type of emotion of the first user. Output to 48. The processing of the video analysis unit 43 is the same as that of the video analysis unit 43 of the first embodiment except that the user to be processed is the first user.

The voice analysis unit 46 receives the voice from the voice decoding unit 53 and identifies the first user who is emitting the voice. In addition, the voice analysis unit 46 determines the emotion and concentration of the first user by analyzing the voice. The voice analysis unit 46 outputs the information for identifying the first user, the calculated emotion score for each type of emotion of the first user, and the degree of concentration to the emotion / concentration determination unit 48. The processing of the voice analysis unit 46 is the same as that of the voice analysis unit 46 of the first embodiment except that the user to be processed is the first user.

The emotion / concentration determination unit 48 includes information for identifying the first user from the video analysis unit 43, a position in the video of the first user, an emotion score for each type of emotion of the first user, and the emotion score of the first user. Receive a degree of concentration. Further, the emotion / concentration determination unit 48 receives information for identifying the first user from the voice analysis unit 46, an emotion score for each type of emotion of the first user, and a degree of concentration of the first user.

The emotion / concentration determination unit 48 determines the emotion of the first user based on the emotion score for each type of emotion of the first user received from the video analysis unit 43 and the voice analysis unit 46. For example, the emotion / concentration determination unit 48 has an emotion score corresponding to the type of the first user received from the video analysis unit 43 and the type of the first user received from the voice analysis unit 46 for each type of emotion. The emotion score of the type is calculated by simply adding or weighting the emotion score corresponding to. The weight of the weighting addition may be set in advance or may be changed according to the two emotion scores.

Further, the emotion / concentration determination unit 48 determines the concentration level of the first user based on the concentration level of each type of emotion of the first user received from the video analysis unit 43 and the voice analysis unit 46. For example, the emotion / concentration determination unit 48 simply adds or weights the concentration of the first user received from the video analysis unit 43 and the concentration of the first user received from the voice analysis unit 46. , Calculate the concentration of the first user. The weight of the weighting addition may be set in advance or may be changed according to the two degrees of concentration.

When there are a plurality of first users, the emotion / concentration determination unit 48 calculates the emotion score and the degree of concentration for each first user.

In addition, the emotion / concentration determination unit 48 creates display data for display on the display. For example, the emotion / concentration determination unit 48 creates display data of a display icon corresponding to an emotion by performing threshold processing of the emotion score for each type of emotion with a predetermined threshold value. For example, the emotion / concentration determination unit 48 displays an icon corresponding to the happy emotion in the vicinity of the position in the image of the first user for the first user having an emotion score of 80 or more for the happy emotion. Create display data for display. In addition, the emotion / concentration determination unit 48 displays an icon corresponding to the unpleasant emotion in the vicinity of the position in the image of the first user for the first user having an emotion score of 80 or more for the unpleasant emotion. Create display data for display.

Further, the emotion / concentration determination unit 48, for example, cuts out the image of the first user from the image and displays the measurement time of the emotion and concentration of the first user, the detected emotion and concentration next to the cut out image. Create display data to do so.

The emotion / concentration determination unit 48 outputs the created display data to the display / output unit 55.

The processing of the display / output unit 55 is the same as that of the display / output unit 55 of the first embodiment.

[Processing flow of providing system 1]
FIG. 9 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the first user from the first device 2 to the second device 4 by the providing system 1 according to the second embodiment of the present disclosure.

The first device 2 executes the same processes of steps S1, S2, and S6 to S8 as shown in FIG.

The first transmission unit 29 of the first device 2 transmits the multiplexed data generated in step S8 to the second device 4, and the second receiving unit 50 of the second device 4 receives the multiplexed data (S16). ).

The separation unit 51 of the second device 4 separates the multiplexed data received in step S16 into coded video and coded audio (S10).

The second device 4 executes the same processes of steps S11 and S12 as shown in FIG.

The video analysis unit 43 of the second device 4 identifies the first user from the video by analyzing the video decoded in step S11, and the emotion score for each position of the first user and the type of emotion of the first user. And the degree of concentration is determined (S17).

The voice analysis unit 46 of the second device 4 identifies the first user from the voice by analyzing the voice decoded in step S12, and determines the emotion score and the degree of concentration for each type of emotion of the first user. (S18).

The emotion / concentration determination unit 48 of the second device 4 determines the emotion score and concentration level for each type of emotion of the first user determined in step S17, and each type of emotion of the first user determined in step S18. Based on the emotional score and the degree of concentration of the first user, the emotional score and the degree of concentration for each type of emotion of the first user are determined. Further, the emotion / concentration determination unit 48 displays the emotion of the first user on the display based on the video decoded in step S11 and the determined emotion score and concentration of each type of emotion of the first user. And display data for displaying the degree of concentration is created (S19).

The display / output unit 55 of the second device 4 superimposes the display data created in step S19 on the video decoded in step S11, and displays the superposed video on the display (S14).

The display / output unit 55 of the second device 4 outputs the sound decoded in step S12 from the speaker (S15).

By executing the process shown in FIG. 9, the emotion and concentration of the first user are provided to the second user.

FIG. 10 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the second user from the second device 4 to the first device 2 by the providing system 1 according to the second embodiment of the present disclosure.

The second device 4 executes the same processes of steps S21, S22, and S26 to S28 as shown in FIG.

The second transmission unit 49 of the second device 4 transmits the multiplexed data generated in step S28 to the first device 2, and the first receiving unit 30 of the first device 2 receives the multiplexed data (S36). ).

The separation unit 31 of the first device 2 separates the multiplexed data received in step S36 into a coded video and a coded audio (S30).

The first device 2 executes the same processes of steps S31 and S32 as shown in FIG.

The video analysis unit 23 of the first device 2 identifies the second user from the video by analyzing the video decoded in step S31, and the emotion score for each position of the second user and the type of emotion of the second user. And the degree of concentration is determined (S37).

The voice analysis unit 26 of the first device 2 identifies the second user from the voice by analyzing the voice decoded in step S32, and determines the emotion score and the degree of concentration for each type of emotion of the second user. (S38).

The emotion / concentration determination unit 28 of the first device 2 determines the emotion score and concentration level for each type of emotion of the second user determined in step S37, and each type of emotion of the second user determined in step S38. Based on the emotion score and concentration of the second user, the emotion score and concentration of each type of emotion of the second user are determined. Further, the emotion / concentration determination unit 28 displays the emotion of the second user on the display based on the video decoded in step S31 and the determined emotion score and concentration of each type of emotion of the second user. And display data for displaying the degree of concentration is created (S39).

The display / output unit 55 of the first device 2 superimposes the display data created in step S39 on the video decoded in step S31, and displays the superposed video on the display (S34).

The display / output unit 55 of the first device 2 outputs the sound decoded in step S32 from the speaker (S35).

By executing the process shown in FIG. 10, the emotion and concentration of the second user are provided to the second user.

[Effects of Embodiment 2 and the like]
According to the second embodiment, the first device 2 can determine at least one of the emotion and the degree of concentration of the second user based on the audio and video of the second user transmitted from the second device 4. Therefore, the first device 2 can accurately synchronize the audio and video of the second user with the emotion or concentration of the second user. As a result, the audio and video of the second user can be accurately associated with the emotion or concentration of the second user and provided to the first user.

In addition, the second device 4 can determine at least one of the emotion and the degree of concentration of the first user based on the audio and video of the first user transmitted from the first device 2. Therefore, the second device 4 can accurately synchronize the audio and video of the first user with the emotion or concentration of the first user. As a result, the audio and video of the first user can be accurately associated with the emotion or concentration of the first user and provided to the second user.

<Embodiment 3>
In the providing system 1 shown in the first and second embodiments, the emotion or concentration of the user who uses the device on the other side can be presented to the user, but the processing based on the emotion or concentration is not performed. ..

In the third embodiment, an example of executing a predetermined process based on the emotion or the degree of concentration of the user will be described. Specifically, the providing system 1 that prompts the user to speak based on the judgment result of at least one of emotion and concentration will be described.

The configuration of the provision system 1 according to the third embodiment is the same as that of the first embodiment.

[Structure of First Device 2]
FIG. 11 is a block diagram showing a functional configuration of the first device 2 according to the third embodiment of the present disclosure. The configuration of the first device 2 according to the third embodiment is the same as that of the first embodiment. However, it differs from the first embodiment in that the processing result of the emotion / concentration processing unit 34 is input to the first transmission unit 29.

Whether the emotion / concentration processing unit 34 functions as a speech promotion unit and prompts the second user to speak based on the emotion score for each type of emotion of the second user received from the separation unit 31 and the degree of concentration. Decide whether or not. For example, the emotion / concentration processing unit 34 compares the degree of concentration with a predetermined threshold value and determines whether or not to prompt the second user to speak. More specifically, the emotion / concentration processing unit 34 determines to prompt a second user whose concentration is less than a predetermined threshold value (for example, 30) to speak.

Further, the emotion / concentration processing unit 34 may compare a predetermined type of emotion with a predetermined threshold value and decide whether or not to prompt the second user to speak. For example, the emotion / concentration processing unit 34 determines that the emotion of anger prompts a second user who has a predetermined threshold value (for example, 90) or more to speak. Further, the emotion / concentration processing unit 34 determines that the emotion of joy prompts the second user who has a feeling of joy less than a predetermined threshold value (for example, 30) to speak.

The emotion / concentration processing unit 34 outputs a speech promotion instruction signal (including the identifier of the second user) that identifies the second user who prompts the speech to the first transmission unit 29.

The first transmission unit 29 receives a speech promotion instruction signal from the emotion / concentration processing unit 34, and transmits the speech promotion instruction signal to the second device 4.

On the other hand, when a speech promotion instruction signal identifying the first user who promotes speech is transmitted from the second device 4, the first reception unit 30 receives the signal and outputs it to the separation unit 31.

When the data received from the first receiving unit 30 includes a speech promotion instruction signal, the separating unit 31 outputs the signal to the emotion / concentration processing unit 34.

When the emotion / concentration processing unit 34 receives the speech promotion instruction signal from the separation unit 31, the emotion / concentration processing unit 34 creates display data for prompting the first user indicated by the speech promotion instruction signal to speak, and displays / outputs the display / output unit. Output to 35. For example, when the user name of the first user is "A", the emotion / concentration processing unit 34 creates data for displaying a message such as "Does Mr. A have any opinion?" ..

The display / output unit 35 receives display data from the emotion / concentration processing unit 34 and displays the display data on the display.

FIG. 12 is a block diagram showing a functional configuration of the second device 4 according to the third embodiment of the present disclosure. The configuration of the second device 4 according to the third embodiment is the same as that of the first embodiment. However, it differs from the first embodiment in that the processing result of the emotion / concentration processing unit 54 is input to the second transmission unit 49.

The emotion / concentration processing unit 54 determines whether or not to prompt the first user to speak based on the emotion score for each type of emotion of the first user received from the separation unit 51 and the degree of concentration. For example, the emotion / concentration processing unit 54 compares the degree of concentration with a predetermined threshold value and determines whether or not to prompt the first user to speak. More specifically, the emotion / concentration processing unit 54 determines to prompt the first user whose concentration is less than a predetermined threshold value (for example, 30) to speak.

Further, the emotion / concentration processing unit 54 may compare a predetermined type of emotion with a predetermined threshold value and decide whether or not to prompt the first user to speak. For example, the emotion / concentration processing unit 54 determines that the emotion of anger prompts a first user who has a predetermined threshold value (for example, 90) or more to speak. Further, the emotion / concentration processing unit 54 determines that the emotion of joy prompts the first user who has a feeling of joy less than a predetermined threshold value (for example, 30) to speak.

The emotion / concentration processing unit 54 outputs a speech promotion instruction signal (including the identifier of the first user) that identifies the first user who prompts the speech to the second transmission unit 49.

The second transmission unit 49 receives a speech promotion instruction signal from the emotion / concentration processing unit 54, and transmits the speech promotion instruction signal to the first device 2.

On the other hand, when a speech promotion instruction signal specifying a second user who promotes speech is transmitted from the first device 2, the second reception unit 50 receives the signal and outputs it to the separation unit 51.

When the data received from the second receiving unit 50 includes a speech promotion instruction signal, the separation unit 51 outputs the signal to the emotion / concentration processing unit 54.

When the emotion / concentration processing unit 54 receives the speech promotion instruction signal from the separation unit 51, the emotion / concentration processing unit 54 creates display data for prompting the second user indicated by the speech promotion instruction signal to speak, and displays / outputs the display / output unit. Output to 55. For example, when the user name of the second user is "B", the emotion / concentration processing unit 54 creates data for displaying a message such as "Does Mr. B have any opinion?" ..

The display / output unit 55 receives display data from the emotion / concentration processing unit 54 and displays the display data on the display.

[Processing flow of providing system 1]
FIG. 13 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the first user from the first device 2 to the second device 4 by the providing system 1 according to the third embodiment of the present disclosure.

The providing system 1 executes the processes from steps S1 to S15 similar to those of the first embodiment shown in FIG.

Whether or not the emotion / concentration processing unit 54 of the second device 4 prompts the first user to speak based on the emotion score for each type of emotion of the first user separated in step S10 and the degree of concentration. Is determined, and a speech promotion instruction signal identifying the first user who prompts the speech is output to the second transmission unit 49 (S41).

The second transmission unit 49 of the second device 4 transmits a speech promotion instruction signal to the first device 2, and the first reception unit 30 of the first device 2 receives the signal (S42).

The separation unit 31 of the first device 2 outputs the speech promotion signal received in step S42 to the emotion / concentration processing unit 34, and the emotion / concentration processing unit 34 is the first user indicated in the speech promotion instruction signal. Create display data to encourage the user to speak, and output it to the display / output unit 35. The display / output unit 35 receives display data from the emotion / concentration processing unit 34 and displays the display data on the display (S43).

FIG. 14 is a sequence diagram showing an example of a procedure for providing the emotion and concentration of the second user from the second device 4 to the first device 2 by the providing system 1 according to the third embodiment of the present disclosure.

The providing system 1 executes the same processes from steps S21 to S35 as in the first embodiment shown in FIG.

Whether or not the emotion / concentration processing unit 34 of the first device 2 prompts the second user to speak based on the emotion score for each type of emotion of the second user separated in step S30 and the degree of concentration. Is determined, and a speech promotion instruction signal identifying the second user who prompts the speech is output to the first transmission unit 29 (S44).

The first transmission unit 29 of the first device 2 transmits a speech promotion instruction signal to the second device 4, and the second reception unit 50 of the second device 4 receives the signal (S45).

The separation unit 51 of the second device 4 outputs the speech promotion signal received in step S45 to the emotion / concentration processing unit 54, and the emotion / concentration processing unit 54 is the second user indicated in the speech promotion instruction signal. Create display data to encourage the user to speak, and output it to the display / output unit 55. The display / output unit 55 receives display data from the emotion / concentration processing unit 54 and displays the display data on the display (S46).

[Effects of Embodiment 3 and the like]
According to the third embodiment, for example, when the second user has a negative feeling toward the utterance content of the first user, or when the second user is not concentrated, the second user is urged to speak. Can be done. Similarly, when the first user has a negative feeling toward the utterance content of the second user, or when the first user is not concentrated, the first user can be urged to speak. This makes the discussion meaningful and supports smooth communication between users.

Although the emotion / concentration processing unit 34 of the first device 2 has determined whether or not to prompt the second user to speak, it may be determined whether or not to prompt the first user to speak. That is, the emotion / concentration processing unit 34 acquires the emotion score and concentration level of the first user from the emotion / concentration determination unit 28, and makes a statement to the first user based on the acquired emotion score and concentration level. Decide whether to encourage. The emotion / concentration processing unit 34 creates display data for prompting the first user to speak based on the determined result, and outputs the display / output data to the display / output unit 35.

Similarly, the emotion / concentration processing unit 54 of the second device 4 may determine whether to prompt the second user to speak. That is, the emotion / concentration processing unit 54 acquires the emotion score and concentration level of the second user from the emotion / concentration determination unit 48, and makes a statement to the second user based on the acquired emotion score and concentration level. Decide whether to encourage. The emotion / concentration processing unit 54 creates display data for prompting the second user to speak based on the determined result, and outputs the display / output data to the display / output unit 55.

<Modified example of the embodiment>
In the above-described embodiment, for each of the first user and the second user who participate in the conference, the degree of contribution of each user in the dialogue between the first user and the second user may be calculated.

For example, in the configuration of the first device 2 shown in FIG. 2 or FIG. 11, the emotion / concentration determination unit 28 contributes to the first user based on the emotion score and the degree of concentration for each type of emotion of the first user. The degree may be calculated. For example, the emotion / concentration determination unit 28 may calculate the average concentration of the first user in the meeting, and calculate the contribution according to a conversion formula such that the larger the average concentration is, the larger the value is. The emotion / concentration determination unit 28 writes the calculated contribution degree of the first user in the emotion / concentration DB 5 together with the identifier of the first user.

Similarly, in the configuration of the second device 4 shown in FIG. 4 or FIG. 12, the emotion / concentration determination unit 48 may calculate the contribution degree of the second user and write the calculation result in the emotion / concentration DB5. ..

Further, in the configuration of the first device 2 shown in FIG. 7, the emotion / concentration determination unit 28 may calculate the contribution degree of the second user and write the calculation result in the emotion / concentration DB5.

Further, in the configuration of the second device 4 shown in FIG. 8, the emotion / concentration determination unit 48 may calculate the contribution degree of the first user and write the calculation result in the emotion / concentration DB 5.

According to this modification, the degree of contribution in the user's dialogue can be calculated based on the judgment result of at least one of the user's emotion and the degree of concentration. For example, it is possible to calculate a high degree of contribution of a user who has concentrated on dialogue, or to calculate a high degree of contribution of a user who has low feelings of contempt and disgust and high feelings of joy and surprise.

[Additional Notes]
Although the provision system 1 according to the embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment.

Each of the above devices may be realized by a plurality of computers.

Some or all the functions of each of the above devices may be provided by cloud computing. That is, some or all the functions of each device may be realized by the cloud server.

Further, at least a part of the above-described embodiment and the above-described modification may be arbitrarily combined.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Provided system 2 1st device 3 Network 4 2nd device 5 Emotion / concentration DB
21 Video acquisition section (1st acquisition section)
22 Video coding unit 23 Video analysis unit (judgment unit)
24 Voice acquisition unit (1st acquisition unit)
25 Voice coding unit 26 Voice analysis unit (judgment unit)
27 Multiplexing Department 28 Emotion / Concentration Judgment Department (Judgment Department)
29 First transmitter (provider)
30 First receiving unit 31 Separation unit 32 Video decoding unit (video acquisition unit)
33 Voice decoding unit (voice acquisition unit)
34 Emotion / Concentration Processing Department (Speech Promotion Department, Calculation Department)
35 Display / output section (providing section, first output section)
41 Video acquisition unit (video acquisition unit, second acquisition unit)
42 Video coding unit 43 Video analysis unit (judgment unit)
44 Voice acquisition unit (voice acquisition unit, second acquisition unit)
45 Voice coding unit 46 Voice analysis unit (judgment unit)
47 Multiplexing section 48 Emotion / concentration judgment section (judgment section)
49 Second transmitter (provider)
50 Second receiving unit 51 Separation unit 52 Video decoding unit 53 Audio decoding unit 54 Emotion / concentration processing unit 55 Display / output unit (second output unit)
60 Video display area 61 Emotion history notification area 71A User 71B User 71C User 72A Icon 72B Icon 72C Icon 73A Judgment result 73B Judgment result 73C Judgment result

Claims (11)

A video acquisition unit that acquires the video of the second user, which is obtained by photographing the second user who listens to the voice of the first user who is the speaker.
A judgment unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user.
A providing system including a providing unit that provides a determination result by the determining unit to the first user. The provided system further includes a voice acquisition unit that acquires the voice of the second user.
The providing system according to claim 1, wherein the determination unit determines at least one of the emotion and the degree of concentration of the second user based on the acquired video and audio of the second user. The provided system includes a first device and a second device connected to each other via a network.
The first device is
The first acquisition unit that acquires the audio and video of the first user, and
A first transmission unit that transmits the acquired audio and video of the first user to the second device, and
A first receiving unit that receives audio and video of the second user from the second device, and
The first output unit as the providing unit, which outputs the received audio and video of the second user and the determination result by the determination unit, is included.
The second device is
The audio acquisition unit, the second acquisition unit as the video acquisition unit, and the second acquisition unit that acquires the audio and video of the second user.
A second transmission unit that transmits the acquired audio and video of the second user to the first device, and
A second receiving unit that receives the audio and video of the first user from the first device, and
The providing system according to claim 2, further comprising a second output unit that outputs the received audio and video of the first user. The determination unit is provided in the first device, and determines at least one of the emotion and the degree of concentration of the second user based on the audio and video of the second user received by the first receiving unit. The provision system according to item 3. The provided system includes a first device and a second device connected to each other via a network.
The first device is
The first acquisition unit that acquires the voice of the first user, and
A first transmission unit that transmits the acquired voice of the first user to the second device, and
A first receiving unit that receives the voice of the second user from the second device, and
The first output unit as the providing unit, which outputs the received voice of the second user and the determination result by the determination unit, is included.
The second device is
The audio acquisition unit, the second acquisition unit as the video acquisition unit, and the second acquisition unit that acquires the audio and video of the second user.
A second transmission unit that transmits the acquired voice of the second user to the first device, and
A second receiving unit that receives the voice of the first user from the first device, and
Including a second output unit that outputs the received voice of the first user.
The determination unit determines at least one of the emotion and concentration of the second user based on the audio and video of the second user acquired by the second acquisition unit, which is provided in the second device.
The second transmission unit further transmits the determination result by the determination unit.
The first receiving unit further receives the determination result by the determination unit, and receives the determination result.
The providing system according to claim 2, wherein the first output unit outputs a determination result by the determination unit received by the first receiving unit. The second device further
The invention according to any one of claims 3 to 5, further comprising a speech promotion unit that prompts the second user to speak based on the determination result of at least one of the emotion and concentration of the second user. Offering system. The provided system further
Any one of claims 3 to 6, further comprising a calculation unit that calculates the degree of contribution of the second user in the dialogue between the first user and the second user based on the judgment result by the determination unit. Provided system described in. The judgment unit further corrects the judgment result of at least one of the emotion and the concentration of the second user based on the history of the judgment result of at least one of the emotion and the concentration of the second user. The providing system according to any one of claims 7. A step of acquiring the video of the second user, which is obtained by photographing the second user who watches the audio and video of the first user who is the speaker.
A step of determining at least one of the emotion and concentration of the second user based on the acquired video of the second user, and
A providing method including a step of providing a determination result in the determination step to the first user. An image acquisition unit that acquires the image of the second user, which is obtained by photographing the second user who views the sound and image of the first user who is the speaker.
A judgment unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user.
A providing device including a providing unit that provides a determination result by the determining unit to the first user. Computer,
An image acquisition unit that acquires the image of the second user, which is obtained by photographing the second user who views the sound and image of the first user who is the speaker.
A judgment unit that determines at least one of the emotion and concentration of the second user based on the acquired video of the second user.
A computer program for causing the determination result by the determination unit to function as a provision unit to be provided to the first user.

Images