JP2021107873A - Voice characteristic change system and voice characteristic change method - Google Patents

Abstract

To efficiently support realization of smooth and comfortable dialogues between an operator and a customer according to the customer's emotions.SOLUTION: In a voice characteristic change system, a receiver which receives and outputs a video and a spoken voice of an operator from an operator terminal, and a server are connected so as to be able to communicate. The receiver is connected to a camera which captures the customer viewing the video and spoken voice, and acquires the captured image of the customer captured by the camera, and sends it to the server. The server derives emotional data indicating emotions for the customer's video and spoken voice based on the captured image of the customer sent from the receiver, and generates and sends processing instructions for changes of the characteristics of the operator's spoken voice, based on the result of deriving the customer's emotional data. The receiver changes and outputs the characteristics of the operator's spoken voice based on the processing instruction sent from the server.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a voice characteristic change system and a voice characteristic change method.

In Patent Document 1, an emotion model set in which a plurality of emotion scores are modeled is stored, an acoustic feature amount is extracted for each frame from the input voice signal of the interlocutor, and a frame is used from the acoustic feature amount using the emotion model set. A device for detecting an empathy / antipathy location that calculates an emotion score for each is disclosed. The empathy / antipathy location detection device calculates the empathy / antipathy location estimation score for each frame based on the calculated emotion score, and estimates the empathy / antipathy location of the interlocutor based on the empathy / antipathy location estimation score.

Japanese Unexamined Patent Publication No. 2015-99304

According to Patent Document 1, it is possible to detect a portion where the emotional state of the interlocutor has changed. However, in the technique of Patent Document 1, the focus is on estimating the emotional state when the customer speaks under the situation of the call center or the like supported by the operator, and the operator or the like is adjusted to the estimation result of the customer's emotional state. It is not considered to change the characteristics of the voice of the information provider. For this reason, it has been difficult to provide some kind of voice information that matches the customer's emotions and is easily accepted by the customer.

The present disclosure is devised in view of the above-mentioned conventional situation, and provides a voice characteristic change system and a voice characteristic change method that efficiently support the realization of voice information provision from the operator to the customer according to the customer's emotions. The purpose is to do.

The present disclosure is an audio characteristic changing system in which a receiver that receives and outputs video and an operator's utterance voice from an operator terminal and a server are communicably connected, and the receiver is the said video and the said utterance. It is connected to a camera that captures the customer who listens to the sound, acquires the captured image of the customer captured by the camera and sends it to the server, and the server sends the captured image of the customer sent from the receiver to the captured image of the customer. Based on this, emotion data indicating the customer's feelings for the video and the spoken voice is derived, and based on the result of deriving the customer's emotion data, a processing instruction regarding a change in the characteristics of the spoken voice of the operator is generated. The receiver provides a voice characteristic changing system that changes and outputs the characteristics of the spoken voice of the operator based on the processing instruction sent from the server.

Further, the present disclosure is a voice characteristic changing method executed by a voice characteristic changing system composed of a receiver composed of a receiver that receives and outputs video and an operator's utterance voice from an operator terminal, and the receiver. The customer has a camera that captures the customer viewing the video and the spoken voice, and acquires the captured image of the customer captured by the camera, and the customer sent from the receiver by the server. Based on the step of deriving the emotion data indicating the customer's feelings for the video and the uttered voice based on the captured image of the above, and the server, the uttered voice of the operator based on the result of deriving the emotion data of the customer. Based on the step of generating a processing instruction related to the change of the characteristic of the above and sending it to the receiver and the processing instruction sent from the server by the receiver, the characteristic of the spoken voice of the operator is changed and output. Provided is a method of changing voice characteristics, which has steps and.

According to the present disclosure, it is possible to efficiently support the realization of voice information provision from the operator to the customer according to the customer's emotions.

The figure which shows an example of the outline of the information display system which concerns on Embodiment 1. Block diagram showing a hardware configuration example of the information display system according to the first embodiment A flowchart showing an example of a basic operation procedure for changing voice characteristics by the information display system according to the first embodiment. A flowchart showing an example of a voice characteristic changing procedure in step S3 of FIG. A flowchart showing an example of an operation procedure by the information display system according to the first embodiment. Diagram showing an example of the registered contents of the emotion / modulation table The figure which shows an example of the outline of the TV viewing system which concerns on Embodiment 2. Block diagram showing a hardware configuration example of the TV viewing system according to the second embodiment A flowchart showing an example of an operation procedure of the TV viewing system according to the second embodiment.

Hereinafter, embodiments in which the voice characteristic changing system and the voice characteristic changing method according to the present disclosure are specifically disclosed will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
In the first embodiment, a use case in which the voice characteristic changing system according to the present disclosure is applied to the information display system shown in FIG. 1 will be described. FIG. 1 is a diagram showing an example of an outline of the information display system 5 according to the first embodiment. The information display system 5 has a configuration including a face-to-face information providing device 10, an operator terminal 50, and a server 80, and the operator op uses the operator terminal 50 to send information to the face-to-face information providing device 10 facing a customer. Present. The face-to-face information providing device 10 and the operator terminal 50 can communicate with each other various data (for example, text data, image data, voice data, or a combination of these data) via the network NW. Further, both the face-to-face information providing device 10 and the operator terminal 50 can access the server 80 connected to the network NW. As the network NW, a wired LAN (Local Area Network), a wireless LAN, a dedicated line, or the like connected to a wide area communication network such as the Internet is used.

The face-to-face information providing device 10 as an example of the receiver is placed on a box-shaped pedestal 31 having a certain height so as to match the height of a person such as a customer in front of the face-to-face information providing device 10. Will be done. The gantry 31 is integrated with the face-to-face information providing device 10 by, for example, being painted in the same color as the housing of the face-to-face information providing device 10 or being covered with a cover common to the face-to-face information providing device 10. .. The face-to-face information providing device 10 includes a first housing 15 provided so as to project upward so as to face the customer's face, and a part of the customer's hands (that is, hands, arms, palms, nails, etc.). It has a second housing 18 extending toward the front so that the periphery of the hand including the portion; the same applies hereinafter) can be contacted.

On the front surface of the first housing 15, a display unit 29 is provided on which the face of the operator op and the image of the upper body imaged by the camera 54 (see FIG. 2) of the operator terminal 50 are projected. The display unit 29 is composed of a display device (for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence)) for displaying an image. The display unit 29 displays an image of the operator op's face and upper body in a size substantially the same as the actual size. As a result, the customer can get a sense of realism as if he / she is actually facing the operator op. A camera 24 that captures images of the customer's face and upper body is provided in the lower front portion of the first housing 15 (that is, the intermediate portion 15z of the housing in which the display unit 29 and the display unit 28 are arranged). Further, a pair of left and right speakers 26 for outputting voice data of the voice spoken by the operator op are provided near both ends of the intermediate portion 15z. A microphone 27 for collecting the sound emitted by the customer is provided near the center of the intermediate portion 15z.

A display unit 28 is provided on the upper surface of the second housing 18. The display unit 28 can display a UI (User Interface) screen, guidance information such as a pamphlet, a website, and the like. The display unit 28 includes a touch panel 14 (see FIG. 2) integrated with an input unit 23 (see FIG. 2) capable of touch input operation.

On the other hand, the operator terminal 50 has an operation desk 60. In front of the operation desk 60, the operator op wearing the headset 73 sits on the chair 71 in a stable posture. The headset 73 has a speaker 55 (see FIG. 2) and a microphone 56 (see FIG. 2) as a part of the operator terminal 50, outputs the voice spoken by the customer from the speaker 55, and the voice spoken by the operator op. Is picked up by the microphone 56.

A gate-shaped support base 61 is fixed to the operation surface of the operation desk 60. The support base 61 supports a display unit 53 on which images of the customer's face and upper body imaged by the camera 24 are projected, and a camera 54 that images the face and upper body of the operator op. The image displayed on the display unit 53 is reflected by the half mirror 75 in the line-of-sight direction of the operator op, and is visually recognized by the operator op.

The face-to-face information providing device 10 receives image data from the operator terminal 50 and displays an image of the operator op's face and upper body on the display unit 29. The face-to-face information providing device 10 outputs the voice data received from the operator terminal 50 from the speaker 26, and transmits the voice data picked up by the microphone 27 to the operator terminal 50.

On the other hand, the operator terminal 50 receives the image data from the face-to-face information providing device 10 and displays the image of the customer's face and the upper body on the display unit 53. Further, the operator terminal 50 receives voice data from the face-to-face information providing device 10, outputs the voice data from the speaker 55 of the headset 73 (see FIG. 2), and collects the sound by the microphone 56 of the headset 73 (see FIG. 2). The voice data is transmitted to the face-to-face information providing device 10.

FIG. 2 is a block diagram showing a hardware configuration example of the information display system 5 according to the first embodiment. The information display system 5 includes a face-to-face information providing device 10, an operator terminal 50, and a server 80.

The face-to-face information providing device 10 is a device in which the operator op can interact with the customer via the operator terminal 50, and is a processor 21, a memory 22, a touch panel 14, a communication unit 20, a display unit 29, a camera 24, and a voice control unit 25. , Speaker 26, and microphone 27. The camera 24 and the microphone 27 may be externally connected separately from the face-to-face information providing device 10.

The processor 21 comprehensively controls the face-to-face information providing device 10. The memory 22 is used as a working memory of the processor 21, and also stores various data, information, and programs. The memory 22 includes a primary storage device (for example, RAM (Random Access Memory) and ROM (Read Only Memory). The memory 22 includes a secondary storage device (for example, HDD (Hard Disk Drive)), SSD (Solid State Drive), or SSD (Solid State Drive). A tertiary storage device (eg, an optical disk, an SD card) may be included.

The touch panel 14 has a configuration in which the display unit 28 and the input unit 23 are integrated. The display unit 28 and the input unit 23 may be configured separately. When configured separately, the display unit 28 is a display device such as an LCD or an organic EL. The input unit 23 is an input device such as a mouse, a keyboard, and a touch pad.

The communication unit 20 is a network I / F circuit that wirelessly or wiredly communicates with the communication unit 57 of the operator terminal 50 and the communication unit 83 of the server 80 via the network NW. The communication method by the communication unit 20 includes, for example, mobile communication such as WAN (Wide Area Network), LAN, LTE (Long Term Evolution), 5G, power line communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), and the like. Communication for mobile phones, etc. The communication unit 20 transmits the image data of the customer's face captured by the camera 24 and the operation information input to the input unit 23 of the touch panel 14 to the operator terminal 50. The communication unit 20 receives the image of the operator op's face and upper body transmitted from the operator terminal 50.

The display unit 29 has an ultra-high resolution display, for example, a 4K (3840 pixel × 2160 pixel) display that displays the face and upper body of the operator op. The display unit 29 may be configured with a touch panel capable of touch input by the customer.

The camera 24 is a built-in camera arranged in the lower part of the first housing 15, and captures an image of a customer's face and upper body standing in front of the face-to-face information providing device 10. The angle of view of the camera 24 may be remotely controlled from the operator terminal 50. As the camera 24, a high-resolution 4K camera, a full high-definition camera, a high-definition camera, a normal camera, and the like are used.

The voice control unit 25 performs compression / decompression processing on the voice data transmitted / received via the communication unit 20, outputs the decompressed voice data from the speaker 26, and compresses the voice data of the voice picked up by the microphone 27. do. In addition, the voice control unit 25 performs noise removal processing, amplification processing, and the like of voice data.

The speaker 26 is a stereo speaker having directivity so that the customer in front of the face-to-face information providing device 10 can easily hear the speaker 26, and outputs the voice of the voice uttered by the operator op.

The microphone 27 is a directional microphone having a directivity direction with respect to the customer, and picks up the voice of the voice spoken by the customer. The microphone 27 and the speaker 26 may be composed of a headset, and the headset is worn on the head when the customer operates the face-to-face information providing device 10.

The operator terminal 50 is a terminal operated by the operator op, and has a processor 51, a memory 52, a display unit 53, a camera 54, a speaker 55, a microphone 56, and a communication unit 57.

The processor 51 controls the operator terminal 50 in an integrated manner. The memory 52 is used as a working memory of the processor 51, and also stores various data, information, and programs. The memory 52 includes a primary storage device (eg, RAM and ROM). The memory 52 may include a secondary storage device (for example, HDD, SSD) or a tertiary storage device (for example, an optical disk, SD card).

The display unit 53 includes an ultra-high resolution display, for example, a 4K (3840 pixel x 2160 pixel) display that displays the customer's face and upper body.

The camera 54 captures an image of the operator op's face and upper body. As the camera 54, a high-resolution 4K camera, a full high-definition camera, a high-definition camera, a normal camera, or the like is used.

The communication unit 57 is a network I / F circuit that wirelessly or wiredly communicates with the communication unit 20 of the face-to-face information providing device 10 and the communication unit 83 of the server 80 via the network NW. The communication method by the communication unit 57 is, for example, mobile communication such as WAN, LAN, LTE, 5G, power line communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), communication for mobile phones, and the like. The communication unit 57 transmits the image data of the operator op's face captured by the camera 54 to the face-to-face information providing device 10. The communication unit 57 receives the image of the customer's face and the upper body transmitted from the face-to-face information providing device 10. The communication unit 57 receives the advice information corresponding to the customer's emotions transmitted from the server 80.

The speaker 55 is a stereo speaker having directivity so that the operator op can easily hear it, and outputs the voice of the voice uttered by the customer. The microphone 56 is a directional microphone having a directivity direction with respect to the operator op, and collects the voice of the voice spoken by the operator op. The microphone 56 and the speaker 55 are composed of a headset 73. The operator op wears the headset 73 on the head when operating the operator terminal 50.

The server 80 changes the characteristics of the voice of the voice spoken by the operator op according to the emotion data of the customer, and has a processor 81, a memory 82, a communication unit 83, and a storage 85. The emotional data shows the customer's emotions toward the video and the spoken voice. Here, changing the characteristics of voice is also referred to as "modulation".

The processor 81 includes a modulation method determination unit 91 and an emotion analysis algorithm 92 as functions realized by executing a program stored in the memory 82. The emotion analysis algorithm 92 estimates the customer's emotion, and the image analysis unit 93 that estimates the customer's emotion based on the customer's face image data, and the customer's emotion based on the voice data of the voice spoken by the customer. Includes a voice analysis unit 94 that estimates. The emotion analysis algorithm 92 may output a time stamp when the customer's emotion is estimated.

The modulation method determination unit 91 modulates the voice corresponding to the estimated customer's emotion using the emotion / modulation table Tb1 registered in the emotion database 95 based on the customer's emotion estimated by the emotion analysis algorithm 92. Select a method.

The memory 82 is used as a working memory of the processor 81, and also stores various data, information, and programs. The memory 82 includes a primary storage device (eg, RAM and ROM).

The communication unit 83 is a network I / F circuit that wirelessly or wiredly communicates with the communication unit 20 of the face-to-face information providing device 10 and the communication unit 57 of the operator terminal 50 via the network NW. The communication method by the communication unit 83 is, for example, mobile communication such as WAN, LAN, LTE, 5G, power line communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), communication for mobile phones, and the like. The communication unit 83 transmits advice information corresponding to the customer's emotions to the operator terminal 50. The communication unit 83 receives the image of the customer's face and the upper body transmitted from the face-to-face information providing device 10, and transmits the method of modulating the voice of the voice spoken by the operator op to the face-to-face information providing device 10.

The storage 85 includes an HDD or SSD and stores an emotion database 95. The emotion database 95 includes an emotion / modulation table Tb1 (see FIG. 6) in which a method of modulating the customer's emotion and the voice of the operator op is registered.

Next, the operation procedure of the information display system 5 according to the first embodiment will be described.

First, the basic operation of changing the voice characteristics will be described. As an example, it is assumed that the operator op transmits voice data to the face-to-face information providing device 10 that the customer watches through the operator terminal 50 and conveys things to the customer by voice. FIG. 3 is a flowchart showing an example of a basic operation procedure for changing the voice characteristics by the information display system 5 according to the first embodiment.

In FIG. 3, the server 80 acquires customer voice data and image data from the face-to-face information providing device 10 (S1). The server 80 estimates the customer's emotions based on the customer's voice data and image data (S2). The server 80 gives an instruction to change the characteristics of the voice of the voice spoken by the operator op according to the estimated customer's emotion. The face-to-face information providing device 10 modulates and outputs the voice of the voice spoken by the operator op according to the instruction from the server 80 (S3). Details of step S3 will be described later with reference to FIG.

FIG. 4 is a flowchart showing an example of a voice characteristic changing procedure in step S3 of FIG. The series of processes shown in FIG. 4 is a subroutine showing details of the voice characteristic changing procedure in step S3 of FIG.

In FIG. 4, the server 80 identifies the characteristics of the biometric information when the customer's emotions estimated in step S2 of FIG. 3 change (for example, when the customer suddenly becomes angry) (S31). As a feature of the biological information, the server 80 performs face recognition based on the image data acquired in step S1 of FIG. 3, and detects the surface emotions of emotions and emotions appearing in the customer's face image. Further, as a feature of the biological information, the heart rate or heart rate variability data derived by the server 80 based on the customer's face image data acquired in step S1 of FIG. 3 may be used. As a technique for analyzing internal emotions (particularly, the degree of stress) based on heart rate variability, for example, in Japanese Patent No. 6358506, an image data captured by a subject is input, and skin color over a plurality of frames of the input image data is input. It is disclosed that the pulse rate is estimated based on the period of the pixel value of the portion. Similarly, according to the International Publication No. 2017/154477, the skin color region is detected from the captured image, the pulse wave signal is detected based on the information extracted from the skin color region, and the pulse of the subject is estimated based on the pulse wave signal. Is disclosed. Further, as biometric information, as shown in Patent Document 1, it is known to estimate a customer's emotion by using voice data of a voice spoken by the customer.

The processor 81 of the server 80 searches for the features of the biometric information similar to the features of the identified biometric information based on the emotion database 95 stored in the storage 85 (S32). In the emotion database 95, features of biometric information corresponding to emotions or changes in emotions are registered. The biological information includes at least one facial expression such as emotions, heart rate, heart rate variability, and voice.

When the features of the biometric information are found as a result of searching the emotion database 95, the processor 81 is a method of modulating the voice of the voice corresponding to the features of the biometric information based on the emotion / modulation table Tb1 registered in the emotion database 95. Is selected (S33). The processor 81 transmits a voice modulation method corresponding to the characteristics of biometric information to the face-to-face information providing device 10 via the communication unit 83.

The face-to-face information providing device 10 modulates and outputs the voice of the operator op transmitted from the operator terminal 50 according to the voice modulation method (S34).

Next, the voice characteristic changing operation in the information display system 5 will be shown more concretely. FIG. 5 is a flowchart showing an operation procedure of the information display system 5 according to the first embodiment. Similar to FIG. 3, it is assumed that the operator op transmits voice data to the face-to-face information providing device 10 that the customer watches through the operator terminal 50 and conveys things to the customer by voice.

In FIG. 5, the server 80 receives and acquires the customer's voice data and image data transmitted from the face-to-face information providing device 10 via the communication unit 83 (S41).

The emotion analysis algorithm 92 of the processor 81 estimates the customer's emotions based on the customer's voice data and image data (S42). At this time, the image analysis unit 93 performs face recognition based on the image data and estimates the surface emotions of emotions appearing in the customer's face image. In addition, the image analysis unit 93 detects the heart rate variability based on the face image data and estimates the inner feelings of the customer. Further, the voice analysis unit 94 estimates emotions such as empathy and antipathy of the customer based on the voice of the voice spoken by the customer.

The modulation method determination unit 91 of the processor 81 creates an instruction for changing the voice characteristic of the voice of the operator op according to the estimated customer's emotion (S43). In creating this instruction, the modulation method determination unit 91 selects a voice modulation method corresponding to the estimated emotion based on the emotion / modulation table Tb1 registered in the emotion database 95. FIG. 6 is a diagram showing an example of the registered contents of the emotion / modulation table Tb1. In the emotion / modulation table Tb1, when the customer's emotion is "normal", "no modulation" of the voice spoken by the operator is registered. When the customer's emotion is "joy", "no modulation" of the voice spoken by the operator is also registered. When the customer's emotion is "anger", the operator speaks "lower the pitch of the end of the word. Decrease the volume according to the degree of anger and change at least one of the lengths of the sound. The greater the degree of anger, the higher the volume. To make the sound louder and the length of the sound longer. ”Is registered. When the customer's emotion is "worry", the voice "strengthen the voice and encourage the purchase or contract" spoken by the operator is registered.

In the emotion / modulation table Tb1, voice data is illustrated as an example of estimating the emotion of the customer, but the emotion may be estimated by combining data such as heart rate and heart rate variability. In addition, when the heart rate variability is combined, the heart rate variability becomes stable in a concentrated state and unstable in a relaxed state. Further, in the emotion / modulation table Tb1, when the emotion estimated by the emotion analysis algorithm 92 is changed from "joy" to "sadness", the operator's voice is registered so as to slow down the utterance speed and lower the pitch. You may. Also, when the estimated emotion is changed from "anger" to "excitement", the operator's voice may be registered to lower the emphasis of a particular word that incites excitement and raise the pitch.

The processor 81 transmits the instruction for changing the voice characteristic of the operator's voice created in step S43 to the face-to-face information providing device 10 via the communication unit 83 (S44).

When the processor 21 of the face-to-face information providing device 10 receives the above instruction via the communication unit 20, the processor 21 modulates the voice of the operator op by the instructed modulation method and outputs it (S45).

Further, the processor 81 of the server 80 transmits the estimated customer emotion data to the operator terminal 50 via the communication unit 83. The processor 51 of the operator terminal 50 displays the customer's facial expression based on the estimated customer's emotion on the display unit 53 (S46). At this time, the processor 51 may select one of the face icons of various emotions registered in the memory 52, for example, and display the facial expression of the customer. Further, the processor 51 may display the facial expression of the customer with text characters, mark images, and the like.

The processor 81 of the server 80 transmits an operator's utterance, for example, advice on whether or not to continue the business of the product currently being introduced, to the operator terminal 50 based on the estimated customer's feelings. The processor 51 of the operator terminal 50 displays this advice on the display unit 53 (S47). For example, if there is customer anger that exceeds expectations (it does not have to exceed expectations), advice may be given to stop the continuation of business. As an example, the processor 81 of the server 80 registers phrases (text data) such as "Bakayaro" and "Come out!" In the memory 82, and assumes the content to be spoken by the customer. If the wording exceeds the limit, we will give advice to stop the continuation of business. The processor 81 may perform machine learning based on data such as estimated customer emotions, customer face images, and customer voice voices, and give advice on whether or not to stop the continuation of business. Here, the advice is determined by the server 80, but may be determined by the operator terminal 50. When the operator terminal 50 does this, the server 80 transmits data representing the estimated customer emotions to the operator terminal 50.

The information display system 5 according to the first embodiment is a case where the operator op transmits voice data of the voice uttered by the operator to the face-to-face information providing device 10 that the customer watches through the operator terminal 50, and conveys things to the customer by voice. , The voice characteristics of the voice spoken by the operator op are changed according to the estimated customer's feelings. As a result, information can be smoothly and efficiently provided to the customer by voice from the operator op according to the customer's emotions.

In this way, in the information display system 5, the face-to-face information providing device 10 that receives and outputs the video and the utterance voice of the operator op from the operator terminal 50 and the server 80 are communicably connected to each other. The face-to-face information providing device 10 is connected to or has a camera 24 that captures a customer who views video and spoken audio, and acquires a captured image of the customer captured by the camera 24 and sends it to the server 80. The server 80 derives emotional data indicating emotions for the customer's video and spoken voice based on the captured image of the customer sent from the face-to-face information providing device 10. Based on the result of deriving the customer's emotion data, the server 80 generates a processing instruction regarding the change in the characteristics of the spoken voice of the operator op and sends it to the face-to-face information providing device 10. The face-to-face information providing device 10 changes the characteristics of the utterance voice of the operator op and outputs it based on the processing instruction sent from the server 80.

As a result, the face-to-face information providing device 10 can adaptively change and output the characteristics of the operator's voice according to the emotions of the customer who has viewed the operator's video. Therefore, the information display system 5 can efficiently support the realization of voice information provision from the operator to the customer according to the customer's emotions.

Further, the face-to-face information providing device 10 is connected to or has a microphone 27 that collects the customer's uttered voice, and acquires the customer's uttered voice collected by the microphone 27 and sends it to the server 80. The server 80 derives the customer's emotion data based on at least one of the customer's captured image and the customer's utterance voice sent from the face-to-face information providing device 10. As a result, the server 80 can easily estimate the customer's emotional data based on the customer's captured image or the customer's utterance voice.

Further, when the server 80 determines that the customer's emotional data indicates anger, the server 80 generates a processing instruction to lower the pitch of the ending portion of the spoken voice of the operator op. As a result, the face-to-face information providing device 10 can lower the pitch of the ending portion of the utterance voice of the operator op so that the customer's anger is calmed down.

Further, when the server 80 determines that the customer's emotional data indicates anger (for example, anger exceeding the expected range), the server 80 generates advice information for urging the operator op to stop continuing the utterance and transmits it to the operator terminal 50. .. The operator terminal 50 receives and displays this advice information. As a result, the operator op can stop the utterance that reverses the customer's anger and wait until the customer's anger subsides.

Further, when it is determined that the customer's emotional data indicates trouble, the server 80 generates a processing instruction to increase the volume of the utterance voice of the operator op. As a result, the face-to-face information providing device 10 can increase the volume of the utterance voice of the operator op, that is, strengthen the vocabulary and encourage the purchase or contract. In addition, the face-to-face information providing device 10 can solve problems and guide customers to regain their energy.

Further, the server 80 derives the customer's emotional data based on both the customer's captured image and the customer's utterance voice sent from the face-to-face information providing device 10. This allows the server 80 to more accurately estimate emotional data using both the customer's captured image and the customer's utterance voice.

In addition, the face-to-face information providing device 10 supports a dialogue between the customer and the operator op. As a result, the face-to-face information providing device 10 changes the characteristics of the utterance voice of the operator op according to the emotions of the customer, so that the customer can enhance the sense of presence as if he / she is directly interacting with the operator.

(Embodiment 2)
In the second embodiment, a use case in which the audio characteristic changing system according to the present disclosure is applied to the TV viewing system shown in FIG. 7 will be described. In a TV viewing system, for example, a customer is a viewer who watches sports (baseball, sumo, etc.) through a TV (television receiver). An operator is a commentator who plays a sport. Here, the sports video spoken by the live broadcaster is a live video, but may be a recorded video.

FIG. 7 is a diagram showing an example of an outline of the TV viewing system 500 according to the second embodiment. The TV viewing system 500 is a viewer vw's feelings of watching a TV program output by the TV 100 with respect to a plurality of television receivers (hereinafter, simply referred to as TVs) placed in each home, business office, or the like. In accordance with this, the characteristics of the voice of the voice spoken by the commentator As are changed for each TV 100 and output. FIG. 7 shows, as an example, a case where the same TV program is viewed by the same live broadcaster at three home HAs, HBs, and HCs. Here, the TV program is distributed to each TV 100 via the network NW, but may be broadcast to each TV by bidirectional communication using a digital broadcast wave.

FIG. 8 is a block diagram showing a hardware configuration example of the TV viewing system 500 according to the second embodiment. In the TV viewing system 500 according to the second embodiment, the same or corresponding reference numerals are used for the same components as the information display system 5 according to the first embodiment, thereby omitting or simplifying the description and discussing different contents. explain.

The TV viewing system 500 includes a plurality of TVs 100, a server 180, and a live broadcaster terminal 150. Each TV 100, the server 180, and the commentator terminal 150 are connected to the network NW and can perform data communication with each other.

The plurality of TV 100s are installed in places such as homes and business establishments, and output TV programs such as sports received from the live broadcaster terminal 150 via the network NW as video and audio. The TV 100 includes a processor 121, a memory 122, a display unit 128, a communication unit 120, a camera 124, a voice control unit 125, a speaker 126, and a microphone 127. The camera 124 and the microphone 127 may be externally connected separately from the TV 100.

The processor 121 controls the TV 100 in an integrated manner. The memory 122 is used as a working memory of the processor 121, and also stores various data, information, and programs. The memory 122 includes a primary storage device (eg, RAM and ROM). The memory 122 may include a secondary storage device (for example, HDD, SSD) or a tertiary storage device (for example, an optical disk, SD card).

The communication unit 120 is a network I / F circuit that wirelessly or wiredly communicates with the communication unit 157 of the commentator terminal 150 and the communication unit 183 of the server 180 via the network NW. The communication method by the communication unit 120 is, for example, mobile communication such as WAN, LAN, LTE, 5G, power line communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), communication for mobile phones, and the like. The communication unit 120 transmits the image data of the face of the viewer vw captured by the camera 124 to the server 180. The communication unit 120 receives the image of the face and upper body of the commentator as transmitted from the commentator terminal 150.

The display unit 128 is a display device such as an LCD or an organic EL. The display unit 128 displays a TV program such as sports, and displays the face and upper body of the live broadcaster as on the wipe screen. The display unit 128 has an ultra-high resolution display, for example, a 4K (3840 pixel × 2160 pixel) display.

The camera 124 is arranged on the front surface of the housing of the TV 100, and captures an image of the face and upper body of the viewer vw to be viewed in a living room or the like in the home. As the camera 124, a high-resolution 4K camera, a full high-definition camera, a high-definition camera, a normal camera, or the like is used.

The voice control unit 125 performs compression / decompression processing on the voice data transmitted / received via the communication unit 120, outputs the decompressed voice data from the speaker 126, and compresses the voice data of the voice picked up by the microphone 127. do. In addition, the voice control unit 125 performs noise removal processing, amplification processing, and the like of voice data.

The speaker 126 is a stereo speaker having directivity so that the viewer vw in front of the TV 100 can easily hear it, and outputs the voice of the voice uttered by the commentator as. The microphone 127 is a directional microphone having a directivity direction with respect to the viewer vw, and collects the sound of the voice spoken by the viewer vw.

Further, the commentator terminal 150 is a terminal that broadcasts a TV program such as sports, and has a processor 151, a memory 152, a camera 154, a microphone 156, and a communication unit 157.

The processor 151 comprehensively controls the live broadcaster terminal 150. The memory 152 is used as a working memory of the processor 151, and also stores various data, information, and programs. The memory 152 includes a primary storage device (eg, RAM and ROM). The memory 152 may include a secondary storage device (for example, HDD, SSD) or a tertiary storage device (for example, an optical disk, SD card).

The camera 154 captures an image of the face and upper body of the commentator as. As the camera 154, a high-resolution 4K camera, a full high-definition camera, a high-definition camera, a normal camera, or the like is used.

The communication unit 157 is a network I / F circuit that wirelessly or wiredly communicates with the communication unit 120 of the TV 100 via the network NW. The communication method by the communication unit 157 is, for example, mobile communication such as WAN, LAN, LTE, 5G, power line communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), communication for mobile phones, and the like. The communication unit 157 transmits the image data of the face and upper body of the commentator as captured by the camera 154 to the TV 100.

The microphone 156 is a directional microphone having a directivity direction with respect to the live broadcaster as, and picks up the voice of the voice spoken by the live broadcaster as.

The server 180 changes the characteristics of the voice of the voice spoken by the commentator As according to the emotional data of the viewer vw who watches the TV program, and changes the processor 181, the memory 182, the communication unit 183, and the storage 185. Has.

The processor 181 includes a modulation method determination unit 191 and an emotion analysis algorithm 192 as functions realized by executing a program stored in the memory 182. The sentiment analysis algorithm 192 estimates the emotions of the viewer vw based on the image analysis unit 193 that estimates the emotions of the viewer vw based on the face image data of the viewer vw, and the voice data of the voice spoken by the viewer vw. Includes a voice analysis unit 194. The modulation method determination unit 191 selects a voice modulation method corresponding to the estimated emotion of the viewer vw based on the emotion / modulation table Tb2 registered in the emotion database 195. The emotion / modulation table Tb2 includes the same registration contents as the emotion / modulation table Tb1 in the first embodiment. For example, when the viewer's emotion is "joy", the modulation of the commentator's voice is "to raise the pitch of the sound to make it louder so that the place becomes an excited atmosphere". Also, when the viewer's emotions are "disappointing", the modulation of the commentator's voice is "lower the pitch of the sound to a lower volume so that the place becomes a sunken atmosphere".

The memory 182 includes a primary storage device (eg, RAM and ROM). The memory 182 may include a secondary storage device (for example, HDD, SSD) or a tertiary storage device (for example, an optical disk, SD card).

The communication unit 183 is a network I / F circuit that wirelessly or wiredly communicates with the communication unit 120 of the TV 100 via the network NW. The communication method by the communication unit 183 is, for example, mobile communication such as WAN, LAN, LTE, 5G, power line communication, short-range wireless communication (for example, Bluetooth (registered trademark) communication), communication for mobile phones, and the like. The communication unit 183 receives the video of the viewer vw's face and upper body transmitted from the TV 100, and transmits to the TV 100 a method of modulating the voice of the voice spoken by the commentator as.

The storage 185 includes an HDD or SSD and stores an emotion database 195. The emotion database 95 includes an emotion / modulation table Tb2 in which the modulation method of the emotion of the viewer vw and the voice of the commentator as is registered. The registered contents of the emotion / modulation table Tb2 are the same as those of the emotion / modulation table Tb1 according to the first embodiment.

Next, an example of the operation procedure of the TV viewing system 500 according to the second embodiment will be described.

FIG. 9 is a flowchart showing an example of an operation procedure of the TV viewing system 500 according to the second embodiment.

In FIG. 9, the processor 181 of the server 180 is the face image data of each viewer vw by the camera 124 transmitted from the TV 100 placed in each home HA, HB, HC via the communication unit 183 and the network NW. The voice data of the voice of each viewer vw by the microphone 127 is received and acquired (S61).

The emotion analysis algorithm 192 of the processor 181 estimates the emotion of each viewer vw based on the face image data and the voice data of each viewer vw (S62). The image analysis unit 193 performs face recognition based on the image data, and estimates the surface emotions of emotions appearing in the face image of each viewer vw. In addition, the image analysis unit 193 detects the heart rate variability based on the face image data and estimates the inner emotion of the viewer vw. Further, the voice analysis unit 194 estimates emotions such as empathy or antipathy of the viewer vw based on the voice of the voice spoken by the viewer vw.

The processor 181 calculates the characteristics of the voice data with respect to the voice of the voice spoken by the live broadcaster from the live broadcaster terminal 150 into the microphone 156 (S63). The characteristics of voice data include, for example, pitch, volume, timbre, and the like. The processor 181 selects (matches) the characteristics of the voice data of the commentator's voice corresponding to (matching) the estimated emotion of the viewer vw based on the emotion / voice feature table (not shown) registered in the emotion database 195. (S64). The processor 181 may perform machine learning on the characteristics of the voice of the commentator corresponding to the estimated emotion of the viewer vw, and may specify the characteristics of the voice of the commentator using the obtained learned model.

The processor 181 of the server 180 creates an instruction (advice) for each TV 100 to change to the characteristics of the voice data of the selected commentator, and transmits the instruction (advice) to each TV 100 (S65). The communication unit 120 of each TV 100 receives an instruction from the server 180. The voice control unit 125 of each TV 100 modulates the voice of the voice spoken by the commentator as according to the instruction of the server 180 (S66). When a plurality of viewers vw are watching the TV 100 in the home, the processor 181 has a predetermined emotion (for example, the most emotions among all the viewers' emotions) as the emotions of the plurality of viewers vw. , Elderly emotions, averaged emotions, etc.) may be modulated. As a result, when a plurality of viewers are watching the same TV, it is possible to make an announcement by a live broadcaster that matches the emotions of the plurality of viewers as much as possible.

In the TV viewing system 500 according to the second embodiment, for example, when the TV broadcasts a baseball broadcast and the D team is approaching victory, in the home HA, the TV is a viewer supporting the D team. On the other hand, the characteristics of the voice of the commentator are changed so that the place becomes an excited atmosphere. On the other hand, in the home HB, the TV changes the characteristics of the voice of the commentator to the viewer who does not support the D team so that the place becomes a sunken atmosphere. In each home, the TV can change the audio characteristics to match the emotions of the viewer and the voice of the commentator.

In this way, the TV 100 is arranged in the home. As a result, the TV viewing system 500 can broadcast a smooth and comfortable live broadcast by the live broadcaster as to the viewer vw according to the emotions of the viewer vw.

Further, for each of the plurality of TVs 100, the server 80 generates a processing instruction regarding the change in the characteristics of the utterance voice of the commentator as for each TV 100 and sends it to each TV 100. As a result, the TV viewing system 500 can broadcast the live audio by the live broadcaster as to a plurality of homes with different audio characteristics for each home.

Further, when there are a plurality of viewers vw who view the video and the spoken sound output from the TV 100, the live broadcaster is based on the derivation result of predetermined emotion data such as the most emotions among the emotions of the plurality of viewers. Generates processing instructions for changing the characteristics of as spoken voice. As a result, the TV viewing system 500 provides a smooth and comfortable live broadcast by the live broadcaster according to the emotions of the viewer vw as much as possible even when there are a plurality of viewers vw viewing one TV 100. It can be broadcast to the viewer vw.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications, modifications, substitutions, additions, deletions, and equality within the scope of the claims. It is understood that it naturally belongs to the technical scope of the present disclosure. Further, each component in the various embodiments described above may be arbitrarily combined as long as the gist of the invention is not deviated.

For example, in the above-described embodiment, the case where the voice characteristic changing system is applied to the information display system 5 and the TV viewing system 400 is shown, but the present invention is not limited to these, and the teacher and the student interact in the communication tutor service. In this case, it can be applied in various fields such as when the president makes a presentation to multiple employees in a video conferencing system. Operators also include sports game commentators and TV presenters.

Further, the estimation of voice emotions based on voice data, the estimation of internal emotions based on heart rate variability data, and the estimation of empathy may be performed by an algorithm using machine learning.

Further, in the above-described embodiment, a camera for obtaining image data and a microphone for obtaining audio data are used, but smart wearable devices such as smart watches and wristbands worn by customers, viewers, etc. are used. Then, data of both the voice spoken by voice and the heart sound (heartbeat signal) may be acquired. By using a smart wearable device, it is possible to further obtain vital data such as blood pressure and blood glucose level, which can be reflected in emotion estimation.

The present disclosure is a voice characteristic change system and a voice characteristic change method that efficiently support the realization of a smooth and comfortable dialogue between the operator and the customer according to the customer's feelings by changing the characteristics of the spoken voice by the operator. It is useful as.

5 Information display system 10 Face-to-face information providing device 50 Operator terminal 80 Server 81 Processor 82 Memory 83 Communication unit 85 Storage 91 Modulation method determination unit 92 Sentiment analysis algorithm 93 Image analysis unit 94 Voice analysis unit 95 Sentiment database 500 TV viewing system

Claims (11)

It is a voice characteristic change system in which a receiver that receives and outputs video and operator's utterance voice from an operator terminal and a server are connected so as to be able to communicate.
The receiver
It is connected to a camera that captures the customer who views the video and the spoken voice, acquires the captured image of the customer captured by the camera, and sends it to the server.
The server
Based on the captured image of the customer sent from the receiver, emotional data indicating emotions of the customer for the video and the spoken voice is derived.
Based on the result of deriving the emotional data of the customer, a processing instruction regarding a change in the characteristics of the spoken voice of the operator is generated and sent to the receiver.
The receiver
Based on the processing instruction sent from the server, the characteristics of the utterance voice of the operator are changed and output.
Voice characteristic change system. The receiver
It is connected to a microphone that picks up the customer's uttered voice, acquires the customer's uttered voice picked up by the microphone, and sends it to the server.
The server
The emotional data of the customer is derived based on the captured image of the customer or the spoken voice of the customer sent from the receiver.
The voice characteristic changing system according to claim 1. The server
When it is determined that the emotional data of the customer indicates anger, the processing instruction for lowering the pitch of the ending portion of the spoken voice of the operator is generated.
The voice characteristic changing system according to claim 1. The server
When it is determined that the emotional data of the customer indicates anger, advice information for urging the operator to stop the continuation of the utterance is generated and transmitted to the operator terminal.
The operator terminal is
Receives and displays the advice information sent from the server.
The voice characteristic changing system according to claim 1. The server
When it is determined that the emotional data of the customer indicates trouble, the processing instruction to increase the volume of the utterance voice of the operator is generated.
The voice characteristic changing system according to claim 1. The server
The emotional data of the customer is derived based on both the captured image of the customer and the spoken voice of the customer sent from the receiver.
The voice characteristic changing system according to claim 2. The receiver is a face-to-face information providing device that supports dialogue with the operator.
The voice characteristic changing system according to any one of claims 1 to 6.
The receiver is a television receiver arranged in the home.
The voice characteristic changing system according to any one of claims 1 to 5. At least one receiver is arranged in each of the plurality of homes.
The server generates, for each receiver in the home, different processing instructions regarding changes in the characteristics of the spoken voice of the operator and sends them to the corresponding receivers.
The voice characteristic changing system according to claim 8. The receiver
When there are a plurality of customers who view the video and the utterance voice output from the receiver, a processing instruction regarding a change in the characteristics of the utterance voice of the operator is generated based on the derivation result of the predetermined emotion data. do,
The voice characteristic changing system according to claim 8. It is a voice characteristic change method executed by a voice characteristic change system composed of a receiver and a server that receives and outputs video and operator's utterance voice from an operator terminal.
A step of having a camera that captures a customer who views the video and the spoken voice by the receiver and acquiring a captured image of the customer captured by the camera.
A step of deriving emotional data indicating emotions of the customer for the video and the spoken voice based on the captured image of the customer sent from the receiver by the server.
A step of generating a processing instruction regarding a change in the characteristics of the spoken voice of the operator based on the result of deriving the emotion data of the customer by the server and sending the processing instruction to the receiver.
The receiver has a step of changing and outputting the characteristics of the utterance voice of the operator based on the processing instruction sent from the server.
How to change voice characteristics.

Images