JP2023131824A - Information processing device, control program, and control method - Google Patents

Abstract

To provide an information processing device, a control program, and a control method that make it easy to talk with a user who is a dialog partner.SOLUTION: In an information processing system in which a user-side terminal communicably connects to an operator-side terminal and a server, the operator-side terminal includes a CPU. The operator looks at an avatar image 210 which is shown to a display device and which corresponds to a user who is a dialog partner, and talks with a user who uses the communicably connected user-side terminal. When the operator is not speaking, the operator-side terminal recognizes the expression of the user and, when the recognized expression is a smiling or a displeased expression, an avatar image having been represented by favorably changing the same expression is displayed. Meanwhile, when the operator is speaking, the operator-side terminal recognizes the expression of the operator and, when the recognized expression is a smiling or a sad expression, an avatar image in which the same expression is represented with a preset level of expression is displayed.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing device, a control program, and a control method, and more particularly to an information processing device, a control program, and a control method used by an operator who interacts by voice with a user of a communicably connected terminal.

An example of this type of conventional information processing device is disclosed in Patent Document 1. In the information processing system disclosed in Patent Document 1, when a user wants to consult about a product or how to use a shopping site, the user can press a call button to call and consult an operator. It is. When the user terminal and the operator terminal are connected, the website displayed on the user terminal is displayed on the operator terminal in the current display mode. Furthermore, an image of the operator or an avatar image synchronized therewith is displayed on the website on the user terminal. Therefore, the operator serves the user while using gestures and gestures.

Patent No. 6937534

In Patent Document 1 mentioned above, screens that are not intended by the user, such as website screens other than the shopping site screen and other personal information, are not reflected on the operator terminal. In other words, the user's image or the avatar image synchronized therewith is not displayed on the operator terminal, and the operator cannot see the face of the user or the avatar synchronized therewith, making it difficult to interact with the operator. For this reason, it is conceivable to display the user's image on the operator terminal, but there is a risk that the user's privacy may be violated. It is also possible to display an avatar image synchronized with the user's image, but there is still room for improvement in order to make it easier for the operator to interact.

Therefore, the main object of the present invention is to provide a novel information processing device, control program, and control method.

Another object of the present invention is to provide an information processing device, a control program, and a control method that make it easier to interact with the other party.

The first invention detects the operator's voice using a voice detecting means for detecting the operator's voice, which is the voice uttered by the operator who interacts with the user, a photographing means for photographing the face image of the operator, and a voice detecting means. a first recognition means for recognizing the operator's facial expression based on a facial image of the operator photographed by the photographing means; the facial expression of the operator recognized by the first recognition means is a predetermined first facial expression; , an operation detected by the avatar display means for displaying on a display device an image of an avatar corresponding to the user who is the other party of the dialogue, expressing the same facial expression as the predetermined first facial expression, and the voice detection means. The information processing apparatus includes a transmitting means for transmitting user voice to a user terminal used by the user.

A second invention is dependent on the first invention, and the predetermined first facial expressions are a smiling facial expression and a sad facial expression.

A third invention is dependent on the first or second invention, and provides a timing judgment for determining whether or not it is time to nod based on the operator's voice when the operator's voice is detected by the voice detection means. The apparatus further includes avatar control means for causing the avatar to perform a nodding motion when the timing determining means determines that it is time to nod.

A fourth invention is dependent on any one of the first to third inventions, and detects operator voice by a receiving means for receiving a user's face image transmitted from a user side terminal and a voice detecting means. The avatar display means further includes a second recognition means for recognizing the user's facial expression based on the user's facial image received by the receiving means when the avatar display means recognizes the facial expression of the user recognized by the second recognition means. When the facial expression of is a predetermined second facial expression, the same facial expression as the predetermined second facial expression is changed favorably by the operator and the avatar is made to express the same facial expression.

A fifth invention is dependent on the fourth invention, wherein the predetermined second expression is a smiling expression and a displeased expression, and the second recognition means further determines the degree of the user's smiling expression and displeased expression. The avatar display means emphasizes the degree of smile and causes the avatar to express it when the user's facial expression is a smiling expression, and reduces the degree of displeasure when the user's facial expression is a displeased expression. and have the avatar express it.

A sixth invention is dependent on any one of the first to third inventions, and further comprises a receiving means for receiving the user's facial expression transmitted from the user side terminal, and the avatar displaying means is a voice detecting means. If the user's facial expression received by the receiving means is a predetermined second facial expression when the operator's voice is not detected by Change it and let the avatar express it.

A seventh invention is dependent on the sixth invention, wherein the predetermined second expression is a smiling expression and a displeased expression, and the receiving means further receives the degree of the user's smiling expression and displeased expression. When the user's facial expression is a smiling expression, the avatar display means emphasizes the degree of smile and causes the avatar to express it, and when the user's facial expression is a displeased expression, the avatar display means reduces the degree of displeasure. Let the avatar express it.

An eighth invention is dependent on any one of the first to fifth aspects, further comprising a line of sight detection means for detecting the user's line of sight based on the user's face image received by the receiving means, and avatar display means. sets the avatar's line of sight to match the user's line of sight detected by the line of sight detection means, and when the voice detection means does not detect the operator's voice, the user's line of sight detected by the line of sight detection means When the avatar faces forward for a first predetermined time period, the avatar's line of sight is averted for a second predetermined time period, regardless of the user's line of sight.

A ninth invention is dependent on the sixth or seventh invention, wherein the receiving means further receives the user's line of sight, and the avatar display means sets the avatar's line of sight in accordance with the received user's line of sight, When the voice detecting means does not detect the operator's voice, if the time during which the user's line of sight detected by the line of sight detection means is facing forward has passed the first predetermined time, regardless of the user's line of sight, Avert the avatar's line of sight for a second predetermined period of time.

A tenth invention is a control program executed by an information processing device, which includes a voice detection step for detecting an operator voice, which is a voice uttered by an operator interacting with a user, in a processor of the information processing device, and an operation. a recognition step for recognizing the facial expression of the operator based on the facial image of the operator photographed in the photographing step when the operator's voice is detected in the photographing step of photographing a facial image of the operator; an avatar displaying step of displaying, when the facial expression of the operator recognized in step is a predetermined facial expression, an image of an avatar that expresses the same facial expression as the predetermined facial expression and corresponds to the user who is the other party of the dialogue; This is a control program that executes a transmitting step of transmitting the operator's voice detected in the voice detecting step to a user terminal used by the user.

An eleventh invention is a method for controlling an information processing device, comprising: (a) detecting operator voice, which is voice uttered by an operator interacting with a user; (b) photographing a face image of the operator. (c) when the operator voice is detected in step (a), recognizing the facial expression of the operator based on the facial image of the operator photographed in the photographing step; (d) step (c) If the facial expression of the operator recognized in ) is a predetermined facial expression, displaying on a display device an image of an avatar corresponding to the user who is the other party of the interaction and expressing the same facial expression as the predetermined facial expression; e) A control method including the step of transmitting the operator voice detected in step (a) to a user-side terminal used by a user.

According to this invention, it is possible to facilitate dialogue with the user who is the other party of the dialogue.

The above objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is a diagram showing an information processing system according to an embodiment of the present invention. FIG. 2 is a block diagram showing the electrical configuration of the user terminal shown in FIG. 1. FIG. 3 is a block diagram showing the electrical configuration of the operator side terminal shown in FIG. 1. FIG. 4 is a diagram showing an example of a screen displayed on a display device of a user-side terminal. FIG. 5 is a diagram showing an example of an avatar image displayed on the display device of the operator side terminal. FIG. 6(A) is a diagram showing an example of an avatar's smiling expression with different degrees, FIG. 6(B) is a diagram showing an example of a displeased expression of an avatar with different degrees, and FIG. 6(C) is a diagram showing an example of a displeased expression of an avatar with different degrees. FIG. 3 is a diagram showing an example of sad facial expressions of different avatars. FIG. 7 is a diagram showing an example of a memory map of the RAM of the operator side terminal. FIG. 8 is a diagram showing an example of specific contents of the data storage area shown in FIG. 7. FIG. 9 is a flow diagram showing a part of an example of control processing of the CPU of the operator side terminal shown in FIG. FIG. 10 is another part of the example of the control process of the CPU of the operator side terminal shown in FIG. 3, and is a flow diagram subsequent to FIG. FIG. 11 is another part of the example of the control process of the CPU of the operator terminal shown in FIG. 3, and is a flow diagram subsequent to FIG. FIG. 12 is a flowchart showing still another part of the control process of the CPU of the operator terminal shown in FIG. 3, and is subsequent to FIGS. 10 and 11. FIG. 13 is a flowchart showing an example of the transmission/reception processing of the CPU of the operator side terminal shown in FIG. FIG. 14 is a flow diagram showing an example of the transmission/reception processing of the CPU of the user terminal shown in FIG. FIG. 15 is a flow diagram showing part of the control processing of the CPU of the operator side terminal in the second embodiment. FIG. 16 is a flow diagram showing the transmission and reception processing of the CPU of the user terminal in the second embodiment.

<First example>
Referring to FIG. 1, an information processing system 10 according to the first embodiment includes a user terminal 12, and the user terminal 12 can communicate with an operator terminal 16 and a server 18 via a network 14. connected to.

Although this first embodiment shows one user terminal 12 and one operator terminal 16, in reality, a plurality of user terminals 12 and a plurality of operator terminals are used. 16, and as described later, one user terminal 12 and one operator terminal 16 selected by the server 18 in response to a request from this one user terminal 12. A chat or talk process takes place between.

The user terminal 12 is used by a user who uses a predetermined service provided by the server 18, and the operator terminal 16 is used by an operator who responds to the user.

The user terminal 12 is an information processing device, for example a general-purpose smartphone, and is equipped with a browser function. In other examples, other general-purpose terminals such as a tablet PC, a notebook PC, or a desktop PC may be used as the user terminal 12.

The network 14 is composed of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. As the access network, a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, CATV (Cable Television), etc. can be used.

The operator-side terminal 16 is another information processing device different from the user-side terminal 12, and is, for example, a general-purpose notebook PC or a desktop PC, but other examples include a smartphone or a tablet PC. Other general-purpose terminals can also be used.

The server 18 is an information processing device different from the user terminal 12 and the operator terminal 16, and may be a general-purpose server. Therefore, the server 18 includes a CPU 18a and a storage section (including an HDD, ROM, and RAM) 18b, as well as components such as a communication interface and an input/output interface. In the first embodiment, the server 18 operates a site that provides predetermined services.

FIG. 2 is a block diagram showing the electrical configuration of the user terminal 12 shown in FIG. 1. As shown in FIG. As shown in FIG. 2, the user terminal 12 includes a CPU 20, and the CPU 20 connects a storage unit 22, a communication interface (hereinafter referred to as "communication I/F") 24, and an input/output interface (hereinafter referred to as "communication I/F") via an internal bus. , ``input/output I/F'') 26.

The CPU 20 is in charge of overall control of the user terminal 12. However, instead of the CPU 20, an SoC (System-on-a-chip) including a plurality of functions such as a CPU function and a GPU (Graphics Processing Unit) function may be provided. Storage unit 22 includes an HDD, ROM, and RAM. However, a nonvolatile memory such as an SSD may be used instead of the HDD or in addition to the HDD, ROM, and RAM.

The communication I/F 24 has a wired interface for transmitting and receiving control signals and data to and from an external computer such as the operator terminal 16 and the server 18 via the network 14 under the control of the CPU 20. . However, as the communication I/F 24, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) can also be used.

An input device 28, a display device 30, a microphone 32, and a speaker 34 are connected to the input/output I/F 26. Input device 28 is a touch panel and hardware buttons. The touch panel is a general-purpose touch panel, and can be of any type, such as a capacitive type, an electromagnetic induction type, a resistive film type, or an infrared type. The same applies to the operator terminal 16, which will be described later.

However, when a notebook PC or a desktop PC is used as the user terminal 12, a keyboard and a computer mouse are used as the input device 28.

Further, the display device 30 is an LCD or an organic EL display device. The above touch panel may be provided on the display surface of the display device 30, or a touch panel display in which the touch panel is integrally formed with the display device 30 may be provided. This also applies to the operator terminal 16, which will be described later.

The input/output I/F 26 outputs the operation data (or operation information) input from the input device 28 to the CPU 20, and also outputs the image data generated by the CPU 20 to the display device 30 to correspond to the image data. A screen or image is displayed on the display device 30. However, image data received from an external computer (for example, operator terminal 16 or server 18) may be output by CPU 20.

In addition, the input/output I/F 26 converts the user's voice detected by the microphone 32 into digital audio data and outputs it to the CPU 20, and also converts the audio data output by the CPU 20 into an analog audio signal. output from the speaker 34. However, in the first embodiment, the audio data output from the CPU 20 is the audio data received from the operator terminal 16.

The user terminal 12 also includes a sensor interface (sensor I/F) 36 and a camera 38. CPU 20 is connected to camera 38 via a bus and sensor I/F 36. The camera 38 is a camera using an imaging device such as a CCD or CMOS.

Note that the electrical configuration of the user terminal 12 shown in FIG. 2 is an example, and does not need to be limited.

Furthermore, when the user terminal 12 is a smartphone, it is provided with a telephone communication circuit for making telephone calls via a mobile telephone communication network or a mobile telephone network and a public telephone network. Since such calls are not conducted, illustrations are omitted. This also applies to the case where the operator terminal 16, which will be described later, is a smartphone.

FIG. 3 is a block diagram showing the electrical configuration of the operator side terminal 16 shown in FIG. 1. As shown in FIG. As shown in FIG. 3, the operator side terminal 16 includes a CPU 50, and the CPU 50 is connected to a storage unit 52, a communication I/F 54, and an input/output I/F 56 via an internal bus.

The CPU 50 is in charge of overall control of the operator terminal 16. However, instead of the CPU 50, an SoC including multiple functions such as a CPU function and a GPU function may be provided. Storage unit 52 includes an HDD, ROM, and RAM. However, a nonvolatile memory such as an SSD may be used instead of the HDD or in addition to the HDD, ROM, and RAM.

The communication I/F 54 has a wired interface for transmitting and receiving control signals and data between the user terminal 12 and an external computer such as the server 18 via the network 14 under the control of the CPU 50. . However, as the communication I/F 54, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) can also be used.

An input device 58, a display device 60, a microphone 62, and a speaker 64 are connected to the input/output I/F 56. The microphone 62 and speaker 64 constitute a headset with a microphone used by the operator to make voice calls with the user.

Further, as the input device 58, a keyboard and a computer mouse are used. However, when a smartphone or a tablet PC is used as the operator terminal 16, a touch panel and hardware buttons are provided as the input device 58. Further, the display device 60 is an LCD or an organic EL display device.

The input/output I/F 56 outputs the operation data (or operation information) input from the input device 58 to the CPU 50, and also outputs the image data generated by the CPU 50 to the display device 60 to correspond to the image data. The screen is displayed on the display device 60.

In addition, the input/output I/F 56 converts the operator's voice detected by the microphone 62 into digital audio data and outputs it to the CPU 50, and also converts the audio data output by the CPU 50 into an analog audio signal. output from the speaker 64. However, in the first embodiment, the audio data output from the CPU 50 is the audio data received from the user terminal 12.

Further, the operator side terminal 16 includes a sensor I/F 66 and a camera 68. CPU 50 is connected to camera 68 via a bus and sensor I/F 66. The camera 68 is a camera using an imaging device such as a CCD or CMOS.

In such an information processing system 10, a user uses the user terminal 12 to view a web screen 100 of a predetermined service provided by the server 18 and perform shopping or the like. On the front of the web screen 100, a button 110 and a button 112 for communicating with an operator via chat or talk are displayed.

However, the web screen 100 is displayed on the display device 30 by starting a web browser and inputting a predetermined URL. The web screen 100 is a screen of a website (or web page) of a predetermined service. In FIG. 4, an example of a web screen 100 for online shopping is shown. As an example, the predetermined service is online shopping, but it is any online service that can respond to inquiries from users through chat or talk.

Further, the button 110 and the button 112 may be displayed when a predetermined condition is satisfied. The predetermined conditions are that the user has instructed the display of the buttons 110 and 112, that there is no user operation for a long time (for example, from 30 seconds to several minutes), and that the web screen 100 being displayed on the display device 30 Repeatedly or continuously instructing the same location or a similar location (nearby location), or returning to the same web screen 100 multiple times (for example, three times) in a predetermined service. .

When the user turns on the button 110, a chat service is provided to the user, and a chat is performed between the user and the operator selected by the server 18. That is, text messages are exchanged between the user terminal 12 and the operator terminal 16. The chat service is already well known and is not essential to the present invention, so a description thereof will be omitted. As an example, a chat service disclosed in Japanese Patent Application Publication No. 2020-86677 can be used. However, instead of an operator responding, a chatbot can also respond.

Further, when the user turns on the button 112, the user can make a voice inquiry to the operator selected by the server 18 about how to use the online shopping site and about the product.

In this case, an image of an avatar corresponding to the operator is displayed on the display device 30 of the user terminal 12. As an example, the image of the avatar corresponding to the operator is a partial or full-body image of a character imitating a human face, and is displayed on the front of the web screen 100. Further, the image of the avatar corresponding to the operator is moved in accordance with the output of the operator's voice. The avatar moves its mouth, moves its head, blinks, and changes facial expressions. However, if the avatar image is an image of the upper body or the whole body of a character imitating a human, the avatar also makes gestures.

Note that the display of the image of the avatar corresponding to the operator and its control are not essential contents of the present invention, and a known technique can be adopted, so a description thereof will be omitted.

However, instead of displaying the image of the avatar corresponding to the operator, an image of the operator may be displayed.

In the information processing system 10 as described above, the operator generally interacts with the user while viewing the user's video, or interacts with the user without viewing the user's video.

In the former case, the operator can easily interact with the user because he or she can see the user's image and recognize the user's facial expressions. However, since the video of the user is displayed, there is room for improvement in order to protect the privacy of the user.

In the latter case, although it is considered that the user's privacy can be protected, there is a problem in that it is difficult for the operator to interact with the user because the operator cannot see the user's image.

Therefore, in the first embodiment, in order to facilitate interaction with the user while protecting the user's privacy, an image of the avatar corresponding to the user is displayed on the display device 60 of the operator terminal 16, and the image of the avatar is displayed on the display device 60 of the operator terminal 16. Expressions (in the first embodiment, facial expressions), the avatar's line of sight, and the avatar's movements are controlled.

FIG. 5 shows an example of an image 210 of an avatar corresponding to the user, which is displayed on the display device 60 of the operator terminal 16. As shown in FIG. 5, an avatar image 210 is displayed within a rectangular display frame 200. The avatar image 210 corresponding to the user is a part of the image including the face of a character imitating a human. More specifically, the avatar image 210 is an image of a portion of a human head (including the face), neck, and shoulders.

However, if the avatar image 210 is an image of the upper body or the whole body of a character imitating a human, the avatar further gestures. Although a detailed explanation will be omitted, in this case, as will be described later, when controlling the facial expressions of the avatar, the body movements are also controlled.

In addition, a web screen 100 that is the same as the web screen 100 displayed on the display device 30 of the user side terminal 12 is displayed on the display device 60, and a display frame 200 and an avatar corresponding to the user are displayed in front of the web screen 100. An image 210 may also be displayed.

In this case, display information including the website URL, window size, and cursor position coordinates displayed on the display device 30 of the user terminal 12 is displayed when the button 112 is turned on on the user terminal 12. The information is transmitted to the server 18 at the time of the request, and further transmitted from the server 18 to the operator terminal 16 when the operator to respond, that is, the operator terminal 16 is selected. However, the display information may be directly transmitted from the user terminal 12 to the operator terminal 16 between the start of communication and the start of talk.

The avatar image 210 is an image of an avatar selected in advance by the user or an avatar selected in advance by the operator. When an image of an avatar selected by the user in advance is displayed, the type of avatar can be changed from the user terminal 12 via the server 18 or directly from the user terminal 12 to the operator terminal 16. will be notified.

Further, in the first embodiment, the operator side terminal 16 causes the avatar corresponding to the user to express a facial expression that is favorable or sympathetic to the operator. Further, the operator side terminal 16 operates the avatar corresponding to the user so as to avoid a situation in which it is difficult for the operator to speak.

Specifically, when the operator is not speaking, the operator terminal 16 displays the facial expressions and expressions of the avatar corresponding to the user based on the user's facial expression or the user's line of sight and head movements. Controls gaze and head (neck) movements. However, some users' facial expressions are emphasized or relaxed and reflected in the avatar corresponding to the user. Furthermore, the line of sight of the avatar corresponding to the user may be controlled independently of the user's line of sight.

In the first embodiment, when the user's facial expression is a smiling expression, the degree of smiling is increased and the avatar corresponding to the user is made to express a smiling expression. In other words, an avatar image 210 in which the user's smile is emphasized is displayed. Further, when the user's facial expression is a displeased expression, the degree of displeasure is reduced and the avatar corresponding to the user is made to express the displeased expression. In other words, an image 210 of the avatar in which the user's moodiness is alleviated is displayed. In this way, by controlling the facial expression of the avatar corresponding to the user, the avatar corresponding to the user is expressed to the extent that the user's facial expression is changed favorably.

The parameters (hereinafter referred to as "setting parameters") for the degree to which a smile is emphasized (hereinafter referred to as "emphasis degree") and the degree to which displeasure is alleviated (hereinafter referred to as "relaxation degree") are set in advance by the operator. Ru.

However, each setting parameter of the degree of emphasis and the degree of relaxation may be automatically set. In this case, as an example, when the degree of the user's smile is low, the degree of emphasis is set high, and when the degree of the user's smile is high, the degree of emphasis is set low. Further, as an example, when the user's degree of moodiness is low, the relaxation degree is set to be small, and when the user's moodiness is high, the relaxation degree is set to be large.

Further, in the first embodiment, the line of sight of the avatar corresponding to the user is controlled according to the user's line of sight, and the user's line of sight is facing the front, and the time period for which the user is facing the front is the first predetermined time ( For example, if it continues for more than a few seconds to 10 seconds, the avatar corresponding to the user averts its line of sight for a second predetermined period of time (for example, a predetermined percentage of the first predetermined time), regardless of the user's line of sight. In other words, by moving (in other words, controlling) the avatar corresponding to the user so as to avert the user's line of sight, the avatar corresponding to the user can move in a way that does not feel intimidating to the operator, making it difficult for the operator to speak. can be avoided. The predetermined ratio parameter is also included in the above setting parameters and is preset by the operator.

However, the user's line of sight during a conversation between the user and the operator is detected based on the user's facial image. Since the method of detecting the line of sight from a face image is already well known, its explanation will be omitted.

Further, the movement of the user's head is detected (estimated) as follows. The orientation of the user's face is based on the orientation of the user's face image directly facing the camera 38, the current face orientation is calculated based on the current face image, and the user's face orientation is calculated based on the current face orientation. head movement is detected (or estimated). However, the direction of the face can be detected by the movement of multiple facial feature points extracted from the face image. Although the explanation will be omitted, the same applies to the case where the direction of the operator's face is calculated and the movement of the head is detected.

As described above, the user's line of sight and head movement are reflected in the line of sight and head (neck) movement of the avatar corresponding to the user. This also applies when the user is speaking.

Further, if the operator is not speaking, the user may speak. When the user is speaking, the lips of the avatar corresponding to the user are moved in accordance with the output of the user's voice. Hereinafter, the movement of the lips of the avatar corresponding to the user in accordance with the output of the user's voice may be referred to as a speaking action.

Further, when the operator is speaking, the operator side terminal 16 controls the facial expression of the avatar corresponding to the user based on the operator's facial expression, and also controls the avatar's facial expression based on the voice uttered by the operator. The behavior of the avatar corresponding to the person may be controlled.

In the first embodiment, the operator terminal 16 recognizes the facial expression of the operator and causes the avatar to express the same facial expression as the operator's facial expression. Specifically, when the operator's facial expression is a smiling expression, the avatar corresponding to the user is caused to express a smiling expression. Furthermore, when the operator's facial expression is a sad expression, the avatar corresponding to the user is made to express a sad expression. In other words, when the operator is speaking, the avatar is made to empathize with the operator.

In this first embodiment, the parameters of the degree of smile and the degree of sadness are set in advance by the operator and are included in the above-mentioned setting parameters. However, the parameters for the degree of smile and the degree of sadness can also be set according to the degree of smile and the degree of sadness of the operator.

As described above, the facial expressions of the user and the operator are recognized based on the facial images, and the recognized facial expressions are expressed in the avatar image 210. The facial expressions of the user and the operator can be recognized based on the facial images of the user and the operator during the interaction, respectively.

Since a method for recognizing facial expressions of people such as users and operators using facial images is already known, a description of the method will be omitted. For example, "Hiroshi Kobayashi, Fumio Hara: Basic human facial expression recognition using neural networks, Proceedings of the Society of Instrument and Control Engineers Vol. 29, No. 1, 112/118 (1993)", "Yosuke Koyanaka, Tsuneyasu Honma, Masao Sakai, Kenichi Abe: Facial expression recognition using neural networks, Bulletin of the Department of Medicine and Health Sciences, Tohoku University 13(1):23-32, 2004” and “Daiki Nishime, Satoshi Endo, Yoshiaki Toma, Koji Yamada, Arihei Akamine It is possible to use known techniques such as those disclosed in "Acquisition of facial expressions and analysis of facial features using convolutional neural networks, Journal of the Japanese Society for Artificial Intelligence, Vol. 32, No. 5, FZ (2017)."

In addition, as another known technique, the technique disclosed in Japanese Patent Application Laid-open No. 2020-163660 can also be used as a method of recognizing human facial expressions based on feature points extracted from facial images.

However, circuit components and data necessary for recognizing facial expressions based on a human face image are provided in the operator terminal 16 as appropriate. Alternatively, a device that recognizes facial expressions based on facial images (hereinafter referred to as a "recognition device") may be installed on the cloud, and the facial images can be sent to the recognition device and the facial expression recognition results can be received from the estimation device. good.

In addition, by having the neural network learn multiple facial expressions with different degrees of intensity (in this first embodiment, a smiling expression, a displeased expression, and a sad expression), it is possible to recognize not only facial expressions but also the degree of expression ( or estimated). Furthermore, it is also possible to recognize the degree of facial expression based on the difference in the output of the neural network when facial expressions are recognized. For example, the degree of facial expression is recognized based on the difference between the output for an expressionless facial image and the output for a recognized facial expression. As an example, the degree of facial expression is recognized between 0-100%.

Furthermore, when recognizing human facial expressions using the method disclosed in Japanese Patent Application Laid-open No. 2020-163660, the degree of facial expression is recognized (or estimated) based on the difference (distance) between feature points extracted from facial images. ) can also be done. For example, the distance between each feature point extracted from an expressionless face image and each feature point extracted from the user's face image used for expression recognition is calculated, and the degree of expression is determined based on the calculated distance. is determined. Since the distance is calculated for each feature point, the degree of facial expression is determined based on, for example, the average value, maximum value, or variance of the plurality of calculated distances.

FIG. 6(A) is a diagram for explaining the difference in the degree of the avatar's smiling expression, and FIG. 6(B) is a diagram for explaining the difference in the degree of the avatar's displeased expression. , FIG. 6(C) is a diagram for explaining the difference in the degree of sad expression of the avatar.

The degree (or magnitude) of each of the smiling, displeased, and sad expressions can be set in multiple levels (for example, 20 levels) from minimum (0%) to maximum (100%). The avatar's facial expressions include each expression and the degree of each expression: eyebrows (position, shape), wrinkles between the eyebrows (how close they are), eyes (size of the iris, how the outer corners of the eyes are raised and lowered), eyelids (how open they are), It is determined by parameters (hereinafter referred to as "facial expression parameters") for each part of the nasolabial folds (degree of approach) and mouth (position, shape, and degree of opening).

As mentioned above, in order to recognize the degree of facial expressions of humans such as users and operators between 0 and 100%, and to express the facial expressions of avatars at a degree of 0 to 100%, in this first embodiment, , facial expression parameters are determined according to the degree of human facial expression. Therefore, the avatar image 210, in which the same facial expression as the human facial expression recognized from the facial image is expressed to the same degree, has the same facial expression as that human.

In the examples shown in FIGS. 6A to 6C, the degree of facial expression increases from the left end (that is, expressionless) toward the right end, and decreases as the left end is approached. 6(A) to 6(C) show avatar images 210 when the degree of expression is medium (normal) and when the degree of expression is maximum for each facial expression.

However, the above expressionless avatar image 210 is the avatar image 210 when the degree of expression is the lowest for each of the smiling expression, the displeased expression, and the sad expression.

Furthermore, although not shown, since the degree of each emotion is set to multiple levels as described above, it is possible to display the image 210 of the avatar having facial expressions at each level.

Furthermore, since each facial expression can be set in 20 levels, the setting parameters for the degree of emphasis, degree of relaxation, degree of smile, and degree of sadness described above can be set in 5% increments.

Note that an expressionless avatar has no wrinkles and a symmetrical face. By designing in this way, it becomes difficult to determine the gender of the avatar, and the avatar becomes acceptable to users who have biased preferences toward men or women. In addition, by creating a featureless face with no wrinkles and symmetrical features, you can easily express facial expressions such as a smile or a frown just by creating a few wrinkles, and the strength of that facial expression can be increased. It can also be easily controlled.

Further, in the first embodiment, when the operator is speaking, the operator terminal 16 causes the avatar corresponding to the user to perform a nodding motion at an appropriate timing. In other words, a situation where it is difficult for the operator to speak is avoided.

Specifically, the avatar corresponding to the user is caused to nod at the timing when the operator's voice is interrupted or at the timing when the operator utters the content for which consent is requested from the user. It is determined that the operator's voice has been interrupted when the volume of the operator's voice remains below a predetermined level for a third predetermined period of time (for example, several milliseconds). In addition, whether the operator has uttered the content for which consent is being requested from the user is determined by recognizing the operator's voice and determining whether or not the operator has uttered the content for which consent is requested from the user. Ru. Contents for which consent is requested include "Isn't it..." and "Are you sure?".

However, if the avatar corresponding to the user does not make a nodding motion and the operator's facial expression is not reflected in the avatar corresponding to the user, the user's facial expression, the user's line of sight, and the user's head movements will be reflected on the avatar corresponding to the user. In addition, if the avatar corresponding to the user does not nod and the operator's facial expression is reflected on the avatar corresponding to the user, the user's line of sight and the movement of the user's head may be It will be reflected on the corresponding avatar. Further, when the avatar corresponding to the user performs a nodding motion, the facial expression of the user is reflected on the avatar corresponding to the user.

Further, even when the operator is speaking, the user may also speak. In this case as well, the avatar corresponding to the user performs a speaking motion in accordance with the output of the user's voice.

FIG. 7 shows an example of a memory map 300 of the storage unit (here, RAM) 52 built in the operator terminal 16. The RAM is used as a work area and a buffer area for the CPU 50. As shown in FIG. 7, the RAM includes a program storage area 302 and a data storage area 304. The program storage area 302 stores a control program for the overall processing executed by the operator terminal 16 of the first embodiment.

The control programs include an operation detection program 302a, a photography program 302b, a sound detection program 302c, a voice recognition program 302d, a communication program 302e, an image generation program 302f, an image output program 302g, an avatar control program 302h, an expression recognition program 302i, and a line of sight and head recognition program. The program includes a motion detection program 302j, a sound output program 302k, and the like.

The operation detection program 302a is a program for detecting operation data 304a input from the input device 58 according to an operation by an operator, and storing the detected operation data 304a in the data storage area 304. The photographing program 302b is a program for photographing an image with the camera 68, storing the photographed image data as transmission data 304b in the data storage area 304, and storing the photographed image data 304i in the data storage area 304.

The sound detection program 302c is a program for detecting the sound input from the microphone 62, storing the corresponding sound data as transmission data 304b in the data storage area 304, and storing the sound data 304k in the data storage area 304. be. The voice recognition program 302d is a program for recognizing the voice detected according to the sound detection program 302c. Dictionary data necessary for voice recognition is not shown, but is stored in the storage unit (HDD or ROM in this case) 52 of the operator terminal 16.

The communication program 302e is a program for wired or wireless communication (data transmission and reception) with external devices, in this first embodiment, the user terminal 12 and the server 18.

The image generation program 302f is for generating image data of images (including the avatar image 210) corresponding to all or part of various screens to be displayed on the display device 60 using the image generation data 304d. It is a program. The image output program 302g is a program for outputting image data generated according to the image generation program 302f to the display device 60.

The avatar control program 302h is a program for controlling the avatar corresponding to the user. In this first embodiment, the CPU 50 changes the facial expression of the avatar, makes the avatar move (speech movement and head (neck) movement), and moves the avatar's line of sight according to the avatar control program 302h. .

The facial expression recognition program 302i is a program for recognizing facial expressions of users and operators. As described above, the user's facial expression and its degree are recognized based on the captured image data received from the user terminal 12. Furthermore, the facial expression of the operator is recognized based on the captured image data 304i captured by the camera 68 of the operator-side terminal 16.

The line of sight and head movement detection program 302j is a program for detecting the user's line of sight and head movement based on the captured image data received from the user terminal 12. The sound output program 302k is a program for outputting the user's voice data received from the user terminal 12 to the speaker 64.

Although not shown, the program storage area 302 also stores other programs such as middleware such as the operating system of the operator terminal 16, programs for executing browser functions, and various application programs.

FIG. 8 is a diagram showing an example of specific contents of the data storage area 304 of the RAM shown in FIG. As shown in FIG. 8, the data storage area 304 includes operation data 304a, transmission data 304b, reception data 304c, image generation data 304d, facial expression parameter data 304e, setting parameter data 304f, user facial expression data 304g, and user gaze. Also stored are head movement data 304h, photographed image data 304i, operator facial expression data 304j, audio data 304k, and the like.

The operation data 304a is operation data detected according to the operation detection program 302a. The transmission data 304b is data to be transmitted to the user terminal 12, and is text data regarding the contents of the operator's response in the chat and audio data regarding the contents of the operator's response in the talk.

The received data 304c is data transmitted and received from the user terminal 12, and includes text data regarding the content of the user's question in the chat, audio data regarding the content of the user's question in the talk, and data from the user terminal 12. This is photographed image data taken by the camera 38. The received data 304c also includes connection information data of the user terminal 12 transmitted from the server 18.

The image generation data 304d is data for generating various screens displayed on the display device 60 of the operator terminal 16, and includes data for generating the avatar image 210. The data for generating the avatar image 210 includes image data of the avatar at rest and data about neck movements. The neck movement is the movement of the neck when speaking and the movement of the neck when nodding. However, since a plurality of types of avatars are provided, the image data of the avatar in a stationary state is stored for each avatar, and the image data of the selected avatar is used.

The facial expression parameter data 304e is data about the facial expression parameters of each part for changing the degree of facial expression from the minimum to the maximum in multiple stages for each of a smiling expression, a displeased expression, and a sad expression. However, since a plurality of avatars are provided, data regarding facial expression parameters are stored for each avatar, and facial expression parameters for the selected avatar are used.

The setting parameter data 304f is data regarding each setting parameter of the degree of emphasis, the degree of relaxation, the degree of smile, the degree of sadness, and a predetermined ratio.

The user facial expression data 304g is data indicating the user's facial expression and its degree recognized from the captured image data received from the user terminal 12. The user's line of sight and head movement data 304h is data indicating the user's line of sight and the movement of the user's head calculated from the captured image data received from the user terminal 12.

The photographed image data 304i is image data photographed by the camera 68. Operator facial expression data 304j is data indicating the facial expression of the operator recognized from photographed image data 304i. The audio data 304k is data about the operator's voice detected according to the sound detection program 302c, and is used to recognize the operator's voice.

Although not shown, the data storage area 304 stores other data necessary for executing control processing, and is provided with a timer (counter) and a flag.

Although not shown, since the user side terminal 12 chats or talks with the operator side terminal 16, the storage unit (RAM in this case) 22 of the user side terminal 12 stores information about the operator side terminal 16. Among the programs and data stored in the RAM of the side terminal 16, the same programs and data as those necessary for chatting or talking are stored.

Specifically, the program storage area of the RAM of the user terminal 12 stores an operation detection program, a photography program, a sound detection program, a communication program, an image generation program, an image output program, a sound output program, and the like.

The operation detection program is a program for detecting operation data input from the input device 28 according to a user's operation, and storing the detected operation data in the data storage area of the storage unit 22. The photographing program is a program for photographing an image with the camera 38 and storing photographed image data of the photographed image in the data storage area as transmission data. The sound detection program is a program for detecting the sound input from the microphone 32 and storing the sound data of the detected sound in the data storage area as transmission data.

The communication program is a program for communicating with external equipment, in this first embodiment, the operator terminal 16 and the server 18, by wire or wirelessly. The image generation program is a program for generating image data corresponding to various screens to be displayed on the display device 30 using image generation data. The image output program is a program for outputting image data generated according to the image generation program to the display device 30. The sound output program is a program for outputting the received operator's voice data.

Further, the data storage area of the storage unit 22 stores operation data, transmission data, reception data, image generation data, and the like.

The operation data is operation data detected according to the operation detection program. The transmission data is data to be transmitted to the operator terminal 16 and the server 18. The data transmitted to the operator side terminal 16 is text data regarding the content of the user's question in the chat, audio data regarding the content of the user's question in the talk, and photographed image data taken by the camera 38. The data transmitted to the server 18 is operation data on the browser regarding online shopping (including operation data regarding buttons 110 and 112).

The received data is data transmitted and received from the operator terminal 16 or the server 18. The data received from the operator side terminal 16 is text data about the operator's response in the chat and audio data about the operator's response in the talk. However, photographed image data photographed by the operator side terminal 16 may also be received. The data received from the server 18 is data to be displayed on the browser and connection information data for the operator terminal 16 selected by the server 18.

The image generation data is data for generating various screens displayed on the display device 30 of the user terminal 12.

Note that, in addition to middleware such as the operating system of the user terminal 12 and a program for executing the browser function, the storage unit 52 also stores other programs and data necessary for chatting or talking with the user. be remembered.

9 to 12 are flowcharts showing control processing of the CPU 50 of the operator terminal 16. FIG. 13 is a flow diagram showing the transmission/reception processing of the CPU 50 of the operator terminal 16. FIG. 14 is a flow diagram showing the transmission and reception processing of the CPU 20 of the user terminal 12.

Although not shown, the CPU 50 executes a process of detecting an operation of the operator, a process of photographing an image of the operator, and a process of detecting the voice of the operator in parallel with the control process and the transmission/reception process.

As shown in FIG. 9, when the CPU 50 of the operator terminal 16 starts the control process, it displays an image 210 of an avatar corresponding to the user in step S1. That is, the CPU 50 generates image data for the display frame 200 including the image 210 of the avatar corresponding to the user, and outputs the generated image data to the display device 60.

In the next step S3, the facial expression of the user is recognized. Here, the CPU 50 acquires the photographed image data included in the received data 304c, recognizes the facial expression and its degree from the user's facial image included in the acquired photographic image data, and stores the corresponding user facial expression data 304g. (Update.

In the next step S5, the user's line of sight and head movement are detected. Here, the CPU 50 detects the user's line of sight and head movement from the user's face image included in the captured image data acquired in step S3, and stores the corresponding user's line of sight and head movement data 304h. (Update.

However, if the acquired photographic image data does not include a facial image of the user, facial expressions will not be recognized, and the user's line of sight and head movements will not be detected.

Subsequently, in step S7, it is determined whether the operator is speaking. Here, the CPU 50 determines whether or not the microphone 62 is detecting audio. If "YES" in step S7, that is, if the operator is speaking, the process advances to step S25 shown in FIG. 11.

On the other hand, if "NO" in step S7, that is, if the operator is not speaking, it is determined in step S9 shown in FIG. 10 whether the user smiles. Here, the CPU 50 determines whether the user's facial expression recognized in step S3 is a smiling face.

If "YES" in step S9, that is, if it is determined that the user has smiled, then in step S11, the avatar image 210 is displayed with an emphasized smiling expression, aligned with the user's line of sight detected in step S5. The process then proceeds to step S49 shown in FIG. In step S11, the CPU 50 generates a facial expression of the avatar so that the avatar smiles at a higher degree than the user smiles. However, the degree of enhancement is determined by a preset emphasis degree parameter.

On the other hand, if "NO" in step S9, that is, if it is determined that the user is not smiling, it is determined in step S13 whether or not the user has a displeased face. Here, the CPU 50 determines whether the facial expression of the user recognized in step S3 is a displeased face.

If "YES" in step S13, that is, if it is determined that the user made a displeased face, in step S15, the avatar image 210 is aligned with the user's line of sight detected in step S5, and the displeasure is alleviated. The screen is displayed with a friendly expression, and the process advances to step S49. In step S15, the CPU 50 generates a displeased facial expression on the avatar at a level lower than the displeased facial expression of the user. However, the degree of reduction is determined by a preset relaxation degree parameter.

On the other hand, if "NO" in step S13, that is, if it is determined that the user does not have a displeased face, then in step S17, the facial expression of the user recognized in step S3 is reflected on the avatar. In other words, the avatar image 210 is displayed with the user's facial expression.

In the following step S19, it is determined whether to avert the avatar's line of sight. Here, the CPU 50 determines whether the user has been facing forward for a first predetermined period of time (for example, about several seconds to 10 seconds) or more. Although not shown, the time is counted when the user's line of sight is facing forward, and is not counted when the user's line of sight is facing other than the front. Furthermore, if the user turns to the front again after facing away from the front, the time spent facing the front from the beginning is counted.

If "YES" in step S19, that is, if it is determined that the avatar's line of sight is to be averted, in step S21, the avatar's line of sight is averted from the front, and the process proceeds to step S49. In step S21, the CPU 50 directs the avatar's line of sight away from the front for a second predetermined period of time. The second predetermined time is set to a predetermined ratio (for example, about 10%) of the first predetermined time. The predetermined ratio is determined by a predetermined ratio parameter set in advance. Further, the direction in which the avatar averts its line of sight, that is, the direction other than the front, is randomly selected from a plurality of preset directions or drawn by lottery with probability.

On the other hand, if "NO" in step S19, that is, if it is not determined to avert the user's line of sight, in step S23, the avatar image 210 is aligned with the user's line of sight, and the process proceeds to step S49.

As described above, if the operator is speaking, the answer is "YES" in step S7, the image taken by the camera 68 is acquired in step S25 shown in FIG. Recognize. However, as described above, the process of photographing the operator is executed in parallel with the control process, and in step S25, the CPU 50 acquires the current photographed image data 304i. Further, in step S27, the CPU 50 recognizes the facial expression and its degree from the facial image of the operator included in the captured image data 304i, and stores (updates) the corresponding operator facial expression data 304j.

Subsequently, in step S29, it is determined whether it is the timing to nod. Here, the CPU 50 determines whether it is the timing at which the operator's voice is interrupted or the timing at which the operator utters the content for which consent is requested from the user.

Although not shown, when executing the process of step S29, the CPU 50 also executes a process of recognizing the voice of the operator.

If "YES" in step S29, that is, if it is time to nod, the avatar is caused to perform a nodding motion in step S31, and in step S33, the avatar corresponding to the user is aligned with the user's line of sight, The user's facial expression is reflected on the avatar corresponding to the user, and the process advances to step S49. In other words, the avatar image 210 is aligned with the user's line of sight and displayed with the user's facial expression.

On the other hand, if "NO" in step S29, that is, if it is not the timing to nod, it is determined in step S35 whether or not the operator has smiled. Here, the CPU 50 determines whether the facial expression of the operator recognized in step S27 is a smile.

If "YES" in step S35, that is, if it is determined that the operator has smiled, in step S37, the avatar is made to express a smiling expression with a preset smile level, and the process proceeds to step S45. In step S37, the CPU 50 generates the facial expression of the avatar so that the avatar smiles at a preset degree of smile. However, the degree of smile is determined by the smile degree parameter included in the setting parameter data 304f.

On the other hand, if "NO" in step S35, that is, if it is determined that the operator is not smiling, it is determined in step S39 whether or not the operator has a sad face. Here, the CPU 50 determines whether the facial expression of the operator recognized in step S27 is sad.

If "NO" in step S39, that is, if it is determined that the operator does not have a sad face, the user's facial expression is reflected in the avatar corresponding to the user in step S41, and the process proceeds to step S45.

On the other hand, if "YES" in step S39, that is, if it is determined that the operator made a sad face, in step S43, the avatar is made to express a sad expression with a preset degree of sadness, and the process proceeds to step S45. . In step S43, the CPU 50 generates a facial expression of the avatar to make the avatar sad to a preset degree of sadness. However, the degree of sadness is determined by the sadness degree parameter included in the setting parameter data 304f.

In step S45, the line of sight of the avatar corresponding to the user is aligned with the line of sight of the user. In the following step S47, the movement of the user's head is reflected on the avatar corresponding to the user, and the process proceeds to step S49 shown in FIG. 12.

As shown in FIG. 12, in step S49, it is determined whether the user is speaking. Here, the CPU 50 determines whether the received data 304c includes the user's voice data.

If "YES" in step S49, that is, if the user is speaking, in step S51, the user's voice is output, and the avatar corresponding to the user is made to perform a speaking action, The process advances to step S55. In step S51, the CPU 50 outputs the received user's voice data from the speaker 64, moves the lips of the avatar image 210 in accordance with this voice data, and moves the head (neck) of the avatar image 210. move. In other words, it appears as if the avatar corresponding to the user is actually speaking.

On the other hand, if "NO" in step S49, that is, if the user is not speaking, the movement of the user's head is reflected in the avatar corresponding to the user in step S53, and step S55 Proceed to.

In step S55, it is determined whether the process is finished. Here, the CPU 50 determines whether the operator has given an instruction to end the control process or whether the user has ended the dialogue.

If "NO" in step S55, that is, if the process is not finished, the process returns to step S1 shown in FIG. Therefore, the avatar image 210 is updated. On the other hand, if "YES" in step S55, that is, if the process ends, the control process ends.

As shown in FIG. 13, when the CPU 50 starts the transmission/reception process, it starts communication with the user terminal 12 in step S71. In the following step S73, it is determined whether the operator's voice has been detected. If "NO" in step S73, that is, if the operator's voice is not detected, the process advances to step S77.

On the other hand, if "YES" in step S73, that is, if the operator's voice is detected, the operator's voice is transmitted to the user terminal 12 in step S75, and the process advances to step S77.

In step S77, it is determined whether data has been received from the user terminal 12. If "NO" in step S77, that is, if data is not received from the user terminal 12, the process advances to step S81.

On the other hand, if "YES" in step S77, that is, if data is being received from the user terminal 12, the received data is stored in step S79, and the process proceeds to step S81.

In step S81, it is determined whether the process is finished. Here, the CPU 50 determines whether the operator has given an instruction to end the transmission/reception process or whether the user has ended the dialogue.

If "NO" in step S81, that is, if the process is not finished, the process returns to step S73. On the other hand, if "YES" in step S81, that is, if it is finished, the transmission/reception process with the user terminal 12 is finished.

As shown in FIG. 14, when the CPU 20 of the user terminal 12 starts the transmission/reception process, it starts communication with the operator terminal 16 in step S101. In the next step S103, it is determined whether the user's voice has been detected.

If "NO" in step S103, that is, if the user's voice is not detected, the process advances to step S107. On the other hand, if "YES" in step S103, that is, if the user's voice is detected, the detected voice is transmitted to the operator terminal 16 in step S105, and the process advances to step S107.

In step S107, a photographed image is acquired. In the next step S109, the photographed image is transmitted to the operator terminal 16. Subsequently, in step S111, it is determined whether the operator's voice has been received.

If "NO" in step S111, that is, if the operator's voice is not received, the process advances to step S115. On the other hand, if "YES" in step S111, that is, if the operator's voice is being received, the operator's voice is output in step S113, and the process proceeds to step S115.

In step S115, it is determined whether the process is finished. Here, the CPU 20 determines whether the user has given an instruction to end the transmission/reception process or whether the operator has ended the dialogue.

If "NO" in step S115, that is, if the process is not finished, the process returns to step S103. On the other hand, if "YES" in step S115, that is, if it is finished, the transmission/reception process with the operator side terminal 16 is finished.

According to the first embodiment, by displaying an avatar corresponding to the user on the display device of the operator side terminal, the privacy of the user is protected, and when the operator is speaking, the avatar corresponding to the user is displayed. Since the avatar corresponding to the user is expressed with the same facial expression as the user's facial expression, in other words, the avatar empathizes with the user, making it easier for the operator to interact with the user. This makes it easier to respond to users.

Further, according to the first embodiment, when the operator is not speaking, the avatar is made to express the user's facial expression with a favorable expression for the operator, so it is possible to create a situation where the operator can easily speak. can. Therefore, it is possible to easily respond to users.

Furthermore, according to the first embodiment, when the operator is not speaking, the avatar averts the avatar's line of sight as appropriate to make the avatar perform non-threatening movements, thereby creating a situation in which the operator can easily speak. be able to. Therefore, it is possible to easily respond to users.

In the first embodiment, when the operator is not speaking, the avatar changes the same facial expression as the user's facial expression to be favorable to the operator, or changes the line of sight as appropriate. If the operator is speaking, the avatar may express the same facial expression as the operator's, or make the avatar nod as appropriate. However, it is not necessary for all of these processes to be performed. Even when one or more processes are executed, the operator can easily interact with the user. The operator sets whether or not to execute each process, and processes that are not executed are skipped in the control process shown in FIGS. 9 to 12.

For example, if the operator is not speaking and the avatar does not want to perform a process of changing the same facial expression as the user's facial expression to make it more favorable to the operator, select "NO" in step S7. In this case, the processes of steps S9 and S13 shown in FIG. 10 are skipped, and the process proceeds to step S17. Therefore, the processes of step S11 and step S15 are never executed. Note that in this case, it is not necessary to transmit the photographed image data from the user terminal 12.

Further, if the operator does not speak and the process of averting the line of sight is not performed, the process of step S19 is skipped and the process proceeds to step S23 when the process of step S17 is executed.

Furthermore, if the operator does not want to perform the process of making the avatar express the same facial expression as the operator's facial expression when the operator is speaking, if "NO" in step S29, the processes of steps S35 and S39 are skipped. The process then proceeds to step S41. Therefore, steps S37 and S43 are never executed.

Furthermore, if the process of causing the operator to nod appropriately while speaking is not executed, once the process of step S27 is executed, the process of step S29 is skipped and the process moves to step S35. . Therefore, the processes of steps S31 and S33 are never executed.

Although the explanation is omitted, if two or more processes are not executed, the corresponding process is skipped as described above.

Note that in the first embodiment, the facial expressions of the user and the operator are recognized from the facial images, but there is no need to be limited to this. The facial expressions of the user and the operator can also be recognized from the voices of the user and the operator, respectively. Known techniques can be used as a method for estimating human facial expressions from voice. For example, the technology disclosed in Japanese Patent Application Laid-Open No. 2021-12285 and "Daiki Mori: Understanding Emotions and Attitudes from Speech, Journal of the Institute of Electronics, Information and Communication Engineers Vol. 101, No. 9, 2018" can be used.

Further, in the first embodiment, when the operator is speaking, the operator's facial expression is recognized, and the avatar corresponding to the user is made to express a smiling expression with a preset smile level, Although the avatar corresponding to the user expresses a sad expression based on a preset degree of sadness, the present invention is not limited to this. Since it is possible to recognize the degree of the operator's facial expression from the operator's face image, the avatar may be made to express a smiling expression or a sad expression based on the recognized degree of the operator's facial expression.

Furthermore, in the first embodiment, the avatar corresponding to the user is made to nod when the operator's voice is interrupted or when the operator utters the content for which the user is requested to consent. However, there is no need to be limited to this. If the number of nods is large or small, it may be difficult for the operator to speak, so it is also possible to set the frequency of nodding. The frequency of nodding is stored as a setting parameter. If the frequency of nodding is reduced, even if the operator's voice is interrupted or the operator has uttered the content for which the user is requesting consent, the frequency of nodding will be reduced. Accordingly, the nodding action is not performed. On the other hand, if the frequency of nodding is reduced, it will occur not only when the operator's voice is interrupted or when the operator utters the content for which the user is requesting consent. Depending on the increasing rate, the nodding motion is also performed when a fourth predetermined time (for example, from 0.5 seconds to several seconds) has elapsed since the previous nodding motion.

Furthermore, in the first embodiment, a situation is created in which it is easy for the operator to interact by having the user express an avatar corresponding to the degree to which the user's facial expression has been changed in a way that is favorable to the operator. However, it is not necessary to be limited to this. If the user's face looks tough or displeased and an angry or displeased expression is recognized, the operator can reduce the degree of anger or displeasure or change the expression to a happy one. It is also possible to create a situation where it is easy for people to interact with each other. The operator may be able to set or cancel the reduction of the degree of anger or displeasure, or the conversion to a happy expression. However, the degree of anger or displeasure to be reduced is set in a configuration parameter. In this way, the degree to which the user's facial expression is changed to be favorable toward the operator is expressed as the corresponding avatar, or the user's facial expression is changed to a favorable facial expression toward the operator. It may also be possible to have the user express a corresponding avatar.

<Second example>
In the second embodiment, the user-side terminal 12 recognizes the user's facial expression and detects the user's line of sight and head movement. The second embodiment is the same as the first embodiment except that the movement of the part is transmitted to the operator terminal 16, so the different contents will be explained and the redundant explanation will be omitted.

Therefore, in the second embodiment, the facial expression recognition program 302i is also stored in the user terminal 12. Furthermore, the line of sight and head movement detection program 302j is stored in the user terminal 12 and deleted from the operator terminal 16.

Specifically, as shown in FIG. 13, part of the control processing of the CPU 50 of the operator terminal 16 is changed, and as shown in FIG. Be changed.

In the second embodiment, the process of recognizing the user's facial expression in step S5 and the process of detecting the user's line of sight in step S7 are deleted from the control process of the CPU 50.

Further, in the second embodiment, the process of transmitting the photographed image in step S109 to the operator terminal 16 is deleted from the transmission/reception process of the CPU 20, and the step S121 of recognizing the user's facial expression is performed between steps S107 and S111. , the process of step S123 for detecting the user's line of sight and head movement, and the process of step S125 for transmitting the user's facial expression, line of sight, and head movement to the operator terminal 16 are added in that order. Ru.

In the second embodiment, as in the first embodiment, it is possible to easily interact with the user who is the other party of the dialog, and to respond to the user.

In each of the above embodiments, the user terminal and the operator terminal communicate via the network, but the user terminal and the operator terminal communicate via the network and the server. It's okay. In this case, when the server receives image data sent from the user terminal to the operator terminal, the server recognizes the user's facial expression based on the received image data, and also recognizes the user's line of sight and head position. , and instead of the received image data, data regarding the user's facial expression, the user's line of sight, and the movement of the user's head may be transmitted to the operator side terminal.

Further, in each of the above-described embodiments, a case has been described in which a user of a shopping site and an operator who serves the user interact, but the present invention is not limited to this. When two or more participants are in a video call or web conference, the device used by each participant displays an avatar image corresponding to each person with whom to interact or other participants participating in the conference. The information may be displayed on the device and each avatar may be individually controlled using the method shown in the above embodiment.

Furthermore, the order of processing of each step in the flowcharts shown in each of the above embodiments can be changed if the same result is obtained.

Furthermore, the various screens and specific numerical values mentioned in each of the above-mentioned embodiments are merely examples, and can be changed as necessary.

10...Information processing system 12...User side terminal 14...Network 16...Operator side terminal 18...Server 18a, 20, 50...CPU
18b, 22, 52...Storage unit 24, 54...Communication I/F
26, 56...Input/output I/F
28, 58...Input device 30, 60...Display device 32, 62...Microphone 34, 64...Speaker 36, 66...Sensor I/F
38, 68...camera

Claims (11)

voice detection means for detecting operator voice, which is voice uttered by an operator interacting with a user;
a photographing means for photographing a face image of the operator;
a first recognition means for recognizing the facial expression of the operator based on a facial image of the operator photographed by the photographing means when the voice of the operator is detected by the voice detecting means;
When the facial expression of the operator recognized by the first recognition means is a predetermined first facial expression, the user who is the other party of the interaction expresses the same facial expression as the predetermined first facial expression. An information processing device comprising: avatar display means for displaying an image of an avatar on a display device; and transmission means for transmitting the operator voice detected by the voice detection means to a user terminal used by the user. The information processing device according to claim 1, wherein the predetermined first expression is a smiling expression and a sad expression. a timing determining means for determining whether or not it is the timing of the nod based on the operator's voice when the voice of the operator is detected by the voice detecting means; 3. The information processing apparatus according to claim 1, further comprising avatar control means for causing the avatar to perform a nodding motion when it is determined that this is the case. a receiving means for receiving the user's face image transmitted from the user-side terminal; and a receiving means for receiving the user's face image transmitted from the user-side terminal; further comprising a second recognition means for recognizing the user's facial expression based on the facial image,
When the user's facial expression recognized by the second recognition device is a predetermined second facial expression, the avatar display means displays the same facial expression as the predetermined second facial expression favorably to the operator. The information processing device according to any one of claims 1 to 3, wherein the information processing device is changed and expressed by the avatar. The predetermined second expression is a smiling expression and a displeased expression,
The second recognition means further recognizes the degree of a smiling expression and a displeased expression of the user,
The avatar display means causes the avatar to emphasize the degree of smile when the user's facial expression is the smiling expression, and displays a moody expression when the user's facial expression is the displeased expression. 5. The information processing apparatus according to claim 4, wherein the avatar is made to express the expression by relaxing the degree of expression. further comprising receiving means for receiving the facial expression of the user transmitted from the user-side terminal,
The avatar display means displays the predetermined second expression when the user's facial expression received by the receiving means is a predetermined second expression when the voice detection means does not detect the operator voice. 4. The information processing apparatus according to claim 1, wherein the avatar is made to express the same facial expression as the second facial expression in a manner favorable to the operator. The predetermined second expression is a smiling expression and a displeased expression,
The receiving means further receives the degree of smiling expression and displeased expression of the user,
The avatar display means causes the avatar to emphasize the degree of smile when the user's facial expression is the smiling expression, and displays a moody expression when the user's facial expression is the displeased expression. 7. The information processing apparatus according to claim 6, wherein the avatar expresses the expression by relaxing the degree of expression. Further comprising a line of sight detection means for detecting the line of sight of the user based on the facial image of the user received by the receiving means,
The avatar display means sets the line of sight of the avatar according to the line of sight of the user detected by the line of sight detection means, and when the voice detection means does not detect the operator voice, the line of sight detection means From claim 1, wherein when the time during which the user's line of sight is facing forward, as detected by the means, has elapsed for a first predetermined time, the line of sight of the avatar is averted for a second predetermined time regardless of the user's line of sight. 5. The information processing device according to any one of items 5 to 5. The receiving means further receives the user's line of sight,
The avatar display means sets the line of sight of the avatar in accordance with the received line of sight of the user, and when the voice detection means does not detect the operator voice, the avatar display means sets the line of sight of the avatar according to the received line of sight of the user, and when the voice detection means does not detect the operator voice, the line of sight detected by the line of sight detection means is set. 8. The information processing apparatus according to claim 6, wherein when the user's line of sight faces forward for a first predetermined period of time, the avatar's line of sight is averted for a second predetermined period of time regardless of the user's line of sight. . A control program executed on an information processing device,
A processor of the information processing device,
a voice detection step of detecting operator voice, which is voice uttered by an operator interacting with a user;
a photographing step of photographing a face image of the operator;
a recognition step of recognizing the facial expression of the operator based on the facial image of the operator photographed in the photographing step when the operator's voice is detected in the voice detecting step;
If the facial expression of the operator recognized in the recognition step is a predetermined facial expression, displaying on a display device an image of an avatar corresponding to the user, who is the other party of the dialogue, and which expresses the same facial expression as the predetermined facial expression. and a transmitting step of transmitting the operator's voice detected in the voice detecting step to a user terminal used by the user. A method for controlling an information processing device, the method comprising:
(a) detecting operator voice, which is voice uttered by an operator interacting with a user;
(b) photographing a face image of the operator;
(c) when the operator voice is detected in the step (a), recognizing the facial expression of the operator based on the facial image of the operator photographed in the photographing step;
(d) If the facial expression of the operator recognized in step (c) is a predetermined facial expression, an image of an avatar corresponding to the user, who is the other party of the dialogue, expressing the same facial expression as the predetermined facial expression. and (e) transmitting the operator's voice detected in step (a) to a user terminal used by the user.

Images