JP2023072111A - Information processing apparatus, control program, control method, and information processing system - Google Patents

Abstract

To display an avatar with stronger presence.SOLUTION: A user-side terminal (12) includes a CPU (20). A user of the user-side terminal and an operator of an operator-side terminal (16) communicate by voice in real time, and when the operator responds to an inquiry from the user, the CPU of the user-side terminal receives voice data of the voice of the operator. The received voice data is added with volume data of the voice of the operator, and the CPU of the user-side terminal calculates a ratio for enlarging or reducing an image of an avatar 120 based on the volume data. The CPU of the user-side terminal displays an image of an avatar enlarged or reduced at the calculated ratio compared to a normal time, and outputs the received voice data. When the image of the avatar is enlarged, the image of the avatar may protrude from a display frame 152.SELECTED DRAWING: Figure 7

Description

The present invention relates to an information processing device, a control program, a control method, and an information processing system, and more particularly to an information processing device, a control program, a control method, and an information processing system in which a user and an operator communicate by chat or talk. .

An example of this type of conventional information processing apparatus is disclosed in Japanese Unexamined Patent Application Publication No. 2002-200012. In the communication system disclosed in Patent Literature 1, an operator terminal converts the operator's expression and gestures into coordinate data in a telexistence mode setting, and transmits the coordinate data together with response voice data to a reception terminal. The response terminal generates an avatar based on the coordinate data sent from the operator terminal, thereby generating character response information in which the expression and gestures of the operator are reflected in the avatar's expression and gestures, and displays the character response information to the user. .

Japanese Patent Application Laid-Open No. 2021-56940

In the above Patent Document 1, an avatar image based on character response information is displayed on the display unit of the operator terminal, but since the screen of the display unit is two-dimensional, the avatar image is generated using three-dimensional CG image data. Even if it is, the avatar is displayed two-dimensionally, and there is room for improvement in order to show the presence of the avatar.

SUMMARY OF THE INVENTION Therefore, a primary object of the present invention is to provide a novel information processing apparatus, control program, control method, and information processing system.

Another object of the present invention is to provide an information processing device, a control program, a control method, and an information processing system capable of displaying an avatar with increased presence.

A first invention comprises a receiving means for receiving a voice uttered by an operator and predetermined information when the operator utters the voice from an operator-side terminal, a sound output means for outputting the voice received by the receiving means, A ratio for enlarging or reducing the image of the avatar corresponding to the operator based on the predetermined information received by the receiving means is calculated with respect to the normal size received by the receiving means when no sound is output. The information processing apparatus is provided with image display means for displaying, on a display, an image of an avatar drawn at the ratio calculated by the ratio calculation means when sound is output by the ratio calculation means and the sound output means.

A second invention is according to the first invention, wherein the image display means further displays a frame image in which the image of the avatar normally fits, and displays the image of the avatar in front of the frame image.

A third invention is according to the first invention, wherein the predetermined information is the volume of the voice uttered by the operator, and the ratio calculating means calculates the ratio based on the volume.

A fourth invention is according to the first or second invention, wherein the predetermined information is a movement of the operator's neck when the operator speaks, and the ratio calculating means comprises: Calculate the ratio based on

According to a fifth aspect of the present invention, there is provided a receiving means for receiving, from an operator-side terminal, a ratio of the voice uttered by the operator and the ratio calculated based on predetermined information when the operator uttered the voice, and the voice received by the receiving means. Sound output means for outputting, and image display means for displaying on a display an image of the avatar drawn at the ratio received by the receiving means when the sound is output by the sound output means, the ratio corresponding to the operator. The information processing device is a ratio for enlarging or reducing an image of an avatar with respect to a normal size when sound received by a receiving means is not output.

A sixth aspect of the present invention is a receiving means for receiving a voice uttered by an operator and an image of an avatar drawn at a ratio calculated based on predetermined information when the operator uttered the voice, and a voice received by the receiving means. and image display means for displaying on a display the image of the avatar received by the receiving means when the sound output means outputs the sound, the ratio being the image of the avatar corresponding to the operator is a ratio of enlarging or reducing the sound received by the receiving means to the normal time when the sound is not output.

A seventh aspect of the present invention is a receiving means for receiving a voice uttered by an operator and predetermined information when the operator utters the voice from an operator-side terminal, and a use for interacting with the operator using the voice received by the receiving means. Sound output means for outputting to the user side terminal used by the operator, based on the predetermined information received by the receiving means, the image of the avatar corresponding to the operator and the voice received by the receiving means are interacted with the operator. Ratio calculation means for calculating the ratio of enlargement or reduction with respect to the normal size that is not output to the user terminal used by the user; The information processing apparatus includes image output means for outputting an image of the avatar drawn at the calculated ratio to a user terminal.

An eighth invention includes receiving means for receiving text or voice uttered by an operator, output means for outputting the text or voice received by the receiving means, and displaying an image of an avatar corresponding to the operator on a display. The image display means displays the image of the avatar on the display in a manner that fits in the frame image in a normal time when the text or voice received by the receiving means is not output, and the output means outputs the text or voice. An information processing device that displays an image of an avatar on a display in such a manner as to protrude from a frame image when outputting.

A ninth aspect of the present invention is a control program executed by an information processing device, which transmits a voice uttered by an operator and predetermined information when the operator uttered to a processor of the information processing device. a sound output step of outputting the voice received in the receiving step; an image of an avatar corresponding to the operator based on predetermined information received in the receiving step; and outputting the voice received in the receiving step. When outputting sound in the ratio calculation step of calculating the ratio of enlargement or reduction with respect to the normal size, and the sound output step, the avatar image drawn at the ratio calculated in the ratio calculation step is displayed on the display It is a control program that causes an image display step to be executed.

A tenth aspect of the present invention is a control program executed by an information processing device, wherein a processor of the information processing device calculates based on a voice uttered by an operator and predetermined information when the operator utters the voice. A reception step of receiving the ratio from the operator-side terminal, a sound output step of outputting the sound received in the reception step, and when outputting the sound in the sound output step, an avatar image drawn at the ratio received in the reception step. The image display step for displaying on the display is executed, and the ratio is the ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to the normal size when the voice received in the receiving step is not output. , is the control program.

An eleventh aspect of the present invention is a control program executed by an information processing device, wherein a processor of the information processing device calculates based on a voice uttered by an operator and predetermined information when the operator utters the voice. a receiving step of receiving an avatar image drawn at a ratio; a sound output step of outputting the sound received in the receiving step; and displaying the avatar image received in the receiving step on a display when outputting the sound in the sound outputting step. and the ratio is the ratio of enlarging or reducing the image of the avatar corresponding to the operator with respect to the normal size when the sound received in the receiving step is not output, the control program is.

A twelfth aspect of the invention is a control program executed by an information processing device, which transmits a voice uttered by an operator and predetermined information when the operator uttered to a processor of the information processing device. a receiving step for receiving from, a sound output step for outputting the voice received in the receiving step to a user-side terminal used by the user who interacts with the operator, and responding to the operator based on the predetermined information received in the receiving step A ratio calculation step for calculating the ratio of the avatar image to be enlarged or reduced with respect to the normal size when the voice received in the reception step is not output to the user-side terminal used by the user who interacts with the operator. , and an image output step of outputting an image of the avatar drawn at the ratio calculated in the ratio calculation step to the user-side terminal when outputting sound in the sound output step.

A thirteenth aspect of the present invention is a control program executed by an information processing device, comprising: a receiving step for receiving text or voice uttered by an operator; and an image display step of displaying the image of the avatar corresponding to the operator on the display. A control program for displaying an image of an avatar on a display in such a manner that the avatar image fits in the display, and displaying the image of the avatar on the display in a manner that protrudes from the frame image when text or voice is output in an output step.

A fourteenth aspect of the invention is a control method for an information processing apparatus having a display, comprising: (a) a step of receiving, from an operator-side terminal, voice uttered by an operator and predetermined information when the operator uttered the voice; (b) a step of outputting the voice received in step (a); (c) an image of an avatar corresponding to the operator based on the predetermined information received in step (a); (d) drawing at the ratio calculated in step (c) when outputting sound in step (b); and displaying an image of the avatar on the display.

A fifteenth aspect of the invention is a control method for an information processing apparatus having a display, comprising: (b) outputting the voice received in step (a); and (c) rendering the voice at the ratio received in step (a) when outputting the voice in step (b). The step of displaying the avatar image on the display is included, and the ratio is the ratio of enlarging or reducing the avatar image corresponding to the operator to the normal size when sound is not output in step (b). There is a control program.

A sixteenth aspect of the invention is a control method for an information processing apparatus having a display, wherein: (a) drawing is performed with a ratio calculated based on a voice uttered by an operator and predetermined information when the operator utters the voice; (b) outputting the sound received in step (a); and (c) when outputting the sound in step (b), the image of the avatar received in step (a). is displayed on the display, and the ratio is the ratio of enlarging or reducing the image of the avatar corresponding to the operator to the size of the normal time when no sound is output in step (b), the control The method.

A seventeenth aspect of the invention is a control method for an information processing apparatus, comprising: (a) a step of receiving, from an operator-side terminal, a voice uttered by an operator and predetermined information when the operator uttered the utterance; ) step of outputting the voice received in step (a) to a user-side terminal used by the user who interacts with the operator; (c) responding to the operator based on the predetermined information received in step (a); A step of calculating the ratio of enlargement or reduction of the avatar image to the normal size when the voice received in step (a) is not output to the user-side terminal used by the user who interacts with the operator. and (d) a step of outputting an image of the avatar drawn at the ratio calculated in step (c) to the user terminal when outputting the sound in step (b).

An eighteenth aspect of the invention is a control method for an information processing apparatus, comprising: (a) a step of receiving text or voice uttered by an operator; and (b) outputting the text or voice received in step (a). and (c) displaying an image of an avatar corresponding to the operator on a display, wherein step (c) fits in the frame image during normal times when the text or voice received in step (a) is not output. and displaying the avatar image on the display in a manner that protrudes from the frame image when text or voice is output in step (b).

A nineteenth invention is an information processing system comprising a server, a user-side terminal and an operator-side terminal communicably connected to the server, wherein a voice uttered by an operator and when the operator utters a voice receiving means for receiving predetermined information from the terminal on the operator side; sound output means for outputting the voice received by the receiving means; A ratio calculation means for calculating a ratio of enlargement or reduction with respect to the normal size when the sound is not output, and when the sound is output by the sound output means, the drawing is performed at the ratio calculated by the ratio calculation means. An information processing system comprising image display means for displaying an image of an avatar on a display of a user terminal.

According to this invention, an avatar with increased presence can be displayed.

The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

FIG. 1 is a diagram showing an information processing system according to one embodiment of the present invention. FIG. 2 is a block diagram showing the electrical configuration of the user terminal shown in FIG. FIG. 3 is a block diagram showing an electrical configuration of the operator-side terminal shown in FIG. 1; FIG. 4 is a diagram showing an example of a screen displayed on the display device of the user terminal. FIG. 5 is a diagram showing another example of a screen displayed on the display device of the user-side terminal. FIG. 6 is a diagram showing another example of the screen displayed on the display device of the user-side terminal. FIG. 7A is a diagram showing an example of a talk screen displaying normal avatar images, and FIG. 7B is a diagram showing an example of a talk screen displaying enlarged avatar images. 7(C) is a diagram showing an example of a talk screen displaying a reduced avatar image. FIG. 8 is a diagram showing an example of a memory map of the RAM of the user-side terminal shown in FIG. 9 is a flowchart showing a first part of an example of control processing of the CPU of the user-side terminal shown in FIG. 2; FIG. 10 is a second part of an example of the control processing of the CPU of the user-side terminal shown in FIG. 2, and is a flow chart subsequent to FIG. 9. FIG. 11 is a flow chart subsequent to FIG. 9, showing a third part of an example of the control processing of the CPU of the user-side terminal shown in FIG. 2; FIG. FIG. 12 is a flow chart showing the ratio calculation processing of the CPU of the user side terminal shown in FIG. FIG. 13 is a block diagram showing the electrical configuration of the operator side terminal of the second embodiment. FIG. 14A is a diagram for explaining the movement distance of the reference plane when the operator nods, and FIG. 14B is a diagram for explaining the movement distance of the reference plane when the operator shakes his head. It is a diagram. FIG. 15 is a flow chart showing the CPU ratio calculation process of the user terminal in the second embodiment. FIG. 16 is a flowchart showing part of the control processing of the CPU of the user terminal in the third embodiment. FIG. 17 is a flowchart showing an example of avatar image generation processing of the CPU of the operator-side terminal in the third embodiment. FIG. 18 is a flowchart showing a first part of an example of control processing of the CPU of the server in the fourth embodiment. FIG. 19 is a flow chart subsequent to FIG. 18, showing a second part of an example of the control processing of the CPU of the server in the fourth embodiment. FIG. 20 is a flow chart subsequent to FIG. 18, showing the third part of an example of the control processing of the CPU of the server in the fourth embodiment. FIG. 21 is a diagram showing an example of an image of an avatar that protrudes from the frame image.

<First embodiment>
Referring to FIG. 1, an information processing system 10 of the first embodiment includes a user terminal 12, and the user terminal 12 can communicate with an operator terminal 16 and a server 18 via a network 14. connected to

The user-side terminal 12 is used by a user who uses a predetermined service provided by the server 18, and the operator-side terminal 16 is used by an operator who responds to the user.

The user-side terminal 12 is an information processing device, such as a smartphone, and has a browser function. In another embodiment, a general-purpose terminal such as a tablet PC, a notebook PC, or a desktop PC can be used as the user terminal 12 .

The network 14 is composed of an IP network (or IP network) including the Internet and an access network (or access network) for accessing this IP network. As the access network, a public telephone network, a mobile telephone network, a wired LAN, a wireless LAN, CATV (Cable Television), or the like can be used.

The operator-side terminal 16 is another information processing device different from the user-side terminal 12. As an example, the operator-side terminal 16 is a notebook PC or a desktop PC. terminal can also be used.

The server 18 is an information processing device different from the user-side terminal 12 and the operator-side terminal 16, and a general-purpose server can be used. Accordingly, the server 18 includes a CPU 18a and a storage unit (including HDD, ROM and RAM) 18b, as well as components such as a communication interface and an input/output interface. In the first embodiment, the server 18 is provided to operate a site that provides a given service.

FIG. 2 is a block diagram showing the electrical configuration of the user terminal 12 shown in FIG. As shown in FIG. 2, the user-side terminal 12 includes a CPU 20. The CPU 20 connects a storage unit 22, a communication interface (hereinafter referred to as "communication I/F") 24, and an input/output interface (hereinafter referred to as "communication I/F") 24 via an internal bus. , “input/output I/F”) 26 .

The CPU 20 manages overall control of the user terminal 12 . However, instead of the CPU 20, an SoC (System-on-a-Chip) including multiple functions such as a CPU function and a GPU (Graphics Processing Unit) function may be provided. Storage unit 22 includes an HDD, a ROM, and a RAM. However, a non-volatile memory such as an SSD may be used instead of the HDD or in addition to the HDD, ROM and RAM.

The communication I/F 24 has a wired interface for transmitting and receiving control signals and data to and from external computers such as the operator terminal 16 and the server 18 via the network 14 under the control of the CPU 20. . However, as the communication I/F 24, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) can also be used.

An input device 28 , a display device 30 , a microphone 32 and a speaker 34 are connected to the input/output I/F 26 . The input device 28 is a touch panel and hardware buttons. The touch panel is a general-purpose touch panel, and any type of touch panel such as an electrostatic capacity type, an electromagnetic induction type, a resistive film type, an infrared type, or the like can be used. The same applies to the operator-side terminal 16, which will be described later.

However, when a notebook PC or desktop PC is used as the user-side terminal 12, a keyboard and a computer mouse are used as the input device 28. FIG.

Also, the display device 30 is an LCD or an organic EL display. The above touch panel may be provided on the display surface of the display device 30, or a touch display in which the touch panel is integrally formed with the display device 30 may be provided. This also applies to the operator-side terminal 16, which will be described later.

The input/output I/F 26 converts the user's voice detected by the microphone 32 into digital voice data and outputs it to the CPU 20, and also converts the voice data output by the CPU 20 into an analog voice signal and outputs it to the speaker 34. output from However, the audio data output from the CPU 20 is the audio data received from the operator-side terminal 16 . In addition, the input/output I/F 26 outputs operation data (or operation information) input from the input device 28 to the CPU 20, and outputs image data generated by the CPU 20 to the display device 30 so that the image data A corresponding screen or image is displayed on the display device 30 . However, image data received from an external computer (for example, operator-side terminal 16 or server 18) may be output by CPU 20. FIG.

Note that the electrical configuration of the user-side terminal 12 shown in FIG. 2 is an example, and does not need to be limited. In another embodiment, the user terminal 12 may be equipped with a camera.

Further, when the user-side terminal 12 is a smart phone, it is provided with a call circuit for making calls via a mobile phone communication network, or a mobile phone network and a public telephone network. Since such a call is not made, illustration is omitted. This is the same when the operator-side terminal 16, which will be described later, is a smart phone.

FIG. 3 is a block diagram showing the electrical configuration of the operator-side terminal 16 shown in FIG. As shown in FIG. 3, the operator-side terminal 16 includes a CPU 50, and the CPU 50 is connected to a storage section 52, a communication I/F 54 and an input/output I/F 56 via an internal bus.

The CPU 50 is in charge of overall control of the operator-side terminal 16 . However, instead of the CPU 50, an SoC including multiple functions such as a CPU function and a GPU function may be provided. Storage unit 52 includes an HDD, a ROM, and a RAM. However, a non-volatile memory such as an SSD may be used instead of the HDD or in addition to the HDD, ROM and RAM.

The communication I/F 54 has a wired interface for transmitting and receiving control signals and data between the user terminal 12 and an external computer such as the server 18 via the network 14 under the control of the CPU 50. . However, as the communication I/F 54, a wireless interface such as a wireless LAN or Bluetooth (registered trademark) can also be used.

An input device 58 , a display device 60 , a microphone 62 and a speaker 64 are connected to the input/output I/F 56 . Microphone 62 and speaker 64 constitute a microphone-equipped headset used by the operator for voice communication with the user.

A keyboard and a computer mouse are used as the input device 58 . However, when a smart phone or a tablet PC is used as the operator-side terminal 16, a touch panel and hardware buttons are provided as the input device 58. FIG. Also, the display device 60 is an LCD or an organic EL display.

The input/output I/F 56 converts the operator's voice detected by the microphone 62 into digital voice data and outputs it to the CPU 50, and converts the voice data output by the CPU 50 into an analog voice signal and outputs it to the speaker 64. output from However, in the first embodiment, the audio data output from the CPU 50 is the audio data received from the user terminal 12 . In addition, the input/output I/F 56 outputs operation data (or operation information) input from the input device 58 to the CPU 50, and outputs image data generated by the CPU 50 to the display device 60 so that the image data The corresponding image is displayed on the display device 60 .

Note that the electrical configuration of the operator-side terminal 16 shown in FIG. 3 is an example, and does not need to be limited. In another embodiment, operator-side terminal 16 may be equipped with a camera.

In such an information processing system 10, when a user uses the user-side terminal 12 to view a web screen 100 of a predetermined service provided by the server 18, and a predetermined condition is satisfied, the operator can operate. An application (hereinafter simply referred to as "application") is launched that allows communication with (operator) via chat or talk.

As an example, the predetermined service is online shopping, but it is any online service that can respond (response) to user inquiries through chat or talk.

FIG. 4 shows an example when an application selection screen 110 is displayed in front of the web screen 100 . However, the web screen 100 is displayed on the display device 30 by activating a web browser and entering a predetermined URL. The web screen 100 is a screen of a website (or web page) of a given service. In FIG. 4, an example of a web screen 100 of some online shopping is shown. Also, the selection screen 110 is an initial screen displayed when the application is activated.

As described above, an application is activated when a predetermined condition is satisfied. In the first embodiment, the predetermined condition is that the user instructs to start (or execute) the application while the web screen of the predetermined service (the web screen 100 in the first embodiment) is displayed. , the user does not operate for more than a first predetermined time (30 seconds in this first embodiment), the same position or a similar place (nearby position) is indicated on the web screen, a predetermined is to return to the same web screen multiple times (for example, three times) in the service.

Note that FIG. 4 shows an example in which the web screen 100 is displayed on the display device 60 when the user-side terminal 12 is a smart phone. Also, the web screen 100 is an example, and screens for services other than online shopping may be displayed.

A selection screen 110 is provided with a display frame 112 in the upper part of the screen, and buttons 114 , 116 and 118 are vertically arranged below the display frame 112 .

A display frame 112 is a frame image for displaying the avatar image 120 . In the first embodiment, the avatar image 120 is a preset character that is the alter ego of the operator who is the partner of the chat or talk (dialogue partner). Thus, the avatar image 120 is a human-like character, and in this first embodiment is an image of the head and neck.

However, the avatar image 120 may be an image of a character imitating an animal or a robot, an animation character, a game character, or the like. Also, the avatar image 120 may be an image of the character's upper body or whole body.

In addition, in the first embodiment, when chatting or talking, the avatar image 120 speaks in accordance with the display of the operator's response content in the chat or the output of the voice of the response content (or utterance content) in the talk. or speak. In this first embodiment, since the head and neck of the avatar image 120 are displayed, the lip portion of the avatar image 120 displays the text of the response in chat or the voice of the response in talk. driven by the output. Therefore, it is represented as if the avatar is actually speaking.

The avatar image 120 is sized to fit within the display frame 112 in a state in which the voice of the response content is not being output, that is, in a state in which the avatar is not speaking or performing a speaking motion (hereinafter referred to as “normal time”). displayed (or drawn) at

Also, on the selection screen 110, the avatar performs a natural action (hereinafter referred to as "unconscious action"). Typical examples of unconscious actions include blinking and breathing. In addition to such physiological actions, habitual actions are also included in unconscious actions. For example, habitual motions include motions of touching hair, motions of touching the face, and motions of biting nails. However, when the selection screen 110 is displayed, the avatar image 120 is first displayed stationary and then displayed greeting (eg, bowing) to the user.

Therefore, in the selection screen 110, after the static avatar image 120 is displayed, the avatar image 120 performing the unconscious action or the greeting action is displayed. Although detailed description is omitted because it is not essential to the present invention, as an example, the display of the static avatar image 120 is performed by outputting (or reproducing) image data stored in advance and static. It is done by Also, the display of the image 120 of the avatar performing the unconscious action and the greeting action is performed by reproducing pre-stored moving image (animation) data.

Button 114 is a button for the user to chat with the operator. Chatting means talking in real time by exchanging texts. Data is sent and received.

Button 116 is a button for the user to talk with the operator. Talk means to talk in real time by exchanging voice. Data is sent and received.

A button 118 is a button for ending the application. When the button 118 is turned on, the application is ended and the selection screen 110 is hidden.

FIG. 5 shows an example of a state in which a chat screen 130 is displayed in front of the web screen 100. As shown in FIG. When button 114 is turned on on selection screen 110, chat screen 130 as shown in FIG. 5 is displayed.

A chat screen 130 is provided with a display frame 132 in the upper part of the screen, and an avatar image 120 is displayed within the display frame 132 . A display frame 134 and a display frame 136 are vertically arranged below the display frame 132 , and a button 138 is provided below the display frame 136 .

The display frame 134 is a frame image for displaying the content of the operator's response. The content of the operator's response is the specific content of the response (or answer) to the user's inquiry (or question), but includes greetings to the user and questions to the user. That is, the text data transmitted from the operator-side terminal 16 is displayed in the display frame 134 . Although detailed description is omitted, in the case of chat, the text data transmitted from the operator-side terminal 16 is data about the contents of key input by the operator using the input device 58 .

The display frame 136 is a frame image for displaying the content of the user's inquiry (question). In the case of chatting, the user's question is input using the input device 28 of the user-side terminal 12 . Although omitted in FIG. 5, on a smartphone or tablet PC, a software keyboard is displayed separately from the chat screen 130, and characters (text) are touch-inputted using the software keyboard, or characters are completed (question content). You can touch input instructions for sending (sending). When the transmission of the question content is instructed, the text data of the question content is sent to the operator side terminal 16 . Therefore, the text of the question content is displayed on the display device 60 of the operator-side terminal 16 . Although not shown, in the chat, the display device 60 of the operator-side terminal 16 displays a chat screen on which the content of the response and the content of the question can be displayed.

A button 138 is a button for ending the chat. When the button 138 is turned on, the chat ends, the chat screen 130 is hidden, and the selection screen 110 is displayed in front of the web screen 100 .

FIG. 6 shows an example of a state in which a talk screen 150 is displayed in front of the web screen 100. As shown in FIG. When button 116 is turned on on selection screen 110, talk screen 150 as shown in FIG. 6 is displayed.

A display frame 152 is provided in the upper part of the screen of the talk screen 150 , and an avatar image 120 is displayed within the display frame 152 . Further, on the talk screen 150, a button 154 is provided below the display frame 152 and at the bottom of the screen.

Although detailed description is omitted, in the case of talk, the voice data transmitted from the operator-side terminal 16 is data about voice input by the operator through the microphone 62 . The voice data transmitted from the operator-side terminal 16 is received by the user-side terminal 12 and output from the speaker 34 .

In the case of talk, the voice data transmitted from the user-side terminal 12 is data about the voice input by the user through the microphone 32 . Also, voice data transmitted from the user-side terminal 12 is received by the operator-side terminal 16 and output from the speaker 64 .

A button 154 is a button for ending the talk. When the button 154 is turned on, the talk ends, the talk screen 150 is hidden, and the selection screen 110 is displayed in front of the web screen 100 .

As described above, voice data corresponding to the voice of the operator is output from the speaker 64 of the user-side terminal 12. At this time, the avatar makes a speaking motion. In this first embodiment, the avatar image 120 is lip-synced to the audio output from the speaker 64 . Therefore, the avatar image 120 is expressed as if it were speaking.

When the operator inputs voice through the microphone 62, the operator-side terminal 16 detects the volume of the voice, and adds data about the detected volume (hereinafter referred to as "volume data") to the voice data. and transmits it to the user-side terminal 12 .

However, the volume data is data about the average value of the volume of the voice detected by the microphone 62 for a second predetermined time period (about 1/10 second in this first embodiment), and is measured every second predetermined time period. Calculated. However, the average value is an example, and the maximum value of the volume in the second predetermined time period may be used.

The user-side terminal 12 calculates a ratio p for determining the size of the avatar image 120 based on the volume data added to the voice data received from the operator-side terminal 16 . However, the ratio p is the ratio of the size after the change when the size of the avatar image 120 in the normal state is set to 1 (100%). In this first embodiment, the ratio p is calculated according to Equation 1 when the volume of the operator's voice is greater than a predetermined value. Further, in the first embodiment, when the volume of the voice of the operator is higher than the predetermined value, the size of the avatar image 120 is increased as the volume increases. However, when the volume is low and the ratio p is less than 1, the size of the avatar image 120 is made smaller than normal.

Here, m is the volume of the operator's voice, M is the preset maximum volume, and P is the ratio (for example, 1.4) when the volume is at the maximum value. Also, the minimum value of the ratio p is set to 0.8, and the volume when the value is smaller than this minimum value is equal to or less than the predetermined value. Note that the initial value of the ratio p is 1, and remains the initial value when the ratio p is not calculated.

[Number 1]
p=P(m/M)
FIG. 7(A) shows an example of a talk screen 150 displaying an avatar image 120 at normal time, and FIG. 7(B) shows an example of a talk screen displaying an avatar image 120 enlarged from the normal time at a ratio p=1.4. An example of the screen 150 is shown, and FIG. 7C shows an example of the talk screen 150 displaying the avatar image 120 reduced from the normal time at a ratio p=0.8.

Although detailed description is omitted, on the talk screen 150 (the same applies to the selection screen 110 and the chat screen 130), the avatar image 120 and images other than the avatar image 120 (screen display frame, screen background, display in the screen) frame and button images) are drawn on separate layers, and the layer on which the avatar image 120 is drawn is placed in front of the layer on which images other than the avatar image 120 are drawn.

Also, in the virtual space, the position of the virtual camera (viewpoint) and the position of the avatar are determined in advance in normal times. The position and/or position of the avatar is moved and the distance between the virtual camera and the avatar is changed.

However, in other embodiments, when enlarging or reducing the avatar image 120, the size of the drawn avatar image 120 may be enlarged or reduced, or the angle of view of the virtual camera may be enlarged or reduced. may

As shown in FIG. 7A, normally, the avatar image 120 is displayed in a predetermined size that fits within the display frame 152, as described above. Normally, the avatar's head and neck are displayed.

As shown in FIG. 7B , when the avatar image 120 is enlarged and displayed, the avatar image 120 may protrude from the display frame 152 . Therefore, the avatar image 120 displayed on the two-dimensional screen looks like it is about to jump out into the three-dimensional real space.

As shown in FIG. 7(C), when the avatar image 120 is displayed in a reduced size, the avatar image 120 appears to move away (or go away) from the user.

Although illustration is omitted, since the ratio p is calculated between 0.8 and 1.4, the avatar image 120 may be enlarged within the display frame 152 without protruding from the display frame 152. be.

In this way, when the avatar image 120 makes a speaking motion, the avatar image 120 is enlarged or reduced in accordance with the ratio p calculated according to the volume. A three-dimensional effect can be obtained even though it is displayed on the screen of That is, the avatar image 120 with increased presence can be displayed.

Also, when the enlarged avatar image 120 is displayed so as to protrude from the display frame 152 (frame image), it looks like it is about to jump out into a three-dimensional real space. In this case also, the avatar image 120 with increased presence can be displayed.

FIG. 8 shows an example of a memory map 300 of the storage unit (here, RAM) 22 built into the user-side terminal 12. As shown in FIG. The RAM is used as a work area and a buffer area for the CPU 20 . As shown in FIG. 8, storage unit 22 includes program storage area 302 and data storage area 304 . A program storage area 302 stores the control program of this embodiment.

The control programs include an activation determination program 302a, a main processing program 302b, an operation detection program 302c, a communication program 302d, an image generation program 302e, an image output program 302f, an avatar control program 302g, a ratio calculation program 302h, a sound detection program 302i, and sound output. Includes program 302j and the like. The application of the first embodiment described above includes a main processing program 302b, an operation detection program 302c, a communication program 302d, an image generation program 302e, an image output program 302f, an avatar control program 302g, a ratio calculation program 302h, a sound detection program 302i, and a sound Contains output program 302j.

However, the application can also use an operation detection program, a communication program, an image generation program, an image output program, a sound detection program, and a sound output program that the user-side terminal 12 has as functions of the terminal body.

The activation determination program 302a is a program for determining whether to activate the application of the first embodiment. The main processing program 302b is a program for executing the main routine processing (overall processing) of the application of the first embodiment.

The operation detection program 302c is a program for detecting operation data 304a input from the input device 28 according to the user's operation and storing it in the data storage area 304. FIG.

The communication program 302d communicates (transmits and receives data) with an external device, in this first embodiment, a server for managing a site that provides a predetermined service and the operator terminal 16 by wire or wirelessly. program.

The image generation program 302e is a program for generating image data corresponding to all or part of various screens to be displayed on the display device 30 using the image generation data 304d.

The image output program 302f is a program for outputting to the display device 30 image data generated according to the image generation program 302e.

The avatar control program 302g is a program for operating the avatar. In this first embodiment, the CPU 20 causes the avatar to speak, make the avatar unconsciously make an avatar, or make the avatar make a greeting (including voice output of the greeting) according to the avatar control program 302g.

The ratio calculation program 302h is a program for calculating the ratio p based on the volume of the voice of the operator. Further, in the first embodiment, the ratio calculation program 302h determines whether the volume of the operator's voice is greater than a predetermined value, and if the volume is greater than the predetermined value, determines to calculate the ratio p. It is also a program for

The sound detection program 302 i is a program for detecting the operator's voice input from the microphone 32 .

The sound output program 302j is a program for outputting the received voice data of the operator.

Although not shown, the program storage area 302 also stores middleware such as the operating system of the user terminal 12, programs for executing browser functions, and application programs other than the application of the present application.

The data storage area 304 also stores operation data 304a, transmission data 304b, reception data 304c, image generation data 304d, ratio data 304e, and the like.

The operation data 304a is operation data detected according to the operation detection program 302c. The transmission data 304b is data to be transmitted to the operator-side terminal 16, and is text data about the content of the user's question in chat and voice data about the content of the user's question in talk. The received data 304c is data transmitted and received from the operator-side terminal 16, and is text data about the content of the operator's response to the chat and voice data about the content of the operator's response to the talk.

The image generation data 304d is data for generating various screens displayed on the display device 30 of the user-side terminal 12, and includes data for generating the image 120 of the avatar. In addition, the data for generating the avatar image 120 includes still image data of the avatar image 120 and animation data about unconscious actions and greeting actions. The ratio data 304e is data about the ratio p. The initial value of the ratio p is 1, and is updated with the ratio p calculated according to the ratio calculation program 302h. Also, when the operator's voice is not being output, that is, when the avatar is not speaking or speaking, the ratio p is reset to its initial value.

Although not shown, the data storage area 304 stores other data necessary for executing control processing, and is provided with timers (counters) and flags.

Although not shown, the operator-side terminal 16 chats or talks with the user-side terminal 12. Therefore, the storage unit (here, RAM) 52 of the operator-side terminal 16 stores user information. Programs and data similar to those stored in the storage unit 22 of the side terminal 12 are stored.

Specifically, the program storage area of the storage unit 52 of the operator-side terminal 16 contains a main processing program, an operation detection program, a communication program, an image generation program, an image output program, a sound detection program, a sound volume detection program, and sound output. Programs and the like are stored.

The main processing program is a program for executing the main routine processing (overall processing) of the application of the operator side terminal 16 that communicates by chat or talk.

The operation detection program is a program for detecting operation data input from the input device 58 according to the operation of the operator and storing it in the data storage area of the storage unit 52 .

The communication program is a program for wired or wireless communication with external devices, in this first embodiment, the user terminal 12 and the server 18 .

The image generation program is a program for generating image data corresponding to various screens to be displayed on the display device 60 using image generation data.

The image output program is a program for outputting image data generated according to the image generation program to the display device 60 .

However, the operator-side terminal 16 does not need to select chat or talk, or display an avatar image. Therefore, a screen such as selection screen 110 is not displayed, and an image of the user's avatar is not displayed on screens such as chat screen 130 and talk screen 150 . However, an image of the user's avatar may be displayed.

The sound detection program is a program for detecting the operator's voice input from the microphone 62 .

The volume detection program is a program for detecting the volume of voice detected according to the volume detection program. As described above, the volume is the average value of the volume of the voice detected by the microphone 62 for the second predetermined time period (about 1/10 second in this first embodiment), and is calculated every second predetermined time period. be done.

The sound output program is a program for outputting received voice data of the user.

The data storage area of the storage unit 52 stores transmission data, reception data, image generation data, audio data, volume data, and the like.

The operation data is operation data detected according to the operation detection program. The transmission data is data to be transmitted to the user-side terminal 12, and is text data about the operator's response to the chat and voice data about the operator's response to the talk. In the first embodiment, volume data is added to audio data. The received data is data transmitted and received from the user-side terminal 12, and is text data about the user's question in the chat and voice data about the user's question in the talk.

The image generation data is data for generating various screens displayed on the display device 60 of the operator-side terminal 16 . The voice data is data corresponding to the operator's voice detected according to the sound detection program. Volume data is data corresponding to the volume detected according to the volume detection program.

Note that the storage unit 52 also stores other programs and data necessary for chatting or talking with the user.

9 to 11 are flowcharts showing control processing of the CPU 20 of the user terminal 12. FIG. Although not shown, the CPU 20 executes processing for detecting operation data and processing for receiving data from the operator-side terminal 16 in parallel with the control processing.

As shown in FIG. 9, when the control process is started, the CPU 20 of the user-side terminal 12 determines in step S1 whether or not the application activation condition is satisfied. As described above, when the user instructs to start (or execute) an application while the web screen of a predetermined service (the web screen 100 in the first embodiment) is displayed, the CPU 20 If there is no operation for a first predetermined time (for example, 30 seconds) or more, if the same location or a similar location (nearby location) is indicated on the web screen, multiple times (for example, 3 times) If it returns to the same web screen, it is determined that the application launch conditions are met.

If "NO" in step S1, that is, if the application activation condition is not satisfied, the process returns to step S1. On the other hand, if "YES" in step S1, that is, if the application activation condition is satisfied, the application is activated in step S3. In the control process, steps after step S3 are the process of the main routine of the application.

In subsequent step S5, the selection screen 110 as shown in FIG. 4 is displayed in front of the web screen 100. FIG. In step S<b>5 , CPU 20 generates image data for selection screen 110 and outputs the generated image data to display device 30 . Hereinafter, the same applies to the case of displaying the screen.

As described above, when the application is initially started, that is, when the selection screen 110 is displayed for the first time after the application is started, the CPU 20 displays the static avatar image 120 and then reproduces the animation data. and let the avatar perform a greeting action.

In the next step S7, the avatar image 120 is caused to perform a standby action. As described above, the CPU 20 causes the unconscious action to be performed. However, when there is no user's operation on the selection screen 110, the CPU 20 causes the avatar to perform an unconscious action every several seconds or several tens of seconds.

Subsequently, in step S9, it is determined whether or not there is a button operation. Here, CPU 20 refers to operation data 304a to determine whether button 114 or button 116 on selection screen 110 has been turned on. Although illustration is omitted, when the button 118 is turned on, the selection screen 110 is closed (hidden) and the application is terminated.

If "NO" in step S9, that is, if there is no button operation, the process returns to step S7. On the other hand, if "YES" in step S9, that is, if there is a button operation, it is determined in step S11 whether or not there is a chat. Here, the CPU 20 determines whether the button 114 is on.

If "NO" in step S11, that is, if button 116 is on, the process proceeds to step S31 shown in FIG. On the other hand, if "YES" in step S11, that is, if button 114 is turned on, chat screen 130 as shown in FIG. 5 is displayed in front of web screen 100 in step S13 shown in FIG.

If a hardware keyboard is not connected to the user terminal 12, a software keyboard is also displayed. Also, when the chat screen 130 is displayed, the selection screen 110 is hidden.

In the next step S15, it is determined whether or not the question content is input. Here, the CPU 20 determines whether or not there is a key input of question content (text). If "YES" in step S15, that is, if the content of the question is input, the content of the input question is displayed in the display frame 136 in step S17, and the process returns to step S15. On the other hand, if "NO" in step S15, that is, if question content is not input, it is determined in step S19 whether or not question content is to be sent. Here, the CPU 20 determines whether or not the content of the question has been finalized.

If "YES" in the step S19, that is, if the content of the question is to be transmitted, the text data of the content of the question is transmitted to the operator-side terminal 16 in a step S21, and the process proceeds to a step S23. On the other hand, if "NO" in step S19, that is, if the content of the question is not transmitted, the process proceeds to step S23.

In step S23, it is determined whether the text data of the content of the response has been received. If "NO" in step S23, that is, if the text data of the response content has not been received, the process proceeds to step S27. On the other hand, if "YES" in step S23, that is, if the text data of the response content is received, in step S25, the avatar is made to speak in accordance with the display of the response content, and the process proceeds to step S27. In step S25, the CPU 20 displays the contents of the response as text in the display frame 134 one by one. indicate. When all the responses are displayed in the display frame 134, all the character strings in the display frame 136 are erased (that is, hidden) so that the question can be entered.

In step S27, it is determined whether or not the chat has ended. Here, the CPU 20 determines whether the button 138 has been turned on or whether the operator terminal 16 has instructed to end the chat.

If "NO" in step S27, that is, if the chat has not ended, the process returns to step S15. On the other hand, if "YES" in step S27, that is, if the chat ends, the chat screen 130 is closed in step S29, and the process returns to step S5 shown in FIG.

Also, as described above, if "NO" in step S11, the talk screen 150 as shown in FIG. 6 is displayed in front of the web screen 100 in step S31 shown in FIG. Note that the selection screen 110 is hidden when the talk screen 150 is displayed. Also, when the talk screen 150 is displayed, that is, when the talk is started, the ratio p is set to the initial value (p=1).

In the subsequent step S33, it is determined whether or not there is voice input. Here, the CPU 20 determines whether or not the microphone 32 has detected sound. If "NO" in step S33, that is, if there is no voice input, the process proceeds to step S37. On the other hand, if "YES" in step S33, that is, if it is a voice input, voice data corresponding to the input voice (that is, voice data of question content) is sent to the operator side terminal 16 in step S35. After transmitting, the process proceeds to step S37.

In step S37, it is determined whether or not voice data of the content of the response has been received. If "YES" in step S37, that is, if voice data of response content is received, in step S39, ratio calculation processing (see FIG. 12), which will be described later, is executed, and in step S41, voice data of response content is , and in step S43, the image data of the image 120 of the avatar performing a speaking action in accordance with the voice data of the response content is generated and output in accordance with the ratio p, and the process returns to step S33. Therefore, the voice of the operator is output from the speaker 34, and the avatar image 120 expressed as if speaking is displayed on the talk screen 150 with a size corresponding to the ratio p.

If "NO" in step S37, that is, if the voice data of the response content has not been received, in step S45, the ratio p is reset, that is, the ratio p indicated by the ratio data 304e is set to the initial value (1 ), and in step S47, the avatar is displayed in normal size, and the process proceeds to step S49. That is, when the operator's voice is not output, the size of the avatar image 120 is returned to the normal size.

In step S49, it is determined whether or not the talk has ended. Here, the CPU 20 determines whether the button 154 has been turned on or whether the operator terminal 16 has instructed to end the talk.

If "NO" in step S49, that is, if the talk has not ended, the process returns to step S33. On the other hand, if "YES" in step S49, that is, if the talk is finished, the talk screen 150 is closed in step S51, and the process returns to step S5.

FIG. 12 is a flowchart showing the ratio calculation process in step S39 shown in FIG. As shown in FIG. 12, when starting the ratio calculation process, the CPU 20 determines in step S71 whether or not the volume is greater than a predetermined value. Here, the CPU 20 determines whether or not the volume indicated by the volume data added to the received audio data exceeds a predetermined value.

If "NO" in step S71, that is, if the sound volume is equal to or less than the predetermined value, the ratio calculation process is ended and the process returns to the control process shown in FIGS. 9-11. On the other hand, if "YES" in step S71, that is, if the volume is greater than the predetermined value, in step S73, the ratio p is calculated according to Equation 1.

Subsequently, in step S75, the calculated ratio p is stored, that is, the ratio data 304e is updated with the calculated ratio p, the ratio calculation process is terminated, and the process returns to the control process.

According to the first embodiment, by displaying the avatar image enlarged or reduced more than usual, it is possible to express a sense of depth. you get a feeling. That is, it is possible to display an avatar with increased presence.

Further, according to the first embodiment, when the enlarged avatar image is displayed so as to protrude from the frame image, it looks like it is about to jump out into the three-dimensional real space. That is, it is possible to display an avatar with increased presence.

In the first embodiment described above, the ratio p is calculated by the user terminal 12, but there is no need to be limited to this. The operator terminal 16 may calculate the ratio p, add the data of the ratio p to the voice data, and transmit the audio data to the user terminal 12 . In this case, in the operator-side terminal 16, when the operator's voice is detected, the CPU 50 executes the ratio calculation process shown in FIG. Send to 12. On the other hand, in the user-side terminal 12, since there is no need to calculate the ratio p, the process of step S39 is omitted, and in step S43, the ratio p indicated by the data of the ratio p added to the received voice data is calculated. Image data of the image 120 of the avatar performing a speech action in accordance with the voice data of the response content is generated and output.

In addition, in the first embodiment, the avatar is enlarged or reduced based on the volume of the voice data of the operator, but it is not necessary to be limited to this. In another embodiment, the avatar may be scaled up or down based on how the operator's eyes open when speaking. However, the degree of opening of the operator's eyes is determined by photographing the operator's face image, and out of a plurality of feature points extracted from the photographed face image, the distance between the feature points for the upper eyelid and lower eyelid of the operator's eye. can be detected by calculating For example, the ratio p is calculated based on the difference between the eye openness when the operator is not speaking and the eye openness when the operator is speaking.

In addition, in the first embodiment, when the volume of the voice of the operator is higher than a predetermined value, the size of the avatar image 120 increases as the volume increases. may be reduced in size. In this case, when the volume is low and the ratio p is less than 1, the size of the avatar image 120 is made larger than normal.

Furthermore, in the first embodiment, in chat and talk, the user terminal 12 and the operator terminal 16 communicate via the network 14, but they may communicate via the server 18. . In such a case, the server 18 calculates the ratio p based on the sound volume indicated by the volume data added to the voice data of the operator, and the server 18 adds the data of the calculated ratio p to the voice data and uses it. It may be transmitted to the person side terminal 12 .

<Second embodiment>
In the second embodiment, in the talk, instead of calculating the ratio p based on the volume of the operator's voice, the ratio p is calculated based on the movement of the operator's neck when the operator speaks. Since it is the same as the first embodiment except that it is changed, redundant description is omitted.

FIG. 13 is a block diagram showing the electrical configuration of the operator-side terminal 16 of the second embodiment. As shown in FIG. 13 , the operator-side terminal 16 of the second embodiment further includes a sensor interface (sensor I/F) 66 and an inertial sensor 68 .

An inertial sensor 68 is connected to the sensor I/F 66 . In this second embodiment, an angular velocity sensor is used as the inertial sensor 68 . An inertial sensor 68 is provided in the headset consisting of the microphone 62 and the speaker 64, and detects vertical and horizontal movements of the operator's neck (in this second embodiment, nodding and shaking movements). Therefore, an angular velocity sensor capable of detecting angular velocity around an axis for detecting vertical movement of the operator's neck and around an axis for detecting horizontal movement of the operator's neck is used. As an example, the axis for detecting vertical movement of the operator's neck is an axis parallel to a straight line through the operator's ears. Also, as an example, the axis for detecting the lateral movement of the operator's neck is the axis extending in the extension direction through the top of the operator's head.

However, as the inertial sensor 68, a triaxial acceleration sensor may be used. In this case, the acceleration of the axes extending in the front direction of the operator's face, the horizontal direction of the head, and the vertical direction of the head is detected.

In the second embodiment, in the talk, angular velocity data ("neck movement data" to be described later) detected by the inertial sensor 68 when the operator speaks is added to the voice data corresponding to the operator's voice. It is added and transmitted to the user side terminal 12 .

However, the neck movement data is data on the average value of a plurality of angular velocities detected by the inertial sensor 68 for a third predetermined time period (about 1/10 second in this second embodiment). Calculated hourly. However, the average value is an example, and the maximum value of the volume in the third predetermined time period may be used. Also, the third predetermined time does not have to be the same as the second predetermined time.

When the user-side terminal 12 receives the voice data from the operator-side terminal 16, it enlarges or reduces the avatar image 120 according to the inertia data added to the received voice data.

In this second embodiment, the avatar image 120 is enlarged based on the vertical movement of the operator's neck, and the size of the avatar image 120 is reduced based on the horizontal movement of the operator's neck. . However, this is only an example, and the size of the avatar image 120 is reduced based on the vertical movement of the operator's neck, and the size of the avatar image 120 is reduced based on the horizontal movement of the operator's neck. may be expanded.

FIG. 14A is a diagram for explaining a method of calculating the ratio p when the operator nods (that is, when the operator's neck moves to face downward), and FIG. 4 is a diagram for explaining a method of calculating the ratio p when the operator shakes his head (that is, when the operator's neck moves so as to turn his face to the right); FIG.

In this second embodiment, data about the movement of the operator's neck (hereinafter referred to as "neck movement data") is used to move the neck of the avatar in the virtual space, and based on the numerical values (parameters) obtained by this movement, , the ratio p is calculated. Although FIGS. 14A and 14B show images of the avatar's head and neck, in reality only calculations are performed and the avatar's image 120 is drawn to calculate the ratio p. does not need to be

Therefore, in the calculation, the model of the avatar's head is set as a sphere or an elliptical sphere, and in the sphere or elliptical sphere, spheres of a size corresponding to the avatar's eyeballs are placed at positions corresponding to the positions of the avatar's eyes. set. For the head, only the rotation axis X for nodding and the rotation axis Y for shaking the head are set.

In FIGS. 14A and 14B, the images of the avatar's head and neck shown on the left side show a state in which the neck is not moved, that is, the avatar faces the virtual camera. However, FIG. 14A is a side view of the avatar, and FIG. 14B is a top view of the avatar. Although illustration is omitted, in FIGS. 14(A) and 14(B), the virtual camera is in the front direction of the avatar and is arranged at a position separated by a predetermined distance. Also, the vertical positions of the virtual camera and the avatar are set so that the line of sight of the virtual camera passes through the center of the avatar's head.

As shown in FIG. 14A, when the operator nods, the head and eyeballs of the avatar are rotated forward (toward the virtual camera) around the rotation axis X. As shown in FIG. Among the eyeballs of the avatar, the plane that includes the part (point) that protrudes most toward the virtual camera side and is perpendicular to the line of sight of the virtual camera is taken as a reference plane, and the moving distance d of the reference plane before and after the movement of the neck is used to calculate the ratio p. Calculate As shown in FIG. 14A, when the operator nods, the reference plane moves closer to the virtual camera by the moving distance d.

Also, as shown in FIG. 14B, when the operator shakes his/her head, the head and eyeballs of the avatar are rotated rightward (not shown, but may be leftward) around the rotation axis Y. . The reference plane is a plane perpendicular to the line of sight of the virtual camera that includes the midpoint of a straight line that connects the points (points) of the avatar's eyeballs that protrude most toward the virtual camera. to calculate the ratio p. As shown in FIG. 14B, when the operator shakes his/her head, the reference plane moves away from the virtual camera by a moving distance d.

However, the movement distance d is the absolute value of the movement amount of the reference plane with respect to the virtual camera.

In the second embodiment, the ratio p is calculated according to Equation 2 when the operator nods, and the ratio p is calculated according to Equation 3 when the operator shakes his head. However, in Equations 2 and 3, D is the maximum moving distance of the reference plane, P is the ratio in the case of the maximum moving distance (enlargement ratio: 1.4), and Q is the ratio in the case of the maximum distance (reduction rate: 0.8). However, when the operator nods, the maximum movement distance D is the movement distance d when the operator nods until the operator's face becomes horizontal. It is the movement distance d when shaken. However, when the operator nods, the avatar image 120 is enlarged (that is, p>1), so in Equation 2, the movement distance d is greater than D/P. Also, when the operator shakes his head, the avatar image 120 is reduced (that is, p<1), so in Equation 3, the movement distance d is greater than DQ.

[Number 2]
p=P(d/D)
However, D/P<d≦D.

[Number 3]
p=Q(D/d)
However, DQ<d≦D.

Thus, in the second embodiment, instead of the volume data, the operator's neck movement data is added to the operator's voice data, and part of the ratio calculation processing shown in the first embodiment is performed. It differs from the first embodiment. However, the neck movement data also includes information that identifies whether the person is nodding or shaking his/her head.

Therefore, in the second embodiment, a motion detection program for detecting neck motion is stored in the program storage area of the storage unit 52 of the operator-side terminal 16 instead of the volume detection program. Further, in the second embodiment, in the data storage area of the storage unit 52 of the operator-side terminal 16, neck movement data is stored in place of volume data.

The motion detection program is a program for storing angular velocity data corresponding to the angular velocity detected by the inertial sensor 68 in the data storage section of the storage section 52 while the user's voice is being detected according to the sound detection program. . Also, the neck motion data is data about the motion of the operator's neck detected according to the motion detection program.

As shown in FIG. 15, in the ratio calculation process of the second embodiment, the process of step S71a is executed instead of the process of step S71. In step S71a, the CPU 20 determines whether or not the parameter indicating the movement of the operator's neck (movement distance d in the second embodiment) is within a predetermined range. That is, when the operator nods, it is determined whether or not the moving distance d is within the range described in Equation (2). Also, when the operator shakes his/her head, it is determined whether or not the moving distance d is within the range described in Equation (3).

If "NO" in step S71a, that is, if the parameter indicating the motion of the operator's neck is not within the predetermined range, the ratio calculation process is terminated and the process returns to the control process. On the other hand, if "YES" in step S71a, that is, if the parameter indicating the motion of the operator's neck is within a predetermined range, the ratio p is calculated in step S73. However, when the operator nods, the CPU 20 calculates the ratio p (the enlargement ratio in the second embodiment) according to Equation (2). Further, when the operator shakes his/her head, the CPU 20 calculates the ratio p (the reduction ratio in the second embodiment) according to Equation (3).

Subsequently, in step S75, the calculated ratio p is stored, that is, the ratio data 304e is updated with the calculated ratio p, the ratio calculation process is terminated, and the process returns to the control process.

In the second embodiment as well, by displaying the avatar image enlarged or reduced more than usual, it is possible to express a sense of depth. is obtained. That is, it is possible to display an avatar with increased presence.

Also in the second embodiment, when the enlarged avatar image is displayed so as to protrude from the frame image, it looks like it is about to jump out into the three-dimensional real space. That is, it is possible to display an avatar with increased presence.

In the second embodiment, the avatar's head model is set in a three-dimensional virtual space, the avatar's neck is moved based on the movement of the operator's neck, and numerical values (parameters) obtained thereby are used. Although the ratio p is calculated, it is not necessary to be limited to this. In another example, the head model of the avatar may be set as a circle or an ellipse in a two-dimensional virtual space, and two-dimensional calculations may be performed separately for nodding and shaking the head. That is, as shown in FIGS. 14(A) and 14(B), they are calculated separately. In this case, a reference line is set instead of the reference plane, and the moving distance d is calculated.

Further, in the second embodiment, the ratio p is calculated by dividing the motion of the operator's neck into the case of nodding and the case of shaking the operator's neck. , that is, the ratio p may be calculated based on the magnitude (maximum value or average value) of the angular velocities around each axis.

In addition, in the second embodiment, in chat and talk, the user terminal 12 and the operator terminal 16 communicate via the network 14, but they may communicate via the server 18. . In such a case, the server 18 calculates the ratio p based on the operator's neck movement indicated by the neck movement data added to the operator's voice data, and the server 18 adds the calculated ratio p to the voice data. Data may be added and transmitted to the user terminal 12 .

Furthermore, in the second embodiment, the motion of the operator's neck is detected by the inertial sensor, but it is not necessary to be limited to this. In another example, neck movement may be detected based on the orientation of the operator's face. In such a case, instead of the inertial sensor, an image sensor (CCD camera, ie web camera) is provided at a position where the operator's face can be photographed. As an example, a CCD camera is provided above the display device 60, and based on the orientation of the face image of the operator facing the CCD camera (or the display device 60), the current face orientation is based on the current face image. , and the motion of the operator's neck is estimated based on the current face orientation. However, the orientation of the face can be detected from the movement of a plurality of facial feature points extracted from the face image.

<Third embodiment>
The third embodiment is the same as the first embodiment except that the operator terminal 16 generates image data corresponding to the image 120 of the avatar, so redundant description will be omitted.

Briefly, in the third embodiment, at least in the talk, the operator-side terminal 16 generates image data corresponding to the avatar image 120, transmits the generated image data to the user-side terminal 12, and uses it. The person-side terminal 12 displays the avatar image 120 on the talk screen 150 using the received image data.

In the third embodiment, when the operator speaks in a talk, the operator-side terminal 16 detects the voice uttered by the operator and its volume, and if the volume of the voice of the operator is greater than a predetermined value, 1 to calculate the ratio p. As described above, when the volume of the operator's voice is equal to or less than the predetermined value, the ratio p is not calculated and remains at the initial value (p=1).

Subsequently, the operator-side terminal 16 generates image data of an avatar image 120 that performs a speaking action in accordance with the voice of the response contents of the operator, with a size corresponding to the ratio p.

The method of generating the image data of the avatar image 120 is the same as the method described in the first embodiment.

The operator terminal 16 transmits the audio data of the detected audio and the generated image data to the user terminal 12 . The user-side terminal 12 receives the audio data and the image data, and uses the image data to display the avatar image 120 on the talk screen 150 in accordance with the output of the audio data. That is, in the user-side terminal 12, the voice of the operator is output from the speaker 34, and the avatar image 120 expressed as if speaking is displayed on the talk screen 150 in a size corresponding to the ratio p. be done.

Therefore, in the third embodiment, the same programs as the avatar control program 302g and ratio calculation program 302h shown in FIG. Therefore, in the third embodiment, the ratio calculation program 302h and the ratio data 304e are deleted in the user terminal 12. FIG. Further, the same data as the ratio data 304e shown in FIG.

Further, in the operator-side terminal 16 of the third embodiment, during talk, voice data transmitted from the user-side terminal 12 is received and output, and voice data and volume data of the operator are detected, In addition to the process of transmitting the voice data to the user-side terminal 12, the process of generating the image data of the avatar image 120 (hereinafter referred to as "avatar image generation process") is executed.

Specifically, the operator-side terminal 16 calculates a ratio p based on the volume of the voice uttered by the operator, and speaks in accordance with the calculated ratio p in accordance with the voice data of the response content. image data of the image 120 of the avatar performing the

The operator-side terminal transmits the detected audio data and the generated image data to the user-side terminal 12 . Further, in the third embodiment, since the operator-side terminal 16 generates the image data of the avatar image 120, volume data is not added to the voice data.

Specific processing will be described below. FIG. 16 is a flowchart showing part of the control processing of the CPU 20 of the user-side terminal 12 in the third embodiment. FIG. 17 is a flowchart showing avatar image generation processing of the CPU 50 of the operator-side terminal 16 in the third embodiment. Hereinafter, the control processing of the CPU 20 will be explained, and the avatar image generation processing of the CPU 50 will be explained, but the explanation of the already explained processing will be omitted.

As shown in FIG. 16, the CPU 20 of the user-side terminal 12, when "NO" in the step S33 or when the process of the step S35 is executed, in a step S91, the voice data and the image of the response content Determine if data has been received.

If "NO" in step S91, that is, if voice data and image data of the response content have not been received, the process proceeds to step S47. On the other hand, if "YES" in step S91, that is, if voice data and image data of the response content have been received, in step S93, the image data is output in accordance with the output of the voice data of the response content, Return to step S33. Therefore, the operator's voice is output from the user-side terminal 12, and an image in which the avatar speaks with a size corresponding to the ratio p is displayed according to the output voice.

Next, avatar image generation processing of the CPU 50 of the operator-side terminal 16 will be described with reference to FIG. Note that the avatar image generation processing is executed when the operator's voice is detected by the microphone 62 .

As shown in FIG. 17, when the avatar image generation process is started, the CPU 50 determines in step S111 whether or not the sound volume is greater than a predetermined value. If "NO" in step S111, that is, if the volume is equal to or less than the predetermined value, the process proceeds to step S115. On the other hand, if "YES" in step S111, that is, if the volume is greater than the predetermined value, in step S113, the ratio p is calculated according to Equation 1, and the process proceeds to step S115.

In step S115, the image data of the image 120 of the avatar performing a speech action in accordance with the voice data of the response content is generated with a size corresponding to the ratio p, and the avatar image generation processing ends.

The image data generated in this manner is transmitted to the user-side terminal 12 together with the voice data of the content of the response.

In the third embodiment as well, by displaying the avatar image enlarged or reduced more than usual, it is possible to express a sense of depth. is obtained. That is, it is possible to display an avatar with increased presence.

Also in the third embodiment, when the enlarged avatar image is displayed so as to protrude from the frame image, it looks like it is about to jump out into the three-dimensional real space. That is, it is possible to display an avatar with increased presence.

In addition, in the third embodiment, the avatar image 120 is enlarged or reduced based on the volume of the voice of the operator, but it is not necessary to be limited to this. The avatar image 120 may be enlarged or reduced based on the degree of eye opening of the operator.

Also, in the third embodiment, as shown in the second embodiment, the avatar image 120 may be enlarged or reduced based on the motion of the operator's neck. In this case, in the avatar image generation process shown in FIG. 17, not only the voice data when the operator speaks but also the movement data of the operator's neck are detected. Further, instead of the process of step S111 shown in FIG. 17, a process of judging whether or not the parameter indicating the movement of the operator's neck (movement distance d in the third embodiment) is within a predetermined range is executed. be done. That is, when the operator nods, it is determined whether or not the moving distance d is within the range described in Equation (2). Also, when the operator shakes his/her head, it is determined whether or not the moving distance d is within the range described in Equation (3). If the parameter indicating the motion of the operator's neck is not within the predetermined range, the process proceeds to step S115. Then, the process proceeds to step S115.

<Fourth embodiment>
The fourth embodiment is the same as the first embodiment except that the control processing is executed on the server 18 side, so redundant description will be omitted.

As described above, in the fourth embodiment, the server 18 executes the control process, so the user terminal 12 functions as an input/output device for the control process. Therefore, in the control process, the user-side terminal 12 transmits to the server 18 the operation data corresponding to the user's operation or input and the voice data corresponding to the user's voice. Output data and audio data.

In the information processing system 10 of the fourth embodiment, the application explained in the first embodiment is stored in the server 18 and executed by the server 18 .

Therefore, in the fourth embodiment, the program storage area of the storage unit (RAM) 18b of the server 18 contains the activation determination program 302a, the main processing program 302b, the communication program 302d, the image generation program 302e, and the image output program shown in FIG. The same programs as program 302f, avatar control program 302g, ratio calculation program 302h and sound output program 302j are stored. However, in the fourth embodiment, the image output program 302f outputs (or transmits) image data generated according to the image generation program 302e to the user terminal 12. FIG. In addition, the sound output program 302j outputs (or transmits) the received voice data of the operator's response to the user-side terminal 12 .

A data storage area of a storage unit (RAM) 18b of the server 18 stores transmission data, reception data, image generation data, and ratio data. The transmission data is data to be transmitted to the user-side terminal 12, and includes image data of the web screen 100, the selection screen 110, the chat screen 130, the talk screen 150, the image data of the avatar image 120, and the content of the user's question in the chat. text data about the operator's response in the chat, voice data about the user's question in the talk, and voice data about the operator's response. The received data is data transmitted and received from the operator-side terminal 16, and includes text data about the content of the operator's response in chat and voice data about the content of the operator's response in talk (in the fourth embodiment, volume audio data to which data is added).

The image generation data is data for generating various screens displayed on the display device 30 of the user-side terminal 12 and includes data for generating the avatar image 120 . In addition, the data for generating the avatar image 120 includes static image data for the avatar image 120 and animation data for unconscious actions and greeting actions. Ratio data is data about the ratio p.

In addition, in the fourth embodiment, since the server 18 executes the control processing, the user terminal 12 can omit the activation determination program 302a, the avatar control program 302g, and the ratio calculation program 302h. Similarly, the user-side terminal 12 does not store the ratio data 304e.

Specifically, the CPU 18a of the server 18 executes control processing shown in FIGS. The control processing executed by the CPU 18a of the server 18 will be described below with reference to FIGS. 18 to 20, but the content that overlaps with the content already described will be briefly described.

As shown in FIG. 18, when the control process is started, the CPU 18a of the server 18 determines in step S201 whether or not the application activation condition is satisfied. If "NO" in step S201, the process returns to step S201. On the other hand, if "YES" in step S201, the application is activated in step S203.

In subsequent step S205, the selection screen 110 as shown in FIG. 4 is displayed on the user terminal 12. FIG. That is, the CPU 18a generates image data for the selection screen 110 as shown in FIG. Therefore, the selection screen 110 is displayed in front of the web screen 100 on the display device 30 of the user-side terminal 12 . The same applies to the case where the screen is displayed on the user-side terminal 12 below.

However, when the application is first started, the CPU 18a generates the image data of the selection screen 110 that displays the static avatar image 120 in the display frame 112, transmits the image data to the user terminal 12, and then displays the avatar. Animation data for performing a greeting action is output to the user-side terminal 12 .

In the next step S207, the avatar image 120 is caused to perform a standby action. Here, the CPU 18a transmits to the user terminal 12 animation data for causing the avatar to perform an unconscious action. However, the CPU 18a transmits animation data for causing the avatar to perform an unconscious action every several seconds or several tens of seconds when there is no user operation on the selection screen 110 .

Subsequently, in step S209, it is determined whether or not there is a button operation. Here, the CPU 18a determines whether operation data indicating that the button 114 or the button 116 of the selection screen 110 has been turned on has been received from the user-side terminal 12 or not. Although illustration is omitted, when operation data indicating that the button 118 is turned on is received, the user-side terminal 12 is instructed to close (hide) the selection screen 110, and the application is executed. finish.

If "NO" in step S209, the process returns to step S207. On the other hand, if "YES" in step S209, it is determined in step S211 whether or not it is a chat. Here, the CPU 18a determines whether or not the operation data indicates that the button 114 is turned on.

If "NO" in step S211, that is, if button 116 is ON, the process proceeds to step S231 shown in FIG. On the other hand, if "YES" in step S211, that is, if button 114 is turned on, chat screen 130 as shown in FIG. 5 is displayed on user terminal 12 in step S213 shown in FIG. That is, the CPU 18a generates image data of the chat screen 130 as shown in FIG.

Therefore, the selection screen 110 is not displayed on the display device 30 of the user-side terminal 12 , and the chat screen 130 is displayed in front of the web screen 100 .

In the next step S215, it is determined whether or not the content of the question is input. Here, the CPU 18a determines whether operation data indicating key input of question content (text) has been received from the user-side terminal 12 or not.

If "YES" in step S215, then in step S217 image data for displaying characters or character strings corresponding to the key input indicated by the operation data as text in display frame 136 is generated and transmitted to user-side terminal 12. Then, the process returns to step S215.

Therefore, on the user-side terminal 12, characters or character strings regarding the content of the question input by the user are sequentially displayed in the display frame 136 of the chat screen 130. FIG.

On the other hand, if "NO" in step S215, it is determined in step S219 whether or not the content of the question is to be transmitted. Here, the CPU 18a determines whether it has received operation data from the user-side terminal 12 indicating that the content of the question has been confirmed (or that the content of the question has been transmitted).

If "YES" in step S219, then in step S221 the text data of the question content is transmitted to the operator terminal 16, and the process proceeds to step S223. However, the text data of the question content is data in which characters or character strings input by the user are arranged in chronological order in this question. On the other hand, if "NO" in step S219, the process proceeds to step S223.

In step S223, it is determined whether the text data of the content of the response has been received from the operator-side terminal 16 or not. If "NO" in step S223, the process proceeds to step S227. On the other hand, if "YES" in step S223, in step S225 the avatar is made to speak in accordance with the display of the response content, and the process proceeds to step S227. In step S225, the CPU 18a generates image data for displaying the content of the response as text in the display frame 134 one character at a time, transmits the image data to the user-side terminal 12, and changes the lips so as to speak according to the display. Image data of the image 120 of the avatar performing the speaking motion is generated and transmitted to the user-side terminal 12 .

Accordingly, on the user-side terminal 12, the response contents are displayed one character at a time in the display frame 134 of the chat screen 130, and the image 120 of the avatar performing the speaking action is displayed in the display frame 132 in accordance with the display of the response contents. be.

In step S227, it is determined whether or not the chat has ended. Here, the CPU 18a determines whether it has received operation data indicating that the button 138 is turned on, or whether it has received operation data indicating the end of the chat from the operator-side terminal 16. FIG.

If "NO" in step S227, the process returns to step S215. On the other hand, if "YES" in step S227, the user terminal 12 is instructed to close the chat screen 130 in step S229, and the process returns to step S205 shown in FIG.

Also, as described above, if "NO" in step S211, the talk screen 150 as shown in FIG. 6 is displayed on the user terminal 12 in step S231 shown in FIG. That is, the CPU 18a generates image data of the talk screen 150 as shown in FIG.

Therefore, on the display device 30 of the user-side terminal 12 , the selection screen 110 is hidden, and the talk screen 150 is displayed in front of the web screen 100 .

In the following step S233, it is determined whether or not voice data of the content of the user's question has been received from the user-side terminal 12 or not. If "NO" in step S233, that is, if voice data of question content has not been received, the process proceeds to step S237. On the other hand, if "YES" in step S233, that is, if the voice data of the question content is received, the voice data of the question content is transmitted to the operator side terminal 16 in step S235, and the process proceeds to step S237.

In other words, the server 18 transmits the received voice data of the content of the question to the operator side terminal 16 . Therefore, the user's voice is output from the speaker 64 of the operator-side terminal 16 . On the other hand, the operator side terminal 16 adds volume data to the voice data of the contents of the operator's response and transmits the data to the server 18 .

In step S237, it is determined whether or not voice data of the content of the operator's response has been received. If "YES" in step S237, that is, if the voice data of the operator's response content is received, in step S239 the ratio calculation process shown in FIG. 12 is executed, and in step S241 The image data of the image 120 of the avatar that speaks according to the voice data of the response content is generated.

However, the image data of the avatar image 120 generated in step S241 is the same as that in the third embodiment (step S115), and has a size corresponding to the ratio p, and is lip-synced to the operator's voice and speaks. This is image data about an image 120 of an avatar performing

In the next step S243, the voice data of the received response content and the image data generated in step S241 are transmitted to the user-side terminal 12, and the process returns to step S233. Therefore, in the user-side terminal 12, the voice of the response content is output from the speaker 34, and on the talk screen 150, the voice of the response content is lip-synced to the voice of the response content at a volume corresponding to the ratio p. image 120 is displayed in the display frame 152 .

If "NO" in step S237, the ratio p is reset in step S245, and the image data of the avatar image 120 in the normal size is transmitted to the user terminal 12 in step S247. The process proceeds to step S249. Therefore, in the user-side terminal 12, when the output of the operator's voice ends, the size of the avatar image 120 is returned to the normal size.

In step S249, it is determined whether or not the talk has ended. Here, the CPU 18a determines whether it has received operation data indicating that the button 154 is turned on from the user-side terminal 12 or whether it has received operation data indicating termination of the talk from the operator-side terminal 16. FIG.

If "NO" in step S249, the process returns to step S233. On the other hand, if "YES" in step S249, the user terminal 12 is instructed to close the talk screen 150 in step S251, and the process returns to step S205.

In the fourth embodiment as well, by displaying the avatar image enlarged or reduced more than usual, it is possible to express a sense of depth. is obtained. That is, it is possible to display an avatar with increased presence.

Also in the fourth embodiment, when the enlarged avatar image is displayed so as to protrude from the frame image, it looks like it is about to jump out into the three-dimensional real space. That is, it is possible to display an avatar with increased presence.

In the fourth embodiment, the voice of the user's question in the talk is transmitted from the user terminal 12 to the operator terminal 16 via the server 18. It may be transmitted from the operator-side terminal 12 to the operator-side terminal 16 .

Also, in the fourth embodiment, as shown in the second embodiment, the avatar image 120 may be enlarged or reduced based on the motion of the operator's neck. In this case, the operator-side terminal 16 transmits to the server 18 voice data to which neck movement data is added. Then, in step S45, the ratio calculation process of the second embodiment shown in FIG. 15 is executed.

As shown in the above embodiments, in the information processing system 10, the ratio p can be calculated by either the user terminal 12, the operator terminal 16, or the server 18. FIG. Further, in the information processing system 10 , the avatar image 120 can be generated by any one of the user terminal 12 , the operator terminal 16 and the server 18 .

In addition, in each of the above-described embodiments, the avatar image 120 having a size corresponding to the ratio p is displayed, and the avatar image 120 performing a speaking action in accordance with the voice of the response content is displayed. The presence of the avatar can be increased by simply displaying the avatar image 120 in a size corresponding to the ratio p without performing any action.

Furthermore, in each of the above-described embodiments, the avatar image 120 having a size corresponding to the ratio p is displayed. 120 may be enlarged to a size that protrudes from the display frame 152 . For example, the avatar image 120 is made 1.4 times larger than normal. However, this is only an example, and the size may be such that it protrudes from the display frame 152 . Specifically, in the first embodiment and the second embodiment, steps S39 and S45 shown in FIG. 11 are deleted, and in step S43, the CPU 20 reproduces the voice of the content of the response in a size magnified 1.4 times. It generates and outputs image data of Aber et al., who speaks according to the data. Further, in the third embodiment, steps S111 and S113 shown in FIG. 17 are omitted, and in step S115, the CPU 50 performs a speech operation in accordance with the voice data of the response contents in a size enlarged by 1.4 times. Aber et al. generate image data. Furthermore, in the fourth embodiment, steps S239 and S245 shown in FIG. 20 are deleted, and in step S241, the CPU 18a performs an utterance operation in accordance with the voice data of the response content in a size enlarged by 1.4 times. Image data for the avatar image 120 is generated.

Thus, when the ratio p is not calculated, the operator terminal 16 transmits only the audio data of the operator's voice to the user terminal 12 or the server 18, and transmits volume data or neck movement data. There is no need to detect volume data or neck movement data.

Further, when the ratio p is not calculated, the volume of the voice when the operator's voice is detected or the movement of the operator's neck is irrelevant. , the avatar image 120 may be enlarged to a size that protrudes from the display frame 152 . Even in this way, it is possible to display an avatar with an increased presence. Specifically, in the first, second, and third embodiments, in step S25 shown in FIG. 10, the CPU 20 displays the response content as text in the display frame 134 one character at a time, and increases the magnification by 1.4. The image 120 of the avatar is displayed in a display frame 132 in a size enlarged to 120, and the lip part is changed so that the avatar speaks according to the display. However, "normal time" in this case includes a state in which the text of the response content is not displayed.

Furthermore, when the ratio p is not calculated, instead of enlarging the size of the avatar image 120, the avatar image 120 is changed (or deformed) so that the avatar image 120 is displayed in the display frame 132 or It can also be displayed so as to protrude from the display frame 152 . As an example, as shown in FIG. 21, it is possible to display an avatar image 120 in which part of the avatar's hands and head protrudes from the frame image (display frame 152). Even in this way, it looks like the avatar is about to jump out into the three-dimensional real space. That is, it is possible to display an avatar with increased presence.

However, when the ratio p is calculated and the ratio p is greater than 1, the avatar image 120 is adjusted so that the degree of protruding from the display frame 152 increases in proportion to the size of the ratio p. may be changed.

As described above, the avatar image 120 is enlarged or reduced, or the avatar image 120 is changed (or deformed) to change the avatar mode, thereby displaying an avatar with increased presence. be able to.

In addition, even if the aspect of the avatar is changed by enlarging or changing the avatar image 120 without calculating the ratio p, as described above, the sense of presence is increased without performing the speaking motion. You can display your avatar.

In each of the above-described embodiments, in the chat, text is exchanged with the operator. may be exchanged.

Further, in each of the above-described embodiments, the image of the avatar on the talk screen is an image of the head and neck of the avatar. The image of the avatar's head and neck is enlarged or reduced accordingly, but when displaying the image of the upper body or the whole body of the avatar, the image of the upper body or the whole body of the avatar is enlarged or reduced. good too. In this case, the speech action of the avatar includes gestures using the upper body or the whole body.

Furthermore, in each of the above-described embodiments, based on predetermined information when the operator speaks, such as volume data or neck movement data included in voice data transmitted from the operator-side terminal, the user-side Although the ratio is calculated by the terminal, the ratio may be calculated by the operator's terminal. In such a case, the operator side terminal adds the calculated ratio data to the voice data and transmits it to the user side terminal. The user-side terminal displays (renders) the image of the avatar at the ratio indicated by the ratio data added to the received voice data. In this way, when the ratio data is added to the voice data transmitted from the operator side terminal, this ratio data is the predetermined information.

Furthermore, in each of the above-described embodiments, the operator's voice is directly output from the speaker of the user-side terminal, but the operator's voice may be converted to be output.

Also, in each of the above-described embodiments, an application capable of executing chat or talk has been described, but it is not necessary to be limited to this. An application capable of executing only talk (that is, exchange of voice) may be used. In another example, it can also be applied when conducting a web conference or video call, and when displaying an avatar image in a web conference or video call, the volume of the voice spoken by the corresponding person or the volume of the voice when speaking It is enlarged or reduced by a ratio p according to the movement of the human neck. In other words, the present invention outputs a voice uttered by a certain person to a terminal used by another person who interacts with the certain person, and outputs an avatar image corresponding to the certain person to the other person. This is applicable when displaying on a terminal that

Furthermore, in each of the embodiments described above, the application is started when the activation condition is satisfied, but the application may be activated when the web screen is displayed.

It should be noted that the order of processing can be changed if the steps in the flowcharts shown in the above-described embodiments yield the same result.

Further, the various screens, angles, and other specific numerical values given in each of the above-described embodiments are merely examples, and can be appropriately changed as necessary. For example, in the case of talk, instead of displaying the talk screen, it is possible to display only the image of the avatar and the display frame (the frame is an image).

DESCRIPTION OF SYMBOLS 10... Information processing system 12... User-side terminal 14... Network 16... Operator-side terminal 18... Server 18a, 20, 50... CPU
18b, 22, 52...storage section 24, 54...communication I/F
26, 56 ... input/output I/F
28, 58 ... input device 30, 60 ... display device 32, 62 ... microphone 34, 64 ... speaker 66 ... sensor I/F
68 ... inertial sensor

Claims (19)

Receiving means for receiving the voice uttered by the operator and predetermined information when the operator uttered from the operator-side terminal;
sound output means for outputting the sound received by the receiving means;
Based on the predetermined information received by the receiving means, the image of the avatar corresponding to the operator is enlarged or reduced relative to the normal size when the voice received by the receiving means is not output. ratio calculation means for calculating a ratio; and image display means for displaying, on a display, an image of the avatar drawn at the ratio calculated by the ratio calculation means when the sound is output by the sound output means. processing equipment. 2. The information processing apparatus according to claim 1, wherein said image display means further displays a frame image in which said image of said avatar fits in said normal state, and displays said image of said avatar in front of said frame image. the predetermined information is the volume of the voice uttered by the operator;
3. The information processing apparatus according to claim 1, wherein said ratio calculating means calculates said ratio based on said volume. the predetermined information is movement of the operator's neck when the operator speaks;
3. The information processing apparatus according to claim 1, wherein said ratio calculating means calculates said ratio based on a movement of said operator's neck. Receiving means for receiving, from an operator-side terminal, a ratio calculated based on the voice uttered by the operator and predetermined information when the operator uttered the voice;
sound output means for outputting the sound received by the reception means; and an image for displaying on a display the image of the avatar drawn at the ratio received by the reception means when the sound is output by the sound output means. comprising display means,
The information processing apparatus, wherein the ratio is a ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to a normal size when the voice received by the receiving means is not output. Receiving means for receiving the voice uttered by the operator and the image of the avatar drawn at a ratio calculated based on predetermined information when the operator uttered the voice;
sound output means for outputting the sound received by the reception means; and image display means for displaying the image of the avatar received by the reception means on a display when the sound output means outputs the sound. prepared,
The information processing apparatus, wherein the ratio is a ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to a normal size when the voice received by the receiving means is not output. Receiving means for receiving the voice uttered by the operator and predetermined information when the operator uttered from the operator-side terminal;
sound output means for outputting the voice received by the receiving means to a user-side terminal used by a user who interacts with the operator;
An image of the avatar corresponding to the operator based on the predetermined information received by the receiving means is transferred to a user terminal used by the user who interacts with the operator by transmitting the voice received by the receiving means. Ratio calculation means for calculating a ratio of enlargement or reduction with respect to the normal size when not being output; An information processing apparatus comprising image output means for outputting an image of the avatar to the user terminal. Receiving means for receiving text or voice uttered by an operator;
An output means for outputting the text or the voice received by the receiving means, and an image display means for displaying an image of an avatar corresponding to the operator on a display,
The image display means displays the image of the avatar on the display in a manner that fits within a frame image in a normal time when the text or the voice received by the receiving means is not output, and the output means outputs the text or the voice. An information processing apparatus that displays an image of the avatar on the display in such a manner as to protrude from the frame image when outputting a sound. A control program executed by an information processing device,
In the processor of the information processing device,
a receiving step of receiving, from the operator-side terminal, the voice uttered by the operator and predetermined information when the operator uttered the voice;
a sound output step of outputting the sound received in the receiving step;
A ratio to enlarge or reduce the image of the avatar corresponding to the operator based on the predetermined information received in the receiving step with respect to the size received in the receiving step during a normal time when the voice is not output. and an image display step of displaying an image of the avatar drawn at the ratio calculated in the ratio calculation step on a display when outputting the sound in the sound output step. A control program executed by an information processing device,
In the processor of the information processing device,
a receiving step of receiving, from the operator-side terminal, the voice uttered by the operator and the ratio calculated based on predetermined information when the operator uttered the voice;
a sound output step of outputting the voice received in the receiving step; and an image display step of displaying, on a display, an image of the avatar drawn at the ratio received in the receiving step when outputting the voice in the sound output step. and
The control program, wherein the ratio is a ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to a normal size when the voice received in the receiving step is not output. A control program executed by an information processing device,
In the processor of the information processing device,
a receiving step of receiving the voice uttered by the operator and the image of the avatar drawn at a ratio calculated based on predetermined information when the operator uttered the voice;
a sound output step of outputting the voice received in the receiving step; and an image display step of displaying the image of the avatar received in the receiving step on a display when outputting the voice in the sound output step. ,
The control program, wherein the ratio is a ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to a normal size when the voice received in the receiving step is not output. A control program executed by an information processing device,
In the processor of the information processing device,
a receiving step of receiving, from the operator-side terminal, the voice uttered by the operator and predetermined information when the operator uttered the voice;
A sound output step of outputting the voice received in the receiving step to a user-side terminal used by a user who interacts with the operator;
outputting an image of an avatar corresponding to the operator based on the predetermined information received in the receiving step, and outputting the voice received in the receiving step to a user terminal used by a user who interacts with the operator; a ratio calculation step of calculating a ratio of enlargement or reduction with respect to the normal size when not in use; and when outputting the voice in the sound output step, the avatar drawn at the ratio calculated in the ratio calculation step A control program for executing an image output step of outputting an image to the user-side terminal. A control program executed by an information processing device,
In the processor of the information processing device,
a receiving step for receiving the text entered by the operator or the voice spoken by the operator;
executing an output step of outputting the text or the voice received in the receiving step, and an image display step of displaying an image of the avatar corresponding to the operator on a display;
In the image display step, the image of the avatar is displayed on the display in a manner that fits within a frame image in a normal time when the text or the voice received in the receiving step is not output, and in the output step, the text or the voice is displayed. is displayed on the display in such a manner that the image of the avatar protrudes from the frame image when the is output. A control method for an information processing device having a display,
(a) a step of receiving a voice uttered by an operator and predetermined information when said operator uttered from an operator-side terminal;
(b) outputting the audio received in step (a);
(c) based on the predetermined information received in step (a), the image of the avatar corresponding to the operator is displayed at a normal size when the voice received in step (a) is not output; and (d) displaying the avatar image drawn at the ratio calculated in step (c) on the display when outputting the sound in step (b). A control method, including the step of A control method for an information processing device having a display,
(a) a step of receiving from the operator-side terminal a ratio calculated based on the voice uttered by the operator and predetermined information when the operator uttered;
(b) outputting the audio received in step (a); and (c) rendering the avatar at the ratio received in step (a) when outputting the audio in step (b). displaying an image on the display;
The control program, wherein the ratio is a ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to a normal size when the sound is not output in step (b). A control method for an information processing device having a display,
(a) a step of receiving a voice uttered by an operator and an image of an avatar drawn at a ratio calculated based on predetermined information when the operator uttered;
(b) outputting the audio received in step (a); and (c) displaying an image of the avatar received in step (a) when outputting the audio in step (b). including steps to display in
The control method, wherein the ratio is a ratio for enlarging or reducing the image of the avatar corresponding to the operator with respect to a normal size when the sound is not output in step (b). A control method for an information processing device,
(a) a step of receiving a voice uttered by an operator and predetermined information when said operator uttered from an operator-side terminal;
(b) a step of outputting the voice received in step (a) to a user-side terminal used by a user who interacts with the operator;
(c) the image of the avatar corresponding to the operator based on the predetermined information received in step (a) is used by the user who interacts with the operator using the voice received in step (a); (d) when outputting the voice in step (b), the step (c) A control method, including a step of outputting the image of the avatar drawn at the ratio calculated in the above to the user-side terminal. A control method for an information processing device,
(a) receiving operator-inputted text or spoken audio;
(b) outputting the text or the voice received in step (a); and (c) displaying an image of an avatar corresponding to the operator on a display;
The step (c) displays the image of the avatar on the display in such a manner that it fits within a frame image during a normal time when the text or the voice received in the step (a) is not output, and in the step (b) A control method, wherein the image of the avatar is displayed on the display in such a manner as to protrude from the frame image when the text or the voice is output. An information processing system comprising a server, and a user-side terminal and an operator-side terminal communicably connected to the server,
Receiving means for receiving the voice uttered by the operator and predetermined information when the operator uttered from the operator-side terminal;
sound output means for outputting the sound received by the receiving means;
A ratio calculation for calculating a ratio for enlarging or reducing an image of the avatar corresponding to the operator based on the predetermined information with respect to a normal size when the voice received by the receiving means is not output. and image display means for displaying the image of the avatar drawn at the ratio calculated by the ratio calculation means on the display of the user-side terminal when the sound is output by the sound output means. processing system.

Images