JP2020136921A - Video call system and computer program - Google Patents

Abstract

To provide a video call system that requires a low processing load and enables a homeworker to provide a service without worrying about the worker's face, environment, and the like.SOLUTION: A video call system comprises: a first user device comprising imaging means for imaging a face image of a first user, a user who operates the first user device, voice input means for inputting voice of the first user, face information generation means for generating face information based on a part of the first user's face from the face image of the first user, and communication means for transmitting the first user's face information and voice to a second user device other than the first user device of a plurality of user devices; and the second user device comprising avatar image generation means for generating an avatar image of the first user on the basis of first user's face information transmitted from the first user device, image output means for outputting the avatar image of the first user, and voice output means for outputting the first user's voice.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video call system and a computer program.

Conventionally, an interpreter service for requesting an interpreter to interpret via a video call has been known (see, for example, Patent Document 1).

In recent years, the number of foreign visitors to Japan has increased, and foreign travelers have more opportunities to enjoy traveling by using cloud interpreters with interpreters who work in crowdsourcing.

In recent years, it has become common for interpreters who are home-based workers to become interpreters in cloud interpretation services by using inbound tourism.

Japanese Unexamined Patent Publication No. 2004-206452

However, as mentioned above, the number of home-based workers is increasing as interpreters involved in cloud interpretation, and there are some situations in which the clothes or background (environment) of the interpreter is not appropriate.

In particular, when the interpreter's background gives the interpreter a sense of life, it is not a favorable situation for both the interpreter and the interpreter.

In addition, when the interpreter was a woman, she had to put on makeup for the interpreter even though she was at home, and she sometimes hesitated to become an interpreter.

Therefore, there has been a demand for a mechanism that allows an interpreter who is a home worker to perform interpreting work without worrying about the face, clothes, environment, and the like. This issue applies not only to cloud interpreters, but also to video calls in other areas.

Furthermore, in a conventional video communication system, since video information with a large amount of data is transmitted and received between the interpreter's device and the interpreter's device, a large amount of communication band is required, and the processing load of the server device is increased. Becomes large. As a result, real-time communication may not be possible, or communication between the interpreter and the interpreter requester may be interrupted.

An object of the present invention is to provide a video call system having a small processing load and capable of providing a service to a home worker without worrying about the face and the environment.

The first invention is
A video communication system including a plurality of user devices that are connected to each other by a communication network and receive operations of each user.
The first user device, which is at least one of the user devices,
A photographing means for capturing a face image of the first user who operates the first user device,
A voice input means for inputting the voice of the first user,
A face information generating means for generating face information regarding a face portion of the first user from the photographed face image of the first user, and at least the face information and the voice of the first user are used by the plurality of users. A communication means for transmitting to a second user device other than the first user device among the devices,
With
The second user device is
An avatar image generation means that generates and outputs an avatar image of the first user based on the face information of the first user transmitted from the first user device, and a voice that outputs the voice of the first user. Output means,
To prepare
It is a video call system.

Further, in the first invention,
The first user device is operated by an interpreter who has been requested to interpret.
The second user device may be operated by an interpreter requester who requests the interpreter to interpret.

Further, in the first invention,
The face information may be at least face recognition information regarding the presence / absence of opening / closing of the mouth, the presence / absence of opening / closing of eyes, and the inclination and rotation of the head, which are the parts of the face of the first user.

Further, in the first invention,
The avatar image generation means may generate the avatar image by varying the positions of the apex of the eyes, mouth, and face corresponding to the avatar image composed of the three-dimensional model based on the face recognition information. ..

Further, in the first invention,
The face information generating means generates emotion information for expressing the emotion of the first user based on at least one of the face information of the first user and the voice.
The avatar image generation means may generate the avatar image of the first user based on the emotion information in addition to the face information.

The second invention is
Computer equipment,
A photographing means for capturing a facial image of a user who operates the computer device,
A voice input means for inputting the user's voice,
A face information generating means for generating face information regarding a part of the user's face from the photographed face image of the user in order to generate an avatar image of the user in another computer device, and at least the face of the user. A communication means for transmitting information and the voice to the other computer device,
To function as
It is a computer program.

The third invention is
Computer equipment,
Regarding the other user who operates the other computer device from the other computer device based on the user's operation, the face portion of the other user generated from the face image of the other user taken by the other computer device. Communication means for receiving face information and voice,
An avatar image generation means that generates and outputs an avatar image of the other user based on the face information, and a voice output means that outputs the voice of the other user.
To function as
It is a computer program.

The fourth invention is
Computer equipment,
A communication means for receiving a face image and voice of the other user taken by the other computer device for another user who operates the other computer device from the other computer device based on the user's operation.
A face information generating means for generating face information regarding a face portion of the other user from the face image of the other user received from the other computer device.
An avatar image generation means that generates and outputs an avatar image of the other user based on the face information, and a voice output means that outputs the voice of the other user.
To function as
It is a computer program.

According to the present invention, it is possible to provide a video call system in which a processing load is small and a home worker can provide a service without worrying about the face and the environment.

It is a conceptual diagram of the video call system in this embodiment. It is a figure which shows the functional structure of the server apparatus in this embodiment. It is a figure which shows the functional structure of the interpreter terminal in this embodiment. It is a figure which shows the functional structure of the interpreter requester terminal in this embodiment. It is a figure which shows the flow of the video call processing in this embodiment. It is a figure which shows the face recognition information of the face of an interpreter in another embodiment. It is a figure which shows the face recognition information of the face of an interpreter in another embodiment.

[Embodiment]
The video communication system 1 according to the embodiment of the present invention will be described with reference to FIGS. 1 to 5.

<Explanation of video call system 1>
As shown in FIG. 1, the video communication system 1 of the present invention includes a server device 2, a plurality of interpreter terminals (first user device) 3, and a plurality of interpreter requester terminals (second user device) 4.

The server device 2 is connected to the interpreter terminal 3 and the interpreter requester terminal 4 via the communication network 5.

Then, the server device 2 transmits / receives video (image) and audio between the interpreter terminal 3 and the interpreter requester terminal 4. As a result, a video call is provided between the interpreter terminal 3 and the interpreter requester terminal 4.

The interpreter terminal 3 includes an operation unit that receives the operation of the interpreter (first user), and is connected to the interpreter requester terminal 4 and the server device 2 via the communication network 5.

The interpreter requester terminal 4 includes an operation unit that receives the operation of the interpreter requester (second user), and is connected to the interpreter terminal 3 and the server device 2 via the communication network 5.

<Hardware configuration>
Hereinafter, with reference to FIGS. 2 to 4, the functional configuration of the server device 2, the functional configuration of the interpreter terminal 3 provided with the video call processing, and the functional configuration of the interpreter requester terminal 4 will be described.

A different account (identification information) is assigned to each interpreter and each interpreter requester.

When each interpreter terminal 3 communicates with the server device 2 via the communication network 5, the interpreter terminal 3 transmits the interpreter's account to the server device 2.

Further, when each interpreter requester terminal 4 communicates with the server device 2 via the communication network 5, the interpreter requester terminal 4 transmits the interpreter requester account to the server device 2.

The transmitted account is subjected to predetermined authentication on the server device 2. As a result, communication between the interpreter terminal 3 or the interpreter requester terminal 4 and the server device 2 becomes possible. As a result, communication between the interpreter terminal 3 and the interpreter requester terminal 4 becomes possible.

<Explanation of server device 2>
As shown in FIG. 2, the server device 2 includes a control unit 20, a storage unit 21, and a network interface 22.

The storage unit 21 and the network interface 22 are connected to the control unit 20 of the server device 2 via the bus 200.

The control unit 20 controls the operation of the server device 2.

The storage unit 21 is mainly composed of an HDD (Hard Disk Drive), a RAM (Random Access Memory), and a ROM (Read Only Memory).

The storage unit 21 stores the registered information of the interpreter, such as the name of the interpreter, the languages that can be interpreted, the time zone in which the interpreter can be in charge, and the detailed area.

Further, the storage unit 21 stores the registration information of the interpreter requester, such as the name and location of the interpreter requester and the service plan selected by the interpreter requester.

The network interface 22 is connected to the communication network 5 in order to send and receive data between the server device 2 and the interpreter terminal 3 or the interpreter requester terminal 4.

<Functional configuration of control unit 20 of server device 2>
The control unit 20 of the server device 2 functions as an interpreter management means 201, an interpreter requester management means 202, a service management means 203, and a communication means 204 by executing a predetermined program.

<Explanation of interpreter management means 201>
The interpreter management means 201 authenticates the interpreter's account by using the interpreter's account transmitted from the interpreter terminal 3.

Further, the interpreter management means 201 stores the registered information of the interpreter transmitted from the interpreter terminal 3 in the storage unit 21. This registration information is preset based on the operation of the interpreter.

Further, the interpreter management means 201 reads the registration information of each interpreter from the storage unit 21 after receiving the information regarding the interpretation request from the interpreter requester terminal 4, and transmits the registration information to the service management means 203.

<Explanation of interpreter requester management means 202>
The interpreter requester management means 202 authenticates the account of the interpreter requester by using the account of the interpreter requester transmitted from the interpreter requester terminal 4.

Further, the interpreter requester management means 202 stores the registration information of the interpreter requester transmitted from the interpreter requester terminal 4 in the storage unit 21 after receiving the information regarding the interpreter request from the interpreter requester terminal 4. This registration information is set in advance or based on the operation of the interpreter requester at the time of interpreter request.

Further, the interpreter requester management means 202 reads the registration information of the interpreter requester from the storage unit 21 and transmits it to the service management means 203.

<Explanation of service management means 203>
The service management means 203 receives information regarding the interpretation request from the interpreter requester terminal 4 via the interpreter requester management means 202. The information regarding the interpretation request includes information such as the location of the interpretation requester and the selected language.

Further, the service management means 203 receives the information related to the interpreter request from the interpreter requester terminal 4, and then receives the registered information of the interpreter via the interpreter management means 201 (interpreterable language, interpreter business charge time zone). Select an interpreter who can handle the situation by referring to. After the interpreter is selected, the service management means 203 transmits an interpreter request (call) from the interpreter requester to the interpreter terminal 3 of the selected interpreter. After that, the communication connection between the interpreter terminal 3 and the interpreter requester terminal 4 is established.

Further, the service management means 203 calculates the service usage time and the like and calculates the service charge and the like when the call is disconnected at either the interpreter terminal 3 or the interpreter requester terminal 4.

In addition, the service management means 203 stores the accounts of the interpreter and the interpreter requester from the storage unit 21 when the communication is interrupted due to deterioration of the communication status of either the interpreter terminal 3 or the interpreter requester terminal 4. Attempts to reconnect, read, automatically, or based on an operation such as an interpreter.

<Explanation of communication means 204>
The communication means 204 receives the interpreter's account, the interpreter's registration information, the information on the interpreter response, the interpreter's face recognition information (described later), the voice information, and the like from the interpreter terminal 3.

Further, the communication means 204 receives the account of the interpreter requester, the registration information of the interpreter requester, the information related to the interpreter request, the video information of the interpreter requester, the audio information, and the like from the interpreter requester terminal 4.

Further, the communication means 204 transmits the face authentication information of the interpreter, voice information, and the like to the interpreter requester terminal 4.

Further, the communication means 204 transmits the video information and audio information of the interpreter requester to the interpreter terminal 3.

<Explanation of interpreter terminal 3>
The interpreter terminal 3 is, for example, a laptop computer.

Further, the interpreter terminal 3 can perform data communication with the server device 2 and the interpreter requester terminal 4 via a communication network 5 such as the Internet or a LAN. As a result, the interpreter terminal 3 provides a video call with the interpreter requester terminal 4 via the server device 2.

Further, as shown in FIG. 3, the interpreter terminal 3 includes a control unit 30, a storage unit 31, a network interface 32, a shooting unit 33, a voice input unit 34, a graphic processing unit 35, an audio processing unit 36, and an operation unit 37. ..

The storage unit 31, network interface 32, photographing unit 33, voice input unit 34, graphic processing unit 35, audio processing unit 36, and operation unit 37 are connected to the control unit 30 of the interpreter terminal 3 via the bus 300. ..

The control unit 30 controls the operation of the interpreter terminal 3.

The storage unit 31 is mainly composed of an HDD, a RAM, and a ROM.

The network interface 32 is connected to the communication network 5 in order to send and receive data between the interpreter terminal 3 and the server device 2 or the interpreter requester terminal 4.

The photographing unit 33 is connected to the camera 330. The image of the interpreter is acquired by the camera 330.

The voice input unit 34 is connected to the microphone 340. The voice of the interpreter is acquired by the microphone 340.

The graphic processing unit 35 is connected to the liquid crystal screen 350. An image of the interpreter requester is displayed on the liquid crystal screen 350.

The audio processing unit 36 is connected to the speaker 360. The voice of the interpreter requester is output from the speaker 360.

The operation unit 37 is connected to a keyboard and a mouse 370 (hereinafter, referred to as “keyboard or the like” 370). In the present embodiment, the operation signal from the interpreter is input to the operation unit 37 via the keyboard or the like 370 which is an input detection device. The interpreter operates the keyboard or the like 370 to answer the interpreter, disconnect the call, or the like.

<Functional configuration of control unit 30 of interpreter terminal 3>
By executing a predetermined program, the control unit 30 of the interpreter terminal 3 executes a call answering means 301, a shooting means 302, a voice input means 303, a face information generating means 304, a video output means 305, a voice output means 306, and It functions as a communication means 307.

<Explanation of call answering means 301>
The call answering means 301 receives a call from the interpreter requester (interpreter answer) distributed by the server device 2 based on the operation of the interpreter. Specifically, the interpreter response is performed by the user operating the "response" button displayed on the liquid crystal screen 350.

<Explanation of photographing means 302>
The photographing means 302 photographs the image of the interpreter based on the fact that the interpreter has made an interpreter response. Specifically, the photographing means 302 transmits information for activating the camera 330 to the photographing unit 33. As a result, the video information of the interpreter is acquired.

<Explanation of voice input means 303>
The voice input means 303 acquires the voice of the interpreter based on the fact that the interpreter has made an interpreter response. Specifically, the voice input means 303 transmits information for activating the microphone 340 to the voice input unit 34. As a result, the voice information of the interpreter is acquired.

<Explanation of face information generation means 304>
The face information generation means 304 analyzes the video information of the interpreter and generates face information regarding the facial part of the interpreter.

In the present embodiment, the face information is, for example, face recognition information regarding the presence / absence of opening / closing of the mouth, which is a part of the face of the interpreter, the presence / absence of opening / closing of eyes, and the inclination and rotation of the head. In generating this face recognition information, for example, an open source library using deep learning can be used. With this open source library, it is determined whether or not the mouth is opened or closed based on the information accumulated by deep running.

Further, the face information generating means 304 generates emotion information of the interpreter from the facial expression of the interpreter and the intonation of the voice of the interpreter based on the face recognition information.

For example, when the tone of the interpreter's voice is high, the face information generating means 304 generates emotional information so that the interpreter is in a state of having fun explaining. Further, in a predetermined case, the face information generating means 304 generates information for generating a heart mark, an exclamation mark, or the like around the avatar image. By reflecting these emotional information in the avatar image, the facial expression of the avatar image can be enriched.

Then, the face information generation means 304 transmits the face authentication information to the server device 2 via the communication means 307 described later.

<Explanation of video output means 305>
The video output means 305 outputs the video of the interpreter requester transmitted from the interpreter requester terminal 4 based on the interpreter's response to the interpreter. Specifically, the video output means 305 transmits information for displaying the video of the interpreter requester on the liquid crystal screen 350 to the graphic processing unit 35. As a result, the image of the interpreter requester is displayed on the liquid crystal screen 350.

<Explanation of audio output means 306>
The voice output means 306 outputs the voice of the interpreter requester transmitted from the interpreter requester terminal 4 based on the interpreter's response to the interpreter. Specifically, the voice output means 306 transmits information for outputting the voice of the interpreter requester from the speaker 360 to the audio processing unit 36. As a result, the voice of the interpreter requester is output from the speaker 360.

<Explanation of communication means 307>
The communication means 307 receives information related to the interpreter request, video information and audio information of the interpreter requester from the server device 2.

Further, the communication means 307 transmits information related to the interpreter response, face authentication information of the interpreter, voice information, and the like to the server device 2.

<Explanation of interpreter requester terminal 4>
The interpreter requester terminal 4 is, for example, a terminal device such as a smartphone or a tablet.

Further, the interpreter requester terminal 4 can perform data communication with the server device 2 and the interpreter terminal 3 via a communication network 5 such as the Internet or a LAN. As a result, the interpreter requester terminal 4 provides a video call with the interpreter terminal 3 via the server device 2.

As shown in FIG. 4, the interpreter requester terminal 4 includes a control unit 40, a storage unit 41, a network interface 42, a shooting unit 43, a voice input unit 44, a graphic processing unit 45, an audio processing unit 46, an operation unit 47, and location information. A detection unit 48 is provided.

The storage unit 41, network interface 42, shooting unit 43, voice input unit 44, graphic processing unit 45, audio processing unit 46, operation unit 47, and position information detection unit 48 are connected to the control unit 40 via the bus 400. Will be done.

The control unit 40 controls the operation of the interpreter requester terminal 4.

The storage unit 41 is mainly composed of an HDD, a RAM, and a ROM.

The network interface 42 is connected to the communication network 5 in order to send and receive data between the interpreter requester terminal 4 and the server device 2 or the interpreter terminal 3.

The photographing unit 43 is connected to the camera 430. The image of the interpreter requester is acquired by the camera 430.

The voice input unit 44 is connected to the microphone 440. The voice of the interpreter requester is acquired by the microphone 440.

The graphic processing unit 45 is connected to the liquid crystal screen 450. An avatar image of the interpreter is displayed on the liquid crystal screen 450.

The audio processing unit 46 is connected to the speaker 460. The voice of the interpreter is output from the speaker 460.

The operation unit 47 is connected to the touch pad 470. In the present embodiment, the operation signal from the interpreter requester is input to the operation unit 47 via the touch pad 470 which is an input position detection device. The interpreter requester operates the touch pad 470 to request an interpreter, disconnect the call, and the like.

The position information detection unit 48 is, for example, a GPS receiver and detects position information (for example, longitude and latitude) indicating the current position of the interpreter requester terminal 4.

<Functional configuration of the control unit 40 of the interpreter requester terminal 4>
By activating the interpreting application, the control unit 40 of the interpreter requester terminal 4 activates the interpreter request means 401, the photographing means 402, the voice input means 403, the avatar image generation means 404, the voice output means 405, the position information detection means 406, And functions as a communication means 407.

After being downloaded by the interpreter requester, the interpreter application can be used by the interpreter requester registering the name, nationality, service plan, etc. in the server device 2. After the interpreter requester registers the necessary information, the server device 2 transmits the ID (account) and password to the interpreter requester terminal 4. The interpreter requester can log in to the interpreter application by entering this ID or the like.

<Explanation of interpreter request means 401>
The interpreter request means 401 starts the interpreter application based on the operation of the interpreter requester, and then makes an interpreter request to the server device 2. Specifically, the interpretation request is made by the user touching the touch pad 470 corresponding to the position where the "interpretation request" button displayed on the liquid crystal screen 450 is displayed.

<Explanation of photographing means 402>
The photographing means 402 captures an image of the interpreter requester based on the interpreter requester's request for interpretation. Specifically, the photographing means 402 transmits information for activating the camera 430 to the photographing unit 43. As a result, the video information of the interpreter requester is acquired.

<Explanation of voice input means 403>
The voice input means 403 acquires the voice of the interpreter requester based on the interpreter requester's request for interpretation. Specifically, the voice input means 403 transmits information for activating the microphone 440 to the voice input unit 44. As a result, the voice information of the interpreter requester is acquired.

<Explanation of avatar image generation means 404>
The avatar image generation means 404 generates an avatar image composed of a 3D model (three-dimensional model) based on the face recognition information of the interpreter transmitted from the interpreter terminal 3.

Specifically, the avatar image generation means 404 makes the face recognition information and emotion information of the interpreter transmitted from the interpreter terminal 3 via the server device 2 correspond to the 3D model of the avatar prepared in advance. Generate an interpreter's avatar image.

The 3D model of the avatar stores, for example, a shape key that moves the apex position of the mouth of the 3D model to a predetermined position when the interpreter's mouth is opened. As a result, the avatar image generation means 404 responds to the interpreter opening his mouth, closing his eyes, and shaking his face, so that the avatar opens his mouth, closes his eyes, and shakes his face. It is possible to draw the state to be done. For example, drawing of the avatar from the open state to the closed state can be realized by performing morphing.

Further, the avatar image generation means 404 outputs the avatar image of the interpreter. Specifically, the avatar image generation means 404 transmits information for displaying the interpreter's avatar image on the liquid crystal screen 450 to the graphic processing unit 45. As a result, the interpreter's avatar image is displayed on the liquid crystal screen 450.

In the present embodiment, the clothes and background of the interpreter's avatar image are clothes and the like preset to match the avatar image. As a result, the interpreter can be involved in the interpreting work without worrying about the clothes and the situation inside the room.

<Explanation of audio output means 405>
The voice output means 405 outputs the voice of the interpreter based on the interpreter's response to the interpreter. Specifically, the voice output means 405 transmits information for outputting the voice of the interpreter from the speaker 460 to the audio processing unit 46. As a result, the interpreter's voice is output from the speaker 360.

<Explanation of position information detecting means 406>
The position information detecting means 406 detects the current position of the interpreter requester based on the interpreter requester's request for interpretation. Specifically, the position information detecting means 406 transmits information for activating the position information detecting unit 48 to the position information detecting unit 48.

<Explanation of communication means 407>
The communication means 407 receives information on the interpreter response, face authentication information of the interpreter, voice information, and the like from the server device 2.

Further, the communication means 407 transmits information related to the interpreter request, video information and audio information of the interpreter requester to the server device 2.

<Explanation of video call processing>
Hereinafter, the video call processing will be described with reference to the flowchart of FIG. The control means and the processing procedure described later are examples, and the embodiments of the present invention are not limited thereto. The design of the processing procedure and the like can be appropriately changed without changing the gist of the present invention.

First, the interpreter request means 401 of the interpreter requester terminal 4 activates the interpreter application installed on the interpreter requester terminal 4 based on the operation of the interpreter requester (step S1).

Then, the interpreter request means 401 registers necessary information based on the operation of the interpreter requester, and also selects the language for which the interpreter is requested (step S2).

Then, the interpreter request means 401 makes an interpreter request based on the operation of the interpreter requester (step S3). Specifically, the interpreter request means 401 transmits the account of the interpreter requester, information on the interpreter request, and the like to the server device 2 via the communication means 407.

Then, the server device 2 starts communication with the interpreter requester terminal 4 (step S4).

Then, the photographing means 402 and the voice input means 403 of the interpreter requester terminal 4 activate the camera 430 and the microphone 440 of the interpreter requester terminal 4, respectively (step S5).

Next, the photographing means 402 acquires the video information of the interpreter requester, and the voice input means 403 acquires the voice information of the interpreter requester (step S6).

Then, the photographing means 402 and the voice input means 403 transmit the video information and the voice information of the interpreter requester to the server device 2 via the communication means 407, respectively (step S7).

Then, the service management means 203 of the server device 2 confirms the connection request necessary for establishing the connection between the interpreter terminal 3 and the interpreter requester terminal 4 (step S8).

Next, the interpreter management means 201 of the server device 2 searches for an interpreter currently available, and distributes the interpreter request to the interpreter terminal 3 of the interpreter that the service management means 203 can handle (step S9). ).

Next, the service management means 203 transmits the video information and audio information of the interpreter requester transmitted from the interpreter requester terminal 4 to the interpreter terminal 3 to which the interpreter request is distributed (step S10).

Then, the interpreter response means 301 of the interpreter terminal 3 performs an interpreter response based on the operation of the interpreter (step S11).

Then, the photographing means 302 and the voice input means 303 of the interpreter terminal 3 activate the camera 330 and the microphone 340 of the interpreter terminal 3, respectively (step S12).

Next, the photographing means 302 of the interpreter terminal 3 acquires the video information of the interpreter, and the voice input means 403 acquires the voice information of the interpreter (S13).

Next, the face information generation means 304 of the interpreter terminal 3 analyzes the video information of the interpreter and generates face recognition information which is information about the face part of the interpreter (step S14). Specifically, the face information generation means 304 extracts the opening and closing of the interpreter's mouth, the inclination of the head, and the like from the video information of the interpreter, and generates face recognition information based on the face of the interpreter.

Then, the face information generation means 304 generates emotion information from the face authentication information and the voice information (step S15). Specifically, the face information generation means 304 generates emotion information of the interpreter from the facial expression of the interpreter based on the face recognition information, the intonation of the interpreter's voice, and the like.

Next, the interpreter terminal 3 transmits the interpreter's face recognition information, emotion information, and voice information to the interpreter requester terminal 4 via the server device 2 (step S16).

Next, the avatar image generation means 404 of the interpreter requester terminal 4 generates an interpreter's avatar image based on the received face recognition information and emotion information of the interpreter (step S17). Specifically, the avatar image generation means 404 applies the interpreter's face recognition information to the 3D model of the avatar prepared in advance, and generates an avatar image so that the avatar moves in the same manner as the interpreter's face recognition information. To do.

Next, the avatar image generation means 404 displays the interpreter's avatar image on the liquid crystal screen 450 of the interpreter requester terminal 4, and the voice output means 405 causes the speaker 460 of the interpreter requester terminal 4 to output the interpreter's voice (step). S18). After that, when the call is disconnected at the interpreter terminal 3 or the interpreter requester terminal 4, the video call processing of the present invention ends.
According to the above procedure, the video call processing of the present invention is executed.

Summarizing the above, the video call system 1 of the present embodiment is
A video communication system 1 including a plurality of user devices connected to each other by a communication network 5 and receiving operations of each user.
The interpreter terminal 3, which is one user device,
Shooting means 302, which captures a face image of an interpreter who is a user who operates the interpreter terminal 3.
Voice input means 303 for inputting the voice of the interpreter,
Face information generating means 304 that generates face information about the face part of the interpreter from the photographed face image of the interpreter, and communication means 307 that transmits the face information and voice of the interpreter to the interpreter requester terminal 4.
With
The interpreter requester terminal 4
An avatar image generation means 404 that generates and outputs an avatar image of an interpreter based on the face information of the interpreter transmitted from the interpreter terminal 3, and a voice output means 405 that outputs the voice of the interpreter.
To be equipped.

<Effect of invention>
According to the video call system of the present embodiment, it is possible to provide a video call system in which the processing load is small and the home worker can provide the service without worrying about the face and the environment.

Specifically, according to the present invention, even if the interpreter is a home-based worker, the interpreter can carry out the interpreting work without worrying about his / her clothes or background.

Moreover, even if the interpreter is a woman, she does not have to bother to put on makeup for the interpreting work, and can easily participate in the interpreting work.

Furthermore, since face recognition information with a small amount of data is transmitted and received between the interpreter terminal and the interpreter requester terminal, the processing load of the server device increases while suppressing the occurrence of a time lag in communication. Can be suppressed.

[Other Embodiments]
In the above embodiment, an example is described in which the avatar image of the interpreter is always displayed on the interpreter requester terminal, but the present invention is not limited to this. For example, the interpreter may be able to select whether to display the actual image of the interpreter or the avatar image (including the type of the avatar image) based on the operation of the interpreter terminal.

Further, unlike the above-described embodiment, the background of the avatar image may be a single color such as black.

Further, the interpreter requester may be able to select the type of the avatar image based on the operation of the interpreter requester terminal.

Further, in the above embodiment, an example is described in which the first user device is an interpreter terminal and the second user device is an interpreter requester terminal, but the present invention is not limited to this. For example, the first user device may be a terminal operated by a clerk at a convenience store, and the second user device may be a terminal operated by a customer. Further, the first user device may be a terminal operated by the tutor, and the second user device may be a terminal operated by the student.

Further, in VR (Virtual Reality), the present invention can be applied even when each avatar talks. Furthermore, the present invention can be applied to holograms and AR (augmented reality).

Further, in the above-described embodiment, a face recognition technique using an open source library using deep learning is described, but the present invention is not limited to this. For example, facial feature points of the interpreter's face may be generated using the face recognition techniques of FIGS. 6 and 7, and face recognition may be performed based on the information of the feature points.

Further, in the above-described embodiment, an example in which emotional information of an interpreter is generated by facial expressions, inflection of voice, etc. is described, but the present invention is not limited to this. For example, emotional information of the interpreter may be generated by performing text mining on the words spoken by the interpreter. For example, when the interpreter utters a positive word such as "happy", the facial expression of the avatar image may reflect the feeling of joy.

Further, the face information generation means can also generate face information and emotion information from the voice information of the interpreter by using a technique such as lip sync. Specifically, for example, vowel information can be extracted from the voice information of the interpreter, and the movement of the avatar's mouth can be generated based on the vowel information.

Further, in the above embodiment, an example in which the face information is generated by the interpreter terminal is described, but the present invention is not limited to this. For example, the face information generation means provided in the interpreter requester terminal that receives the interpreter's face image (live video) from the interpreter terminal after the interpreter request operation by the interpreter requester is the interpreter from the interpreter's face image. The face information regarding the part of the face of the interpreter may be generated, and the avatar image generation means of the interpreter requester terminal may generate the avatar image of the interpreter based on the face information.

Further, in the above-described embodiment, an example is described in which the face information generating means generates only face information based on the face of an interpreter, but the present invention is not limited to this. For example, the face information generating means may generate information based on the movement of the hand, the movement of the upper body, and the like in addition to the face.

Further, in the example of FIG. 5, an example is described in which the camera and the microphone of the interpreter requester terminal are activated after the communication between the server device and the interpreter requester terminal is started, but the present invention is limited to this. I can't. The camera and microphone of the interpreter requester terminal may be activated before the communication between the server device and the interpreter requester terminal is started (for example, when the interpreter requester starts the interpreter application).

Further, the computer program used for the interpreter terminal and the computer program used for the interpreter requester terminal may each be composed of two or more computer programs (applications).

Further, in the above embodiment, the interpreter terminal is described as a laptop computer, but it may be a desktop computer, a smartphone, a tablet, or the like.

Similarly, in the above-described embodiment, the interpreter requester terminal is described as a portable terminal such as a smartphone or tablet, but may be a desktop computer, a laptop computer, or the like.

1 Video call system 2 Server device 201 Interpreter management means 202 Interpreter requester management means 203 Service management means 204 Communication means 3 Interpreter terminal 301 Interpreter response means 302 Shooting means 303 Voice input means 304 Face information generation means 305 Video output means 306 Voice output means 307 Communication means 4 Interpreter requester terminal 401 Interpreter request means 402 Shooting means 403 Voice input means 404 Avatar image generation means 405 Voice output means 406 Position information detection means 407 Communication means 5 Communication network

Claims (8)

A video communication system including a plurality of user devices that are connected to each other by a communication network and receive operations of each user.
The first user device, which is at least one of the user devices,
A photographing means for capturing a face image of the first user who operates the first user device,
A voice input means for inputting the voice of the first user,
A face information generating means for generating face information regarding a face portion of the first user from the photographed face image of the first user, and at least the face information and the voice of the first user are used by the plurality of users. A communication means for transmitting to a second user device other than the first user device among the devices,
With
The second user device is
An avatar image generation means that generates and outputs an avatar image of the first user based on the face information of the first user transmitted from the first user device, and a voice that outputs the voice of the first user. Output means,
To prepare
Video calling system. The first user device is operated by an interpreter who has been requested to interpret.
The second user device is operated by an interpreter requester who requests the interpreter to interpret.
The video call system according to claim 1. The face information is at least face recognition information regarding the presence / absence of opening / closing of the mouth, which is the portion of the face of the first user, the presence / absence of opening / closing of eyes, and the inclination and rotation of the head.
The video calling system according to claim 1 or 2. The avatar image generation means generates the avatar image by varying the positions of the apex of the eyes, mouth, and face corresponding to the avatar image composed of the three-dimensional model based on the face recognition information.
The video call system according to claim 3. The face information generating means generates emotion information for expressing the emotion of the first user based on at least one of the face information of the first user and the voice.
The avatar image generation means generates the avatar image of the first user based on the emotion information in addition to the face information.
The video call system according to any one of claims 1 to 4. Computer equipment,
A photographing means for capturing a facial image of a user who operates the computer device,
A voice input means for inputting the user's voice,
A face information generating means for generating face information regarding a part of the user's face from the photographed face image of the user in order to generate an avatar image of the user in another computer device, and at least the face of the user. A communication means for transmitting information and the voice to the other computer device,
To function as
Computer program. Computer equipment,
Regarding the other user who operates the other computer device from the other computer device based on the user's operation, the face portion of the other user generated from the face image of the other user taken by the other computer device. Communication means for receiving face information and voice,
An avatar image generation means that generates and outputs an avatar image of the other user based on the face information, and a voice output means that outputs the voice of the other user.
To function as
Computer program. Computer equipment,
A communication means for receiving a face image and voice of the other user taken by the other computer device for another user who operates the other computer device from the other computer device based on the user's operation.
A face information generating means for generating face information regarding a face portion of the other user from the face image of the other user received from the other computer device.
An avatar image generation means that generates and outputs an avatar image of the other user based on the face information, and a voice output means that outputs the voice of the other user.
To function as
Computer program.

Images