JP6901190B1 - Remote dialogue system, remote dialogue method and remote dialogue program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a technique for easily confirming the identity of participants at the time of remote dialogue such as a WEB conference. SOLUTION: A management server 2 has an information input unit 13 for inputting participation information output from a participating terminal 1, and a participant image generation unit 14 for generating images of individual participants according to the participation information. , A dialogue video generation unit 15 that generates a dialogue video displaying participant videos and remarks in a virtual dialogue space set as a stage for remote dialogue, and a behavior information database 16 that records past behavior information of each participant. , The behavior information comparison unit 19 that compares the past behavior information with the current behavior, the identity determination unit 20 that determines the identity of the participants based on the comparison result in the behavior information comparison unit 19, and the identity determination. It includes a warning information generation unit 21 that generates warning information as needed based on the determination result of the unit 20, and an information output unit 22 that outputs an interactive video and warning information. [Selection diagram] Fig. 1

Description

The present invention relates to a technique for performing remote dialogue between participants who are not in the same place.

In recent years, for multi-person two-way communication such as management meetings at commercial enterprises and lectures at educational institutions such as universities, participants are not held in a single venue but in different places. A technique has been proposed in which a conference or the like is held remotely as if there are participants in the same venue after connecting each other with a network.

For example, Patent Document 1 discloses a technique for holding a remote conference after allowing an actual participant and a fictitious character created by an information processing device to enter and exit a Web conference room formed in a virtual space. Unlike the case where participants are simply connected to each other by communication, the reality of the meeting is improved and the discomfort felt by the participants is reduced.

Further, Patent Document 2 discloses a technique for performing image division on a screen and distortion correction for a participant's image in order to appropriately display the image of a participant in a remote conference. In meetings, lectures, etc., not only the content of the speaker's remarks but also the nonverbal communication mode such as gestures at the time of remarks is important information, and the nonverbal reaction of the listener is also important information. In Patent Document 2, by appropriately displaying the video of the participants, not only the linguistic information such as the content of remarks but also the non-linguistic information can be obtained among the participants in the remote conference as well as the conference at the same venue. It discloses technologies that can be shared.

Japanese Unexamined Patent Publication No. 2020-194343 Japanese Unexamined Patent Publication No. 2020-160538

However, in Patent Documents 1 and 2, it is difficult to determine whether or not a person who participates in a remote conference or the like is a specific person who is truly qualified to participate. In a teleconference, for example, when an industrial espionage disguised as an officer infiltrates at a management meeting of a commercial company, or a person other than a student pretends to participate in a university lecture for the purpose of so-called "return". However, it is difficult to properly eliminate these frauds with the techniques of Patent Documents 1 and 2.

And, in view of recent technological developments, the person who is considered to be a "participant" is a true participant or pretended to be a participant only by the information on the image showing the face and the whole body and the sound spoken. It is becoming difficult to determine whether it is a third party. For example, it is technically possible to create a moving image of a specific person by using a person image synthesis technique or a voice synthesis technique using deep learning called deep fake. When such technology is abused, it is determined whether or not the person who is considered to be a "participant" is a true participant, depending on the video and audio of the participant displayed on the screen of the remote conference. That is very difficult.

In addition, identification by entering ID and password is not limited to cases where information is leaked due to hacking, etc., but also when a third party colludes with a person who has regular participation qualifications as in the case of "return" in the lecture. Unfortunately, it's useless.
The present invention has been made in view of the above problems, and an object of the present invention is to realize a technique for easily confirming the identity of participants at the time of remote dialogue such as a WEB conference.

In order to achieve the above object, the remote dialogue system according to claim 1 is a remote dialogue system that realizes dialogue between a plurality of participants located at different locations, and is based on information on the past behavior of the participants. A reference behavior information generating means that detects the above characteristic behavior and generates reference behavior information that is information on the frequency of occurrence of the characteristic behavior and the duration of the characteristic behavior at the time of occurrence, and a participant at the time of the current dialogue. With respect to the one or more characteristic behaviors of the above, the current behavior information generating means for generating the current behavior information which is information on the frequency of occurrence of the characteristic behavior and the duration of the characteristic behavior at the time of occurrence, and the reference behavior information. The behavior information comparison means for comparing the current behavior information, the identity determination means for determining the identity of the participants based on the comparison result of the description behavior information comparison means, and the participation subject to the determination in the identity determination means. It is characterized in that it is provided with an output means for outputting a determination result by the identity determination means to one or more participants other than the person.

Further, in order to achieve the above object, in the remote dialogue system according to claim 2, in the above invention, the reference behavior information generating means is the reference behavior information, which is numerical information regarding the occurrence frequency, the reference expression frequency and the duration. The current behavior information generation means generates information including a reference duration which is numerical information about the current behavior information, and the current behavior information generating means is numerical information about the current occurrence frequency and the duration as the current behavior information. Is generated, and the behavior information comparison means relates to a value obtained by dividing the absolute value of the difference value between the reference occurrence frequency and the current occurrence frequency for the same characteristic behavior by the reference occurrence frequency and the same characteristic behavior. The absolute value of the difference between the reference duration and the current duration is divided by the reference duration to derive the value as the comparison result, and the identity determination means is used when the sum of the comparison results is less than the first threshold value. It is characterized in that it is determined that there is identity, it is determined that there is a possibility of identity when it is equal to or greater than the first threshold value and less than the second threshold value, and it is determined that there is no identity when it is equal to or greater than the second threshold value. To do.

Further, in order to achieve the above object, the remote dialogue system according to claim 3 is a display means for displaying a dialogue video, which is a video of a plurality of participants participating in the remote dialogue, to a specific participant in the above invention. Based on the positional relationship between the photographing means for photographing the image of the specific participant, the display means, the photographing means, and the specific participant, the specific participant may be present at the time of photographing by the photographing means. The attention target information generation means for generating attention target information including information on which participant the eyes are directed to, and the attention target information as an image of the specific participant to be displayed on the display means. It is characterized in that it is further provided with a dialogue image generation means for generating the dialogue image so as to have a mode in which the line of sight is directed to the participant shown in the above.

Further, in order to achieve the above object, the remote dialogue method according to claim 4 is a remote dialogue method that realizes a dialogue between a plurality of participants located at different places, and provides information on the past behavior of the participants. Based on this, a reference behavior information generation step that detects one or more characteristic behaviors and generates reference behavior information that is information on the frequency of occurrence of the characteristic behavior and the duration of the characteristic behavior at the time of occurrence, and a reference behavior information generation step at the time of the current dialogue. Regarding the one or more characteristic behaviors of the participants, the current behavior information generation step for generating the current behavior information which is information on the frequency of occurrence of the characteristic behavior and the duration of the characteristic behavior at the time of occurrence, and the reference behavior. The behavior information comparison step for comparing the information with the current behavior information, the identity determination step for determining the identity of the participants based on the comparison result of the behavior information comparison step, and the determination target in the identity determination step. It is characterized by including an output step for outputting a determination result by the identity determination step to one or more participants other than the participants.

Further, in order to achieve the above object, the remote dialogue method according to claim 5 is a display step of displaying a dialogue video which is a video of a plurality of participants participating in the remote dialogue to a specific participant in the above invention. And, based on the shooting step of shooting the image of the specific participant, the display location in the display step, the shooting point in the shooting step, and the positional relationship between the specific participants, the specification at the time of shooting in the shooting step. Of the attention target information generation step for generating the attention target information including information on which of the plurality of participants the participants are looking at, and the specific participant to be displayed in the display step. The image is further characterized by including a dialogue image generation step of generating the dialogue image so as to have an aspect in which the line of sight is directed to the participant indicated by the attention target information.

Further, in order to achieve the above object, the remote dialogue program according to claim 6 is a remote dialogue program that allows a computer to realize a dialogue between a plurality of participants located at different places, and relates to the past behavior of the participants. A reference behavior information generation function that detects one or more characteristic behaviors based on the information and generates reference behavior information that is information on the frequency of occurrence of the characteristic behavior and the duration of the characteristic behavior at the time of occurrence, and the current dialogue. With respect to the one or more characteristic behaviors of the participants at the time, the current behavior information generation function for generating the current behavior information which is information on the frequency of occurrence of the characteristic behaviors and the duration of the characteristic behaviors at the time of occurrence, and the above-mentioned A behavior information comparison function that compares the reference behavior information with the current behavior information, an identity determination function that determines the identity of participants based on the comparison result of the behavior information comparison function, and a determination target in the identity determination function. It is characterized in that an output function for outputting a determination result by the identity determination function is realized for one or more participants other than the participant who has become.

According to the present invention, there is an effect that the identity of participants can be easily confirmed at the time of remote dialogue such as a WEB conference.

It is a schematic diagram which shows the structure of the remote dialogue system which concerns on Embodiment 1. FIG. It is a flowchart for demonstrating the operation of the remote dialogue system which concerns on Embodiment 1. FIG. It is a schematic diagram which shows the structure of the remote dialogue system which concerns on Embodiment 2. FIG. It is a flowchart for demonstrating the operation of the remote dialogue system which concerns on Embodiment 2.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments describe examples that are considered to be the most appropriate embodiments of the present invention, and as a matter of course, the contents of the present invention are limited to the specific examples shown in the present embodiments. Should not be understood. It goes without saying that any configuration other than the specific configuration shown in the embodiment is included in the technical scope of the present invention as long as the configuration exhibits the same action and effect.
(Embodiment 1)

First, the remote dialogue system according to the first embodiment will be described. As shown in FIG. 1, the remote dialogue system according to the first embodiment generates and outputs participation information including information on the speech content and behavior content of the participants, and displays a dialogue video between the participants. A dialogue video is generated based on the participation terminal 1 and the participation information output from the participation terminal 1 and output to the participation terminal 1, and the behavior information of the participants included in the participation information output from the participation terminal 1 is included. Based on this, the management server 2 for determining whether or not the subject of the participation information is the same person as the participant registered in advance, and the information communication network 3 for connecting between the participating terminal 1 and the management server 2 are provided.

Participating terminals 1 are arranged in response to participants in remote dialogue, and in the first embodiment, in principle, participating terminals 1 composed of the number of participants are used. The participating terminal 1 includes a shooting unit 4 that shoots a participant's image, a speech recording unit 5 that records the participant's remarks, and a behavior information generation unit that generates behavior information from the video captured by the shooting unit 4. 6, the participation information generation unit 7 that generates the participation information including the behavior information and the speech data recorded by the voice recording unit 5, the information output unit 8 that outputs the participation information to the management server 2, and the management server 2. The information input unit 9 for inputting the dialogue video transmitted from the above and the display unit 10 for displaying the input dialogue video are provided.

The photographing unit 4 is for acquiring a video of a participant at the time of holding a remote dialogue. The specific configuration of the photographing unit 4 in the first embodiment is a 3D scanner capable of acquiring a three-dimensional stereoscopic image of the whole body of the participant.

The remark recording unit 5 is for recording the remark contents of the participants. Specifically, the speech recording unit 5 is provided with a microphone function for performing voice input and a voice recognition function for converting the input voice into character information. However, it is not necessary to limit the configuration of the speech recording unit 5 to this, and for example, as the simplest configuration, the audio data input via the microphone function may be output to the management server 2 as it is. However, instead of the microphone function, an input mechanism such as a keyboard or a pointing device may be provided, and character information may be directly input as the content of remarks.

The behavior information generation unit 6 is for generating the behavior information of the participants based on the video of the participants acquired by the shooting unit 4. Specifically, the behavior information generation unit 6 has a function of generating information on position fluctuations of predetermined feature points in a participant's image with the passage of time.

The feature points mean the feature points on the outer surface and the feature points on the skeletal structure of the object to be photographed. For example, the feature points corresponding to the outer surface include specific points in each part such as eyes, nose, and mouth. (Outer corners of the eyes, eyes, nose tip, corners of the mouth, etc.), and the feature points corresponding to the skeletal structure are joints such as the neck, shoulders, elbows, wrists, fingertips, hips, knees, and ankles. The behavior information generation unit 6 has the meaning and content of the participant's feature points extracted from the video acquired by the shooting unit 4 (feature points A are points corresponding to the outer corners of the right eye, etc.) and positions with the passage of time. It has a function of deriving changes and generating behavior information that is information including these.

The participation information generation unit 7 is for generating information about participants who participate in the remote dialogue by using the participation terminal 1. Specifically, the participation information generation unit 7 has a function of generating participation information which is information including the speech information recorded by the speech recording unit 5 and the behavior information generated by the behavior information generation unit 6. Have. It should be noted that the speech information and the behavior information may be treated as generating the participation information as information that is independent of each other, but from the viewpoint of useful for more realistic dialogue video generation and accurate behavior judgment, the speech information and the behavior It is desirable to have a format in which information is synchronized in time.

The information output unit 8 is for outputting the participation information generated by the participation information generation unit 7 to the management server 2. Although the specific configuration of the information output unit 8 differs depending on the form of the information communication network 3 and the connection protocol, it may be either a wired connection or a wireless connection, and even in the case of a wireless connection, any wireless LAN, Bluetooth (registered trademark), etc. It can also be adopted in the form.

The information input unit 9 is for inputting the dialogue video generated by the management server 2. Although the specific configuration of the information input unit 9 differs depending on the form of the information communication network 3 and the connection protocol, it may be either a wired connection or a wireless connection, and even in the case of a wireless connection, any wireless LAN, Bluetooth (registered trademark), etc. It can also be adopted in the form.

The display unit 10 is for displaying the dialogue video input from the management server 2. Participants participate in discussions in remote dialogues, etc. by viewing the dialogue video generated based on each participant's remarks and behavior while outputting their own remarks and behavior. It becomes possible.

Next, the management server 2 will be described. The management server 2 includes an information input unit 13 for inputting participation information output from the participation terminal 1, a participant image generation unit 14 for generating images of individual participants according to the input participation information, and a participant image generation unit 14. A dialogue video generation unit 15 that generates a dialogue video displaying participant videos and remarks in a virtual dialogue space set as a stage for remote dialogue, a behavior information database 16 that records past behavior information of each participant, and A behavior information comparison unit 19 that compares past behavior information with the current behavior, an identity determination unit 20 that determines the identity of participants based on the comparison results in the behavior information comparison unit 19, and an identity determination unit. A warning information generation unit 21 that generates warning information as needed based on the determination result of 20 and an information output unit 22 that outputs an interactive video and warning information are provided.

The information input unit 13 is for inputting the participation information generated by the participating terminal 1. Although the specific configuration of the information input unit 13 differs depending on the form of the information communication network 3 and the connection protocol, it may be either a wired connection or a wireless connection, and even in the case of a wireless connection, any wireless LAN, Bluetooth (registered trademark), etc. It can also be adopted in the form.

The participant video generation unit 14 generates a participant video to be displayed in the virtual dialogue space based on the participation information input via the information input unit 13 and the character video information for each participant set in advance. Is for. Character video information is information about the appearance of a character expressing a participant, such as a figure (skin, etc.) appearing on the outer surface of the character expressed as a participant, and an internal skeletal structure (bone joint, etc.). ) And information (weights, etc.) that defines the relationship between the figure and the skeletal structure. In addition, the character video information also records information about feature points used for processing in the behavior information generation unit 6.

That is, in the character video information, specific parts (outer corners of eyes, eyes, nose tips, corners of mouth, etc.) on the outer surface of the participants' characters such as eyes, nose, and mouth, neck, shoulders, elbows, wrists, fingertips, hips, and knees. , Positions corresponding to specific points in the skeletal structure such as joints such as ankles are recorded as feature points. The specific mode of the participant's character may be a three-dimensional computer graphic that imitates the person himself / herself, a person completely different from the person himself / herself, an animal, a character appearing in an animation, or the like. Preferably, it includes information about feature points that correspond to those used in the behavior information generation unit 6.

By using the character video information having such a configuration, the participant video generation unit 14 applies the position change mode of the feature points included in the behavior information input from the participating terminal 1 to the feature points in the character video information. By operating the character image, the character image has a function similar to that of the participant photographed by the photographing unit 4.

In addition, the participant video generation unit 14 synchronizes the content of the participant's remarks with the participant video based on the remark information input from the participating terminal 1. The data format of the remark content to be synchronized may be voice data or character data. In the case of voice data, it may be data to which the voice characteristics of the participant himself / herself are added, or it may be a separate voice completely different from the voice of the participant himself / herself.

The dialogue video generation unit 15 is for generating a dialogue video composed of virtual images as if each participant actually gathers in the same space and has a dialogue. Specifically, the dialogue video generation unit 15 arranges the participant video generated by the participant video generation unit 14 in the virtual dialogue space prepared in advance, so that each participant is placed in the virtual dialogue space. It has a function to generate a dialogue video as if they were meeting in person and having a discussion. The contents of the dialogue video may be the same video for all the participating terminals 1 owned by each participant, or may be different for each participating terminal 1. Although the basic video is the same, the display angle can be changed by operating the participating terminal 1 or the display angle can be adjusted to the field of view of the participant (to be exact, the character video of the participant placed in the virtual dialogue space). It may be a mode in which the image can be switched to. On the contrary, the virtual dialogue space may be simply displayed side by side with the participant images.

The behavior information database 16 is for recording the past behavior of the participants. The behavior information database 16 may record the behavior video as it is when the participant participated in the dialogue in the past as the past behavior, or may record the position variation mode of the feature point based on the behavior video.
The reference behavior information generation unit 17 is for generating reference behavior information from the information on the past behavior of the participants recorded in the behavior information database 16. Specifically, the reference behavior information generation unit 17 determines the characteristic behavior, which is the characteristic behavior of the participants (identified by typifying the change in the position of the feature point), based on the information on the past behavior. It has a function to detect one or more and generate reference behavior information including the reference occurrence frequency and the reference duration in addition to the aspect of the characteristic behavior (position fluctuation mode of the corresponding feature point) for each of the detected characteristic behaviors. ..

As characteristic behaviors, for example, the behavior of raising the index finger on the left arm and bringing the fingertip into contact with the nose tip (the presence or absence of such behavior can be specified by the position change of the feature point corresponding to the index finger of the left arm, the left wrist, the left elbow, etc.) and , Place both elbows on the desk and put both palms together in front of the face (identified by the position change of the feature points corresponding to each part of the elbows and palms of both arms), and close both eyes strongly It includes all behaviors as long as it can be specified by the position change of the feature point, such as the facial expression change such as the left and right corners of the mouth rising (it can be specified by the position change of the feature point corresponding to each part of the face). From the viewpoint of improving the accuracy of identity determination, it is desirable that there are multiple types of characteristic behaviors to be memorized for each participant, and on the other hand, "characteristic behaviors" can be referred to as "characteristic behaviors" by other participants. Does not have to be a unique behavior that is not seen. Even if the behavior is similar, there is usually a difference in the frequency of occurrence, duration, etc., and when the identity is judged based on a plurality of characteristic behaviors, it is rare that all the behaviors match. Therefore, it is possible to determine the identity with high accuracy even if the behavior is not unique.

Regarding the reference occurrence frequency and the reference duration, the behavior information database 16 may be a simple average value of the occurrence frequency and the duration in the past dialogue, but the value in the dialogue held near the present is more important. It is preferable to use a weighted average (weighted average).

The current behavior information generation unit 18 is for generating the current behavior information based on the current behavior information actually input from the participating terminal 1. Specifically, the current behavior information generation unit 18 detects the presence or absence of the same characteristic behavior as that in the reference behavior information generation unit 17 from the current behavior information, and for each of the detected characteristic behaviors, the characteristic behavior. In addition to the above-mentioned mode (positional variation mode of the corresponding feature point), it has a function of generating current behavior information which is information on the expression frequency and duration. In the first embodiment, the current behavior information includes the current expression frequency, which is numerical information regarding the expression frequency, and the current duration, which is numerical information regarding the duration.

In determining whether or not the characteristic behavior is present at the present time, the behavior information generation unit 18 is completely different from the position change of the feature point group corresponding to the characteristic behavior detected by the reference behavior information generation unit. It is judged that the characteristic behavior is exhibited not only when the above is the case but also when the total difference value of the position change in each of the corresponding feature point groups is less than the predetermined threshold value. That is, even if the heights of both palms are slightly different from those in the past in a slight change in position, for example, the behavior of putting both palms together in front of the face, it is determined that the same characteristic behavior is exhibited.

The behavior information comparison unit 19 is for comparing the reference behavior information with the current behavior information. Specifically, the behavior information comparison unit 19 first calculates a value obtained by dividing the absolute value of the difference value between the reference appearance frequency and the current appearance frequency by the reference appearance frequency for the appearance frequency and duration of each characteristic behavior. Then, the absolute value of the difference between the reference duration and the current duration is divided by the reference duration to calculate the value. Then, the calculated values are added for all characteristic behaviors (more preferably, each value is multiplied by a coefficient according to its importance and then added). The addition result is output to the identity determination unit 20 as a comparison result.

The identity determination unit 20 determines the identity of the participants (whether the pre-registered "participant" and the person who is actually participating are the same, depending on the addition result obtained by the behavior information comparison unit 19. It is for making a judgment regarding (whether or not). In the first embodiment, when the addition result is a value less than the first threshold value, it is determined that there is "identity", and when the value is equal to or more than the first threshold value and less than the second threshold value. It is determined that "there is a possibility that they are not the same", and when the value is equal to or greater than the second threshold value, it is determined that there is no identity.

The warning information generation unit 21 is for generating warning information regarding the identity of the participants according to the determination result by the identity determination unit 20. Specifically, if the identity determination unit 20 determines that there is "identity", no warning information is generated, and if it is determined that "there is a possibility that they are not the same", the participant who is the determination target. Generate warning information to other participants with the content "Recommend confirmation of identity." In addition, when the identity determination unit 20 determines that there is no identity, the warning information generation unit 21 sends the remote dialogue operation manager "recommends disconnection from the participant". Generate warning information consisting of the contents of.

The information output unit 22 is for outputting the dialogue video generated by the dialogue video generation unit 15 and the warning information generated by the warning information generation unit 21 to the participating terminals 1. As for the warning information, since the information about the destination is also specified in the information, it is configured to be output only to the participating terminal 1 designated as the destination. Further, although the specific configuration of the information output unit 22 differs depending on the form of the information communication network 3 and the connection protocol, it may be either a wired connection or a wireless connection, and even in the case of a wireless connection, a wireless LAN or Bluetooth (registered trademark) may be used. It is possible to adopt any format such as.

Next, the warning information generation process for a specific participant in the remote dialogue system according to the first embodiment will be described with reference to FIG. First, the reference behavior information generation unit 17 detects the presence or absence of a specific characteristic behavior from past behavior information, and determines the reference expression frequency, which is the frequency of occurrence of the characteristic behavior, and the reference, which is the duration at the time of occurrence. The duration is calculated (step S101). Then, the current behavior information generation unit 18 calculates the current expression frequency and the current duration, which are the expression frequency and duration in the currently held dialogue, for the same characteristic behavior as in step S101 (step S102). Specifically, the behavior information input from the participating terminal 1 is analyzed, and the position change of the participant's feature point is changed in the same manner as the specific characteristic behavior detected by the reference behavior information generation unit 17. If so, it is determined that the same characteristic behavior has occurred, and the frequency of expression and the duration at the time of expression are calculated.

After that, the behavior information comparison unit 19 calculates the absolute value of the difference between the reference expression frequency and the current expression frequency, and divides the calculation result by the reference appearance frequency (step S103). Then, similarly, the behavior information comparison unit 19 calculates the absolute value of the difference value between the reference duration and the current duration, and divides the calculation result by the reference duration (step S104). If the processing for all the characteristic behaviors is not completed (step S105, No), the process returns to step S102 again and the same processing is performed for the other characteristic behaviors. If the processing for all the characteristic behaviors is completed (step S105, Yes), the calculated values are added and the addition result is output (step S106).

After that, the identity determination unit 20 performs the identity determination process. First, the identity determination unit 20 determines whether or not the addition result obtained in step S106 is less than the first threshold value (step S107). When the addition result is less than the first threshold value (step S107, Yes), the identity of the participants is affirmed as the characteristic behavior appears with the same frequency and duration as the past ones, and a special warning is given. The process ends without any information being generated.

When the addition result obtained in step S106 is equal to or greater than the first threshold value (step S107, No), whether or not the addition result is less than or equal to the second threshold value which is equal to or greater than the first threshold value. The determination is made (step S108). When the addition result is less than the second threshold value (step S108, Yes), information to that effect is output to the warning information generation unit 21, and the warning information generation unit 21 is the specific determination target. For participants other than the participants, warning information with the content "It is recommended to confirm the identity" is generated (step S109).

If the addition result is equal to or greater than the second threshold value (step S108, No), information to that effect is output to the warning information generation unit 21, and the warning information generation unit 21 is responsible for the operation of the remote dialogue. A warning information with the content "It is highly possible that they are not the same and it is recommended to cut off the connection" is generated (step S110). The warning information generated in steps S109 and S110 is output to the addressee through the information output unit 22, and all the processing is completed.

Next, the advantages of the remote dialogue system according to the first embodiment will be described. First, in the remote dialogue system according to the first embodiment, the behavior information is used when confirming the identity of the participants, which enables highly accurate identity confirmation.

Humans often behave in a seemingly meaningless manner during activities, such as placing the right hand on the forehead at a moment's notice, raising the index finger of the left hand and shaking it to the left or right when speaking, and the mode, frequency, and duration of such behavior. It is usually not possible for all of them to match exactly with another person. Therefore, by recording the behavior information of the person who is scheduled to participate in the dialogue in advance (for example, in the case of a dialogue held multiple times, the behavior information at the time of the dialogue immediately after the start is recorded), and comparing with the current behavior. It has an advantage that it is possible to determine whether or not a person who is currently participating as a specific participant is really the participant.

As a method of identity verification in a service used from a remote place, a method of inputting an ID and a password, a face image, and voice data has been used. However, the ID / password input does not function as an identity verification means if information is leaked due to hacking or the like, or if the participant himself / herself communicates with a third party and provides the ID or the like. In addition, it is easy to forge a face image with the current technology. For example, a three-dimensional shape is estimated for one two-dimensional image and feature points are extracted to change the facial expression according to the content of the statement. It is possible to create a video that has been applied. In such a case, the identity verification procedure using the conventional face recognition technology or the like does not make sense. The same applies to voice data. With the current technology, it is possible to create voice data as if the person spoke by only speaking about the keyword for about 20 seconds. The reality is that it is difficult to sufficiently verify the identity even with voice recognition technology.

On the other hand, in the first embodiment, the identity of the participants is confirmed by using the behavior information, and a third party obtains the behavior information including the specific behavior mode, the frequency of occurrence, and the duration in advance. (On the other hand, it is possible to synthesize a face image with only one photo, and to synthesize voice data with about 20 seconds of voice information). Further, even if the behavior information can be acquired, it is technically not easy to incorporate the specific behavior into the conversation developed in real time at the frequency and duration according to the content of the behavior information. Therefore, it is almost impossible for a third party who participates in the dialogue under the guise of a participant to disguise the behavior information. Therefore, in the remote dialogue system according to the first embodiment, the remote dialogue system is efficient and accurate. It has the advantage of being able to determine the identity of the participants.

Further, the remote dialogue system according to the first embodiment has an advantage that more accurate identity determination can be performed by generating behavior information for a plurality of characteristic behaviors and using it as a material for identity determination. .. That is, in the case of a single characteristic behavior, the difference in the frequency of occurrence and the duration becomes slight by chance depending on the circumstances such as the characteristic behavior being a typical behavior, and the person is actually a different person. Nevertheless, there is a risk of falsely determining that they are identical. In the first embodiment, since behavior information is generated and used for a plurality of characteristic behaviors, there is an advantage that the possibility of such an erroneous determination can be suppressed and a more accurate identity determination can be realized. There is.

Further, the remote dialogue system according to the first embodiment determines that the identity is suspected in addition to the two determinations of identity and non-identity, and at the time of the determination, the participants other than the target participant are determined. On the other hand, we have adopted a configuration that notifies warning information that "confirmation of identity is recommended." For example, in the case of a management meeting that handles confidential information, it is desirable to refrain from disclosing confidential information just in case, even if it cannot be determined that there is no identity. In the first embodiment, by treating the suspected case as issuing warning information, there is an advantage that each participant can be given a warning regarding information retention and the like. In addition, by receiving such warning information, each participant brings up the facts that only the participant can know to the "participant" who is suspected of being the same, and confirms the reaction of the "participant". It is possible to verify the identity by a flexible method such as.

Further, as an operational advantage, the remote dialogue system according to the first embodiment has an advantage that the identity of the participants can be confirmed without using personal information. That is, information such as ID, password, face, and voice is protected as personal information or does not correspond to personal information, but careful handling is required from the viewpoint of privacy, and management is complicated. On the other hand, the behavior information is only information about the "behavior" of the participants and is not subject to privacy protection. Therefore, there is an operational advantage in that it can be managed by a normal information management system.
(Embodiment 2)

Next, the remote dialogue system according to the second embodiment will be described. Unless otherwise specified, the components having the same name and the same reference numerals as those in the first embodiment in the second embodiment shall exhibit the same functions as the components in the first embodiment.

As shown in FIG. 3, the remote dialogue system according to the second embodiment is newly provided with the attention target information generation unit 24 for generating the attention target information regarding the target to be noticed by the participants, in addition to the components in the first embodiment. In addition to the participating terminals 23 and the components of the first embodiment, the participant image generation unit 26 that highlights the behavior of the participants and the dialogue image generation unit 27 that generates the dialogue image that matches the information of interest are newly added. A management server 25 is provided in preparation for the above.

The attention target information generation unit 24 is for generating attention target information indicating which of the other participants the participants participating in the dialogue are paying attention to. Specifically, the attention target information generation unit 24 grasps in advance the relative positional relationship between each display on the display unit 10, the photographing unit 4, and the participants to be photographed, and is photographed by the photographing unit 4. From the video, the line-of-sight direction information indicating which of the other participants displayed on the display unit 10 the line of sight is directed, and which target person each of the participants' arms is assigned to. Generates attention target information including arm direction information indicating whether or not it is aimed. The attention target information generated by the attention target information generation unit 24 is output to the management server 25 via the information output unit 8.

Similar to the participant image generation unit 14 in the first embodiment, the participant image generation unit 26 is basically based on the behavior information output from the participating terminal 23, and the position change mode of the feature points included in the behavior information. Has a function to generate a participant image by applying to a feature point in the character image information. In addition, the participant image generation unit 26 corresponds to the case where the position fluctuation amount per unit time shown in the behavior information is amplified by a predetermined ratio (for example, 10%) with respect to the feature points corresponding to both arms and the face. It has a function to apply it to the feature points in the character video information after converting it into the obtained position information.

The dialogue video generation unit 27 has a function of generating a dialogue video in which participant videos are arranged in a virtual dialogue space prepared in advance, similarly to the dialogue video generation unit 15 in the first embodiment. Then, the dialogue video generation unit 27 generates the dialogue video after changing the direction of the line of sight and the directions of both arms in the participant video so as to match the attention target information output from the participating terminal 1. Has the function of Specifically, the dialogue image generation unit 27 makes the line-of-sight direction of the participant image in the dialogue image the direction toward the position of the target person specified by the line-of-sight direction information included in the attention target information. It has a function to rotate the entire participant image (for example, the bone corresponding to the spine is used as the rotation axis). Further, the dialogue image generation unit 27 sets the positions of both arms so that the directions of both arms of the participant image are directed toward the position of the target person specified by the arm direction information included in the attention target information. It has a function to change.

Next, with respect to the remote dialogue system according to the second embodiment, the process of generating the dialogue video by the management server 25 will be described with reference to FIG. First, in the participant image generation unit 26, the position fluctuation amount per unit time derived from the behavior information for the feature points corresponding to the face and both arms is multiplied by a predetermined coefficient consisting of a value greater than 1, and the multiplication result is obtained. The position of each feature point is calculated so as to correspond to the position fluctuation amount obtained as (step S201). Then, the participant image generation unit 26 generates the participant image by changing the position of the feature point in the character image information so as to be the position determined in step S201 (step S202).

After that, the dialogue video generation unit 27 arranges the generated participant video in the virtual dialogue space prepared in advance (step S203). When the processing of steps S201 to S203 is completed for all the participants (steps S204, Yes), the process proceeds to step S205.

Then, the dialogue image generation unit 27 determines whether or not the line-of-sight direction of the arranged participant image matches the line-of-sight direction information generated by the attention target information generation unit 24 (step S205), and if they match (step S205). S205, Yes) shifts to step S207, and if they do not match (step S205, No), the line-of-sight direction of the participant image and the direction indicated by the line-of-sight direction information are the directions indicated by the line-of-sight direction information about the bone corresponding to the backbone of the participant image. Participant images are rotated by a predetermined angle so that they match (step S206).

After that, the dialogue video generation unit 27 determines whether or not the directions of both arms of the arranged participant video match the arm direction information generated by the attention target information generation unit 24 (step S207), and if they do not match. (Step S207, No) changes the positions of both arms so as to match (step S208). When the position change is completed or the directions are the same in the first place (step S207, Yes), it is determined whether or not the processing has been performed for all the participants (step S209), and if it is not completed (step S209). S209, No) Return to step S205 and repeat the same process, and if it is completed (step S209, Yes), the process of creating the dialogue video is completed, and the participant terminal 23 of each participant is contacted via the information output unit 22. On the other hand, a dialogue video is output.

Next, the advantages of the remote dialogue system according to the second embodiment will be described. First, in the remote dialogue system according to the second embodiment, the position information of the feature points of the face and both arms is converted so as to be a position amplified (for example, about 10%) from the amount of position fluctuation shown in the behavior information. By performing the above, there is an advantage that a participant image that emphasizes the behavior of the participant can be generated. By adopting such a configuration, other participants who view the video of the remote dialogue on the participating terminal can easily recognize the behavior of the participant.

In remote dialogue, where the dialogue video is viewed through the screen, it is difficult to realistically recognize the behavior of other participants as in the case of directly gathering and holding a dialogue, which makes it difficult to recognize the behavior of other participants. Interfering with linguistic communication. In the second embodiment, by highlighting the behavior of the participants, it is possible to understand the behavior of other participants as much as when they gather directly, and thereby through not only linguistic communication through remarks but also behavior. It also has the advantage of facilitating nonverbal communication and enabling lively discussions.

Further, the remote dialogue system according to the second embodiment generates attention target information indicating the attention status to other participants, and matches the attention target information in the participant video displayed in the dialogue video. The line of sight and the direction of both arms are adjusted so as to do so. The video of the participants taken by the shooting department is only taken from the viewpoint of the shooting department, and from the video itself, what the participants specifically see and express in the video, the behavior It is difficult to understand what is being directed at. In view of this point, in the second embodiment, which person is speaking while looking at which of the other participants displayed on the display unit 10 at the time of shooting by the imaging unit 4. The line-of-sight direction information and the arm direction information regarding whether or not the person is pointing or the like are generated, and the participant image in the virtual dialogue space is adjusted so as to match these information. As a result, each participant who views the dialogue video through the display unit 10 can easily recognize to which participant the remarks and gestures of a certain participant are directed, thereby only linguistic communication. However, there is an advantage that nonverbal communication can be further facilitated.

Although the contents of the present invention have been described above in the embodiments, the technical scope of the present invention should not be construed as being limited to the specific configurations described in the embodiments, and the functions of the present invention can be realized. As long as it is, it goes without saying that various modifications and applications to the above-described embodiment also belong to the technical scope of the present invention.

For example, in the first and second embodiments of the present invention, the content of the present invention is explained by taking so-called remote dialogue as an example. It can also be used in a videophone system or the like in between (may be one-to-one). Further, the mode of "dialogue" is not limited to voice dialogue, but also in the case of "dialogue" by exchanging text information such as so-called "chat", the technique of the present invention can be utilized and "remote". However, it is possible to apply the present invention to, for example, dialogue between adjacent rooms in the same building.

Further, the specific configuration of the photographing unit 4 can be other than the 3D scanner, and it is not necessary to interpret the image of the participant to be photographed only in the three-dimensional image. It may be a two-dimensional image, or may be not only a moving image but also a plurality of still images (for example, still images taken at 1-second intervals). It is technically possible to extract feature points even in a two-dimensional image, and a three-dimensional participant image (not to mention two-dimensional ones) is created based on the positional fluctuation of the feature points in the two-dimensional image. It is possible to generate.

Further, the behavior information generation unit 6 generates behavior information, which is information on the behavior, in a mode of expressing the behavior information by the position change of the feature point, but it is not necessary to limit the behavior information to such a mode. The behavior may be directly detected from the video by deep learning or the like and converted into information, or the contour of the participant projected on the video may be cut out and the behavior information may be generated according to the deformation of the contour.

Further, the participant video generation units 14 and 26 do not necessarily have to generate the participant video as a three-dimensional video, and may be a two-dimensional video. The same applies to the dialogue video generated by the dialogue video generation units 15 and 27.

Further, the comparison method in the behavior information comparison unit 19 may be any method of comparing the reference value and the current value, and the absolute value of the difference between the reference value and the current value is divided by the reference value to calculate the value. It may be other than the method of doing. For example, only the difference value between the reference value and the current value may be used, or the comparison may be made based on the ratio of the reference value and the current value.

Further, regarding the identity determination method in the identity determination unit 20, the addition result of a plurality of comparison results (in the first and second embodiments, the absolute value of the difference value between the reference value and the current value is divided by the reference value). Is compared with the first threshold value and the second threshold value, but the weighted average of a plurality of comparison results and the threshold value may be compared, or a threshold value is set for each of the plurality of comparison results and the threshold value is exceeded. The presence or absence of identity may be determined based on the number of (or less) comparison results. Further, the identity determination unit 20 uses two threshold values to indicate three determination results (possible with identity, none, and no identity), but one threshold value and three or more threshold values are used. Etc., and the judgment results may be derived in two ways with or without the same, or four or more kinds of judgment results may be derived depending on the probability of having the same (or "not"). ..

Further, the content / mode of the warning information generated by the warning information generation unit 21 is not limited to the mode described in the first embodiment. For example, when it is determined that there is no identity, the person in charge of the remote dialogue Instead of providing warning information to the user, an instruction to "block the communication status" may be sent directly to the part that controls the communication function of the devices constituting the dialogue system.

Further, in the second embodiment, the dialogue image generation unit 27 not only rotates about the bone corresponding to the spine when matching the line-of-sight direction of the participant image with the line-of-sight direction information, but also the neck portion of the participant image. The line-of-sight directions may be matched by turning, or only the portion corresponding to the eyes of the participant image may be changed. Similarly, when matching the direction of the arm in the participant image with the arm direction information, not only the arm may be moved, but other parts of the body may be moved in conjunction with the arm, and virtual dialogue may be performed. Depending on the positional relationship in the space, the direction of the arm may be changed to the back direction of the participant image, but in that case, instead of changing the position of only the arm, the upper body should be turned backward. The position may be changed.

The present invention can be used as a technique for performing remote dialogue between participants who are not in the same place.

1, 23 Participating terminals 2, 25 Management server 3 Information communication network 4 Shooting unit 5 Speech recording unit 6 Behavior information generation unit 7 Participation information generation unit 8, 22 Information output unit 9, 13 Information input unit 10 Display unit 14, 26 Participation Person video generation unit 15, 27 Dialogue video generation unit 16 Behavior information database 17 Reference behavior information generation unit 18 Current behavior information generation unit 19 Behavior information comparison unit 20 Identity judgment unit 21 Warning information generation unit 24 Attention target information generation unit

Claims (6)

A remote dialogue system that enables dialogue between multiple participants located in different locations.
A reference behavior that detects one or more characteristic behaviors based on information on the past behaviors of participants and generates reference behavior information that is information on the frequency of occurrence of the characteristic behaviors and the duration of the characteristic behaviors at the time of occurrence. Information generation means and
With respect to the one or more characteristic behaviors of the participants at the time of the current dialogue, the current behavior information generating means for generating the current behavior information which is the information on the frequency of occurrence of the characteristic behavior and the duration of the characteristic behavior at the time of occurrence. ,
A behavior information comparison means for comparing the reference behavior information with the current behavior information,
An identity determination means for determining the identity of participants based on the comparison result of the behavior information comparison means, and
An output means for outputting the determination result by the identity determination means to one or more participants other than the participant who was the determination target in the identity determination means, and an output means.
A remote dialogue system characterized by being equipped with. The reference behavior information generating means generates information including the reference duration, which is numerical information regarding the expression frequency and the reference duration, as the reference behavior information.
The current behavior information generation means generates information including the current duration, which is numerical information regarding the current expression frequency and the duration, which is numerical information regarding the expression frequency, as the current behavior information.
The behavior information comparison means is a value obtained by dividing the absolute value of the difference value between the reference expression frequency and the current expression frequency for the same characteristic behavior by the reference expression frequency, and the reference duration and the current duration for the same characteristic behavior. The absolute value of the difference value divided by the reference duration is derived as the comparison result.
The identity determination means determines that there is identity when the sum of the comparison results is less than the first threshold value, and determines that there is a possibility of identity when the sum of the comparison results is equal to or more than the first threshold value and less than the second threshold value. The remote dialogue system according to claim 1, wherein it is determined that there is no identity when the value is equal to or greater than the second threshold value. A display means for displaying a dialogue video, which is a video of multiple participants participating in a remote dialogue, to a specific participant, and
A shooting means for shooting a video of the specific participant, and
Information on which of the plurality of participants the specific participant is looking at at the time of photographing by the photographing means based on the positional relationship between the display means, the photographing means, and the specific participant. Attention target information generation means for generating attention target information including
As the image of the specific participant to be displayed on the display means, the dialogue image generation means for generating the dialogue image so as to direct the line of sight to the participant indicated by the attention target information, and the dialogue image generation means.
The remote dialogue system according to claim 1 or 2, further comprising. A remote dialogue method that enables dialogue between multiple participants located in different locations.
A reference behavior that detects one or more characteristic behaviors based on information on the past behaviors of participants and generates reference behavior information that is information on the frequency of occurrence of the characteristic behaviors and the duration of the characteristic behaviors at the time of occurrence. Information generation steps and
With respect to the one or more characteristic behaviors of the participants at the time of the current dialogue, the current behavior information generation step for generating the current behavior information which is information on the frequency of occurrence of the characteristic behaviors and the duration of the characteristic behaviors at the time of occurrence. ,
A behavior information comparison step for comparing the reference behavior information with the current behavior information,
An identity determination step for determining the identity of participants based on the comparison result of the behavior information comparison step, and
An output step for outputting the determination result by the identity determination step to one or more participants other than the participant who was the determination target in the identity determination step, and an output step.
A remote dialogue method characterized by including. A display step that displays a dialogue video, which is a video of multiple participants participating in a remote dialogue, to a specific participant, and
The shooting step of shooting the video of the specific participant, and
Based on the display location in the display step, the shooting point in the shooting step, and the positional relationship between the specific participants, the specific participant looks at which of the plurality of participants during shooting in the shooting step. Attention target information generation step to generate attention target information including information on whether or not the user is pointing
As the image of the specific participant to be displayed in the display step, the dialogue image generation step of generating the dialogue image so as to direct the line of sight to the participant indicated by the attention target information, and the dialogue image generation step.
4. The remote dialogue method according to claim 4, further comprising. A remote dialogue program that enables a computer to interact with multiple participants located in different locations.
A reference behavior that detects one or more characteristic behaviors based on information on the past behaviors of participants and generates reference behavior information that is information on the frequency of occurrence of the characteristic behaviors and the duration of the characteristic behaviors at the time of occurrence. Information generation function and
With respect to the one or more characteristic behaviors of the participants at the time of the current dialogue, a current behavior information generation function that generates current behavior information which is information on the frequency of occurrence of the characteristic behaviors and the duration of the characteristic behaviors at the time of occurrence. ,
A behavior information comparison function that compares the reference behavior information with the current behavior information,
An identity determination function that determines the identity of participants based on the comparison result of the behavior information comparison function, and
An output function that outputs a judgment result by the identity judgment function to one or more participants other than the participant who is the judgment target in the identity judgment function, and an output function.
A remote dialogue program characterized by realizing.

Images