JP2016048855A - Remote communication device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote communication device capable of avoiding influences of delay in a remote communication.SOLUTION: An analysis part 2 acquires a sound and an action from one user as data. In accordance with data of an action transmitted from a device of another user, a presentation part 5 displays the action of the other user. On the basis of the data acquired by the analysis part 2, a predictive action part 3 discriminates whether the one user starts talking and, in the case where it is discriminated that the one user starts talking, performs control so as to replace the action display of the other user on the presentation part 5 with a predetermined predictive action. A reaction control part 4 performs control in such a manner that, after the action display of the other user on the presentation part 5 is replaced with the predetermined predictive action, the action display is returned to display according to the data of the action transmitted from the device of the other user.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a remote communication device and program capable of avoiding the effect of delay in communication performed remotely via a network.

  There are benefits such as environmental protection that reduces carbon dioxide emissions and business trip expenses, and remote conference calls via videophones have increased in the workplace. However, since most videophone systems on the market have a configuration in which the video captured at each remote location is displayed as it is together with the recorded audio on the other party's side, there is a problem that the line of sight is difficult to match and a problem that it is impossible to speak anonymously. For example, when a client company conducts a group interview on a product as a case where anonymous speech is preferable, a more flexible response can be achieved if a panelist can participate remotely. Furthermore, anonymous participation is desirable for client companies to participate in discussions.

  In relation to the above two problems, there is Patent Document 1 as a prior art that enables anonymous participation by reflecting the line of sight in a remote conference. In Patent Document 1, a three-dimensional computer model (avatar) representing an attendee is animated according to the movement of the attendee in the real world. Hereinafter, the remote conference based on the animation is referred to as an avatar remote conference.

  Specifically, in the avatar remote conference, an image of a three-dimensional computer model including an avatar representing an attendee is displayed at each user station. A pair of cameras is used to process image data of a user wearing headphones with body markers and lights to determine the user's movement and the point in the display image being watched. This information is transmitted to other user stations and animated so that the movement of the avatar's head corresponds to the converted movement of the user's head.

  On the other hand, face-to-face, that is, actual communication between humans facing each other proceeds while looking at other people's feedback. As discussed in Non-Patent Document 1, when communication is performed remotely as in a remote conference, a delay occurs in network transmission or calculation processing, and thus a response such as feedback is delayed. The delay may cause the smoothness of communication such as “I feel uncomfortable in communication”, “I am disturbed by communication”, “I don't understand the other party's story”, “I don't enjoy talking”, etc. It is said.

  In order to reduce the hindrance to the smooth progress of the remote conference due to delay, in Patent Document 2, when a reaction signal is detected at the other party in the videophone, only the message that there is a reaction signal is sent from the other party and the message is received. Then, the reaction time image prepared in advance by interrupting the streaming image is displayed.

JP 2000-244886 A JP 2010-154387 A JP 2013-089170 A

Karen Ruhleder, Brigitte Jordan, Meaning-Making Across Remote Sites: How Delays in Transmission Affect Interaction, ECSCW'99, pp 411-429, 1999 Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011.Realtime performance-based facial animation.ACM Trans.Graph. 30, 4, Article 77 (July 2011), 10 pages.

  Here, when analyzing the delay of the avatar remote conference as in Patent Document 1, the delay is the sum of the time for detecting a reaction such as whispering and the display time of the avatar in addition to the bidirectional network transmission time. FIG. 1 shows an image diagram of the delay.

  In FIG. 1, from the moment t1 when the participant A in the place LA starts speaking, the response of the participant B in the remote place LA is displayed to the participant A in the place L1 in response to the start of the statement. The fact that the total delay DL7 up to the instant t9 is the sum of the delays DL1 to DL6 as elements is shown on the time axes of both A and B. Specifically, it is as follows.

  First, the participant A starts speaking at time t1 and performs speaking SA1 until time t3. Reaction detection delay DL1 occurs as a delay for converting the state of participant A at the instant t1 when the speech starts to data for network transmission (data for displaying an avatar at location LB), from time t2. Transmission of the data is started. As indicated by an arrow AR2, when the data is transmitted from the location LA to the location LB, a net delay DL2 occurs, and the data of the participant A's speech start instant t1 arrives at the location LB at time t4.

  Next, a display delay DL3 occurs in order to display the arrived data at the location LB as an avatar, and at the time t5, the participant t Is displayed as an avatar. Accordingly, the participant B makes a response or other response RB1 from the time t5 to the time t7 in response to the speech SA1 of the participant A. The data at the start instant t5 of the response RB1 is converted to data for displaying an avatar at the place LA, with a reaction detection delay DL4, and transmission of the data toward the place LA is started at time t6.

  Furthermore, as indicated by the arrow AR5, the data transmitted at the time t6 is accompanied by the net delay DL5, and after reaching the place LA at the time t8, accompanied by the display delay DL6 for displaying the avatar, at the time t9. For the first time, the response of the participant B to the moment when the participant A in the place LA starts speaking at time t1 is displayed and transmitted to the participant A in the place LA. Accordingly, a delay D7 obtained by adding the above delays D1 to D6 occurs as a series of periods from time t1 to t9.

  In FIG. 1, participant A's utterance SA1 is drawn as having already ended at the time t9, but the length may vary arbitrarily depending on the actual utterance. It is. Accordingly, the utterance SA1 may continue even at time t9 when the participant A can first see the state of the response from the participant B to his utterance SA1.

  As described above, when the delay in the avatar remote conference as in Patent Document 1 is analyzed as shown in FIG. 1, even if the method of Patent Document 2 is applied to the avatar remote conference, the problem that the effect of delay reduction is small appears. . In other words, when the technique of Patent Document 2 is applied to a videophone such as a live-action image, there is an effect of reducing delay by greatly reducing transmission data and omitting encoding and decoding time of the image. Is not enough.

  The reason why the effect is not sufficient is that the data transmitted in the avatar remote conference is only human joint motion data and is sufficiently smaller than the image data, and there is no encoding and decoding of the image. In other words, considering the case where the method of Patent Document 2 is applied to an avatar remote conference, the change due to the application is that the transmission data from the place LB to the place LA reacts to the participant B from the motion data of the joint of the participant B It only replaces the data that there is. Therefore, the effect is that the net delay DL5 in FIG. 1 is only slightly shortened by slightly reducing the data amount, and the effect is small in reducing the total delay.

  In the first place, even if the method of Patent Document 2 is applied to the avatar remote conference, the net delay DL2 in FIG. 1 for transmitting the motion data of the joint of the participant A itself is smaller than the image. Can not be avoided.

  That is, the technique of Patent Document 2 transmits and receives a reaction signal with a small amount of data instead of transmitting and receiving large data such as video in both directions, and displays a response self-portrait prepared in advance until there is an actual reaction. By substituting, the effect of delay is reduced compared to the case of video. However, since it is necessary to exchange some signal (that is, a reaction signal with a small amount of data) between the remote locations LA and LB as shown in FIG. The same kind of delay (ie, net delays DL2 and DL5) itself cannot be avoided.

  In view of the above-described problems of the prior art, an object of the present invention is to provide a remote communication device and a program that can avoid the influence of delay in remote communication such as an avatar remote conference.

  In order to achieve the above object, the present invention is a remote communication device that mediates communication between another user and the user by bidirectionally transmitting and receiving voice and operation data to and from another user's device. Based on the data acquired by the analysis unit, the analysis unit that acquires the voice and the operation as data from the own user, the presentation unit that displays the other user according to the operation data transmitted from the device of the other user, and the data acquired by the analysis unit. Determining whether or not the user has started speaking, and determining that the user has started speaking, the prediction operation unit that controls to replace the other user's operation display in the presenting unit with a predetermined prediction operation; and the predetermined prediction A reaction control unit that controls to return the operation display of the other user in the presenting unit to the display according to the operation data transmitted from the other user's device after being replaced with the operation. It is characterized by.

  In addition, the present invention is a program that causes a computer to function as the remote communication device.

  According to the present invention, when it is determined that the user has started speaking, the operation display of other users can be switched to a predetermined prediction operation. Here, whether or not the user has started speaking is information that can be independently determined by the user's remote communication device without transmitting or receiving information via the network, and therefore the influence of delay can be avoided. It becomes possible.

It is the image figure which analyzed the delay in the avatar remote conference in patent documents 1. It is a figure which shows the structural example of the remote communication system which concerns on one Embodiment. It is a figure which shows the example of the screen shown in the remote communication apparatus which a certain participant uses. It is a functional block diagram of the remote communication apparatus which concerns on one Embodiment. It is a flowchart of the interruption process by the prediction operation | movement part and reaction control part with respect to the operation display of the other user in a presentation part. It is a figure which shows notionally the example of the operation data of the other user utilized in an interruption process on a time-axis. It is a functional block diagram drawn including data exchange at the time of exchanging bidirectionally between two remote communication devices. It is a figure which shows the example which defines several prediction operation | movement according to the content of a statement in a table format. It is a conceptual diagram for demonstrating a blending process.

  FIG. 2 is a diagram illustrating a configuration example of a remote communication system according to an embodiment. The remote communication system 10 includes a plurality of remote communication devices 1A to 1D for allowing users A to D to perform remote communication such as a remote conference. As shown in the figure, the users A to D exist in remote locations LA to LD, respectively, and communicate with each other via the network N by using their own remote communication devices 1A to 1D.

  FIG. 2 shows an example in which four users A to D use the remote communication devices 1A to 1D as an example, but the remote communication system 10 is realized by an arbitrary number of remote communication devices. Can do. Further, one remote communication device existing in one remote location may be shared by two or more users.

  In order to realize the mutual communication in the remote, each of the remote communication devices 1A to 1D transmits the voice and operation information of the own user to the other user's device, and from the other user's device to the other Continuously receiving the user's voice and operation information. For example, as shown in the figure, in the remote communication device 1A used by the user A, the voice and operation information of the own user A are transmitted to the remote communication devices 1B to 1D of the other users B to D, and the other The voice and operation information of each of the other users B to D is received from the remote communication devices 1B to 1D of the users B to D.

  FIG. 3 shows a screen displayed by the remote communication device 1A for the user A in order to realize remote communication with other users B to D in the remote communication device 1A of the user A in FIG. It is an example. As illustrated, on the screen D [A], users A to D who communicate remotely are displayed as avatars AB [A] to AB [D], respectively. In the screen D [A], the avatar AB [A] of the own user A moves according to the movement of the own user A acquired by the remote communication device 1A, and the avatars AB [B] to AB [D] of other users B to D Move according to the movements of the other users B to D acquired and transmitted by the respective remote communication devices 1B to 1D.

  In this way, user A's own avatar AB [A] and remote user B ~ D's avatars AB [B] -AB [D] meet together on screen D [A] and move according to each user's actual movement. You can communicate remotely while watching the situation. Other users B to D are also provided with the same screen as the screen D [A], and can communicate remotely. At this time, as described with reference to FIG. 2, the voice (the remarks of each user) is also reproduced on each user's device together with the screen, thereby enabling remote communication.

  Here, as described with reference to FIG. 1, assuming that each remote communication device in the remote communication system configured as shown in FIG. 2 is configured by the prior art, an operation transmitted from a remote user In addition, there is a problem that inevitable delay occurs in the voice. For example, in the screen D [A] as shown in FIG. 3, the avatar AB [B] to AB [D] of other users B to D is started even though the avatar AB [A] linked to the own user A has started to speak. A situation occurs in which no response is given to the remarks, and communication inconvenience occurs such that the user A looking at the screen D [A] feels uncomfortable. According to the present invention, the inconvenience can be avoided. Details will be described below.

  FIG. 4 is a functional block diagram of the remote communication device 1 according to an embodiment. As described with reference to FIG. 2, the remote communication device 1 exists for each user or two or more users who use the device. In FIG. Is explained. The remote communication device 1 includes an analysis unit 2, a prediction operation unit 3, a reaction control unit 4, and a presentation unit 5. The outline of each part is as follows.

  The analysis unit 2 is configured to include a microphone, a camera, a sensor, etc., so that the voice and body movements of the own user (user using the remote communication device 1) are acquired as data, and the data is predicted. To the unit 3, the presentation unit 5, and the other user's device (another user's remote communication device). When the data is transferred to another user's device, data transmission is performed via the network N as described with reference to FIG.

  The presentation unit 5 receives the voice and operation data of the own user sent from the analysis unit 2, and the voice and operation data of the other user sent from one or more other users' remote communication devices. Thus, information for the user to communicate remotely with other users is presented.

  If a remote conference is realized as a remote communication, the presentation unit 5 can present the user with a screen and a voice for performing the avatar remote conference as described in FIG. In this case, the presentation unit 5 performs control processing so that each avatar in the screen is operated according to the operation data of the corresponding user (self user or other user). In addition, the presentation unit 5 performs a process of playing back the voice spoken by each user in conjunction with the screen display. In the following, the present invention will be described by taking as an example the case of realizing the avatar remote conference.

  In the predictive action unit 3 and the reaction control part 4, when it is determined that there is an influence of delay from the standpoint of the own user on the action display of the other user in the presenting part 5, that is, the avatar action display, The operation display processing of other users in the presentation unit 5 is controlled so that a predetermined operation display in which the influence of delay is not felt from the standpoint is performed by interruption. While the interruption process is not being performed, the presentation unit 5 displays the corresponding avatar according to the operation data of the user and other users obtained at each time point as described above.

  The outline of the interrupt processing is as follows. First, the prediction operation unit 3 determines whether or not the other user's operation display in the presentation unit 5 is affected by delay from the standpoint of the user. Specifically, if it is determined that the user starts to speak by examining the data of the user sent from the analysis unit 2, the following predetermined period is assumed to be affected by the delay from the user's standpoint. judge.

  Specifically, this determination is performed in advance to detect the occurrence of an inconvenient situation for communication as described with reference to FIG. That is, even if the user has started speaking, if the data of the delay is used as it is and the operation of another user is displayed on the presentation unit 5, the first predetermined period will be This is an inconvenient situation where a screen on which no other users are responding is displayed. The details of the process for determining whether or not the user has started speaking will be described later.

  Further, when it is determined that the user has started speaking as described above, the prediction operation unit 3 performs a predetermined prediction operation instead of displaying the operation sent from the remote communication device of the other user as it is. The display processing of other users in the presentation unit 5 is controlled so as to be temporarily displayed. In this way, from the user's standpoint, as soon as he / she starts speaking, other users displayed on the presentation unit 5 can perform a predetermined predictive action so that the state of the response to his / her speech can be seen. You can continue to speak without feeling uncomfortable.

  Here, in particular, the display of the predetermined prediction operation is performed without waiting for an actual response operation from another user and between the remote communication device of the other user and the remote communication device 1 of the own user. Since it can be implemented immediately without performing transmission / reception of specific information such as transmission / reception of reaction signals in Patent Document 2, the present invention can avoid the influence of network delay. That is, the information as to whether or not the own user has started speaking is information that can be independently determined within the remote communication device 1 on the own user's side. It is unnecessary information to communicate with the communication device. In the present invention, since the prediction operation is displayed using such information as a trigger, the influence of the network delay can be avoided.

  Next, the reaction control unit 4 presents the response by the prediction operation unit 3 when an actual response operation from another user in response to the start of speech by the own user is received (with a network delay or the like). The presenting unit 5 is controlled to switch the operation of the other user who is controlled to move according to the predetermined prediction operation in the unit 5 to the actual operation of the other user who has received the operation. After the switching, the operation of the other user is displayed on the presentation unit 5 according to the actual operation of the other user. Further, when the switching is performed, a process of smoothly connecting the predicted operation and the actual operation is performed, and details thereof will be described later.

  FIG. 5 is a flowchart showing the interrupt processing described above by the prediction operation unit 3 and the reaction control unit 4 for the operation display of other users in the presentation unit 5. FIG. 6 is a diagram conceptually showing an example of operation data of other users used in the interrupt process on the time axis.

  Hereafter, each step of FIG. 5 is demonstrated, referring FIG. 6 suitably. In FIG. 6, for the sake of explanation, it is assumed that the user A is the user A, and the other user who is interrupted with respect to the operation display is the user B. In FIG. 6, (1) is the time series D1, D2, P1, D7, D8 of the data of other user B displayed in the own user A, and (2) is the other user B's original data to be displayed. Time series D1 to D8 of actual operation data.

  In FIG. 6, the data displayed in the interrupt processing is P1 of (1), but the data P1 is an example for briefly explaining the interrupt processing, and the operation is displayed immediately at the interrupt timing (time T5, T9). This is an example in which the process for allowing the user A to see the actions of the other user B naturally and continuously is omitted. The data P10 in (3) is obtained by partially correcting the data P1 by performing the process of making the image appear natural without being omitted. That is, from the viewpoint that the operation looks natural, it is preferable that the data P10 is actually displayed by the interrupt process.

  In the flow of FIG. 5, the time is referred to as the time t by the counter variable t. The overall structure of the flow is as follows. That is, by dividing the case according to the situation placed at each time t, at each time t, in any one of steps S11, S20, or S30, the presentation unit 5 determines the actions of other users to be displayed at the time t. The overall structure is that the display of the corresponding frame image is continued while updating the time t to the next latest time point.

  When the flow starts in step S1, first, in step S10, the prediction operation unit 3 analyzes the data of the own user of the analysis unit 2 at the time t to determine whether or not the own user has started speaking. To do. If it is determined that it has started, the process proceeds to step S20, and if it is determined that it has not started, the process proceeds to step S11. Details of the determination in step S10 will be described later.

  In step S11, the presenting unit 5 displays other users based on the actual data of other users received and displayed at the current time t, and proceeds to step S12. In step S12, the time t is updated to the next latest time point, and then the process returns to step S10.

  Thus, while the other users at each time t are continuously displayed in the loop of steps S10, S11, and S12, the actual operation transmitted from the other users is continuously displayed on the presentation unit 5. It becomes. In the case of an avatar remote conference, the actual operation continues to be displayed via the avatar.

  In the example of FIG. 6, the state in which the operations of other users are displayed with the data D1 and D2 in the series of time periods T3 to T5 in (1) remains in the loop of steps S10, S11, and S12 in FIG. Corresponds to the state. In this case, as shown in (2), the other users moved the data D1 and D2 during the time T1 to T3 before the time T3 to T5, with a delay such as a network delay. Although the actions of other users are displayed on the presentation unit 5, the user himself / herself has not started speaking and is in a state where there is no need to see the immediate response from other users. Absent.

  Returning to FIG. 5, in step S <b> 20, the presentation unit 5 receives the control from the prediction operation unit 3 and shifts to the prediction operation, or if the state has already been transferred to the prediction operation, the prediction operation after the transfer The frame image of the other user at the time t (displayed as an avatar image in the case of an avatar remote conference) is displayed in conformity with the above, and the process proceeds to step S21. Details of the display process in step S20 will be described later.

  In step S21, the reaction control unit 4 determines whether or not a response has been received from another user at the time t. If a response has been received, the process proceeds to step S30. If not, the process proceeds to step S22. In step S22, time t is updated to the next latest time point, and then the process returns to step S20.

  In order to determine whether or not the response control unit 4 has received a response from another user in step S21, the following may be performed. That is, when the prediction operation unit 3 determines the start of the user's speech in the latest step S10, the operation data of the own user transmitted from the analysis unit 2 to the remote communication device of the other user at the time t100 In addition, flag information indicating that the start of speech has been determined by the user is added. In the remote communication device of another user, when the flag information is added to the operation data of the own user (another user for another user) to be displayed at time t100 + ΔT1 with a delay ΔT1 In addition, flag information indicating that the response to the start of the user's speech is started by the other user is added to the operation data of the other user himself / herself at the time t100 + ΔT1 as reply flag information. Reply to the communication device 1. Further, the response control unit 4 of the own user who has received the operation data of the other user to which the flag information indicating that the response has been started is added at time t100 + ΔT1 + ΔT2 with the delay ΔT2, the time t100 + ΔT1 It becomes possible to make the affirmative determination in step S21 at + ΔT2.

  The above transmission / reception of flag information is illustrated as arrows AR10 and AR11 in FIG. The own user A makes an affirmative determination in step S10 at time T5, that is, a determination to start speech, and adds a speech start flag to his / her operation data at time T5 to be transmitted to the other user B. As indicated by the arrow AR10, the other user B displays the operation data of the own user A to which the flag is actually assigned at time T7 after receiving the operation of the own user A to which the speech start flag is assigned. Accordingly, a flag indicating that a response has been started is added to the operation data D7 of the other user B himself / herself to be transmitted at the time T7 and transmitted. As indicated by the arrow AR11, the own user A who displays the operation data D7 of the other user B to which the response has been started at time T9 may make a positive determination in step S21 at the time T9. it can.

  Returning to FIG. 5, in step S30, the presentation unit 5 receives control from the reaction control unit 4, and from the display by the prediction operation realized in the loop of the latest steps S20, S21, S22, the other user's remote The frame image of the other user at the time t is displayed while gradually returning to the display in the form according to the actual operation data of the other user sequentially transmitted from the communication device, and the process proceeds to step S31. Details of the display process in step S30 will be described later.

  In step S31, the reaction control unit 4 determines whether or not the process of gradually returning to the actual operation of the other user and displaying in the latest step S30 is completely completed. If completed, the process returns to step S10. If not completed, the process proceeds to step S32. In step S32, the time t is updated to the next latest time point, and then the process returns to step S30.

  As described above, each step of FIG. 5 has been described. However, the display process of other users' actions by the presentation unit 5 while remaining in the loop of steps S20, S21, and S22 or the loop of steps S30, S31, and S32 is performed. This corresponds to (substantially) a state in which the predetermined prediction operation is replaced by the interrupt process. Here, when the presentation unit 5 is in the loop of the former steps S20, S21, and S22, it is controlled by the prediction operation unit 3, and when it is in the latter loop of S30, S31, and S32, the reaction control Controlled by part 4.

  In particular, this is an unpreferable embodiment from the viewpoint of smoothly and naturally realizing the operation display of other users. However, immediately after reaching step S10 from step S10, the display based on the motion data of other users is displayed by the prediction operation. Similarly, conversely, when switching from the display based on the prediction operation to the display based on the actual motion data of another user immediately after reaching step S21 to step S30, the data P1 in (1) of FIG. Is the operation data (predicted operation data) of another user displayed by the interrupt process.

  However, since there is a high possibility that the operation display is discontinuous in the embodiment in which the display data is switched instantaneously, an example of display data in the embodiment in which the display is switched smoothly is the data P10 in (3) of FIG. It is.

  In the case of data P10, instead of immediately switching at time T5 (time when affirmative determination is obtained in step S10 in FIG. 5) as in data P1, after detecting time T51 that can be switched smoothly after time T5, The prediction operation is started at the time T51. Similarly, instead of switching immediately at time T9 (the time when the affirmative determination was obtained in step S21 in FIG. 5) as in data P1, until the time T91 thereafter, the predicted operation and the actual operation D7 of other users The operation in the hybrid state is set so that the operation completely returns to the operation D7 at time T91. Details of the configuration of the data P10 will be described later.

  FIG. 7 is a functional block diagram of the remote communication devices 1A and 1B drawn including data exchange when the remote communication devices communicate bidirectionally between the two users A and B. The configuration of each device 1A, 1B is the same as that shown in FIG. 4, as shown in FIG. However, in order to distinguish between the devices 1A and 1B of the users A and B, A and B are given at the end of the reference numbers of the functional units. That is, the analysis unit 2A that acquired the voice and operation data of the own user A passes the data to the presenting unit 5A and the prediction operation unit 3A, and the other user B's prediction operation unit 3B and the reaction control unit Pass to 4B. The analysis unit 2B of the other user B also performs data exchange symmetrical to this as shown in the figure.

  For example, the flag information described as arrows AR10 and AR11 in FIG. 6 is added to the operation data of the analysis unit 2A and then added to the corresponding function unit of the remote communication device 1B and to the operation data of the analysis unit 2B. Then, the data is transmitted to the corresponding functional unit of the remote communication device 1A.

  In addition, the remote communication such as the remote conference in the present invention can be realized even when there is one or more other users who are counterparts to the own user. In this case, each of the remote communication devices of a plurality of other users Bidirectional data exchange as shown in FIG. Thus, for example, if a total of four users each conduct a remote conference using one remote communication device, a screen as shown in FIG. 3 is displayed on the presentation unit 5 of each user's remote communication device 1. Can be displayed.

  The overall operation of the remote communication device 1 has been described above. Below, each part of FIG. 4 is demonstrated regarding the detail of the element technology which implement | achieves the said whole operation | movement. In particular, the details of the operation data to be displayed on the screen as shown in FIG. 3 will be described.

<About Analysis Unit 2>
The analysis unit 2 obtains the user's voice and motion with the device. If there are multiple users, the data is obtained for each user. Available devices include cameras and microphones, Kinect (registered trademark), or Google Glass (registered trademark).

For example, in the embodiment using Kinect, the posture of the head, the facial expression, the motion of the upper body, the presence or absence of voice, and whether the target person is speaking are estimated. At the same time, video, audio and depth data are recorded with Kinect. The SDK (software development kit) provided by Microsoft (registered trademark), the posture of the head Pitch, yaw, roll (pitch, yaw, roll) and face parts AU (AU0 to AU5) (action unit) from Kinect data ). Furthermore, the facial expression is determined by the facial part AU. For example, when AU4 (Lip Corner Depressor) is used, three types are determined: 0 = neutral, -1 = pos, and + 1 = neg. The tracking of the head posture and face part AU is disclosed in the following URL.
[URL] http://msdn.microsoft.com/en-us/library/jj130970.aspx

Also, human motion is obtained from Kinect data using the SDK provided by Microsoft. In the case of a meeting, it is difficult to take the motion of the lower body, but since it is not important, it is only necessary to acquire the motion of the upper body in the Seated mode. Details of the motion acquisition are disclosed at the following URL.
[URL] http://msdn.microsoft.com/en-us/library/hh973077.aspx

  In addition, the presence / absence of the subject's speech is determined based on Kinect audio data. First, when the volume exceeds a threshold value, it is determined that there is a speech. Conversely, when the volume is equal to or less than the threshold value, it is determined that there is no speech. Here, when one remote communication device 1 is used by a plurality of users, when there is a speech, a known speaker verification technology based on voiceprint verification is further used to Determine whether or not. Here, the voice print data of each user is registered in advance.

  Further, as another embodiment for specifying which user the utterance is, it is possible to use a method in which a microphone is provided for each user and the relative difference in sound volume is used. That is, the recording of a plurality of microphones may be analyzed at each time, and the user using the microphone having the maximum volume may be determined as the speaker at that time.

<About Presentation Unit 5>
The presentation unit 5 receives the data from the analysis unit 2 described above, and causes the own user and other users' avatars to animate. Here, regarding the control when operating the other user's avatar, the control by the prediction operation unit 3 and the reaction control unit 4 unique to the present invention is performed when a predetermined condition is satisfied as described in FIG. 5 and FIG. Will follow. However, a well-known technique can be used as the technique for causing the avatar to perform an animation operation regardless of whether the control is followed or not.

For example, an avatar's facial expression and motion can be animated in real time using Kinect. As a technique for animating an avatar expression in real time using Kinect, for example, a technique as described in Non-Patent Document 2 described above is known. In addition, an SDK that animates the upper body of an avatar in real time using Kinect is provided by Microsoft and is disclosed at the following URL.
[URL] http://msdn.microsoft.com/en-us/library/hh973077.aspx

  Note that it is the prediction operation unit 3 and the reaction control unit 4 that control what is used for the “real-time” data regarding the avatars of other users that are animated by the presentation unit 5 of the own user. The animation operation of the own user's avatar in the own user's presentation unit 5 always uses "real time" data in the own user's position.

<About Predictive Action Unit 3>
As described with reference to FIGS. 5 and 6, the prediction operation unit 3 unnaturally displays other users' avatars to be displayed on the presentation unit 5 when it is determined that the user has started speaking. It shifts to an animation operation according to a predetermined prediction operation so that it cannot be seen.

  The determination of the start of speech may be made according to the presence / absence of speech by the threshold determination in the analysis unit 2 described above. For example, when it is determined that there is no utterance for a predetermined period or longer, when it is determined that there is a utterance, or when the presence of utterance continues for a predetermined period or longer, the start of the utterance is determined. That's fine.

  As for the data of the predetermined prediction operation, the same kind of data as the data acquired by the analysis unit 2 may be extracted in advance from a predetermined operation that does not look unnatural from the standpoint of the user who started the speech. . The prediction operation unit 3 controls the animation operation of another user in the presentation unit 5 using the data extracted in advance.

  Specifically, the following prediction operation can be employed. For example, after starting to speak, predictive action is adopted that automatically makes the displayed other user's avatar appear or thinks even when not speaking or intentionally staring. You can do it.

  However, since specific motion data regarding the whispering and thinking posture has the personality of the person, it is preferable to acquire a motion having personality in advance for each other user. For example, if another user has participated in a past meeting, it is only necessary to manually cut out a thinking or thinking attitude in the motion data acquired at that time. Also, if there is no such data, you can instruct the participants and record the data for whispering and thinking.

  Moreover, after the speech start determination in the prediction operation unit 3, the timing of switching the avatar operation of the other user in the presentation unit 5 from the one according to the actual data to the one according to the data of the prediction operation may be determined as follows.

  In one embodiment, the frame having the shortest distance from the start frame of the predicted motion data is received in the motion data of other users that have been received at that time but are waiting for display processing in the presentation unit 5 (so-called buffer data). The search may be switched to the prediction operation data from the time of the search.

  For example, in the example of (3) in FIG. 6, if the buffer data is D3 at the speech start time T5, the frame within the data D3 that has the smallest distance from the disclosed frame of the prediction operation is the time T51. As a result, the display P10 based on the prediction operation is started from time T51.

  In another embodiment, the display may be switched to the display based on the prediction operation as soon as the prediction operation unit 3 determines to start speaking. However, the reproduction start position of the predicted operation to be switched is determined as the one closest to the posture of the other user who is the display target at the time of the determination. In this case, in the example of FIG. 6, as in (1), the display P1 by the prediction operation is immediately disclosed at the determination time T5.

  In each of the above embodiments, the distance between the posture in the frame in the predicted motion data and the posture of the frame in the actual data of other users is calculated. The calculation of the distance can be performed in the same manner as Equation (1) in the description of the reaction control unit 4 described later.

  Note that only one type of prediction operation may be prepared for each other user, or two or more types of prediction operations appropriate for the content of the user's speech that has started speaking may be prepared. For example, as shown in a table form in FIG. 8, two types of predictive actions “think” and “buzz” are prepared, and if the content of the speech of the user who started speaking is a question, The “predictive action” may be used, and if it is something other than the question, the “nodding” predictive action may be used.

  Here, regarding the determination of the content of the utterance such as whether or not the question is made in FIG. 8, the prediction operation unit 3 uses a well-known utterance content analysis method (voice analysis and It may be determined by applying (text analysis).

<About Reaction Control Unit 4>
When the response signal of the actual listener (other user) is obtained, the reaction control unit 4 displays the operation display of the other user in the presentation unit 5 so as to smoothly connect the predicted operation and the actual response operation. Control. Note that the reaction signal can be acquired as flag information indicating that there is a response in the arrow AR11 described in FIG.

  Regarding the processing for smoothly connecting, first, the case where a plurality of prediction operations are set according to the content of a statement as described in FIG. 8 in the prediction operation unit 3 will be described.

  First, when a response signal is received, the predicted action determined from a plurality is different from the actual response action (case 1: when the result of the prediction action unit 3 is wrong) (case 2: prediction). The two cases are divided into two cases: when the result of operation part 3 hits.

  Specifically, in the prediction operation reproduced by the presentation unit 5 under the control of the prediction operation unit 3, the motion for a certain period of time from now (hereinafter referred to as motion data M1) and the actual response operation of the other user received from the analysis unit of the other user In (hereinafter referred to as motion data M2), the distance of each frame is calculated by the following equation (1) to obtain the smallest distance. The motion data M2 may be data having a predetermined length from the time when the response signal is obtained.

Here, the distance d (F ⁱ _B , F ^j _B ) between the frame F ⁱ _B with the motion data M1 and the frame F ^j _B with the motion data M2 can be calculated by the following equation (1).

However, q _{i, k} is a quaternion of the k-th joint of the frame F ⁱ _B , and represents the posture as disclosed in the above-mentioned Patent Document 3. w _k is a weight related to the k-th joint. The weight w _k is preset.

  When the minimum distance among all the combinations (i, j) in the above formula (1) becomes a certain value or more, the case 1 is determined. In the case 1, when the motion data M1 continues the predicted motion for a while and reaches the neutral pose, the motion data M2 may be reproduced from the neutral pose.

  As for the neutral pose, data that can be determined as a neutral pose from within each of the motion data M1 and M2 by defining the joint quaternion with a predetermined value as in the above equation (1). Can be determined as that of the minimum distance as in the equation (1).

  Further, when the minimum distance among all (i, j) combinations of the above formula (1) is less than a certain value, it is determined as case 2. In Case 2, the frame pair (i, j) with the shortest distance in the above equation (1) is selected as the optimum branch point (connection point). Then, it is only necessary to perform motion blending (hereinafter referred to as blending) for a frame of a certain time around the optimum branch point. Blending is a well-known technique disclosed in the above-mentioned Patent Document 3 and the like, and is a technique for smoothly connecting different operations.

  FIG. 9 is a conceptual diagram for explaining the blending process. In the blending process, interpolated data (blending data) in which the motion data M1 having the frame i and the motion data M2 having the frame j are mixed with each other so that the motion connection is not unnatural. MB1_2 is generated.

  In one embodiment, the connected portions may be interpolated using spherical linear interpolation by quaternions using frames for a certain period of time. Specifically, as shown in FIG. 9, the blending data MB1_2 in the connection section (section length m, where m is a predetermined value) connecting the motion data M1 and the motion data M2 is centered on the frame i of the motion data M1. Are generated using the data M2_m of the section length m around the node j of the surrounding section length data M1_m and the motion data M2. (The data M1_m and M2_m are not shown in FIG. 9)

  At this time, according to the ratio (u / m) of the distance u from the head of the connection section to the section length m of the connection section, the frame i corresponding to the distance u in the data M1_m corresponds to the distance u in the data M2_m. Mix frame j. Specifically, each frame constituting the blending data MB1_2 is generated by the following equations (2) and (3). Equation (2) is an equation for one bone.

  Where m is the total number (predetermined value) of the frames (blending frames) constituting the blending motion data MB1_2, u is the order from the top of the blending frame (1 ≦ u ≦ m), and q (k, u) is the uth The quaternion of the kth bone in the blending frame, q (k, i) is the quaternion of the kth bone in frame i, and q (j) is the quaternion of the kth bone in frame j. However, blending is not performed on the route. Equation (3) is a calculation formula of slerp (spherical linear interpolation).

The blending data MB1_2 is data of a connection portion between the motion data M1 and the motion data M2. Thus, the finally reproduced data is M1 _[front] , MB1_2 and M2 _[rear] in FIG.

  Hereinafter, supplementary matters in the present invention will be described.

  (1) The voice data sent from another user may be played back simultaneously with the display of the operation data sent from the other user at the same time when played back by the presenting unit 5 of the user. Here, even when the prediction operation unit 3 and the reaction control unit 4 control the operation display of other users, the audio data received at that time is always reproduced independently of the control. Alternatively, a predetermined predicted sound may be reproduced in the same manner as when a predetermined prediction operation is interrupted by the operation display.

  (2) Although this invention demonstrated as an example the case where an avatar remote conference is implement | achieved, it is necessary to eliminate the unnaturalness by the delay at the time of displaying the response action of another user with respect to a user's own speech. The present invention is applicable to any remote communication as described above. In other words, the present invention can be applied not only to meetings in business and others, but also to communication between teachers and students in remote education via a network, and when a game etc. is executed between participants who exist remotely. It is.

  (3) Similarly, the present invention is applicable not only to avatar display based on animation display but also to displaying a user in live action. In this case, only when the prediction operation unit 3 and the reaction control unit 4 interrupt and control the display of the presentation unit 5, the “actual avatar” of other users is displayed by using the operation information when using the avatar display. In other cases, a live-action video of another user may be displayed. However, when the interrupt display is switched, various switching processes using the above-described equation (1) may be realized by associating operation information as an avatar with a live-action video.

  Thus, for example, when relaying to a remote location in live broadcasting, when displaying the other party in the remote location on the screen, by displaying the “live-action avatar”, unnaturalness when issuing a question to the other side May be eliminated.

  It should be noted that, when realizing the live-action display, image information is transmitted from the partner side, but by receiving the image information, the operation data of the partner is also received. In addition, when performing an interrupt display, it is necessary to use operation data for causing an avatar to perform an animation operation in a form associated with image information. Therefore, the analysis unit 2 performs the association process and then transmits its own operation data to the other user's device.

  (4) In the example of the avatar remote conference, the presentation unit 5 always displays the avatar of the own user in real time, but displays the own user on the screen according to other uses to which the present invention is applied. This may be omitted. For example, in the screen display of the remote party in the live broadcast, the user does not need to display it.

  (5) “Remote” communication realized by the remote communication device 1 does not matter whether the physical distance is large or small. The remote communication device 1 of the present invention can be used in any case where voice and motion data are exchanged between devices via a network in both directions. For example, the remote communication device 1 of the present invention can be used even when the physical distance is sufficiently small, such as communication between adjacent rooms in the same building via a LAN. Mediate communication between each other.

  (6) Predictive motion data used in the predictive motion unit 3 is preferably prepared as data that can repeatedly reproduce a predetermined unit motion. For example, in the case of a “whipping” operation, it is preferable to prepare data that can be reproduced smoothly and repeatedly, with one or a plurality of whispering operations as unit operations.

  Similarly, predictive motion data may be prepared over a predetermined maximum length in which the user's speech is expected to continue. In this case, it is preferable that a pause determined periodically as neutral appears in the predicted motion data.

  (7) The present invention can also be provided as a program that causes a computer to function as the remote communication device 1. The computer can be configured with known hardware such as a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU that reads and executes the program functions as each part of the remote communication device 1. Become.

  1 ... Remote communication device, 2 ... Analysis unit, 3 ... Predictive action unit, 4 ... Reaction control unit, 5 ... Presentation unit

Claims (10)

A remote communication device that mediates communication between another user and the user by bidirectionally transmitting and receiving voice and operation data to and from another user's device,
An analysis unit that acquires voice and motion as data from the user;
A presentation unit for displaying an operation of another user according to the operation data transmitted from the device of the other user;
Based on the data acquired by the analysis unit, it is determined whether or not the user has started speaking, and when it is determined that the speaking has started, the operation display of other users in the presenting unit is replaced with a predetermined prediction operation A predictive motion unit to control;
A reaction control unit configured to control the display of the other user's operation in the presenting unit to return to the display according to the operation data transmitted from the other user's device after being replaced with the predetermined prediction operation. Remote communication device. A plurality of types of the predetermined prediction operations are prepared in advance,
The prediction operation unit, when determined, selects a predetermined prediction operation according to the content of the user's utterance determined to have started the utterance from the plurality of types, and then the other user in the presentation unit The remote communication device according to claim 1, wherein the operation display is controlled to be replaced with the selected predetermined prediction operation. The predetermined prediction operation is prepared for each other user in advance,
3. The control according to claim 1, wherein, when the determination is made, the prediction operation unit performs control to replace an operation display of another user in the presentation unit with a predetermined prediction operation corresponding to the other user. Remote communication device.   The prediction operation unit starts reproduction of the predetermined prediction operation from the state determined to be closest to the posture of the other user displayed by the presentation unit at the time of the determination when the determination is made. 4. The remote communication device according to claim 1, wherein the remote communication device controls the operation display of the other user in the presentation unit to be replaced with a predetermined prediction operation.   The reaction control unit receives operation data transmitted from another user's device, and is associated with response information indicating that there is a response from the other user in response to the start of the user's own speech. 5. The remote communication device according to claim 1, wherein after the time point, the operation display of the other user in the presenting unit is returned to the display according to the operation data transmitted from the device of the other user. .   The reaction control unit includes each frame in the prediction operation data for a certain period of time that will be displayed by the control of the prediction operation unit after the time when the response information is received, and other frames in a certain period after the time point. By comparing each frame in the motion data received from the user's device, the closest frames are identified, and the identified frame is used as a connection point when controlling to replace the frame. The remote communication device according to claim 5, wherein:   The reaction control unit may perform motion blending of the motion data predicted for a certain period and motion data received from another user's device for a certain period after the time point with the connection point as a center. Item 7. The remote communication device according to Item 6.   The reaction control unit identifies the time point when the posture is determined to be neutral in the data of the predicted motion that is to be displayed by the control of the prediction motion unit after the time point when the response information is received, 6. The remote communication device according to claim 5, wherein the operation display of the other user in the presenting unit is returned to the display according to the operation data transmitted from the other user's device.   The remote communication device according to claim 1, wherein the presenting unit displays an operation of the other user as an avatar according to the operation data transmitted from the device of the other user.   A program for causing a computer to function as the remote communication device according to any one of claims 1 to 9.