JP2013009111A - Communication system and communication apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a communication system for enhancing presence given to a user.SOLUTION: The communication system enables communication between a first user and a second user. The communication system includes: means for holding video data on a video obtained by imaging the second user by using a camera; a display for screen-displaying the video depending on the held video data for displaying it to the first user; a direction control mechanism turning a screen direction of the display to a direction corresponding to control; means for acquiring physical position information of the first user or the second user; and means for determining and controlling the new screen direction of the display in accordance with the physical position information of the first user or the second user.

Description

  The present invention relates to a communication system and a communication apparatus, and can be applied to, for example, a communication system that realizes remote communication by voice and video calls.

  The importance of remote communication is increasing, and a communication system that visually and audibly conveys the presence of a remote conversation person and the realism of the place to the remote place is drawing attention.

  As technologies related to conventional communication systems, there are technologies described in Patent Documents 1 to 4.

  The technology described in Patent Document 1 is a highly realistic communication device having a high visual and auditory sense of realism by displaying the expression of a remote conversation person on a dummy head simulating the shape of a human head. Is to provide. However, in order to provide a sense of realism, a large-scale device such as a dummy head must be prepared, and the realism cannot be easily produced. On the other hand, in Patent Document 2, the face of a remote conference attendee is projected on a video monitor almost in real size, giving a feeling of staring directly with the eyes, and the video monitor automatically or manually speaks. A device is described that can provide a sense that a remote conference attendee is turning his face.

  Further, in Patent Document 3, an information processing terminal that analyzes a sound source signal from an earphone microphone to identify the position of the earphone microphone, changes the orientation of the display of the mobile phone terminal based on the position information, and directly faces the earphone microphone. Is described. Furthermore, Patent Document 4 describes a short-range communication device that detects the approach of a human body and directs the display in the approaching direction.

JP 2005-269549 A Special table 2001-524286 JP 2003-169161 A JP 2003-289273 A

  However, the technology described in Patent Document 2, like the technology described in Patent Document 1, requires use of a large-scale device to provide a high sense of realism, and the moving rotation of the video monitor and the image from a remote location. These changes are not integrated.

  In addition, both of the technologies described in Patent Documents 3 and 4 are characterized in that the orientation of the display is changed in accordance with the user's operation, but the purpose is to reduce the user's labor. It is not intended to display any other talkers or to support remote communication. In the techniques described in Patent Documents 3 and 4, the rotation of the display and the change of the image from a remote place are not integrated.

  Therefore, a communication system and a communication device that can improve a sense of presence given to a user are desired.

  A first aspect of the present invention is a communication system that enables communication between a first user and a second user. (1) Using the one or a plurality of cameras, the second user is Video data holding means for holding video data of the captured video; (2) a display for displaying a video based on the video data held by the video data holding means for presentation to the first user; Image display means comprising a screen direction control mechanism for directing the screen direction of the display in a direction according to control; and (3) a body for acquiring physical position information of the first user or the second user. And (4) determining a new screen direction of the display according to the physical position information of the first user or the second user, and As a screen direction in which direction is determined, and having a screen direction control means for controlling the screen direction control mechanism.

  According to a second aspect of the present invention, in the first communication device used by the first user, constituting a communication system that enables communication between the first user and the second user ( 1) Video data acquisition means for acquiring video data obtained by imaging the second user using one or a plurality of cameras from the second communication device used by the second user; (2) A display for displaying the video of the video data acquired by the video data acquisition means to be presented to the first user, and a screen direction control mechanism for directing the screen direction of the display in a direction according to the control; Video display means, (3) physical position information acquisition means for acquiring physical position information of the first user or the second user, and (4) the first user A new screen direction of the display is determined according to the physical position information of the user or the second user, and the screen direction control mechanism is set so that the screen direction of the display becomes the determined screen direction. And a screen direction control means for controlling.

  ADVANTAGE OF THE INVENTION According to this invention, the communication system which improves the realistic feeling given to a user can be provided.

It is the block diagram shown about the functional structure of the communication apparatus (communication apparatus main body) which concerns on 1st Embodiment. It is the block diagram shown about the whole structure of the communication system which concerns on 1st Embodiment. 1 is a perspective view of a communication device according to a first embodiment. It is explanatory drawing shown about the user head direction which the video processing part which concerns on 1st Embodiment grasps | ascertains as user information. It is explanatory drawing shown about the process of the image | video synthetic | combination part and rotation control part which concern on 1st Embodiment. It is the flowchart shown about operation | movement of the communication apparatus which concerns on 1st Embodiment. It is explanatory drawing shown about the user gaze direction which the video processing part which concerns on 2nd Embodiment grasps | ascertains as user information. It is explanatory drawing shown about the process which the image | video synthetic | combination part and rotation control part which concern on 2nd Embodiment perform. It is the flowchart shown about operation | movement of the communication apparatus which concerns on 2nd Embodiment. It is explanatory drawing shown about the structure of the camera system which comprises the communication apparatus which concerns on 3rd Embodiment. It is the flowchart shown about operation | movement of the communication apparatus which concerns on 3rd Embodiment. It is the flowchart shown about operation | movement of the communication apparatus which concerns on 4th Embodiment. It is the block diagram shown about the functional structure of the communication apparatus (communication apparatus main body) which concerns on 5th Embodiment. It is the flowchart shown about operation | movement of the communication apparatus which concerns on 5th Embodiment.

(A) First Embodiment Hereinafter, a first embodiment of a communication system and a communication device according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of the First Embodiment FIG. 2 is a block diagram showing the overall configuration of the communication system 1 of this embodiment.

  As shown in FIG. 2, in the communication system 1, two communication devices 10 (10-1 and 10-2) are connected via a network N. The communication device 10 is a device that provides a user with a remote communication function via the network N such as a video conference function.

  In the following, it is assumed that the communication device 10-1 is used by the first user U1, and the communication device 10-2 is used by the second user U2.

  Hereinafter, in each communication device 10, the user of the own device is referred to as “own device user”, and the user of the communication device 10 facing through the network N is referred to as “remote side user”. And Further, in each communication device 10, the communication device 10 facing the own device via the network N is represented as a “remote device”. Here, in order to simplify the description, the communication device 10-1 and the communication device 10-2 are described as having the same configuration, but may be configured differently.

  Next, the configuration of the communication device 10 will be described.

  FIG. 3 is a perspective view of the communication device 10. As illustrated in FIG. 3, the communication device 10 includes a communication device body 11, a camera 12, and a display 13.

  The communication device body 11 has a function of controlling the communication device 10 as a whole. In the communication device 10, the camera 12 and the display 13 are controlled by the communication device body 11. The communication device main body 11 is realized, for example, by installing the communication program of the embodiment in an information processing device (for example, a PC, a dedicated device, or the like) including a processor that executes a program and an interface connected to a network. Even in such a case, the functional configuration of the communication device 10 can be illustrated as shown in FIG.

  The camera 12 includes a camera main body 121 for capturing an image of a user of the device itself, and a camera rotating plate 122 on which the camera main body 121 is placed.

  The camera body 121 supplies video data of a video imaged according to the control of the communication apparatus body 11 to the communication apparatus body 11. As the camera body 121, for example, a CCD camera used for an existing video conference terminal or the like can be applied.

  The camera rotating plate 122 is rotated and placed clockwise or counterclockwise in the left-right direction (horizontal direction) by a rotating mechanism (not shown, for example, a motor) that operates under the control of the communication apparatus body 11. The imaging direction of the camera body 121 is changed. In other words, in the communication device 10, the communication device body 11 controls the direction in which the camera body 121 captures an image by operating the above-described rotation mechanism included in the camera rotation plate 122. As the camera 12, for example, a camera (for example, a robot camera) that can change the imaging direction used in an existing video conference terminal or the like can be used. In the first embodiment, the means for controlling the imaging direction of the camera body 121 is not limited, and means other than the rotation mechanism described above (for example, a stand having a swing mechanism) is applied. May be. In FIG. 3, the physical interface between the camera 12 and the communication apparatus main body 11 is not limited, and an existing wired (for example, USB cable) or wireless interface can be applied.

  The display 13 includes a display main body 131 for displaying and outputting the video (video signal) output from the communication apparatus main body 11 to the first user U1, and a display rotating plate 132 on which the display main body 131 is placed. is doing.

  As the display body 131, for example, a liquid crystal display used in an existing PC or the like can be used.

  The display rotation plate 132 is rotated and placed in a horizontal direction (left and right direction) clockwise or counterclockwise by a rotation mechanism (not shown, for example, a motor) operated by control of the communication device main body 11. The screen direction of the display main body 131 is changed. In other words, in the communication device 10, the communication device main body 11 operates the above-described rotation mechanism included in the display rotation plate 132 to control the direction in which the screen of the display main body 131 is directed. In the first embodiment, means for controlling the screen direction of the display main body 131 is not limited, and means other than the rotation mechanism described above can be applied as in the camera 12. In FIG. 3, the physical interface between the rotation mechanism (not shown) of the display 13 and the communication apparatus main body 11 is not limited, and an existing wired (for example, USB cable) or wireless interface can be applied. it can. Further, the physical interface for the display main body 131 to receive the supply of the video signal from the communication apparatus main body 11 is not limited. For example, the same interface as that of a display such as an existing PC can be applied.

  Next, the functional configuration of the communication apparatus main body 11 will be described with reference to FIG.

  Functionally, the communication device main body 11 constituting the communication device 10 functionally includes a video acquisition unit 101, a video processing unit 102, an information transmission unit 103, an information reception unit 104, a video synthesis unit 105, a video display unit 106, and a rotation control unit. 107 and a control unit 108. In addition, although illustration is abbreviate | omitted in FIG. 1, the communication apparatus main body 11 capture | acquires the voice which the remote side user uttered by the microphone which captures the voice which the own apparatus user utters, or the remote side apparatus supplied. It is assumed that a configuration similar to that of an existing video conference terminal such as an output speaker is also provided.

  The control unit 108 controls the operation of the communication device 10 as a whole. For example, the control unit 108 exchanges control information with each component in the communication device 10 or calls other communication devices 10. It is responsible for functions such as connection. In addition, the control unit 108 also performs processing of sending a voice signal (voice data) of a voice uttered by the user of the own apparatus captured from the microphone (not shown) to the remote apparatus. Further, the control unit 108 also performs processing for outputting the voice of the voice signal (voice data) uttered by the remote user supplied from the remote apparatus to a speaker (not shown). In this embodiment, the communication device 10 is described as a terminal capable of communicating (calling) with a remote user using audio and video. However, the communication device 10 is adapted to communication using only video and audio is transmitted. The configuration to be processed may be omitted.

  The video acquisition unit 101 has a function of acquiring video data obtained by capturing the user of the device from the camera 12. In addition, it is assumed that the video acquisition unit 101 controls the shooting direction and shooting magnification of the camera 12 so as to follow the movement of the user of the apparatus. Here, as an example, it is assumed that the video acquisition unit 101 controls the direction of the camera 12 and the like so that the head of the user of the apparatus itself becomes the center of the captured video. A method for controlling the direction of the camera 12 and the like so that the video acquisition unit 101 continues to capture the user of the device itself is not limited, but a method similar to that for an existing video conference terminal or the like may be applied. it can. Then, the video acquisition unit 101 transmits video data (moving image data) taken by the camera 12 to the remote device in real time. The method in which the video acquisition unit 101 transmits the video data to the remote device in real time is not limited, and various methods used in existing video conference terminals and the like can be applied.

  The video processing unit 102 has a function of processing the video of the video data acquired by the video acquisition unit 101 (the video of the own device user). Specifically, the video processing unit 102 separates a video to be processed into a background video and a person video (video of the own device user) and user information of the own device user (details will be described later). ) Is calculated.

Here, the user information is, for example, non-linguistic information of a user who uses the communication device 10 and includes physical position information of the user (hereinafter also referred to as “physical position information”). It is assumed to be information. The user's physical position information included in the user information includes, for example, the position and orientation of the user's head, the direction of the line of sight, and the direction of the shoulder as seen from the rotation axis that rotates the display main body 131. . Here, as an example, a part of the user's head (for example, the center position between the user's eyes) viewed from the rotation axis of the display main body 131 (hereinafter, “user head direction θ _H ”). Information) is included in the user information. The method for the video processing unit 102 to grasp the user's head direction of the user's device user using the user's device user's video is not limited, but for example, Reference 1 (Japanese Patent Laid-Open No. 2008-102902). The technology described in Japanese Laid-Open Patent Publication No. 2001-228707 can be used. Reference 1 describes that the position and orientation of the user's head, the direction of the line of sight, the direction of the shoulder, and the like are grasped from the video of the user. In the following, the user head direction of the first user U1 who uses the communication device 10-1 is “user head direction θ _H1 ”, and the second user U2 who uses the communication device 10-2. The user head direction is expressed as “user head direction θ _H2 ”.

  FIG. 4 is an explanatory diagram showing an example of user information grasped by the video processing unit 102 constituting the communication device 10.

Here, as an example, a vertical plane S with respect to the screen display main body 131 is facing in the initial state is referred to as the direction of 0 °, about the rotation axis P _A of the display body 131, a positive a direction shifted clockwise It is assumed that the direction is expressed with a negative angle that is a direction shifted counterclockwise. That is, it is assumed that the screen of the display main body 131 is oriented at 0 degrees in the initial state. Thereafter, the screen direction of the display main body 131 changes according to the operation of the display rotating plate 132. Hereinafter, the current screen direction of the display main body 131 is represented as “screen direction θ _S ”. Hereinafter, the screen direction of the communication device 10-1 is represented as “screen direction θ _S1 ”, and the screen direction of the communication device 10-2 is represented as “screen direction θ _S2 ”.

And here, the image processing unit 102, a direction from the rotational axis P _A of the display body 131, to the position P _E of a predetermined portion of the user's head _(e.g., center position between the eyes of the user) It shall be understood as a use's head direction θ _H.

Then, the video processing unit 102 determines the amount of rotation of the display main body 131 (hereinafter referred to as “rotation amount θ _T ”) based on the user information (user head direction θ _H ) of the user of the apparatus. decide. Then, the image processing unit 102, the determined information amount of rotation theta _T, gives the rotation control unit 107. In the following, the rotation amount determined by the communication device 10-1 is represented as “rotation amount θ _T1 ”, and the rotation amount determined by the communication device 10-2 is represented as “rotation amount θ _T2 ”.

Here, as a general rule, the video processing unit 102 has a rotation amount for causing the screen of the display main body 131 to face the user of the apparatus itself (that is, to make the user head direction θ _H and the screen direction θ _S coincide). Assume that θ _T is calculated. Specifically, in order to match the user head direction θ _H and the screen direction θ _S , it is necessary to set the rotation amount θ _T = θ _H −θ _S. For example, in the state of FIG. 4, when θ _S = 0 degrees and θ _H = 45 degrees, in order to make the user head direction θ _H coincide with the screen direction θ _S , the rotation amount θ _T = 45 degrees-0 degrees = 45 degrees.

The video processing unit 102, the user head direction theta _H and screen direction theta _S and as the amount of rotation theta _T for matching, may be applied to the rotation controller 107, range screen direction theta _S of predetermined Alternatively, the rotation control unit 107 may be provided after conversion to keep within (for example, −45 degrees to 45 degrees). For example, when the screen direction θ _S is 40 degrees and the rotation amount θ _T for matching the user head direction θ _H and the screen direction θ _S is 10 degrees, the video processing unit 102 In order to keep the screen direction θ _S in the range of −45 degrees to 45 degrees, the rotation amount θ _T given to the rotation control unit 107 may be determined to be 5 degrees.

The rotation control unit 107 controls the rotation of the display main body 131 and the camera main body 121. Rotation control section 107, in accordance with the rotation amount theta _T supplied from the video processing unit 102, is rotated by controlling the motor of a display rotation plate 132 (not shown). That is, the rotation control unit 107 controls the rotation mechanism provided in the display rotation plate 132 (not shown) so that the screen direction θ _S = θ _S + θ _T of the display main body 131 is obtained.

The information transmission unit 103 transmits the user information (user head direction θ _H ) of the own device user acquired by the video acquisition unit 101 to the remote device via the network N.

The information receiving unit 104 acquires user information (user head direction θ _H ) of the remote user from the remote device via the network N.

  The video composition unit 105 performs processing (for example, composition processing) on the video data supplied from the remote device and supplies the processed data to the video display unit 106.

Here, the video composition unit 105 does not transform the person video of the remote user (hereinafter referred to as “remote person video”), and does not modify the background video of the remote person (hereinafter “remote background video”). It is assumed that only the screen direction θ _{S of the} device itself is distorted and synthesized (for example, by projective transformation).

  The process of dividing the video data received from the remote device into the remote background video and the remote person video is not limited, and an existing processing method can be applied. For example, the technique described in Reference 3 (Japanese Patent Laid-Open No. 2001-14474) can be applied to the process of extracting the remote person video from the video data received from the remote side. In Reference 3, a background image captured and stored in advance and a mixed image in which a background and a person are captured are compared using a background subtraction method to extract a person image. When the video composition unit 105 uses the technique described in Reference 3, the remote background video is stored in advance, and the stored remote background video and the video data (background and person received from the remote device) are stored. And a remote person image can be acquired. When the technique described in Reference 3 is used, what is stored in advance may be used without extracting the remote background video from the video data received from the remote device. The method in which the video composition unit 105 holds the remote-side background video in advance is not limited. For example, it may be registered in advance by the user, or a video in a state where the remote user is not imaged (that is, the remote background video) is captured in response to the operation of the remote user or the user of the device. And may be held.

As described above, the video composition unit 105 synthesizes the remote background video by distorting it in the direction opposite to the screen direction θ _{S of the} device itself. The method for distorting the background image on the remote side according to the screen direction θ _S (for example, projective transformation) is not limited, and an existing processing method can be used. For example, Reference 4 (Japanese Patent Laid-Open No. 2009-260819) The described technique may be used. Reference 4 describes a projective transformation of a rectangular image according to a projective transformation parameter. When the video composition unit 105 performs the projective transformation of the remote background video using the technique described in Reference 4, the projection transformation parameter corresponding to the screen direction θ _S is determined to perform the projective transformation. good. Note that Reference 4 describes a process of converting a trapezoidal image into a rectangle, but in this embodiment, it is necessary to perform the process as a process of converting a rectangle into a trapezoid. Then, the video composition unit 105 synthesizes the remote-side person video with the projective-converted remote-side background video and supplies it to the video display unit 106.

  Next, a specific example of composition processing performed by the video composition unit 105 will be described.

  FIG. 5 is an explanatory diagram showing processing performed by the video composition unit 105 and the rotation control unit 107 in the communication device 10-1.

FIG. 5A-1 shows an image synthesized by the video composition unit 105 when the screen direction θ _S1 is 0 degree (in the state of FIG. 5A-2). FIG. 5B-1 shows an image synthesized by the video composition unit 105 when the screen direction θ _S1 is a positive angle (for example, 45 degrees) (in the state of FIG. 5B-2). Yes. FIG. 5C-1 shows an image synthesized by the video composition unit 105 when the screen direction θ _S1 is a negative angle (for example, −45 degrees) (in the state of FIG. 5C-2). ing.

  5A-1, 5B-1, and 5C-1, areas A111, A121, and A131 indicate remote person images. In addition, areas A112, A122, and A132 indicate remote side background images.

As shown in FIG. 5 (a-1), when the screen direction θ _S1 is 0 degree, the video composition unit 105 synthesizes the remote background video as it is without performing projective conversion on the remote background video. Generate.

As shown in FIG. 5B-1, when the screen direction θ _S1 is a positive angle (for example, 45 degrees), the video composition unit 105 moves the screen in the direction rotated to the left (toward the screen). The background image of the remote side is distorted by projective transformation so that the end of) is the base of the trapezoid and the end of the opposite direction (right side toward the screen) is the top of the trapezoid. Further, as shown in FIG. 5C-1, when the screen direction θ _S1 is a negative angle (for example, −45 degrees), the video composition unit 105 rotates the screen (towards the screen). It is distorted by projective transformation so that the edge on the right side is the base of the trapezoid and the edge in the opposite direction (left side toward the screen) is the top of the trapezoid. Furthermore, the video composition unit 105 performs projective transformation on the remote-side background video so that the distortion of the remote-side background video increases (the difference between the bottom and top sides of the trapezoid increases) as the absolute value of the screen direction θ _S increases. To do.

  Further, the video composition unit 105 performs remote side background video projective transformation on the screen of the display main body 131 such that areas on which no video is displayed (for example, remote backgrounds such as the areas A121 and A123 in FIG. 5A-1). For example, a region where the image is missing may be complemented by filling with a predetermined single color pixel, a predetermined pattern, or a pixel having a color (hue) closest to the remote background image. You may make it complement.

(A-2) Operation of the First Embodiment Next, the operation of the communication system 1 of the first embodiment having the above configuration will be described.

  Below, it demonstrates centering around operation | movement of the communication apparatus 10-1 which the 1st user U1 uses.

  FIG. 6 is a flowchart showing the operation of the communication device 10 constituting the communication system 1.

  First, it is assumed that a video call is started between the communication device 10-1 and the communication device 10-2 (S100).

Next, using the video of the first user U1 acquired from the video acquisition unit 101 in the video processing unit 102 of the communication device 10-1, the user head as user information of the first user U1. The part direction θ _H1 is calculated (S101).

Next, the video processing unit 102 determines the screen direction θ _S1 and the rotation amount θ _T1 based on the user head direction θ _H1 calculated in step S101 described above. Then, the information on the determined rotation amount θ _T1 is given from the video processing unit 102 to the rotation control unit 107. Then, the rotation control unit 107, in accordance with a given amount of rotation theta _T1, rotating operating a display body 131 (display rotating plate 132) (S102).

  Then, the video composition unit 105 acquires the video data from the communication device 10-2 (S103), performs a synthesis process on the video of the video data, and gives the video data to the video display unit 106 (S104).

In step S104, the remote person video is extracted from the video data of the video data from the communication device 10-2 by the video compositing unit 105, and the latest screen direction θ _{S1 is} stored for the remote background video that has been held in advance. Projective transformation is performed in accordance with. Then, the video composition unit 105 synthesizes the extracted remote person video and the remote background image after projective transformation, and provides the composited video data to the video display unit 106.

  Then, the video display unit 106 displays the video data provided from the video synthesis unit 105 on the display main body 131 (S105). The processes in steps S103 to S105 are then repeated until the user information is updated.

  And in the control part 108 of the communication apparatus 10-1, whether the communication with the communication apparatus 10-2 is continuing, ie, the line between the communication apparatuses 10-1, 10-2 is connected. Is confirmed (S106). If it is determined in step S106 that communication with the communication device 10-2 is continuing (when the line is connected), the control unit 108 returns to the process in step S101 described above and operates. Each component is controlled as follows. On the other hand, when it is determined in step S106 that the communication with the communication device 10-2 is continuing (when the line is disconnected), the control unit 108 ends the above-described series of processing ( S107).

(A-3) Effects of First Embodiment According to the first embodiment, the following effects can be achieved.

In the communication device 10 of the first embodiment, the screen direction of the display main body 131 of the own device is made to follow the user information (user head direction θ _H ) of the own device user. Specifically, the communication device 10 controls the screen direction of the display main body 131 so that the user of the device itself faces the screen of the display main body 131. Further, in the communication device 10 according to the first embodiment, the remote background image to be displayed on the screen of the display main body 131 according to the screen direction θ _S of the display main body 131 of the own device is displayed. (Projection transformation process) to distort in the opposite direction. Thereby, in the communication apparatus 10, a remote image | video can be shown to a self-device user as a pseudo-stereoscopic image, the presence of the other party (remote user) is shown, and a realistic feeling is heightened. Can do.

  In addition, the communication device 10 of the first embodiment can be constructed at a low cost because it can be built simply by providing a mechanism that can control the screen direction of the display in a conventional video conference terminal or the like. Is possible.

(B) Second Embodiment Hereinafter, a second embodiment of a communication system and a communication device according to the present invention will be described in detail with reference to the drawings.

(B-1) Configuration of Second Embodiment The configuration of the communication system 1A of the second embodiment can also be shown in FIG. 2 described above. Hereinafter, the second embodiment will be described focusing on the differences from the first embodiment.

  In the communication system 1A according to the second embodiment, the communication device 10 (10-1, 10-2) is replaced with the communication device 10A (10A-1, 10A-2). Is different.

  The communication device 10A of the second embodiment can also be shown by the above-described FIGS.

  The communication device main body 11A constituting the communication device 10A of the second embodiment functionally includes a video acquisition unit 101, a video processing unit 102A, an information transmission unit 103, an information reception unit 104, a video synthesis unit 105, and a video display. Unit 106, rotation control unit 107A, and control unit 108. Functionally, in the communication apparatus main body 11A of the second embodiment, since the configuration other than the video processing unit 102A and the rotation control unit 107A is the same as that of the first embodiment, detailed description thereof is omitted.

The video processing unit 102A grasps not the user head direction θ _H as the user information, but the direction of the user's line of sight (hereinafter referred to as “user line-of-sight direction θ _E ”). This is different from the first embodiment.

7, the image processing unit 102A is an explanatory diagram showing a user line-of-sight direction theta _E grasped as user information.

As shown in FIG. 7, the user's line-of-sight direction θ _E is changed from the position P _E (for example, the center position between the user's eyes) of the user U's head to the direction of the line of sight of the user U. extending the line L _E is the direction indicated. Then, the direction of 0 degrees the user viewing direction theta _E when the vertical plane S with respect to the screen display main body 131 is facing in the initial state (the screen direction theta _S is 0 degrees), and a line L _E becomes parallel It shall be expressed as And, in other cases (if a vertical plane S, and a line L _E not parallel), the angle between L _E and the vertical plane S, the user viewing direction theta _E shall be represented . As described above, the user head direction θ _H and the screen direction θ _S are expressed as directions viewed from the rotation axis of the display main body 131, but the user line-of-sight direction θ _E is from the user U. It shall be expressed as the viewing direction. The user's line-of-sight direction θ _E represents a direction as viewed from the user U, with the direction deviating counterclockwise from the 0 degree direction as a positive angle and the direction deviating clockwise as a negative angle. To do. In the case of the example in FIG. 7, the user U's line of sight is deviated from the 0 degree direction in the counterclockwise direction, and thus is represented as a positive angle.

In the following, the user gaze direction of the first user U1 who uses the communication device 10A-1 is represented as “user gaze direction θ _E1 ”, and the second user U2 who uses the communication device 10A-2. The user gaze direction is expressed as “user gaze direction θ _E2 ”.

The rotation control unit 107 of the first embodiment determines the direction (screen direction θ _S ) in which the display main body 131 of the own device is directed based on the user head direction θ _H of the own device user. On the other hand, the rotation control unit 107A of the second embodiment determines the screen direction θ _S of the own device based on the user gaze direction θ _E of the remote user.

For example, in the communication device 10A-1, when the user information (user viewing direction θ _E2 ) of the second user U2 is supplied from the communication device 10A-2, the rotation control unit 107A causes the user viewing direction. A rotation amount θ _T1 based on θ _E2 is calculated. Specifically, the rotation control unit 107A of the communication device 10A-1 has a rotation amount for causing the screen direction θ _S1 of the display main body 131 of the own device to coincide with the user line-of-sight direction θ _E2 in which the sign is reversed. _Is calculated as the rotation amount θ _T1 . That is, in this case, it is necessary to set the rotation amount θ _T1 = −θ _E2 −θ _S1 . Then, the rotation control unit 107A, in accordance with the calculated amount of rotation theta _T1, rotates the display body 131 of the apparatus.

In the second embodiment, as in the first embodiment, the video composition unit 105 does not transform the remote person video, but only the remote background video in the direction opposite to the screen direction θ _S in its own device. It is assumed that the image is synthesized by distortion (for example, it is synthesized by projective transformation).

  FIG. 8 is an explanatory diagram showing processing performed by the video composition unit 105 and the rotation control unit 107A in the communication device 10A-1.

FIG. 8A-2 shows the case where the user gaze direction θ _{E2 of} the second user U2 grasped by the communication device 10-2 is 0 degree (in the case of the state of FIG. 8A-1). An image synthesized by the image synthesis unit 105 is shown. FIG. 8B-2 shows an image synthesized by the image synthesizing unit 105 when the user's line-of-sight direction θ _E2 is a negative angle (for example, −45 degrees) (in the state of FIG. 8B-1). Shows about. FIG. 8C-2 shows an image synthesized by the video composition unit 105 when the user's line-of-sight direction θ _E2 is a positive angle (for example, 45 degrees) (in the state of FIG. 8C-1). Show. In FIG. 8A-2, FIG. 8B-2, and FIG. 8C-2, areas A211, A221, and A231 indicate remote person images. In addition, areas A212, A222, and A232 indicate remote side background images.

FIG. 8A-3 shows the direction in which the rotation control unit 107A directs the screen of the display main body 131 when the user's line-of-sight direction θ _E2 is 0 degree (in the state of FIG. 8A-1). ing. FIG. 8B-3 shows the case where the rotation control unit 107A is displayed on the display body when the user's line-of-sight direction θ _E2 is a negative angle (for example, in the state of FIG. 8B-1). The direction in which the screen 131 is directed is shown. FIG. 8C-3 shows the case where the rotation control unit 107A displays the display main body 131 when the user's line-of-sight direction θ _E2 is a positive angle (for example, in the state of FIG. 8C-1). The direction in which the screen is directed is shown.

As shown in FIG. 8A-3, when the user's line-of-sight direction θ _E2 is 0 degree, the rotation control unit 107A rotates the rotation amount θ _T1 so that the screen of the display main body 131 is directed to the 0 degree direction. Determine and control. Also, as shown in FIG. 8B-3, when the user's line-of-sight direction θ _E2 is −45 degrees, the rotation control unit 107A displays the screen of the display main body 131 with the screen direction θ _S1 = −θ _S2. The rotation amount θ _T1 is determined and controlled so as to be +45 degrees. Further, as shown in FIG. 8C-3, when the user's line-of-sight direction θ _E2 is +45 degrees, the rotation control unit 107A displays the screen of the display main body 131 with the screen direction θ _S1 = −θ _S2 = The rotation amount θ _T1 is determined and controlled so as to be −45 degrees.

(B-2) Operation of Second Embodiment Next, the operation of the communication system 1A of the second embodiment having the above configuration will be described.

  Hereinafter, the operation of the communication device 10A-1 used by the first user U1 will be mainly described.

  FIG. 9 is a flowchart showing the operation of the communication device 10A constituting the communication system 1A.

  First, it is assumed that a video call is started between the communication device 10A-1 and the communication device 10A-2 (S200).

Then, by using the video of the second user U2 acquired from the video acquisition unit 101 by the video processing unit 102A of the communication device 10A-2, the user information of the second user U2 (user gaze direction θ _E2 ) is calculated. Then, it is assumed that the user information (user gaze direction θ _E2 ) calculated by the communication device 10A-2 is supplied to the communication device 10A-1 (S201).

In the communication device 10A-1, the rotation control unit 107A determines the screen direction θ _S1 and the rotation amount θ _T1 based on the user line-of-sight direction θ _E2 supplied in step S201 described above. Then, the rotation control unit 107A, according to the determined amount of rotation theta _T1, the display body 131 (display rotating plate 132) is controlled to operate rotation (S202).

  Then, the video composition unit 105 acquires the video data from the communication device 10A-2, and further performs a composition process on the video of the video data (S203).

In step S203, the video composition unit 105 extracts a remote person video image from the video data from the communication device 10A-2, and further displays the latest screen direction θ _S1 for the remote background video image stored in advance. Projective transformation is performed in accordance with. Then, the video composition unit 105 synthesizes the extracted remote person video and the remote background image after projective transformation, and provides the composited video data to the video display unit 106.

  Then, the video display unit 106 displays the video of the video data provided from the video synthesis unit 105 on the screen of the display main body 131 (S204). The processes in steps S203 and S204 are then repeated until the user information of the remote user is updated.

  Then, in the control unit 108 of the communication device 10A-1, whether or not communication with the communication device 10A-2 is continued, that is, whether or not the lines between the communication devices 10A-1 and 10A-2 are connected. Is confirmed (S205). When it is determined in step S205 that communication with the communication device 10A-2 is continuing (when the line is connected), the control unit 108 returns to the processing in step S201 described above and operates. Each component is controlled as follows. On the other hand, when it is determined in step S205 that communication with the communication device 10A-2 is continuing (when the line is disconnected), the control unit 108 ends the above-described series of processing ( S206).

(B-3) Effects of Second Embodiment According to the second embodiment, the following effects can be achieved.

In the communication apparatus 10A of the second embodiment, the screen direction of the display main body 131 of the own apparatus is made to follow the user information (user line-of-sight direction θ _E ) of the remote user. Specifically, for example, in the communication device 10A, when the remote user rotates the head direction to the right, the screen direction of the display main body 131 of the own device is rotated to the right correspondingly. I am letting. Further, in the communication device 10A of the second embodiment, the remote-side background image displayed on the screen of the display main body 131 according to the screen direction θ _S of the display main body 131 of the own device is displayed. (Projection transformation process) to distort in the opposite direction. Thereby, in the communication apparatus 10A, it is possible to make it appear to the user of the apparatus as if the remote person image is stationary. Thus, in the communication device 10A, it is possible to express to the user device user a trivial operation of the remote user by emphasizing it as a rotation of the screen of the display main body 131. The presence of the side user can be transmitted.

  In addition, the communication device 10A of the second embodiment can be constructed at a low cost because it can be constructed simply by providing a mechanism that can control the screen direction of a display in a conventional video conference terminal or the like. Is possible.

(C) Third Embodiment Hereinafter, a third embodiment of a communication system and a communication device according to the present invention will be described in detail with reference to the drawings.

(C-1) Configuration of Third Embodiment The configuration of the communication system 1B of the third embodiment can also be shown in FIG. 2 described above. Hereinafter, the third embodiment will be described with a focus on differences from the first embodiment.

  The communication system 1B of the third embodiment is different from the first embodiment in that the communication device 10 (10-1, 10-2) is replaced with the communication device 10B (10B-1, 10B-2). Is different.

  The hardware configuration of the third communication device 10B is different from that of the first embodiment in that the camera 12 is replaced with a camera system 14 (details will be described later) in the configuration shown in FIG. . On the other hand, the functional configuration of the communication device main body 11B constituting the communication device 10B of the third embodiment can be shown in FIG. 1 described above.

  In the third embodiment, the communication device main body 11B constituting the communication device 10B functionally includes a video acquisition unit 101B, a video processing unit 102, an information transmission unit 103, an information reception unit 104, a video synthesis unit 105B, and a video. A display unit 106, a rotation control unit 107, and a control unit 108 are provided. Functionally, in the communication apparatus main body 11B of the third embodiment, since the configuration other than the video acquisition unit 101B and the video synthesis unit 105B is the same as that of the first embodiment, detailed description thereof is omitted.

  Next, a camera system 14 including a plurality of cameras 15 in the communication device 10B of the third embodiment will be described.

  FIG. 10 is an explanatory diagram showing the configuration of the camera system 14 included in the communication device 10B of the third embodiment. FIG. 10 shows a configuration in which the positional relationship among the plurality of cameras 15, the user U, and the display main body 131 constituting the camera system 14 is viewed from above.

  As shown in FIG. 10, the camera system 14 includes three cameras 15 (15-1 to 15-3). As shown in FIG. 10, each camera 15 (15-1 to 15-3) includes a camera main body 151 (151-1 to 151-3) and a camera rotation plate 152 (152-1 to 152-3). ). Each camera 15 can be the same as the camera 12 of the first embodiment. And the camera 15-1 shall be arrange | positioned in the same position as the camera 12 of 1st Embodiment. The camera 15-2 and the camera 15-3 are arranged so that the user U can be imaged from a direction different from the camera 15-1 when viewed from the user U.

  In FIG. 10, when the user U is located at a predetermined reference position in the initial state, the camera 15-1 is viewed from the user U in the front direction of the user U (the same direction as the rotation axis of the display main body 131). The camera 15-2 is arranged on the left side when viewed from the user, and the camera 15-3 is arranged on the right side when viewed from the user. In the camera system 14, the number of cameras 15 to be arranged and the positions to be arranged (imaging direction) are not limited. Then, the camera system 14 supplies the video data captured by each camera 15 to the video acquisition unit 101B.

  The video acquisition unit 101B transmits the video data supplied from each camera 15 to the remote device in real time.

In the communication device 10 of the first embodiment, there is one type of video data supplied from the remote side device to the own device. However, in the communication device 10B of the third embodiment, as described above, there are three types of video data. Data is supplied. Therefore, the video composition unit 105B of the third embodiment needs to select video data to be processed. Here, the image combining unit 105B in response to the user head direction theta _H of the apparatus user shall select the video data. For example, when the user head direction θ _H of the user's own device is 90 degrees, the user's own device has moved to the right as viewed from the display main body 131. At this time, by displaying an image captured by the camera 15-3 that can capture a remote user from the right side on the display main body 131, the user of the device itself is as if the remote user exists on the spot. Can give a sense of realism.

Here, it is assumed that information on the user head direction θ _H corresponding to each camera 15 is registered in the video composition unit 105B. Here, it is assumed that the video composition unit 105B selects the video data of the camera 15-1 as a processing target when the user head direction θ _H is within a range of −45 degrees to +45. The video synthesizing unit 105B is assumed to select the video data of the camera 15-3 when the user head direction theta _H is within the range of +45 degrees to +90 for processing. Further, the video compositing unit 105B is assumed to select the video data of the camera 15-2 as the processing target when the user head direction theta _H is within the range of -45 ° to -90.

  In this case, the video composition unit 105B supplies the selected video data to the video display unit 106 as it is without processing.

(C-2) Operation of the Third Embodiment Next, the operation of the communication system 1B of the third embodiment having the above configuration will be described.

  Hereinafter, the operation of the communication device 10B-1 used by the first user U1 will be mainly described.

  FIG. 11 is a flowchart showing the operation of the communication device 10B constituting the communication system 1B.

  Below, the case where communication apparatus 10B-1 operate | moves according to a motion of the 1st user U1 to utilize is demonstrated using FIG.

  First, it is assumed that a video call is started between the communication device 10B-1 and the communication device 10B-2 (S300).

Next, the video processing unit 102B of the communication device 10B-1 uses the video of the first user U1 acquired from the video acquisition unit 101B, and uses the user's head as the user information of the first user U1. The part direction θ _H1 is calculated (S301).

Next, the video processing unit 102B determines the screen direction θ _S1 and the rotation amount θ _T1 based on the user head direction θ _H1 calculated in step S301 described above. Then, the information of the determined rotation amount θ _T1 is given from the video processing unit 102B to the rotation control unit 107. Then, the rotation control unit 107, in accordance with a given amount of rotation theta _T1, rotating operating a display body 131 (display rotating plate 132) (S302).

  Then, the video composition unit 105B acquires video data (a plurality of types of video data) from the remote device (communication device 10B-2) (S303).

In the video composition unit 105B, based on the user information (user head direction θ _H1 ) of the user of the own device, one to be processed from a plurality of types of video data supplied from the remote device is 1 Are selected and given to the video display unit 106 (S304).

  Then, the video display unit 106 displays the video of the video data provided from the video synthesis unit 105B on the display main body 131 (S305). Note that the processes in steps S303 to S305 are repeated until the user information is updated next time.

  Then, whether or not communication with communication device 10B-2 is continued in control unit 108 of communication device 10B-1, that is, whether or not the line between communication devices 10B-1 and 10B-2 is connected. Is confirmed (S306). If it is determined in step S306 that communication with the communication device 10B-2 is continuing (when the line is connected), the control unit 108 returns to the processing in step S301 described above and operates. Each component is controlled as follows. On the other hand, when it is determined in step S306 that communication with the communication device 10B-2 is continuing (when the line is disconnected), the control unit 108 ends the above-described series of processing ( S307).

(C-3) Effects of Third Embodiment According to the third embodiment, the following effects can be achieved.

In the communication apparatus 10B of the third embodiment, the screen direction of the display main body 131 of the own apparatus is made to follow the user information (user head direction θ _H ) of the own apparatus user. Specifically, in the communication device 10B, the screen direction of the display main body 131 is controlled so that the user of the own device faces the screen of the display main body 131 directly.

Further, in the communication device 10B of the third embodiment, a remote-side video (remote-side user) displayed on the screen of the display main body 131 according to the user information (user head direction θ _H ) of the user of the own device. The angle at which the image is captured is switched. Thereby, in the communication apparatus 10B, it can be shown to the own apparatus user that the remote user is following the operation | movement of self. As a result, in the communication device 10B, it is possible to give a sense that the remote device can be looked around from various angles to the user of the device, and the presence of the partner (remote device) is enhanced. be able to.

  In addition, the communication device 10B according to the third embodiment can be constructed simply by providing a mechanism capable of controlling the screen direction of a display or a plurality of cameras in a conventional video conference terminal or the like in terms of hardware. Can be realized at low cost.

(D) Fourth Embodiment Hereinafter, a communication system and a communication device according to a fourth embodiment of the present invention will be described in detail with reference to the drawings.

(D-1) Configuration of Fourth Embodiment The configuration of the communication system 1C of the fourth embodiment can also be shown in FIG. 2 described above. Hereinafter, the fourth embodiment will be described focusing on differences from the third embodiment.

  In the communication system 1C of the fourth embodiment, the communication device 10B (10B-1, 10B-2) is replaced with the communication device 10C (10C-1, 10C-2). Is different.

  The hardware configuration of the fourth communication device 10C is the same as that of the third embodiment. That is, the fourth communication device 10C includes the camera system 14 as in the third embodiment. Further, the functional configuration of the communication device main body 11C constituting the communication device 10C of the fourth embodiment can be shown in FIG. 1 described above.

  The communication device main body 11C constituting the communication device 10C of the fourth embodiment functionally includes a video acquisition unit 101B, a video processing unit 102C, an information transmission unit 103, an information reception unit 104, a video synthesis unit 105C, and a video display. Unit 106, rotation control unit 107C, and control unit 108. The communication device main body 11C of the fourth embodiment is functionally the same as the third embodiment except for the video processing unit 102C, the video synthesis unit 105C, and the rotation control unit 107C. Omitted.

The video processing unit 102C is different from the third embodiment in that the user information is not the user head direction θ _H but the user line-of-sight direction θ _E as user information.

In a third embodiment, the video processing unit 102B is based on the use's head direction theta _H of the apparatus user, a plurality of kinds of video data supplied to the own device from the remote side apparatus, one type of Video data was selected. In contrast, in the image processing unit 102C of the fourth embodiment, in response to user viewing direction theta _E of the remote side user shall select the video data. Here, the image combining unit 105C, based on user viewing direction theta _E, assumed to select the video data taken from possible front face of the remote-side user.

As shown in FIG. 10, three cameras 15-1 to 15-3 are arranged in the camera system 14 of the remote device. Then, the information of the remote-side user of the user viewing direction theta _E corresponding to each camera 15 is assumed to have been held by the video synthesis section 105C. For example, the image combining unit 105C is assumed to select the video data of the camera 15-1 as the processing target when the user viewing direction theta _E in the range of -45 degrees to +45. The video composition unit 105C selects the video data of the camera 15-2 as a processing target when the user's line-of-sight direction θ _E is in the range of +45 degrees to +90. Further, the image combining unit 105C is assumed to select the video data of the camera 15-3 as the processing target when the user viewing direction theta _E in the range of -45 ° to -90.

  Here, the video composition unit 105C supplies the selected video data to the video display unit 106 as it is without processing.

As in the first embodiment, the rotation control unit 107 according to the third embodiment is configured to direct the display main body 131 of the own device based on the user head direction θ _H of the own device user (screen direction θ _S ) was determined. In contrast, in the fourth embodiment of the rotation control section 107C, similar to the rotation control section 107A of the second embodiment, based on user viewing direction theta _E of the remote side user, own device screen The direction θ _S shall be determined. That is, the rotation control unit 107C performs the same processing as the rotation control unit 107A of the second embodiment, and detailed description thereof is omitted here.

(D-2) Operation of the Fourth Embodiment Next, the operation of the communication system 1C of the fourth embodiment having the above configuration will be described.

  Hereinafter, the operation of the communication device 10C-1 used by the first user U1 will be mainly described.

  FIG. 12 is a flowchart showing the operation of the communication device 10C constituting the communication system 1C.

  First, it is assumed that a video call is started between the communication device 10C-1 and the communication device 10C-2 (S400).

Then, by using the video of the second user U2 acquired by the video acquisition unit 101 by the video processing unit 102 of the communication device 10C-2, user information regarding the state of the second user U2 (user gaze) The direction θ _E2 ) is calculated. Then, it is assumed that the user information (user line-of-sight direction θ _E2 ) calculated by the communication device 10C-2 is supplied to the communication device 10C-1 (S401).

In the communication device 10-1, the rotation control unit 107C determines the screen direction θ _S1 and the rotation amount θ _T1 based on the user line-of-sight direction θ _E2 supplied in step S401 described above. Then, the rotation control unit 107C, according to the determined amount of rotation theta _T1, the display body 131 (display rotating plate 132) is controlled to operate rotation (S402).

Then, in the video composition unit 105C, based on the user's line-of-sight direction θ _E2 of the remote user, one to be processed is selected from a plurality of types of video data supplied from the remote device, and the video display is performed. Is given to the unit 106 (S403).

  Then, the video display unit 106 displays the video of the video data provided from the video composition unit 105C on the display main body 131 (S404). Note that the processes in steps S403 and S404 are repeated until the user information is updated next time.

  Then, in the control unit 108 of the communication device 10C-1, whether or not communication with the communication device 10C-2 is continued, that is, whether or not the line between the communication devices 10C-1 and 10C-2 is connected. Is confirmed (S405). If it is determined in step S405 that communication with the communication device 10C-2 is continuing (when the line is connected), the control unit 108 returns to the process of step S401 described above and operates. Thus, each component is controlled. On the other hand, when it is determined in step S405 that communication with the communication device 10C-2 is continued (when the line is disconnected), the control unit 108 ends the above-described series of processing ( S406).

(D-3) Effects of the fourth embodiment According to the fourth embodiment, the following effects can be achieved.

In the communication device 10 </ b> C of the fourth embodiment, the screen direction of the display main body 131 of the own device is made to follow the user information (user gaze direction θ _E ) of the remote user, as in the second embodiment. ing. Further, in the communication device 10C of the fourth embodiment, the remote-side video (remote-side user is displayed on the screen of the display main body 131 in accordance with the user information (user gaze direction θ _E ) of the user of the own device. The angle of imaging) is switched. For example, in the communication device 10C, video data of a video in which the remote user is facing the front as much as possible is selected from video data obtained by photographing the remote user from a plurality of angles and displayed on the screen of the display main body 131. is doing. Thereby, in the communication apparatus 10C, it is possible to always present the remote user photographed from the front direction to the own apparatus user. As a result, the communication device 10C can transmit the presence as in the second embodiment to the user of the device itself, and at the same time, can simultaneously show the facial expression of the remote user's face, making the communication more realistic. Can be provided.

  In addition, the communication device 10C of the fourth embodiment can be constructed simply by providing a mechanism capable of controlling the screen direction of a display or a plurality of cameras in a conventional video conference terminal or the like in terms of hardware. Can be realized at low cost.

(E) Fifth Embodiment Hereinafter, a fifth embodiment of a communication system and a communication device according to the present invention will be described in detail with reference to the drawings.

(E-1) Configuration of Fifth Embodiment The configuration of the communication system 1D of the fifth embodiment can also be shown in FIG.

  The communication system 1D of the fifth embodiment includes a communication device 10D (10D-1, 10D-2).

  The communication device 10D according to the fifth embodiment includes a configuration that realizes the processing according to any one of the first to fourth embodiments by switching the operation mode. That is, the communication device 10D of the fifth embodiment has an operation mode (hereinafter referred to as “mode M1”) capable of performing the same operation as that of the first embodiment, and an operation similar to that of the second embodiment. An operation mode (hereinafter referred to as “mode M2”), an operation mode capable of performing the same operation as in the third embodiment (hereinafter referred to as “mode M3”), An operation mode (hereinafter referred to as “mode M4”) capable of performing the same operation as in the embodiment is provided, and the operation mode can be switched.

  Hereinafter, a difference between the fifth embodiment and the first to fourth embodiments will be described.

  The communication device 10D is assumed to have the same configuration as the communication devices 10B and 10C of the third and fourth embodiments in terms of hardware. Here, the communication device 10D includes the camera system 14 as in the third and fourth embodiments. When operating in the mode M1 and the mode M2, one camera 15 constituting the camera system 14 is provided. (Here, it is assumed that it is the camera 15-1) is assumed to operate as the camera 12 in the first and second embodiments.

  FIG. 13 is an explanatory diagram showing a functional configuration of a communication device main body 11D constituting the communication device 10D.

  The communication device main body 11D constituting the communication device 10D functionally includes a video acquisition unit 101D, a video processing unit 102D, an information transmission unit 103, an information reception unit 104, a video synthesis unit 105D, a video display unit 106, and a rotation control unit. 107D, the control part 108, and the communication estimation part 109 are included. Since the information transmission unit 103, the information reception unit 104, the video display unit 106, and the control unit 108 can be the same as those in the first to fourth embodiments, detailed description thereof is omitted.

  The communication estimation unit 109 selects the operation mode of the own device according to the state of communication between the own device user and the remote user. Then, the communication estimation unit 109 performs a process of controlling each component of the own device so as to operate in the selected operation mode. In this embodiment, the communication device 10D is described as corresponding to the modes M1 to M4 as described above, but the number and combination of the corresponding operation modes are not limited. For example, in the communication device 10D, only a part of the operation modes among the modes M1 to M4 may be supported, or another operation mode may be added.

  Then, the video acquisition unit 101D, the video processing unit 102D, the video synthesis unit 105D, and the rotation control unit 107D perform processing according to the operation mode selected by the communication estimation unit 109. For example, when the mode M1 is selected by the communication estimation unit 109, the video acquisition unit 101D, the video processing unit 102D, the video synthesis unit 105D, and the rotation control unit 107D perform the same processing as in the first embodiment ( For example, the process of the flowchart shown in FIG. 6 is performed.

  The operation mode to be initially selected by the communication estimation unit 109 is not limited. For example, the operation may be started in a preset operation mode and then switched to another operation mode.

  Further, the opportunity for the communication estimation unit 109 to switch the operation mode is not limited. For example, the communication estimation unit 109 may switch the operation mode based on an operation (for example, an operation using a keyboard or a touch panel (not shown)) of the user of the apparatus or the remote user. good. Further, the communication estimation unit 109 estimates the communication status between the user of the apparatus and the remote user based on, for example, user information or voice spoken by the user, and based on the estimation result. Thus, the operation mode of the device itself may be switched. Specific examples include the following methods.

[Switching according to communication time]
As a method for the communication estimation unit 109 to select the operation mode of the own device, for example, communication time can be cited.

  For example, the communication estimator 109 immediately starts a conversation between the user of the own device and the remote user (for example, until a certain time elapses after the call connection with the remote device is started). ) May be set to mode M1. Immediately after the conversation between the user of the device and the remote user starts, it is necessary to recognize each other's existence. Therefore, depending on the operation of the partner (remote user), the screen direction and This is because it can be estimated that the remote background video conversion should not fluctuate. When a certain time has elapsed since the conversation between the user of the apparatus and the remote user has started (for example, when a certain time has elapsed since the call was connected), the communication estimation unit 109 The operation mode of the device itself may be set to mode M2. When a certain amount of time has passed since the conversation between the user of the device and the remote user has started, the conversation is on and the reaction of the other party (remote user) is particularly important for communication. Therefore, it can be estimated that it is better to change the screen direction and the remote side background video according to the operation of the other party.

[Switching according to change of speaker]
As a method for the communication estimation unit 109 to select the operation mode of the own device, for example, a switching method according to the change of the speaker is cited.

  Specifically, the communication estimation unit 109 may perform mode switching according to the change of the speaker and the listener. For example, the communication estimation unit 109 holds and holds the voice signal of the own device user captured by the microphone included in the own device and the voice signal of the remote user supplied from the remote communication device 10B. Based on one audio signal, it may be determined which user the current speaker is. The method for determining which user is a speaker by the communication estimation unit 109 is not limited. For example, when any user continues to speak for a certain period of time (for example, for a certain period of time). As described above, when the power of the audio signal is equal to or higher than the threshold value, the user may be determined as a speaker and the other as a listener. As another method for the communication estimation unit 109 to determine which user is a speaker, other speaker switching as described in Reference 5 (Japanese Patent Laid-Open No. 2010-32792) is detected. By doing so, the change of the speaker may be recognized, or the number of nods may be measured and determined.

  Then, the communication estimation unit 109 may set the mode M2 when it is determined that the current speaker is the user of the apparatus. This is because if the user of the device is a speaker, the user (remote user) will want to know the reaction of the other person (the remote user) to what he is talking about. This is because it can be estimated that it is better to control the remote background video conversion. Further, the communication estimation unit 109 may set the mode M1 when it is determined that the current listener is the user of the own apparatus. This is because when the user of the device is the listener, it can be estimated that a feeling that the other party is in front of the eyes is desired.

[Switching according to the user's physical location information]
As a method for the communication estimation unit 109 to select the operation mode of the own device, for example, switching of the mode according to the physical position information of the remote user or the user of the own device can be mentioned. Here, the physical position information includes, for example, parameters included in the user information such as the user head direction θ _H and the user line-of-sight direction θ _E. The communication estimation unit 109 switches the mode when, for example, the user information (user head direction θ _H or user gaze direction θ _E ) of the remote user or the character device user fluctuates beyond a predetermined range. May be performed.

For example, when the user's head direction θ _H of the remote user is changing at an angle within a certain range (for example, within a range of −45 degrees to 45 degrees) when viewed from the front of the display main body 131. Alternatively, the communication estimation unit 109 may select the mode M1 or the mode M2. Further, when the user head direction θ _H of the remote user becomes an angle outside the above range, the communication estimation unit 109 may select the mode M3 or the mode M4. This is because it can be estimated that there is an object of interest in the side direction of the remote user when the user's head direction θ _H of the remote user deviates from the angle within the above-mentioned fixed range.

[Switching corresponding to video from remote locations]
As a method for the communication estimation unit 109 to select the operation mode of its own device, for example, an object other than the remote background image and the remote person image is included in the video data image supplied from the remote device. The mode can be switched according to the case.

  For example, when a remote user presents an object that attracts the interest of the user of the device, the user of the device can estimate that he or she wants to see the object (for example, a transaction product) from various angles. . At this time, the communication estimation unit 109 switches the operation mode of the own device to the mode M3, so that the user of the own device can see the object while leaning on the object from various angles. In order for the communication estimation unit 109 to determine an object other than the remote background and the remote person image based on the video data supplied from the remote device, for example, an object tracking method such as pattern matching is conceivable. Specifically, when the remote user presents an object with a predetermined pattern (for example, a product with a two-dimensional barcode), the communication estimation unit 109 displays the video supplied from the remote device. The operation mode may be switched to the mode M3 by recognizing the predetermined pattern from the data contents.

(E-2) Operation of Fifth Embodiment Next, the operation of the communication system 1D of the fifth embodiment having the above configuration will be described.

  FIG. 14 is a flowchart showing the operation of the communication device 10D (communication estimation unit 109) constituting the communication system 1. Below, it demonstrates centering around operation | movement of communication apparatus 10D-1 (communication estimation part 109) which the 1st user U1 utilizes.

  First, it is assumed that a video call is started between the communication device 10D-1 and the communication device 10D-2 (S500). Here, it is assumed that communication device 10D-1 is set to operate in mode M1 as an initial state.

  Then, in the communication device 10D-1, the communication estimation unit 109 determines whether or not the operation mode of the own device needs to be changed. If the communication estimation unit 109 determines that the operation mode needs to be changed, the communication estimation unit 109 controls each component in the own device so as to operate according to the changed mode (S501).

  Here, as an example, it is assumed that the communication estimation unit 109 determines whether or not it is necessary to change the operation mode of the own device according to the communication time. In this case, the communication estimation unit 109 starts a timer (not shown) for measuring time at the timing when the video call is started (timing in step S500 described above), and the measurement time of the timer is a fixed time. In the following cases, it may be determined that the operation mode change is unnecessary. Then, the communication estimation unit 109 may determine that the operation mode of the device itself is changed to the mode M2 when the measurement time of the timer is abnormal for a certain time. Note that the determination processing (communication status estimation processing) performed simultaneously by the communication estimation unit 109 may be one type or a plurality of types, and the types and number thereof are not limited.

  Then, in control unit 108 of communication device 10D-1, whether or not communication with communication device 10D-2 is continued, that is, whether or not the line between communication devices 10D-1 and 10D-2 is connected. Is confirmed (S502). When it is determined in step S502 that communication with the communication device 10D-2 is continuing (when the line is connected), the control unit 108 returns to the processing in step S501 described above and operates. To do. On the other hand, when it is determined in step S502 that communication with the communication device 10D-2 is continuing (when the line is disconnected), the control unit 108 ends the above-described series of processing ( S503).

(E-3) Effects of Fifth Embodiment According to the fifth embodiment, the following effects can be achieved.

  In the communication devices 10D-1 and 10D-2, communication between users is performed using an interlocking method such as an operation of the display main body 131 of the user and a processing / deformation operation of a remote image displayed on the screen of the display main body 131. It is dynamically switched according to the situation. Thereby, in communication system 1D, compared with the 1st-4th embodiment, a further realism can be provided to a user.

(F) Other Embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments as exemplified below.

(F-1) In each of the embodiments described above, when the video composition unit separates the remote background video and the remote human video from the video data supplied from the remote device, the video processing (background The example using the difference method has been described. However, the separation method of the video composition unit is not limited to this. For example, by capturing a user using a camera or the like having a depth sensor, a group of videos separated according to the measured depth can be obtained. It may be used to separate the remote background video and the remote person video. For example, the video composition unit is a video of the object closest to the camera having the above-described depth sensor is the video of the user (remote person video), and the video of the object farther than that is the background video. You may make it recognize that it is (remote side background image | video).

(F-2) In each of the above embodiments, the display main body has been described as a configuration that can rotate the screen only in the left-right direction (horizontal direction) by the display rotating plate, but the configuration that can rotate the screen in the vertical direction (horizontal direction). Alternatively, the screen may be rotated in both the horizontal direction and the vertical direction. The camera that captures the user may be configured to be rotatable in the left-right direction and / or the up-down direction, similar to the display body.

(F-3) In the fifth embodiment, the communication device switches the operation mode using the user information supplied from the remote device and the speech voice obtained from the audio signal supplied from the remote device. Explained that to estimate the communication for. However, the communication device can also be used to estimate communication by constructing user information in an integrated manner from user position information that can be acquired by distance sensors such as ultrasonic sensors and infrared cameras. It may be possible to improve the accuracy of the communication estimation process by combining it with the above embodiment.

(F-4) Which process performs which information processing (except for input / output processing for a user) performed by the communication device in each of the above embodiments is not limited to the example of each of the above embodiments It is. For example, information processing other than input / output processing for a user may be performed only by one of the communication devices used by the first or second user. Each communication device performs only input / output processing for the user, and collects other processing (acquisition of user information, video composition, control of the orientation of the screen of the display main body, etc.) collectively as ASP (Application). Processing by a server may be performed in a service provider format. In this case, each communication device is configured to connect to the remote device via the above-described server.

(F-5) In each of the above embodiments, the system has been described as a system that provides communication between two parties. However, the present invention may be applied to communication between three or more users. For example, only a voice may be a three-party conference, and any one remote user may be selected (for example, selected by the user's operation) as a target for changing the screen of the display body and the screen orientation. ) May be applied.

(F-6) In the third and fourth embodiments, the video composition unit does not particularly process the remote side background video, but according to the screen orientation of the display main body, as in the first embodiment. You may make it process it. In the first and second embodiments, the video composition unit performs projective transformation and the like on the remote side background video, but it may be displayed on the screen of the display main body without any particular processing. good.

  DESCRIPTION OF SYMBOLS 1 ... Communication system 10, 10-1, 10-2 ... Communication apparatus, N ... Network, 11 ... Communication apparatus main body, 12 ... Camera, 121 ... Camera main body, 122 ... Camera rotating plate, 13 ... Display, 131 ... Display Main unit 132 ... Display rotation plate 101 ... Video acquisition unit 102 ... Video processing unit 103 ... Information transmission unit 104 ... Information reception unit 105 ... Video composition unit 106 106 Video display unit 107 ... Rotation control unit 108: Control unit.

Claims (5)

In a communication system that enables communication between a first user and a second user,
Video data holding means for holding video data of a video image of the second user using one or a plurality of cameras;
A display for displaying a video based on the video data held by the video data holding means for presenting to the first user, and a screen direction control mechanism for directing the screen direction of the display in a direction according to the control; Video display means comprising:
Physical position information acquisition means for acquiring physical position information of the first user or the second user;
In accordance with the physical position information of the first user or the second user, a new screen direction of the display is determined, and the screen direction of the display becomes the determined screen direction. And a screen direction control means for controlling the direction control mechanism. In accordance with the screen direction of the display, it further comprises video data processing means for processing the video data held by the video data holding means,
The communication system according to claim 1, wherein the video display means displays the video of the video data processed by the video data processing means on the screen of the display.   3. The communication system according to claim 2, wherein the video data processing unit performs projective conversion of video of video data held by the video data holding unit according to a screen direction of the display. The video data holding means includes a plurality of cameras that capture images of the second user from different directions, stores video data of images captured by the respective cameras,
According to the physical position information of the first user or the second user, further comprising video data selection means for selecting any video data among the video data captured by the camera,
The communication system according to claim 1, wherein the video display means displays a video based on the video data selected by the video data selection means on the screen of the display. In the first communication device used by the first user, which constitutes a communication system that enables communication between the first user and the second user,
Video data acquisition means for acquiring video data of the second user using one or a plurality of cameras from the second communication device used by the second user;
A display for displaying the video of the video data acquired by the video data acquisition means to be presented to the first user, and a screen direction control mechanism for directing the screen direction of the display in a direction according to the control; Video display means comprising;
Physical position information acquisition means for acquiring physical position information of the first user or the second user;
In accordance with the physical position information of the first user or the second user, a new screen direction of the display is determined, and the screen direction of the display becomes the determined screen direction. And a screen direction control means for controlling the direction control mechanism.

Images