JP2006352749A - Video image communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide small communication terminals by receiving processing by the communication terminals. <P>SOLUTION: Video images photographed by two communication terminals 1<SB>11</SB>and 1<SB>12</SB>are transmitted to a video image composition device 2 through a network 3. The video image composition device 2 superimposes and composes the video images and transmits it back to the respective communication terminals 1<SB>11</SB>and 1<SB>12</SB>. The respective communication terminals 1<SB>11</SB>and 1<SB>12</SB>display the transmitted video image to the display of the video display part. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention is a video communication system that enables users to communicate with each other by temporarily synthesizing videos captured by users at remote locations with their communication terminals into a single video and displaying the result. It is about.

  There is a technique described in Non-Patent Document 1 as a technique that enables communication between users in remote locations. This is a system that has the effect that users in various locations are in the same room by superimposing user images at multiple points on the display, including self-images, and displaying the images to each user. .

  In Non-Patent Document 2, by using a real-space mirror image (self-displayed image) on which a self-image is displayed, a real object can be accessed not only through a CG image but also through an object image displayed on the screen. A real-world interface "mirror interface" is proposed.

  Non-Patent Document 3 proposes a system in which the above-described “mirror interface” is a bidirectional type, and communication is performed while viewing images superimposed on each other at remote locations.

In addition, there is a video conference system as a system in which a plurality of points can be connected and a video at each point can be viewed.
Morikawa et al., “Video Dialogue with Pointing to the Other's Space: Ultra Mirror”, Society of Instrument and Control Engineers 50th Pattern Measurement Subcommittee, pp. 382-387, 2000 Hosoya et al., “Real World Interaction between Remote Locations Using Mirror Interface”, FIT (Information Science and Technology Forum) 2003, pp. 591-592 Hosoya et al. “Realization of interactive interactive communication using mirror interface”, IEICE 2005 General Conference 2005, A-16-20

  However, the conventional method has the following problems.

  The technique described in Non-Patent Document 1 can obtain the effect of being in the same room virtually by superimposing the image of a person or object in a remote place and the user's self-image. The video is limited to a fixed background (blue background, etc.) because the video is overlaid (superimposed) on the remote video after the user's image part is cut out using the chroma key method. There was a problem that could only be realized at the shooting location and conditions. In addition, when a video of a person in a remote area and a user in the own room overlap, there is a problem that one of them is hidden. In addition, video composition is performed by a single common device, and each user at a remote location has to view the same composite result video.

  In the method described in Non-Patent Document 2, although it was possible to operate or display information on a remote person or object, the communication was limited to a single direction. Therefore, there is a problem that access (interaction) such as reverse operation and information display on the user operation side cannot be executed.

  In Non-Patent Document 3, two-way communication is realized, but it is premised on a stationary type in which a user always displays a self-image superimposed on a large display for interaction. Further, it is assumed that a self-image is always reflected, and the feasibility of superimposing the image of the communication partner on the image where the self-image is not reflected has not been considered.

  The conventional TV conference system can connect a plurality of points and watch the video at each point, but cannot display the video at different points in a superimposed manner.

  An object of the present invention is to provide a video communication system having a portable communication terminal.

  Another object of the present invention is to provide a video communication system capable of displaying different synthesis results for each communication terminal.

  Still another object of the present invention is to provide a video communication system having a communication terminal that is easier for a user to use.

  In order to achieve the above object, in the present invention, by arranging the communication terminal and the video composition device separately on the network, the processing of the communication terminal can be reduced, and a smaller communication terminal can be realized. It can also be used as a portable type.

  Also, by arranging the same number of video composition devices as the communication terminals on the network, it is possible to display different superimposed composite videos for each communication terminal. If a different combination result can be displayed for each communication terminal, for example, it is possible to emphasize and view a portion of interest for each communication terminal.

  Also, in the embodiment of the present invention, by arranging the lens and the display of the communication terminal on the opposite side, the user using the terminal is as if the communication partner appeared in the scenery that he / she is looking at. You can see images like The communication partner can enter the video that the user is watching and give a work instruction to the object reflected in the video.

The present invention has the following effects.
1. The communication terminal processing can be reduced to realize a smaller communication terminal, which can be used as a portable type.
2. Different superimposition synthesis result videos can be displayed for each communication terminal. For example, it is possible to emphasize and highlight a portion of interest for each communication terminal.
3. By attaching the lens and the display in the opposite directions, it can be used for work support and work instructions for receiving support and instructions from the communication partner for the image shown on the camera.

  Next, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
Referring to FIG. 1, the video communication system according to the first embodiment of the present invention is composed of communication devices 1 ₁₁ , 1 ₁₂ , a video composition device 2, and a network 3 that connects them to each other.

As shown in FIG. 2, each of the communication terminals 1 ₁₁ and 1 ₁₂ transmits a video capturing unit 11 that captures a video captured by the lens, and transmits the video to the video synthesizer 2 via the network 3. It has a video transmission / reception unit 12 that receives a synthesis result video transmitted from the synthesis device 2 via the network 3 and a video display unit 13 that displays the synthesis result video on a display.

As shown in FIG. 3, the video composition apparatus 2 receives video transmitted from the communication terminals 1 ₁₁ and 1 ₁₂ via the network 3 and transmits a composite result video to each communication terminal. 21, a mirror image inversion unit 22 that mirrors the image received by the video transmission / reception unit 21, a translucent image of the mirror image inversion, superimposes the original mirror image inversion video on one video, and a video transmission / reception unit as a composite result video 21 has a video superimposing and synthesizing unit 23 that outputs the video to 21.

  FIG. 4 shows an example of an image of composition by translucency.

  It is easy to set the degree of translucency freely. The translucent method can be realized by providing transparency (also referred to as an alpha value, which is a general method) information for each pixel of image data in a video. By changing the transparency, it is possible not to be translucent but to be completely transparent (that is, not displayed) or opaque.

  At the time of superimposition processing, one image (superimposed image) is generated by superimposing (superimposing) two translucent images. In that case, the pixels in the two images may be overlapped in accordance with the transparency ratio between the two. For example, if the transparency of one side is 0.7, the pixel value of the image is 30%, the pixel information of the other image at the same position is 70%, and the values may be added.

At this time, for example, an image as shown in FIG. 5 is displayed on each of the communication terminals 1 ₁₁ and 1 ₁₂ . Here, the user A is in communication terminal 1 ₁₁ side as shown in figure, cage user B is in communication terminal 1 ₁₂ side, a lens of the communication terminal 1 _11, 1 ₁₂ of the image photographing unit 11, respectively, image It is placed immediately above the display of the display unit 13, and the users A and B communicate with each other toward the display. The display image in this case is an image in which user A and user B are standing side by side as shown in the figure. At this time, if the user A who is in the communication terminal 1 ₁₁ side has an object to the left hand, it appears to have the right hand on the display image of each communication terminal 1 _11, 1 _12, the communication terminal 1 ₁₂ If the user B on the side raises his right hand, it is displayed as if his left hand is raised on each of the communication terminals 1 ₁₁ and 1 ₁₂ . At this time, each user A, since B sees the same image to each other, for example, points to the part of the shelf by the user B is in communication terminal 1 ₁₁ side, whether pointed where the shelf to the user A Can be understood, and vice versa. In addition, the transparency can be changed (initial setting can be made in advance or setting can be changed during operation), so that the state of each input video (photographed lighting conditions, background image, object / user color) The transparency can be changed to a video that is easy for the user to see depending on the status.

  In addition, by superimposing the entire video on the user side in a semi-transparent manner, the user can operate objects in his / her own room or a remote place in any background without the need for a special environment such as chroma key composition using a blue background. is there. Further, even when a person in a remote place and a user in the own room overlap each other, the image can be seen without being hidden from each other.

Further, in the mirror image inversion unit 22 of the video composition device 2, an image as shown in FIG. 6 can be generated if only one video is inverted. In the figure, only the image of user A is mirror-inverted. In this case, in order to create an image in which the users on both sides are standing side by side, the user B needs to move to the left rather than in the case of FIG. It is also conceivable that the presence or absence of mirror image inversion is performed by providing a changeover switch or a changeover command and changing the designation by changing the designation. In this case, user A Sashishimeseru easily the object in the video communication terminal 1 _12.

  As a method of sending video via the network 3, it is possible to use a video data transmission technology using the existing general Internet, and also to use wireless transmission by general video broadcasting or the like. The above transmission is also easily possible.

[Second Embodiment]
Referring to FIG. 7, in the present embodiment, two video synthesizers 2 are provided in the first embodiment, and each video synthesizer 2 transmits a synthesized result video to a communication terminal designated in advance.

[Third Embodiment]
Referring to FIG. 8, the video communication system according to the third embodiment of the present invention includes communication terminals 1 ₂₁ and ₁₂₂ , video composition devices 2 ₁ and 2 _2, and a network 3.

Communication terminal 1 _21, 1 _22, as shown in FIG. 9, the communication mirror-reversing section 14 to output so that the analog video data captured by the lens of the image pickup unit 11 in the reverse order of the scan lines of FIG. 2 It has been added to terminals 1 ₁₁ and 1 ₁₂ . As shown in FIG. 10, the video composition apparatuses 2 ₁ and 2 ₂ are configured by omitting the mirror image inversion unit 22 of the video composition apparatus 2 of FIG. In the video superimposing combiner 23, by the background difference process, to extract the region where the person is reflected by superimposing the single image with it, mirror-reversing image transmitted from the communication terminal 1 _21, 1 ₂₂ Synthesize each other. Thereafter, the synthesized video is returned to the video transmission / reception unit 21 and transmitted to a communication terminal designated in advance.

FIG. 11 shows an image of superposition synthesis in the video synthesis apparatuses 2 ₁ and 2 ₂ in the present embodiment. In this embodiment, on the communication terminal ₁₂₁ side, an image when the user A is not present is captured in advance as a background image, and the user area is determined based on the difference in pixel value between this and the current video in which the user A is reflected. As a foreground image. Thereto by superimposing the image sent from the communication terminal 1 _22, the user A has created a video that entered the space are the user B. Thus, user A, the image first photographed by the communication terminal 1 _22, can be compared with the case of the second embodiment to confirm more clearly, acts becomes easier to point. Further, in the video synthesizing apparatus 2 _2, by performing an operation reverse to the above, act also becomes as easy user B to point to the space of the user A.

  In the above, the video composition operation is not limited to this. As in the first and second embodiments, it may be possible to make a semi-transparent composition over the entire captured image, or by reversely performing the background clipping operation, it is possible to easily indicate the space in which the subject is reflected. become able to. In addition, by expressing the difference between the pixel values of the background image and the video as transparency, it is also possible to synthesize by superimposing two translucent videos as in the first and second embodiments.

Furthermore, the cable for the image or the like without going through the network 3, may be connected to the image receiving unit 12 and the video synthesizing apparatus 2 _first video transceiver 21 of the communication terminal 1 ₂₁ directly, the image transmission and reception unit of the communication terminal 1 ₂₂ it is also possible to connect 12 and the video synthesizer 2 _second video transmitting and receiving unit 21 directly. In this case, the configuration is as shown in FIG.

[Fourth Embodiment]
Referring to FIG. 13, the present embodiment is a single video composition apparatus in the third embodiment.

[Fifth Embodiment]
Referring to FIG. 14, in the third embodiment, a mirror image inverting unit is provided on only one of the communication terminals in the third embodiment.

[Sixth Embodiment]
Referring to FIG. 15, in the fourth embodiment, a mirror image inverting unit is provided on only one of the communication terminals in the fourth embodiment.

[Seventh Embodiment]
FIG. 16 is a diagram illustrating an image of a communication terminal according to the sixth embodiment of the present invention. FIG. 17 shows a case where user A uses the communication terminal of FIG. 16 and user B uses the same communication terminal as shown in FIGS. 5 and 6 with the lens and the display facing the same direction. Represents an image. Here, captured image of the communication terminal 1 ₂₁ remains normal image, the captured image of the communication terminal 1 ₂₂ are overlapped synthesized by mirror-reversing. The method described in the first embodiment or the third embodiment is used for superposition synthesis.

At this time, as shown in FIG., The communication terminal 1 ₂₁ is able to display an image that enters the user B in the landscape being viewed yourself now display of the communication terminal 1 _21. Further, similarly in the communication terminal 1 _22, and displays an image he has entered into landscape (taken by that the communication terminal 1 ₂₁₎ the user A is viewing. At this time, user B things in a video communication terminal 1 _21, it is possible to point with a stick that has or their fingers. As a result, the user A can directly give work instructions and work support to the user B while pointing to the actual video.

Also in this case, the system configuration may be FIG. 1 or FIG. In the case of FIG. 7, the display video can be changed between the communication terminals 1 ₂₁ and 1 ₂₂ as in the third embodiment. Also, the configuration of the communication terminal and the video composition device may be a combination of FIG. 2 and FIG. 3 or a combination of FIG. 9 and FIG.

  In these embodiments, as described in Non-Patent Documents 2 and 3, it is conceivable to operate a remote object by superimposing a CG icon and superimposing a marker image thereon.

[Eighth Embodiment]
As shown in FIG. 18, in this embodiment using the network system of the mobile telephone service, it is assumed the form that uses the mobile phone terminal with a camera as the communication terminal 4 ₁ in FIG. Examples of the camera-equipped mobile phone described here include FOMA terminals such as NTT Docomo's FOMA (registered trademark) 901i series. As shown in FIG. 19, this FOMA terminal has two cameras, an inner camera 42 on the display 41 side and an outer camera 43 on the opposite side of the display 41, and is used as a video phone. In some cases, you can switch between the images from these two cameras. Therefore, the FOMA terminal can be used as a communication terminal of the type shown in FIG.

In this case, the communication terminal 4 _2, as shown in FIG. 20, may be configured, for example data card type FOMA terminal P2401, Hagiwara Sys-Com Co. viewer port 210 · 51, A / D converter 52, and PC 53. In reception the video, to receive the image sent from the communication terminal 4 ₁ P2401, converts it to NTSC video at viewer port 210 · 51. The converted video is input via the A / D converter 52 to the PC 53 to which a camera as a video shooting function is connected. The PC53 can also function as an image synthesizing apparatus, an image sent from the communication terminal 4 _2, an image captured by the camera connected, superimposed synthesized in the manner described in the first embodiment, it Has the function to display. The image superimposed and synthesized in the PC 53 goes through the reverse process of reception and transmits the superimposed image from the data card type FOMA terminal to the original mobile phone. As a result, the communication terminal 4 ₁ displays the video superimposed and synthesized by the video synthesis device.

  The URL of the viewer port 210 of Hagiwara Cisco is shown below.

http: // www. hscjpn. co. jp / products_s. php? idno = 216
In the above, the method for capturing the video from the mobile phone into the PC is not limited to the above. For example, a method using a video phone Moppet released from NTT East or West Japan is possible, and any method can be used as long as it has a function of transmitting and receiving video with a FOMA terminal and a function of transmitting and receiving video with a PC. Also, a card-type FOMA terminal can be used as long as it is compatible with the viewer port and Moppet.

In the present embodiment, as described in the seventh embodiment, it is possible to display the composite image as shown in FIG. 17 With the outer camera 43, the user of the communication terminal 4 ₂ side while pointing to the real image, to the user of the communication terminal 4 ₁ side, in that it can perform the work instruction and work support, the same advantages can be obtained.

It is a block diagram of the video communication system of the 1st Embodiment of this invention. It is a block diagram of the communication terminal in 1st Embodiment. 1 is a configuration diagram of a video composition device in a first embodiment. FIG. It is an image figure of the semi-transparent superimposition composition in a 1st embodiment. It is an image figure of the mirror image video superimposition composition in a 1st embodiment. It is an image figure of the mirror image video superimposition synthesis only on one side of the first embodiment. It is a block diagram of the video communication system by the 2nd Embodiment of this invention. It is a block diagram of the video communication system by the 3rd Embodiment of this invention. It is a block diagram of the communication terminal in 3rd Embodiment. It is a block diagram of the video synthesizing | combining apparatus in 3rd Embodiment. It is an image figure of the background difference superimposition composition in a 3rd embodiment. It is a block diagram of the video communication system by 3rd Embodiment. It is a block diagram of the video communication system by 4th Embodiment. It is a block diagram of the video communication system by 5th Embodiment. It is a block diagram of the video communication system by 6th Embodiment. It is an image figure of the communication terminal in 7th Embodiment. It is a usage image figure in a 7th embodiment. It is a block diagram of the video communication system by the 8th Embodiment of this invention. It is an image figure of the FOMA terminal in 8th Embodiment. It is a block diagram of the communication terminal and video synthesizing | combining apparatus in 8th Embodiment.

Explanation of symbols

1 ₁₁ , 1 ₁₂ , 1 ₂₁ , 1 ₂₂ , 4 ₁ , 4 ₂ Communication terminals 2, 2 ₁ , 2 ₂ , 5 Video composition device 3 Network 6 Mobile phone network 11 Video shooting unit 12 Video transmission / reception unit 13 Video display unit 14 Mirror image inversion unit 21 Video transmission / reception unit 22 Mirror image inversion unit 23 Video superposition / synthesis unit 31 Lens 32 Display 41 Display 42 Inner camera 43 Outer camera

Claims (10)

Having two communication terminals, one video composition device, and a network connecting them to each other;
Each of the communication terminals transmits a video shooting unit and a video shot by the video shooting unit to the video synthesizing device via the network, and has been transmitted from the video synthesizing device via the network. A video transmission / reception unit for receiving the synthesis result video, and a video display unit for displaying the received synthesis result video,
The video composition device receives the video transmitted from the communication terminals via the network, and transmits the synthesis result video to the communication terminals via the network; The received video images are mirror-inverted, or a mirror image inversion unit that inverts only a pre-specified video image and the mirror-inverted video images are superimposed or synthesized, or the mirror-inverted video image is mirror-inverted. A video communication system comprising: a video superimposing and synthesizing unit that synthesizes a video that has not been output and outputs the synthesized video as the synthesized result video to the video transmitting / receiving unit. Having two communication terminals, two video composition devices, and a network connecting them to each other;
Each of the communication terminals transmits a video shooting unit and a video shot by the video shooting unit to each of the video synthesizing devices via the network, and is transmitted from the video synthesizing device via the network. A video transmission / reception unit for receiving the synthesized result video, and a video display unit for displaying the received synthesized result video,
The video composition device receives the video transmitted from the communication terminals via the network, and sends the composite result video to the different communication terminals specified in advance via the network. A video transmission / reception unit to be transmitted and a mirror image inversion of both the received images, or a mirror image inversion unit to mirror only a predetermined video, and a combination of the mirrored image and the mirror image inversion A video communication system comprising: a video superimposing / synthesizing unit that synthesizes a video that has not been mirror-inverted and a video that has not been mirror-inverted and outputs the resultant video to the video transmission / reception unit. Having two communication terminals, one video composition device, and a network connecting them to each other;
Each of the communication terminals transmits a video image capturing unit, a mirror image reversing unit that mirror-inverts the video imaged by the video image capturing unit, and the mirror-inverted video image to the video synthesizing device via the network. A video transmission / reception unit that receives a synthesis result video transmitted from the video synthesis device via the network; and a video display unit that displays the received synthesis result video.
The video composition device receives the video transmitted from the communication terminals via the network, and transmits the synthesis result video to the communication terminals via the network; A video communication system comprising: a video superimposing and synthesizing unit that superimposes and synthesizes both received videos and outputs the synthesized result video to the video transmitting / receiving unit. Having two communication terminals, two video composition devices, and a network connecting them to each other;
Each of the communication terminals transmits a video image capturing unit, a mirror image reversing unit that mirror-inverts the video imaged by the video image capturing unit, and the mirror image-inverted video to each video composition device via the network, A video transmission / reception unit for receiving a synthesis result video transmitted from the video synthesis device via the network; and a video display unit for displaying the received synthesis result video.
Each of the video composition devices receives the video transmitted from each communication terminal via the network, and each of the different communication terminals designated in advance via the network as a result of the synthesis A video communication system comprising: a video transmission / reception unit that transmits to the video signal; and a video superposition / synthesis unit that superimposes and synthesizes both the received videos and outputs the resultant video to the video transmission / reception unit. Having two communication terminals, one video composition device, and a network connecting them to each other;
One of the communication terminals transmits a video capturing unit, a mirror image reversing unit that mirrors the video captured by the video capturing unit, and the mirrored video to the video synthesizing device via the network, A video transmission / reception unit that receives a synthesis result video transmitted from the video synthesis device via the network; and a video display unit that displays the received synthesis result video. A video shooting unit, and a video shot by the video shooting unit is transmitted to the video synthesizing device via the network, and a synthesis result video transmitted from the video synthesizing device via the network is received. And a video display unit for displaying the received composite result video,
The video composition device receives the video transmitted from the communication terminals via the network, and transmits the synthesis result video to the communication terminals via the network; A video communication system comprising: a video superimposing and synthesizing unit that superimposes and synthesizes both received videos and outputs the synthesized result video to the video transmitting / receiving unit. Having two communication terminals, two video composition devices, and a network connecting them to each other;
One of the communication terminals transmits a video capturing unit, a mirror image reversing unit that mirrors the video captured by the video capturing unit, and the mirrored video to each video compositing device via the network. And a video transmission / reception unit that receives the synthesis result video transmitted from the video synthesis device via the network, and a video display unit that displays the received synthesis result video. The other is a video shooting unit, and a video captured by the video shooting unit is transmitted to the video synthesizing device via the network, and a synthesized result video transmitted from the video synthesizing device via the network. And a video display unit for displaying the received composite result video,
Each of the video composition devices receives the video transmitted from each communication terminal via the network, and each of the different communication terminals designated in advance via the network as a result of the synthesis A video communication system comprising: a video transmission / reception unit that transmits to the video signal; and a video superposition / synthesis unit that superimposes and synthesizes both the received videos and outputs the resultant video to the video transmission / reception unit.   The video communication system according to claim 2, wherein the two sets of communication terminals and the video composition device are each connected by a dedicated network different from a network connecting the communication terminals.   The video communication system according to any one of claims 1 to 7, wherein the video superimposing and synthesizing unit of each video synthesizing device makes the received video semi-transparent and performs superposition and synthesizing processing by superimposing them.   8. The video communication system according to claim 1, wherein the video superimposing and synthesizing unit of each video synthesizing device cuts out the foreground video by image processing and performs superimposing and synthesizing processing by translucent superimposition of the foreground video with the normal image. .   The video communication system according to any one of claims 1 to 9, wherein in at least one of the communication terminals, the lens of the video photographing unit and the display of the video display unit are on opposite sides of the terminal main body.

Images