JPH11266443A - Picture and sound transmission-reception equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a low-cost and small-sized equipment by composing two or more image pickup signals selected from image pickup signals of plural image pickup parts so that the sight of an image of the other speaker of a dialogue may be matched with that of a user. SOLUTION: Plural image pickup parts 1A, 1B, 1C, and 1D are attached to the peripheral frame of a display part 212, and they are provided in upper, lower, right, and left frame parts of the display part 212 respectively. Only picture signals from plural specific image pickup parts are selected from picture signals obtained from plural image pickup parts 1A to 1D by an input picture selection part 201. Selected picture signals are synthesized by a picture synthesis part 202 to generate a facing picture which cannot be obtained from image pickup parts 1A to 1D. The picture signal outputted from the picture synthesis part 202 is transmitted to a picture and sound transmission/reception equipment on the receiver side. The picture signal transmitted from the other party is processed by a signal processing part 211, and the picture is displayed on a display part 212.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image and sound transmitting and receiving apparatus, and more particularly to an image and a sound transmitting / receiving apparatus, such that the sender's line of sight is directly facing the receiver in an image displayed on a display screen of the receiver. It is a device that can send the video of the above, and is also a device that enables miniaturization and cost reduction.

[0002]

2. Description of the Related Art Hitherto, image and voice transmitting and receiving apparatuses, so-called TV conference apparatuses and TV telephone apparatuses have been developed. FIG.
6 (a) shows a personal computer 11 with a camera 12;
Is an example (showing a front view and a side view) in which a TV telephone device is configured. FIG. 16 (b) shows the telephone 13
This is an example in which a display device 14 and a camera 15 are attached to a TV telephone device.

In any of the TV telephone apparatuses, since the display unit and the camera unit are located at different places, the image 16 of the partner who looks at the screen and interacts with the image 16 There is no. For this reason, the interlocutors cannot obtain an environment having a sense of presence with each other.

As a hint to solve this,
A TV conference apparatus as shown in FIG. 16C has been developed (a front view and a side view are shown). That is, the display device 2
1, a half mirror 22 is arranged in front of the half mirror 2,
In this configuration, the camera 23 is disposed below the camera 2. According to this configuration, the interlocutor looks at the screen of the display device 21 through the half mirror 22, but at the same time, the face is imaged by the camera 23 disposed below the half mirror 22, and the image is transmitted to the other party. Transmitted. This way,
Since the image capturing side and the display screen face the interlocutor directly, the line of sight of the image of the other party appearing on the display screen is projected as if looking at this face.

[0005]

As described with reference to FIG. 16C, a TV conference apparatus has been developed in which the line of sight of the image of the interlocutor on the screen is adjusted to this line of sight. However, there is a problem that the configuration is large-scale and is not suitable as a TV telephone device, and the price increases.

Accordingly, the present invention provides an image and voice transmission apparatus which can synthesize an image so that the line of sight of the image of the interlocutor on the screen matches the line of sight of the image, and which can be reduced in size at low cost. The purpose is to do.

[0007]

According to the present invention, there is provided a display device, a plurality of image pickup units provided around the display device for picking up an image in front of a screen of the display device, and an image pickup signal from the plurality of image pickup units. And selecting means for selecting at least two or more arbitrary image signals, and synthesizing means for synthesizing a plurality of image signals by the selecting means to obtain image signals for transmission. Features.

[0008] According to the above means, the image pickup signals obtained from the plurality of image pickup units are combined, and the line of sight of the image of the interlocutor on the screen of the other party is adjusted to the line of sight of the other party facing the screen. Images can be combined as if they were. In addition, this device can be realized at a small size and at low cost.

[0009]

Embodiments of the present invention will be described below with reference to the drawings. From here, the embodiment will be described. For ease of explanation, a communicating party who wants to send an image will be referred to as a sender and a communicating party will be referred to as a receiving party.

FIG. 1 shows an embodiment of the present invention. 1A and 1B are external views of the apparatus of the present invention, showing a front view and a side view. As shown in the front view, the display device 1
00, a plurality of image pickup units 1A,
1B, 1C and 1D are attached. In this embodiment, one is provided in each of the upper and lower frames and the left and right frames of the display unit.

FIG. 2 shows the image pickup units 1A, 1B, 1
C and 1D and a circuit block for processing the output are shown. That is, for the image signals obtained from the plurality of imaging units 1A, 1B, 1C, and 1D, the input image selection unit 201 selects only the image signals from the specific imaging units (a plurality of imaging units from 1A to 1C). .

Next, the selected image signal is supplied to the image synthesizing section 20.
2 to generate a directly facing image which cannot be obtained from each imaging unit. This image synthesizing unit 20
2 is transmitted to the image and voice transmitting / receiving apparatus 203 on the receiver side.

The image signal sent from the other party is processed by the signal processing unit 211, and the image is displayed on the display unit 212. FIG. 3 is a diagram shown to explain the operation of the device of FIG. Although it is preferable for the sender to transmit an image facing the receiver (a state in which the eyes are facing each other) in order to achieve smooth communication,
A face-to-face image cannot be obtained only from an image obtained from a single imaging unit (1A in this example) (see the figure in FIG. 3 in which the face is facing sideways).

Therefore, in the present invention, image information lacking in the image obtained from the image pickup section 1A is obtained by the image signals from the image pickup sections 2B and 2C and synthesized. Thereby, in the image on the receiver side, an image facing the receiver is generated, and the line of sight of the receiver matches the line of sight of the image of the sender (FIG. 3).
(See the figure in which the face is facing the front) to create a realistic environment.

The present invention is not limited to the above embodiment. FIG. 4 shows another embodiment of the present invention. In the embodiment of FIG. 1, the sender transmits the transmitted image after the image synthesis without confirming the transmitted image. However, in some cases, it is desired to confirm the image to be transmitted before transmitting.

Therefore, as shown in FIG.
2 can be temporarily stored in the image memory 213. When the confirmation button from the operation unit 214 such as a keyboard or a remote controller is operated, the switch 2
15 is switched to the terminal X side. Then, the output image signal of the image memory 213 is input to the signal processing unit 211, and the image of the transmission image signal can be confirmed on the display unit 212. When the confirmation is completed, the switch 215 is switched to the terminal Y side, and the image signal sent from the other side is processed by the signal processing unit 211. Further, the switch 216 is turned on, and the image signal of the sender is transmitted to the image and voice transmitting / receiving device of the receiver.

In the above apparatus, if the sender does not like the transmission image, the transmission image is not transmitted, and the synthesized image is changed by changing the combination of selection of input images used for image synthesis or by changing conditions such as the direction of the subject itself. Can be changed. When selecting an input image to be used for image synthesis, the combination can be arbitrarily changed through the operation unit 214. By repeating this, the image can be transmitted after the sender understands the image to be transmitted on the monitor. This embodiment is particularly effective when transmitting a still image.

The present invention is not limited to the above embodiment. FIG. 5 shows still another embodiment of the present invention. In this example, a plurality of windows 301 and 30 are displayed on the screen of the display device 100 like a personal computer.
2, 303 is opened, and the image of the recipient is displayed in one of the windows.

In this case, it is assumed that the other party is in the same state, and the window is
It is assumed that the recipient's line of sight is also directed from the center front of the screen. Then, when transmitting the image of the sender, the window is set so that the line of sight is directed, that is, an imaging device is selected, and a composite image signal is created and transmitted.

FIG. 6 shows a signal processing circuit block of the above device. The window 203 in which the image of the previous recipient appears can be detected by the window position detection unit 311 using the signal of the signal processing unit 211. This detection can be obtained from, for example, XY coordinate information on the screen. The window 203 in which the image of the receiver appears is determined in advance and can be obtained by acquiring the display coordinate information. In addition, in the image signals obtained from each of the imaging units 1A to 1D, for example, a signal of a common pattern, for example, a signal of an eye part is added, and the gaze direction is detected. Various embodiments are possible as a gaze detection method. When the direction of the line of sight is detected, it is determined whether the direction of the line of sight is in the direction of the window position.

Then, a selection signal for selecting an input image is output from the visual line detection unit 312 so that the visual line direction is directed toward the window position. The image signal selected by the input image selection unit 201 is synthesized by the image synthesis unit 202 and transmitted. Since the image signal on which the same processing is performed is transmitted to the other party, the line of sight of the other image displayed in each other's window is directed to the user in front of the screen.

By adopting such a configuration, it is possible to realize smooth communication in which the line of sight coincides with the communication partner. FIG. 7 shows still another embodiment of the present invention.

The basic configuration of FIG. 7 is the same as the configuration described in FIG. This is an embodiment in which the processing and control of each unit are processed in the multi-interlocked signal processing device 400 and completed.

Most of the image pickup units used in conventional image and sound transmission devices are made based on video cameras, and are generally used in terms of interfaces. The personal computer dedicated product is made in accordance with an interface (I / F) 401 generally provided.

Therefore, in the present invention, it is more important to match the interface with these.
In the case of the present invention, unlike a normal imaging unit, dedicated signal processing is required in addition to general video signal processing to synthesize and output a plurality of images.

It is conceivable that this signal processing is performed on the host side (such as a personal computer), that is, on the external device 404, but the operation speed may be reduced in some cases because the CPU occupies a part of the CPU performance. This increases the processing load on the computer and the like.

Therefore, as in the present invention, the signal processing device 40
If the configuration is such that the image processing is completed within 0, it can be realized without applying a signal processing load to the external device 404, so that it can be applied to many systems that do not have an image processing device from the imaging unit. It is possible. Dedicated hardware will be added, but it will not be a problem in terms of price given the recent advances in semiconductors.

FIG. 8 shows still another embodiment of the present invention. This embodiment is also applied to a case where a plurality of windows (such as 502 in the figure) are open, including a window 501 for displaying a recipient image on the screen of the display device 100.

A window used in a personal computer is usually activated by clicking a pointing device (mouse or the like). Although this method is a reliable means, when the number of windows is large or the windows are overlapped, it is complicated or time-consuming to search for and activate a target window. This is particularly problematic when checking while checking the face of the receiver.

Therefore, in this embodiment, in order to solve this problem, the image of the receiver can be automatically displayed in the front row only by the line of sight of the sender. In other words, as shown on the left side of FIG. 8, when the window 501 fits behind the window 502, the user's line of sight is detected, and the display state of the window 501 is brought to the front row as shown on the right side.

FIG. 9 shows a circuit block for realizing the above functions. The positions of the windows 501 and 502 can be detected by the window position detection unit 311 using the signal of the signal processing unit 211. Next, the direction of the user's line of sight is detected using the image signals obtained from the imaging units 1A to 1D. When the direction of the line of sight is detected, it is determined which direction of the window position the line of sight is facing. This determination is performed, for example, by the comparison and priority display window determination unit 320.

If the direction of the line of sight is in the direction of the position of the window 501, a control signal is given to the signal processing unit 211 so that the window 501 is displayed in the front row.

In this operation, control is performed such that the window in which the sender's line of sight coincides with the front row. By changing the control mode, for example, only when the window 501 coincides with the window 501, the window 501 becomes the front row. May be controlled so that If they do not match, the composite image is displayed on the screen of the display device, but the image may be hidden if another window is activated. In this way, smooth communication can be achieved.

FIG. 10 shows still another embodiment of the present invention. In this embodiment, a plurality of imaging units are arranged around the display device 100, and use these to detect a line of sight, or to perform image synthesis using images from some of the imaging units after detecting the line of sight. However, the use of an imaging unit not used for image synthesis is released to the communication partner.

FIG. 10 shows an example in which there are four image pickup units. If the three imaging units 1A, 1B, and 1C are used for image synthesis, the other imaging units 1D are opened to the communication partner so that images can be freely captured by a command from the communication partner. It is. In the example shown in the figure, an example is shown in which the images of the face of the sender are captured by the imaging units 1A, 1B, and 1C, and the image of the interior planting is captured by the imaging unit 1D.

FIG. 11 is a circuit block diagram for realizing the above functions. Here, a description will be given of a case where a directly facing image is combined with a window in which a communication partner is displayed and transmitted.

First, the window position detecting unit 311 detects the position of the window in which the communication partner is displayed from the display unit 212 on the sender side. Also, in the line-of-sight detection unit 312,
A gaze direction using a plurality of imaging units is detected. Then, the gaze detection unit 312 compares the gaze detection result with the window position information in which the partner image is displayed. Selected. The output image signal of the input image selection unit 201 is synthesized in the image synthesis unit 202. This composite image signal causes the image of the sender facing the receiver to be displayed on the receiver-side display.

Here, as for the image pickup unit not selected for image synthesis, the image designation unit 611 from the receiver side.
Command can be sent. The command sent from the image specifying unit 611 is analyzed by the image specifying unit 612 via the signal processing unit 211 on the sender side. Then, the image designating unit 612 controls the image signals from the remaining image capturing units other than the image capturing unit used for the image synthesis to be extracted by the arbitrary image extracting unit 613. Then, the image signal extracted by the arbitrary image extracting unit 613 is transmitted to the receiver side so as to form a separate screen from the composite image signal. As a result, as shown in FIG. 10, it is possible to monitor the image around the sender on the display device of the receiver.

FIG. 12 shows still another embodiment of the present invention. This embodiment is also the same as the previous embodiment, until a line of sight is detected using a plurality of imaging units and an imaging unit used for image synthesis is selected accordingly.

In this embodiment, however, the window 7 in which the image of the recipient given as the initial condition is displayed
In the case where a synthetic image is generated at the position No. 01 and the processing time required for the synthesis takes longer than necessary (a large number of imaging units are selected or a large amount of information is transmitted from a plurality of imaging units). When the processing takes time, for example), the window position on the screen of the display device is changed to the position of the window 702 (the position where the processing information is small and the processing is quick and smooth) and the feedback is performed. And then re-synthesize again.

That is, in the case where the image of the receiver appears at the position of the window 701 shown by the dotted line in FIG. 12, image synthesis is performed so that the image of the sender also appears at the same position as the receiver. Will be However, in this case, in the example shown in the drawing, the imaging unit 1C is used for the subject images of the imaging units 1A and 1B.
Subject image is greatly shifted. When the gaze direction is adjusted by image synthesis, it is desirable that the same sender's face be imaged from substantially the same direction, and that there be little deviation between images of the imaging units, and the processing time of signal processing be short. However, in an example in which the image of the recipient is arranged at the position of the window 701, the subject image of the imaging unit 1C is largely displaced from the subject images of the imaging units 1A and 1B, and thus the line-of-sight direction is adjusted. Signal processing takes time.

Therefore, in this embodiment, the position of the window is shifted to the position 702, and each of the imaging units 1A, 1B,
The displacement of the subject image of 1C is reduced, and the load of signal processing is reduced.

This sequence control (automatic control in which each stage of control is sequentially advanced in a predetermined order)
If the image is transmitted in a state where the composition is completed within a predetermined processing time after passing through, a smoothly moving image can be transmitted to the receiver.

FIG. 13 shows a transmission / reception device having a function of changing the position of the window when it takes more than a predetermined processing time for image synthesis for adjusting the direction of the line of sight.

In the initial state, the sender's own composite image is shown in FIG.
2 is displayed at a position 701 in FIG. 2, and using all (in this case, four) imaging units (1A to 1D),
When it takes a very long time to display (processing), the solid-state image sensor, that is, the image pickup unit (1A to 1D) may not be entirely used, or the position where the information from the solid-state image sensor may be reduced even when all the images are used. (702 in the figure) so that the image display of the sender's confidence can be smoothly displayed.

In FIG. 13, the same parts as those in the embodiment of FIGS. 6 and 11 are denoted by the same reference numerals and described. In this embodiment, when the image combining section 202 performs image combining, it is determined whether or not the combining processing time is within a predetermined time. This determination is performed by the synthesis processing time determination unit 711. If it is within the predetermined time, the image synthesis processing is executed as it is, and the synthesized image signal is transmitted.

In the determination result of the combining processing time determining section 711, if the combining processing is spent for a predetermined time or more, a command signal is given from the window position change command section 712 to the signal processing section 211, and the window position is corrected. The window position is corrected in a direction that does not require much time for the combining process. Therefore, in this case, the window position change command unit 712
Fetches information indicating which imaging device output is currently selected from the input image selection unit 201. Then, control is performed so that the window moves toward the center position of each imaging unit (that is, the window moves in a direction at the same distance from each imaging unit).

FIG. 14 shows still another embodiment of the present invention. In this embodiment, a plurality of correspondents are displayed on the same display device with a plurality of windows (801, 802, 803, 804 in the illustrated example) opened.

FIG. 14 shows a case where the sender's line of sight (see the dotted arrow) is displayed in the window 804. In such a case, the confronting image is synthesized and transmitted to the communicating party shown in the window 804 as in the above embodiment. However, for other communicators, an image from the imaging unit near the window where the image of each communicator is displayed other than the imaging unit used for the face-to-face image synthesis is transmitted to each communicator. ing.

This is because, for a receiver (a receiver corresponding to the windows 801, 802, and 803) that the sender does not see, an image that is obliquely sent is sent, and the sender turns his or her eyes to the receiver. This is to indicate that it is not. In the example of the figure, the imaging units used for the facing image synthesis are 1A, 1C,
1H, and the remaining imaging units 1B, 1E, 1F, 1D, 1
G is not used. Therefore, these imaging units are used for a communication person who is not looking at the line of sight.

By adopting such a configuration, it is possible to clearly identify the person who is speaking, and it is possible to smoothly carry out a plurality of conversations. FIG. 15 shows circuit blocks for realizing the above functions. The same parts as those in the above embodiment are denoted by the same reference numerals. Window position detection unit 311
Detects the position of each window. Eye-gaze detector 3
Numeral 12 detects the direction of the line of sight of the sender. The line-of-sight window detection unit 911 detects a window (804 in the above example) in which the sender points to the line of sight, and matches the line of sight of the image of the sender with the receiver of the window. An image signal from an appropriate direction is selected and synthesized by the image synthesis unit 202.

An image signal not used for image synthesis (a single image signal used) may be transmitted to the transmission image distribution unit 9.
It is led to 12. The window position information from the previous window position detection unit 311 is also given to the transmission image distribution unit 912. As a result, the image signals from the imaging unit near the position of each window are associated with each other and transmitted to the receiver corresponding to the window.

In the present embodiment, the window 80
If it is detected that the line of sight is directed to 4, and if the facing images are combined using the image pickup units 1B, 1C, and 1H among the plurality of arranged imaging units, it is possible to notify another communication person whose line of sight is not directed. On the other hand, an image obtained from an imaging unit other than the solid-state imaging device used for image composition arranged at a position close to each window is transmitted to each communication party.

According to the present embodiment, smooth communication can be realized even by a plurality of communicating parties. As described above in detail, it is possible to have a conversation in which the eyes are matched, which is difficult to realize with the conventional small-sized and low-cost image and sound transmission device, and it is possible to perform smooth communication. Further, since a communication party or a plurality of communication partners can automatically detect a line of sight and automatically control a window, communication is possible without being conscious of switching windows. In addition, non-contact gaze detection can be realized by disposing a plurality of imaging units around the information display device.

[0055]

As described above, according to the present invention,
The image can be synthesized so that the line of sight of the image of the interlocutor on the screen matches this line of sight, and the size can be reduced at low cost.

[Brief description of the drawings]

FIG. 1 is a diagram showing the appearance of an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a circuit block according to the embodiment of FIG. 1;

FIG. 3 is a view showing an example of an image shown for explaining the operation of the embodiment of FIG. 1;

FIG. 4 is a diagram showing an example of a circuit block according to another embodiment of the present invention.

FIG. 5 is a diagram showing the appearance of still another embodiment of the present invention.

FIG. 6 is a diagram illustrating an example of a circuit block according to the embodiment of FIG. 5;

FIG. 7 is a diagram showing a circuit block according to still another embodiment of the present invention.

FIG. 8 is a diagram showing the appearance of still another embodiment of the present invention.

9 is a diagram illustrating an example of a circuit block according to the embodiment in FIG. 8;

FIG. 10 is a diagram showing the appearance of still another embodiment of the present invention.

11 is a diagram illustrating an example of a circuit block according to the embodiment in FIG. 10;

FIG. 12 is a diagram showing the appearance of still another embodiment of the present invention.

FIG. 13 is a diagram illustrating an example of a circuit block according to the embodiment of FIG. 12;

FIG. 14 is a diagram showing the appearance of still another embodiment of the present invention.

15 illustrates an example of a circuit block according to the embodiment in FIG. 14;

FIG. 16 is a diagram showing a conventional video and audio transmitting / receiving device.

[Explanation of symbols]

 1A, 1B, 1C, 1D ... Imaging unit 100 ... Display device 101 ... Frame around display unit 201 ... Input image selecting unit 202 ... Image combining unit 203 ... Image and sound transmitting / receiving device 211 ... Signal processing unit 212 ... Display unit 213 ... Image memory 214 ... Operation unit 215, 216 ... Switch

Claims (7)

[Claims] 1. A display device, a plurality of imaging units provided around the display device, and imaging the front of a screen of the display device, and at least two or more imaging signals from the plurality of imaging units Image and audio transmission and reception, comprising: a selection unit that can select any one of the imaging signals described above; and a combination unit that combines a plurality of imaging signals with the selection unit to obtain an imaging signal for transmission. apparatus. 2. When an image of a recipient as a communication partner is displayed in a part of a window of the display device, the selection unit is configured to output a signal from the imaging unit in accordance with a position of a window in which the communication partner is displayed. 2. The image and sound transmitting / receiving apparatus according to claim 1, wherein a combination to be combined with the image signal is selected. 3. The image and sound transmitting / receiving apparatus according to claim 1, wherein the signal processing for generating a composite image from the plurality of imaging units is performed in an integrated signal processing device. 4. A signal processing device for displaying a plurality of windows, including a specific window for displaying an image of the communication partner, on the display device, wherein a signal processing device of the display device Detects the line of sight of the sender located in front, and if the line of sight is the direction of the window of the image of the communication partner, controls the signal processing device to display the specific window in the front row. 2. The image and sound transmitting / receiving apparatus according to claim 1, further comprising means. 5. An imaging unit used for generating an image facing the communication partner, wherein the plurality of imaging units include:
2. The image and sound transmitting / receiving apparatus according to claim 1, wherein there is an imaging unit that can be controlled by a command sent from the communication partner. 6. The image pickup unit arranged in plural numbers,
2. The apparatus according to claim 1, further comprising: a unit for detecting a line of sight of a sender located in front of the display device and moving a window in which the communication partner is displayed in the line of sight.
Image and sound transmitting and receiving apparatus. 7. When a plurality of communication partners displayed on the display device are displayed in a plurality of windows, a line of sight of a sender is detected by a plurality of solid-state imaging devices arranged around the display device, A directly-composed image is sent to the correspondent displayed in the window determined to be closest to the line-of-sight direction, and images of the imaging units other than the imaging unit used for the facing image composition are sent to other correspondents. 2. The image and voice transmitting / receiving apparatus according to claim 1, further comprising means for transmitting the used image or the composite image.