JP2001268532A - Pictorial communication equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide pictorial communication equipment capable of easily switching and transmitting an object arranged on different focal distances without requiring complicated operation such as the focal control of a camera. SOLUTION: A luminance area extraction part 12B extracts the image area of a speaker P from a luminance image based on distance images obtained by photographing a projection pattern projected to the speaker P by 1st and 2nd cameras 3A, 3B.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image communication apparatus, and more particularly, to an image communication apparatus capable of transmitting and receiving voice and images to and from a remote party such as a videophone or a video conference.

[0002]

2. Description of the Related Art In recent years, videophones for transmitting images together with sound have become popular in ordinary households. These videophones can easily communicate with each other because the display can display the expression of the other party or an object that is a topic.

As such an image communication device, for example,
There is one disclosed in JP-A-5-91504.
According to this image communication device, a background portion is detected along a contour of a subject from a video signal including a subject and a background portion of a caller or the like photographed by a camera focused on a position where a subject to be communicated is located. The video signal of only the subject is transmitted after removing the detected video signal of the background portion.

[0004]

However, according to the conventional image communication apparatus, the object to be transmitted is arranged so as to be in focus of the camera, and the out-of-focus portion is separated as a background portion. If the user tries to switch between different subjects, the user must move the subject to be in focus or adjust the focus of the camera, which causes a problem that the shooting operation becomes complicated.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image communication apparatus which does not require a troublesome operation such as a focus adjustment of a camera and which can easily switch and transmit subjects arranged at different focal lengths. Is to do.

[0006]

In order to achieve the above object, the present invention provides an image communication apparatus for receiving and transmitting a brightness image of a subject taken by a camera through a communication line, wherein the distance from the camera to the subject is obtained. Distance information obtaining means, area extracting means for determining an area of the subject located at a predetermined distance based on the distance, and object image extraction for generating an object image by extracting a luminance image of the area from the luminance image And an image communication apparatus having control means for transmitting the generated object image via the communication line.

According to the image communication apparatus described above, the distance from the camera to the subject is obtained, and a luminance image of a region located at a predetermined distance from the luminance image is extracted to generate an object image. Can be eliminated. In addition, the extraction range can be easily switched by specifying the value of the distance when extracting the area.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image communication apparatus according to the present invention will be described in detail with reference to the drawings.

FIG. 1 shows an image communication apparatus according to a first embodiment of the present invention, wherein a projector 1 for projecting a projection pattern coded with respect to intensity to a caller P, and a projector 1 provided on an optical path of the projection pattern. A half mirror 2, a first camera 3A provided coaxially with the optical axis of the projector 1 and taking an image reflected by the half mirror 2, and a caller P from a direction different from the optical axis direction of the projector 1. Camera 3B that captures a distance image, a processing unit 4 that acquires a distance image and a luminance image based on image information captured by the first camera 3A and the second camera 3B, and a distance image that stores the distance image. Memory 5
And a luminance image memory 6 for storing a luminance image, a transmission image extraction unit 7 for inputting a distance image and a luminance image and extracting image data to be transmitted, and a telephone 8 for receiving and transmitting voice.
And a receiver 9 for transmitting and receiving voice, transmitting transmission data based on the transmission image data input from the transmission image extracting unit 7 and the voice data input from the telephone 8, and receiving the transmission image data and the voice data from the telephone line. Transmission unit 10
And a display 11 such as a CRT or a liquid crystal for displaying the received transmission image data.

In the above configuration, the processing unit 4, the distance image memory 5, the luminance image memory 6, the transmission image extraction unit 7, and the transmission / reception unit 10 are connected to the first camera 3A and the second camera 3B. An image control unit 31 for extracting an image of the person P and performing a transmission / reception process is formed.

The processing section 4 inputs a photographing pattern obtained by photographing a slit pattern as a projection pattern projected from the projector 1 on the optical axis of the projector 1 with the first camera 3A. This photographing pattern is sent to the processing unit 4
Is compared with the projection pattern to compare the intensity difference. The comparison of the intensity difference is performed, for example, by decomposing the slit pattern and the photographing pattern into cells of several pixels and comparing the average intensity of the cells. When the intensity difference is equal to or greater than the threshold value and has a new intensity, the re-encoding unit generates and assigns a new intensity code, thereby re-encoding the cell. Also,
If it is not a new strength, the cell is recoded by assigning a code of the corresponding strength.

The transmission image extraction unit 7 includes a distance image acquisition unit 12
A, and a luminance area extracting unit 12B.
2A, a threshold value processing unit 13 that determines a distance value in each pixel of the distance image based on the threshold value and extracts the distance value as a binary distance image, and a threshold value that stores a threshold value used for determining the distance. A value information memory 14 and an interpolation processing unit 1 for performing an interpolation process for removing noise included in acquiring a distance image
5

When the binary distance image acquired by the distance image acquiring section 12A is input, the luminance area extracting section 12B extracts a luminance image area having a pixel value of 1 from the luminance image stored in the luminance image memory 6. Then, the image data is output to the transmission / reception unit 10 as transmission image data.

The operation of the image communication apparatus according to the present invention will be described below with reference to the drawings.

In the following operation, the distance image of the caller P is obtained based on the pattern projection method. First, the caller P falls within the photographable range of the first camera 3A and the second camera 3B. The camera angle is adjusted as described above, and the projection pattern is projected from the light projector 1 and photographed by the first camera 3A. The second camera 3B captures an image of the caller P from a direction different from the optical axis direction of the projector 1. First camera 3A
By processing the video signal of the second camera 3B with the processing unit 4, a distance image and a luminance image are obtained. The procedure for measuring the distance by the pattern projection method will be described below.

FIG. 2 shows a distance measuring method based on the pattern projection method, in which the projector 1 projects a vertical stripe pattern as a projection pattern to the caller P, and the first camera 3
A and the second camera 3B photograph the caller P. The processing unit 4 extracts a stripe pattern from a photographing pattern input from a first camera (not shown), divides each stripe into cells of several pixels, and re-codes them by a re-coding unit (not shown). The code of each cell is detected using the code, and the slit angle θ from the projector 1 is calculated based on the detected code. From the slit angle θ of the cell to which each pixel belongs, the x coordinate on the image taken on the image plane of the second camera 3B, the focal length F and the base length (distance) L as camera parameters, the following equation The distance Z to the measurement point A is calculated according to (1). Z = (F × L) / (x + F × tan θ) (1)

FIG. 3 shows a luminance image obtained by photographing the caller P. An image obtained by photographing the caller P and the background B by the first camera 3A is stored in the luminance image memory 6 as a luminance image. . This luminance image is obtained by classifying the luminance of light entering the light receiving section of the first camera 3A from the surface of the caller P for each pixel.

FIG. 4 shows a distance image obtained by photographing the caller P. The processing unit 4 generates a projection pattern based on the photographing of the projection pattern projected on the background 13 and the caller P by the second camera 3B. A distance image is formed by classifying the image into regions corresponding to the distance based on the deformation amount and assigning a distance code to a pixel of each classified region. This distance image is
The distance from the light receiving section of the camera 3B to each point of the object is defined as each pixel value, and is arranged two-dimensionally. In the figure, the caller P is located at the forefront in the paper direction, the area 19 is located behind the area P, the area 16 is located further behind the area P, the area 17 is located further behind the area P, and the area An area 18 is located. As a method of displaying these on the screen, for example, they can be represented by the magnitude of luminance,
The luminance may be expressed such that the luminance in the area on the near side of the paper is increased and the luminance is reduced in the depth direction of the paper.

FIG. 5 shows an image obtained by separating the caller P and other parts based on the distance image. The pixel value of the caller P is 1 (white), and the pixel value of the other background part 20 is 0.
(Black). When noise or the like is superimposed when acquiring a distance image, a pixel that should be 1 in the distance image may become 0 in some cases. In the figure, a dot 21 having a pixel value of 0 is formed due to noise in the part of the talker P.

FIG. 6 shows a distance image on which interpolation processing is performed by the interpolation processing unit 15. When a 0 pixel on which noise is superimposed in a part of the caller P is surrounded by 1 pixel, By performing the interpolation process of converting the 0 pixel into 1 to generate binary image data in which defective pixels due to noise or the like included in the part of the caller P are interpolated.

FIG. 7 shows a transmission image in which the part of the caller P is extracted and the others are formed as a background having a pixel value of 0. The luminance area extraction unit 12B calculates the distance from the luminance image stored in the luminance image memory. Interpolation processing unit 15 of image acquisition unit 12A
The region where the pixel value in the binary image data input from is 1, that is, the region corresponding to the part of the caller P is extracted,
The transmission image data is formed with the other portion as the background having the pixel value of 0.

The voice of the caller P is input to the telephone 8 via the receiver 9, converted into a voice signal, and input to the transmission / reception unit 10. The transmission / reception unit 10 multiplexes the transmission image data input from the luminance area extraction unit 12B and the audio signal input from the telephone 8 based on a known multiplexing method, and transmits the transmission data via a telephone line.

The transmission / reception unit 10 separates transmission data received from the other party into transmission image data and an audio signal, displays the transmission image received on the display 11, and outputs sound to the receiver 9 via the telephone 8. I do.

According to the image communication apparatus of the first embodiment, a call is made from a luminance image based on a distance image obtained by photographing a projection pattern projected on a caller with the first and second cameras. Since the image area of the caller is extracted, the caller can see only the caller or only the object to be displayed, and does not give the information that he does not want to see to the other party, so privacy protection and security Allow for guarantees. In addition, since no special equipment or operation is required, usability can be improved.

For example, in FIG. 3, a display device C is provided near the left side of the caller P. When it is desired to display an area where the display device C is arranged together with the caller P, FIG. By setting a threshold value stored in the threshold information memory so that the area 19 of the distance image is selected together with the caller P, the image is transmitted as transmission image data including the caller P and the display device C. be able to.

As a method of acquiring a distance image, for example, stereo image processing using the principle of triangulation can be performed. Further, it is desirable that each pixel of the luminance image and the distance image correspond to each other on a one-to-one basis, but it may be one-to-many. Also, the form of transmission and reception is not limited to the multiplexing of images and sounds, and the transmission and reception of images and sounds may be switched at a predetermined time.

FIG. 8 shows an image communication apparatus according to a second embodiment of the present invention. The presence confirmation unit 23 for confirming whether a caller is present in front of the camera, and the presence confirmation unit 23 Communication control unit 2 that controls connection and disconnection of the line based on the determination result
4 is different from the image communication apparatus shown in FIG. 1, and other configurations and functions are the same.

The presence confirmation unit 23 determines that the pixel value of the luminance image input from the luminance image memory 6 to the luminance region extraction unit 12B is 1
Area calculation unit 25 for calculating the area of a region, and a determination process of outputting a control signal for connecting or disconnecting a telephone line based on an area calculation result input from the area calculation unit 25 and an area threshold described later. A unit 26 and an area threshold memory 27 storing an area threshold of luminance for determining whether a caller is enrolled in front of the camera.

The area calculator 25 calculates the area of the area whose pixel value is 1 extracted by the luminance area extractor. The area of the region can be determined, for example, by summing up the number of pixels in the region having a pixel value of 1. The total area increases when the caller enters the field of view from outside the camera, and decreases when the caller goes out of the field of view from the camera.

The judgment processing unit 26 is provided with an area threshold memory 2
7 is compared with the area threshold value stored in the memory 7 and when the total area changes from the threshold value or less to the threshold value or more, it is determined that the talker has appeared in front of the camera, and the communication control unit 24
An N signal is output, and when the total area changes from the threshold value or more to the threshold value or less, it is determined that the caller has left the camera and an OFF signal is output.

When the caller appears in front of the camera, the communication control unit 24 connects the telephone line based on the ON signal input from the judgment processing unit 26. Further, when the caller leaves from in front of the camera, the telephone line is disconnected based on the OFF signal input from the determination processing unit 26.

According to the image communication apparatus of the second embodiment, the caller can easily receive and transmit images and voices without being aware of the start and end of the call, and can perform special operations. It is unnecessary and the usability can be improved.

FIG. 9 shows an image communication apparatus according to a third embodiment of the present invention, in which a background memory 28 in which desired background information (such as an image) is stored in advance, transmission image data, an audio signal, and a background image. The configuration having an image synthesizing unit 29 for synthesizing information is different from the image communication device shown in FIG.
The other configurations and functions are the same, and duplicate description will be omitted.

The image synthesizing section 29 includes a luminance area extracting section 12B
The image of the region corresponding to the part of the caller P extracted by the above is synthesized with the background information stored in the background memory 28.
The background information may be monochrome color information or an image.

The transmission / reception unit 10 receives from the image combining unit 29 a composite image obtained by combining the background information stored in the background memory 28 and the extracted image, and multiplexes with the audio signal input from the telephone 8 for transmission.

According to the image communication apparatus of the third embodiment, an image of a caller and background information such as an image prepared in advance can be combined and transmitted, and the display can be diversified. Can be planned.

FIG. 10 shows an image communication apparatus according to a fourth embodiment of the present invention, in which transmission image data input from the luminance area extraction unit 12B and audio signals input from the telephone 8 are multiplexed to establish a telephone line. A transmitting / receiving unit 10 for transmitting via
A communication control unit 24 connected to the transmission / reception unit 10, a background memory 28 for storing background information such as an image, an image synthesis unit 29 for synthesizing background information and transmission image data, and a synthesized image output from the image synthesis unit 29 Is different from the image communication apparatus shown in FIG. 1 in the configuration having the image display unit 30 having the display 11 for displaying the same, and the other configurations and functions are the same, so that the duplicated description will be omitted.

In the fourth embodiment, the transmission / reception unit 10 receives the reception data reflecting the other party via the communication control unit 24, separates the audio signal from the received image, and outputs the audio signal to the telephone 8. And outputs the received image to the image synthesizing unit 29. The image combining unit 29 combines background information such as an image stored in the background information memory 22 with a received image. The background information may be monochrome color information. The composite image thus formed is displayed on the display 11.

According to the image communication apparatus of the fourth embodiment, a composite image obtained by compositing the received image of the other party and the background image prepared in advance on the receiving side is displayed. Accordingly, the size of the data to be transmitted can be reduced, and the time for receiving and transmitting can be reduced.

FIG. 11 shows an image communication apparatus according to a fifth embodiment of the present invention. The presence confirmation unit 23 for confirming whether a caller is present in front of the camera, and the presence confirmation unit 23 The configuration having a communication control unit 24 for controlling connection and disconnection of the line based on the determination result is different from the image communication apparatus shown in FIG. 10, and other configurations and functions are shown in FIG.
Since the image communication apparatus is the same as the image communication apparatus shown in FIG.

In the fifth embodiment, transmission and reception are controlled by determining whether or not a caller is present in front of the camera and connecting and disconnecting the line as described with reference to FIG.

According to the image communication apparatus of the fifth embodiment, the caller can easily synchronously transmit and receive images and voices without being aware of the start and end of the call. This eliminates the need for complicated operations and improves usability.

[0043]

As described above, according to the image communication apparatus of the present invention, distance information is simultaneously obtained at the time of photographing a caller, and an image to be transmitted is extracted and transmitted based on the distance information. This eliminates the need for troublesome operations such as adjusting the focus of the camera, and can easily switch and transmit subjects arranged at different focal lengths.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram showing an image communication device according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a distance measuring method by a projection pattern method according to the embodiment of the present invention;

FIG. 3 is an explanatory view of the caller and the background thereof according to the first embodiment of the present invention;

FIG. 4 is an explanatory diagram showing a distance image according to the first embodiment of the present invention;

FIG. 5 is an explanatory diagram showing an image obtained by separating a part of a caller P based on a distance image according to the first embodiment of the present invention;

FIG. 6 is an explanatory diagram showing a distance image on which interpolation processing has been performed according to the first embodiment of the present invention;

FIG. 7 is an explanatory diagram showing a transmission image in which a part of a caller P is extracted according to the first embodiment of the present invention;

FIG. 8 is an explanatory diagram showing an image communication device according to a second embodiment of the present invention.

FIG. 9 is an explanatory diagram showing an image communication device according to a third embodiment of the present invention.

FIG. 10 is an explanatory diagram showing an image communication device according to a fourth embodiment of the present invention.

FIG. 11 is an explanatory diagram showing an image communication device according to a fifth embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Projector 2 Half mirror 3A camera 3B camera 4 Processing part 5 Distance image memory 6 Luminance image memory 7 Transmission image extraction part 8 Telephone 9 Receiver 10 Receiver / transmission part 11 Display 12A Distance image acquisition part 12B Luminance area extraction part 13 Threshold processing Unit 14 threshold information memory 15 interpolation processing unit 16 area 17 area 18 area 19 area 20 background part 21 dot 22 background information memory 23 presence confirmation unit 24 communication control unit 25 area calculation unit 26 judgment processing unit 27 area threshold Memory 28 Background memory 29 Image synthesis unit 30 Image display unit 31 Image control unit

Claims (13)

[Claims] 1. An image communication apparatus for receiving and transmitting a brightness image of a subject taken by a camera through a communication line, comprising: a distance information obtaining unit configured to obtain a distance from the camera to the subject; Area extraction means for determining an area of the subject located at a distance; object image extraction means for extracting a luminance image of the area from the luminance image to generate an object image; and transmitting the generated object image to the communication line. An image communication device, comprising: a control unit that transmits the image communication via the image communication apparatus. 2. The distance information acquiring unit includes: a plurality of cameras that photograph the subject; and a projector that is provided on an optical axis of at least one of the plurality of cameras and projects a predetermined pattern onto the subject. 2. The image communication device according to claim 1, further comprising: a distance image memory configured to store a distance image to the subject based on a captured image obtained by capturing the projected pattern with the plurality of cameras. 3. 3. A distance area extraction unit for extracting the area as a distance area for each pixel based on a threshold value, a threshold area memory for storing the threshold value, 2. The image communication device according to claim 1, further comprising an interpolation unit that interpolates pixel values of the distance area. 4. The object image extracting means includes: a luminance image memory for storing a luminance image having luminance information for each pixel of the luminance image; and extracting a luminance region corresponding to the distance from the luminance image as the object image. 2. The image communication apparatus according to claim 1, wherein the image communication apparatus has a luminance area extracting unit. 5. The image communication apparatus according to claim 1, wherein said control means transmits an audio signal together with said object image. 6. An area calculating unit for calculating a total area of an object image at the predetermined distance, wherein the control unit turns on the communication line when the total area is equal to or more than a predetermined threshold. 2. The image communication device according to claim 1, wherein the image communication device has a line control unit for executing an operation of transmitting an object image. 7. The control means includes: a background image memory storing a background image arranged behind the object image; and an image synthesizing unit for forming a transmission image obtained by synthesizing the background image and the object image to be transmitted. 2. The image communication device according to claim 1, wherein the image communication device has a configuration including: 8. The control unit, comprising: a receiving unit that receives the object image via the communication line; a background image memory that stores a background image arranged behind the received object image; The image communication device according to claim 1, further comprising: an image combining unit configured to form a received image obtained by combining the received object images; and a display unit configured to display the received image. 9. An area calculating unit for calculating a total area of an object image to be transmitted at the predetermined distance, wherein the control unit turns on the communication line when the total area is equal to or more than a predetermined threshold value. A line controller that executes the transmission operation of the object image, a receiver that receives the object image via the communication line, and a background image memory that stores a background image arranged behind the received object image. The image communication device according to claim 1, further comprising: an image combining unit configured to form a received image obtained by combining the background image and the received object image; and a display unit configured to display the received image. 10. The image communication apparatus according to claim 7, wherein said background image memory stores an arbitrary image stored by a sender. 11. The background image memory according to claim 7, wherein said background image memory stores color information such as a color palette having a predetermined number of gradations for forming said background image. The image communication device according to the item. 12. The image combining unit according to claim 7, wherein the image combining unit arranges the object image at a predetermined position when combining the image with the background image based on the position stored in the object image.
Item 10. The image communication device according to Item 8, Item 9, or Item 9. 13. The image synthesizing unit according to claim 1, wherein a predetermined image is read from the background image memory and synthesized at the time of synthesizing with the background image based on background image information stored in the object image. Item 7, Item 8, or Item 9
The image communication device according to the item.