JPH0832947A - Image communication equipment - Google Patents

Abstract

PURPOSE:To provide image communication equipment capable of displaying the abundant action of an image as suppressing the increment of the signal quantity of a transmission/reception signal. CONSTITUTION:Video information photographed by a television camera 100 is transmitted at every plural frames, and action information discriminated by an action discrimination processing part 103 is transmitted at every frame. An action processing part 111 generates an interpolation image by using the video information and the action information, and supplies an action to a display image, and a display image with abundant action can be obtained from a small quantity of signals.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image communication device for communicating image information such as a video telephone device and a video conference device.

[0002]

2. Description of the Related Art For example, Japanese Patent Publication No. 2-366887 discloses a communication method in which a signal quantity of image information is transmitted and received in a video telephone apparatus as much as possible. This communication method uses still image data for parts other than the mouth of the face of the communicator, and the mouth generates a moving image of the face by combining processing on the receiving side according to the audio information transmitted during communication. However, it is devised so that the information signal amount of the face moving image can be significantly compressed and the transmitted / received signal amount can be significantly reduced.

[0003]

However, in the videophone apparatus for transmitting and receiving such image information, the communicator moves the neck vertically to nod, or shakes the neck sideways to express "no". However, since the entire face is a still image, there is an inconvenience that such an operation is not transmitted to the other party.

An object of the present invention is to solve the above problems and to provide an image communication device capable of enriching the motion of an image while suppressing an increase in the amount of transmitted / received signals.
For example, in a videophone device or a videoconference device, it is intended to communicate the intention of the communicator and his / her facial expression to the other party so that the caller can expressively talk.

[0005]

One of the features of the present invention is as follows.
A television camera, an extracting means for extracting partial image information about a predetermined portion of the correspondent from the video information captured by the television camera, and a motion of the predetermined portion based on the extracted partial image information to determine motion information. An operation determining means for generating, a transmitting / receiving means for transmitting / receiving the video information in every plural frames and transmitting / receiving the motion information in every frame, and an image generation for generating a display image based on the received video information and the motion information. It is equipped with processing means.

Another feature of the present invention is based on a television camera, extraction means for extracting partial image information of a composing portion of a face of a correspondent from video information taken by the television camera, and based on the extracted partial image information. And a motion determining unit that determines motion of the component to generate motion information, a coding unit that codes the video information and the motion information, and transmits and receives the coded video information for every plurality of frames. A transmitting / receiving unit that transmits / receives motion information for each frame, a restoring unit that restores the video information and the motion information from the encoded signal received by the transmitting / receiving unit, and a display image is generated based on the restored video information and the motion information. The present invention is provided with an image generation processing means for performing the above.

Still another feature of the present invention is based on a television camera, extraction means for extracting partial image information relating to a predetermined portion of a correspondent from video information taken by the television camera, and the extracted partial image information. Transmission / reception is performed with an operation determination unit that determines the movement of the predetermined portion and generates operation information, with a transmission / reception unit that transmits / receives the video information for each predetermined frame and transmits / receives the partial image information and the operation information for each frame It is provided with a memory means for storing the image information and an image transforming means for transforming the video information read from the memory means according to the partial image information and the motion information.

[0008]

On the transmitting side, the video information is transmitted for each plurality of frames, and the motion information determined from the partial image information extracted from the video information is transmitted for each frame. Then, on the receiving side, an interpolation image is generated by using the video information or the partial image information and the motion information to give a motion to the display image, and a display image with rich motion is obtained from a small amount of signal.

[0009]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram of a videophone device according to an embodiment of the present invention. Reference numeral 100 is a television camera, 101 is an extraction processing unit, 102 is a control circuit, and 103 is an operation determination processing unit. This operation determination processing unit 103
A motion detection processing unit 104 and a motion processing unit 105 are provided. 1
Reference numeral 06 is an encoding processing unit, 107 is a multiplexing process, and 108 is a transmission / reception unit. 109 is a separation processing unit, 110 is a decoding processing unit, and 111 is an operation processing unit. This motion processing unit 11
1 includes a motion determination processing unit 112 and a motion processing unit 113. Reference numeral 114 is a combination processing unit, 115 is an image display unit, and 116.
Is a zoom control mechanism section of the television camera 100.

The television camera 100 photographs a correspondent or the like and converts it into video information. The extraction processing unit 101 pays attention to information such as the luminance signal of the video information, and the eyes, mouth,
Extract face parts such as nose, eyebrows, and ears. Control circuit 1
02 controls the zoom control mechanism unit 116 of the television camera 100 so that the image of the face is always a certain size. The motion detection processing unit 104 determines whether or not the extracted portion has moved vertically or horizontally by a predetermined amount or more. If you are moving in the vertical direction, it is determined that you have shaken your head vertically,
If it is moving in the lateral direction, it is determined that the head has been shook. If neither of the above applies, it is determined that there is no neck movement. The motion processing unit 105 performs signal processing for expressing the motion content determined by the motion detection processing unit 104, and the coding unit 106 codes the motion information and the video information.
The multiplexing processing unit 107 creates a transmission information signal by combining the encoded operation information and the video information, and the transmission / reception unit 108 transmits the transmission information signal.

The transmission / reception section 108 gives the reception information signal received from the other party to the separation processing section 109. The separation processing unit 109 separates the received information signal into respective parts. The decoding processing unit 110 decodes the separated reception information signal, gives video information to the synthesis processing unit 114, and gives motion information to the motion processing unit 111. In the operation processing unit 111, the operation determination processing unit 112 determines the operation content from the operation information, and the operation processing unit 113 performs the interpolation image generation processing according to the operation content. This interpolation image generation process is performed based on the image information stored in the synthesis processing unit 114. The synthesis processing unit 114 stores the restored video information, synthesizes the interpolated image information generated by the operation processing unit 113, stores the synthesized image information, and causes the image display unit 115 to display the display image signal.

Next, a specific example of the motion detection processing unit 104 in the motion discrimination processing unit 103 will be described with reference to FIG. Two
Reference numeral 01 is a coordinate value calculation processing unit, 202 is a center of gravity position calculation processing unit, 203 is a discrimination processing unit, and 204 is a memory.

The coordinate value calculation processing unit 201 includes an extraction processing unit 1.
The coordinate values of both eyes and mouth extracted in 01 are calculated. The center-of-gravity position calculation processing unit 202 obtains the coordinate value of the center of gravity of the triangle connecting the three coordinates calculated by the coordinate value calculation processing unit 201.
The determination processing 203 compares the coordinate value of the center of gravity calculated in the center-of-gravity position calculation processing 202 with the average value of the coordinate values of the center of gravity recorded in the memory 204 a few frames before, and determines the center-of-gravity position in the vertical or horizontal direction. When a movement of a predetermined amount or more is detected, a nod or yes motion is determined in the case of vertical movement, and a nod or no motion is determined in the case of horizontal movement. Then, the calculated coordinate value of the center of gravity is stored in the memory 2
Record in 04.

Next, another specific example of the motion detection processing unit 104 in the motion discrimination processing unit 103 will be described with reference to FIG. Reference numeral 301 is a coordinate value calculation processing unit, 302 is a determination processing unit, and 303 is a memory.

The coordinate value calculation processing unit 301 performs the extraction processing 10
The coordinate value of the nose extracted in 1 is calculated. Judgment processing unit 30
2 detects the movement of the nose by comparing the coordinate value with the average value of the coordinate values of the nose several frames before stored in the memory 303, and performs the same discrimination processing as described above. Then, the calculated nose coordinate values are recorded in the memory 303. This discrimination processing can also be performed by focusing on one of the eyes, mouth, eyebrows, and ears and detecting the movement of the coordinates.

FIG. 4 and FIG. 5 show an example of interpolation image generation processing of the operation processing unit 111 in this videophone device. In FIG. 4, (a) is a figure (image) viewed from the front of the communicator, and alternate long and short dash lines 501 and 502 are reference axes for performing rotation processing of the image. (B) is a figure (image) viewed from the side of the communicator, and a black circle 501 is the reference axis for performing rotation processing. Further, (c) is a figure (image) as seen from above the communicator, and a black circle 502 is the reference axis for performing rotation processing.

In the videophone apparatus, the movement of moving the neck vertically or shaking it horizontally must be expressed by the image change on the display screen. When the operation processing unit 111 on the receiving side receives the operation information indicating the operation content of the neck determined by the operation determination processing unit 103 on the transmitting side, the operation processing unit 111
The action determination processing unit 112 determines the content of the action, and the action processing unit 113 performs image processing so that the action matches the action. In other words, when the motion information when the neck moves vertically is received, the motion processing unit 113
As shown in (b), image processing is performed such that the image is rotated around the reference axis 501. Similarly, when the motion information when the head is shaken sideways is received, as shown in (c), image processing for rotating the image around the reference axis 502 is performed. By sequentially generating images when the neck faces front, diagonally downward, diagonally left, and diagonally right, image processing for displaying an image of nod and swing is executed. In order to express the movement smoothly, it is necessary to display the state in the middle of the movement using multiple images.
Here, the simplest operation expression example is shown.

FIG. 5 shows an example of image processing for expressing a motion of vertically moving a neck. An image of a face, which is shown by a thick line, which faces the front, and a face, which is shown by a thin line, which faces obliquely downward are shown. It shows how to generate. This image processing is processing for rotating the face (image) around the reference axis 501. Reference numeral 601 denotes an N frame of the face facing the front, which is indicated by a thick line, and 602 is an N + 1 frame after the processing of the face facing the diagonally downward, which is indicated by a thin line.

Image processing for the eyes will be described as an example. The image data b of the N + 1th frame (nodded) is the image data a of the Nth frame (front facing).
Generated in relation to. The coordinates of the image data b can be calculated from the coordinates of the image data a using the deflection angle of the neck and the three-dimensional shape model of the neck. That is, by performing three-dimensional mapping for each pixel of the two-dimensional image, it is possible to generate a two-dimensional image after operation. The other components of the face can be obtained by processing in the same manner. In this way, an image of the face that faces diagonally downward is created from the image data of the face that faces the front.

An image of a face facing diagonally left or diagonally right can also be created by the same method.

By combining the front-facing image and the two diagonally-downward images obtained by performing such processing, the neck is moved vertically to express a nod motion, or the diagonal left is displayed. It is possible to express the action of shaking the head sideways by combining the three images of the front and diagonally right.

In this example, the vertical and horizontal movements of the neck are discriminated and reproduced, but by applying this, it is possible to express the movement of tilting the neck.

FIG. 6 shows an example in which a stereoscopic screen is used as the image display unit 115 in order to make a realistic videophone device. Reference numeral 701 is a communicator, 702 is a television camera, 703 is a processing device, 704 is a communication network, 705 is a stereoscopic screen, and 706 is a stereoscopic screen drive unit. The image projection on the stereoscopic screen is performed by the projection device.

The television camera 702 photographs the subject on the transmitting side and converts it into video information. The processing device 703 includes a processing unit similar to that in the above embodiment, processes the video information in the same manner as in the above embodiment, encodes it, and then transmits it via the communication network 704. Further, the information sent from the other party is restored and displayed on the stereoscopic screen 705, and the stereoscopic screen drive unit 706 moves the stereoscopic screen 705.

FIG. 7 shows a specific example of operation control in the videophone device using the stereoscopic screen 705. In the case of a positive motion in which the head is vertically moved to nod, it is desirable that the partner's three-dimensional screen 705 also operates in the same manner. Therefore, when the motion determination processing unit (103) determines that the user has shaken his head vertically to nod and the motion information is transmitted, the motion processing unit (113) displays the stereoscopic screen 705 as shown in (a). A control signal is given to the three-dimensional screen drive unit 706 so as to drive to move in the vertical direction. Further, when the motion information which is determined to be a negative motion by shaking the head horizontally is transmitted, the motion processing unit (113) transmits the stereoscopic screen 705 as shown in (b).
Controls the three-dimensional screen drive unit 706 so that the person shakes his / her neck.

FIG. 8 shows another embodiment of the video telephone device according to the present invention. 901 is a television camera that generates video information, 902 is an encoding processing unit, 903 is an extraction processing unit, 904 is a memory, 905 is an operation determination processing unit, and 906.
Is a transmitting / receiving unit, 907 is a decoding processing unit, 908 is a memory,
Reference numeral 909 is a transformation processing unit.

A part of the transmission information is coded by the coding processing unit 902 of the video information obtained by photographing with the television camera 901, and is transmitted by the route 912 every plural frames. The extraction processing unit 903 extracts area information such as eyes, mouth, and face from the video information, calculates the coordinate values of the extracted eyes and mouth, and uses them as the vertices of a triangle to determine the barycentric value, as in the above-described embodiment. Is stored in the memory 904. Motion discrimination processing unit 905
Determines the content of the movement by comparing the newly obtained barycentric value with the average value of the coordinate values of the barycentric value stored several times before and stored in the memory 904. This operation information is transmitted for each frame by the route 914. The extracted area information (coordinate information) is also transmitted by the route 913.

As the received information, the video information received for every plural frames is transmitted to the decoding processing unit 907 by the route 916 to be restored, and stored in the memory 908. Deformation processing unit 909
Transforms the image of the region extracted from the video information stored in the memory 908 based on the motion information and the region information transmitted by the route 915. This transformation process
Similar to the embodiment described above, the operation (deformation) is performed by performing the three-dimensional mapping for each pixel of the two-dimensional image.
This is a process for generating a subsequent two-dimensional image. And display unit 9
10 displays the image after the transformation process.

It is also possible that the extraction target is not the area but the face constituent portion, and the image of the corresponding portion is read out from the image information stored in the memory 908 and the transformation processing is performed.

The result of the operation determination process according to the present invention can be used in place of the keyboard operation and the mouse click operation in the computer. A nod operation by moving the neck vertically can be used as an input operation equivalent to "yes", and a motion of moving the neck horizontally to negate can be used as an input equivalent to an input operation equivalent to "no".

[0032]

As described above, according to the present invention, motion information is transmitted and received between video information to generate an interpolated image so as to give a motion to a display image. Images can be displayed.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a videophone device according to the present invention.

FIG. 2 is a block diagram showing a specific example of a motion detection processing unit in FIG.

FIG. 3 is a block diagram showing another specific example of the motion detection processing section in FIG.

4 is an example of an interpolation image generation process executed by an operation processing unit in FIG.

FIG. 5 is an example of an interpolation image generation process executed by paying attention to an eye in the operation processing unit in FIG.

FIG. 6 is a block diagram showing another embodiment of the videophone device according to the present invention.

FIG. 7 is an operation explanatory diagram of the stereoscopic screen in FIG.

FIG. 8 is a block diagram showing still another embodiment of the videophone device according to the present invention.

[Explanation of symbols]

100 ... TV camera, 101 ... Extraction processing unit, 103 ...
Motion discrimination processing unit, 104 ... Motion detection processing unit, 105 ... Motion processing unit, 106 ... Encoding processing unit, 107 ... Multiplexing processing unit, 108 ... Transceiver unit, 109 ... Separation processing unit, 110 ...
Decryption processing unit, 111 ... Motion processing unit, 112 ... Motion discrimination processing unit, 113 ... Motion processing unit, 114 ... Combining processing unit.

Front page continuation (72) Inventor Takeshi Ichige 292 Yoshida-cho, Totsuka-ku, Yokohama-shi, Kanagawa, Ltd. Inside Hitachi Media Media Research Laboratories

Claims (9)

[Claims] 1. A television camera, an extracting means for extracting partial image information relating to a predetermined portion of a correspondent from video information captured by the television camera, and a movement of the predetermined portion is determined based on the extracted partial image information. Motion determining means for generating motion information, a transmitting / receiving means for transmitting / receiving the video information in every plural frames and transmitting / receiving the motion information in every frame, and a display image based on the received video information and the motion information. An image communication apparatus comprising an image generation processing unit for generating 2. A television camera, extracting means for extracting partial image information of a composing portion of a face of a correspondent from video information photographed by the television camera, and movement of the composing portion based on the extracted partial image information. And a motion determining unit for generating motion information, a coding unit for coding the video information and the motion information, and transmitting and receiving the coded video information for every plurality of frames and the motion information for each frame. A transmitting / receiving means for transmitting / receiving the image information, a restoring means for restoring the video information and the motion information from the encoded signal received by the transmitting / receiving means, and an image generation processing means for generating a display image based on the restored video information and the motion information. An image communication device characterized by being provided. 3. A television camera, extraction means for extracting partial image information relating to a predetermined portion of a correspondent from the whole video information taken by the television camera, and movement of the predetermined portion based on the extracted partial image information. An operation determination means for determining and generating operation information, a transmission / reception means for transmitting / receiving the whole video information for each predetermined frame and transmitting / receiving the partial image information and operation information for each frame, and storing the transmitted / received image information. An image communication apparatus comprising: a memory unit; and an image transforming unit that transforms video information read from the memory unit according to the partial image information and the motion information. 4. The image communication apparatus according to claim 1, wherein the extracting means extracts image information of an area including a predetermined portion of a correspondent from the video information. 5. The image communication apparatus according to claim 1, wherein the extracting means extracts image information of a predetermined portion of a correspondent from the video information. 6. The image communication apparatus according to claim 1, wherein the operation discriminating means has a memory means for storing information on the constituent parts. 7. The image communication device according to claim 1, wherein the motion discriminating means discriminates a vertical or horizontal movement of the neck. 8. The operation discriminating means according to claim 1, wherein the operation discriminating means obtains coordinate values of the extracted constituent parts and discriminates the operation of the constituent parts based on the change of the coordinate values. An image communication device comprising: 9. The method according to claim 1, wherein the extracting means has means for extracting image information of both eyes and a mouth of a correspondent, and the motion determining means has the extracted both eyes. And means for obtaining the coordinate values of the mouth, means for obtaining the coordinate values of the center of gravity of the area surrounded by the coordinate values, and means for determining the movement of the neck based on the change in the coordinates of the center of gravity. Image communication device.