EP0581593B1

EP0581593B1 - Image communication apparatus and method therefor

Info

Publication number: EP0581593B1
Application number: EP19930306003
Authority: EP
Inventors: Yasuji C/O Canon Kabushiki Kaisha Hirabayashi
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 1992-07-31
Filing date: 1993-07-29
Publication date: 1999-09-29
Anticipated expiration: 2013-07-29
Also published as: DE69326586D1; DE69326586T2; US5666154A; EP0581593A1; JPH0654310A; JP3526057B2

Description

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image communication apparatus and method therefor, and more particularly to a moving image transmission apparatus adapted for use in a television telephone apparatus or a television conference system.

Related Background Art

In such moving image communication system, it is preferable to be able to transmit as many images as possible per second, while maintaining satisfactory image quality in each image frame. In the conventional moving image encoding method, since the available amount of codes is by far smaller than the entropy of the image, it has been customary to maintain the image quality in a state of semi-moving image in which the number of image frames per second is limited, or to maintain a sufficient number of image frames per second for the moving image, while sacrificing the image quality. Stated differently, in the conventional method, the image quality and the number of image frames per second are in the trade-off relationship.
Also in the television telephone or the like, there is proposed a configuration of automatically detecting a part of the object, for example the face of a person, and preferentially allotting the codes to the part, thereby enabling to transmit a sufficient number of image frames per second for a moving image, while securing the image quality in a part of the image frame.
However, in such conventional method of preferentially assigning the codes to the face portion of the object, there may eventually result an error in the automatic detection of the face portion, and such face portion may not necessarily be the most important part in the image frame. For example the transmitting person or the receiving person may wish to observe the background or the article held by the person, rather than the person himself, and the above-mentioned conventional method is unable to cope with such complex and varying requirement.
In consideration of the foregoing, the present invention is concerned with providing an image communication apparatus and method potentially capable of resolving such drawbacks.
The present invention is also concerned with providing an image transmission method capable of sufficient data compression while maintaining the image quality, and an apparatus adapted therefor.
EP-A-0,449,555 discloses an image communication apparatus in which a portion of a generated image can be detected and prior to transmission the detected portion encoded with encoding parameters which are different from the encoding parameters used for the rest of the image.
Thus according to one aspect of the present invention, there is provided an image communication apparatus as set out in claim 1 or claim 6.
In accordance with a second aspect of the present invention there is provided an image communication method as set out in claim 11.
An embodiment of the present invention provides an image communication method capable of satisfactory moving image transmission, and an apparatus adapted therefor.
Another embodiment of the present invention provides an image communication apparatus which is easy to use.
Still other advantages of the present invention, and the features thereof, will become fully apparent from the following description of embodiments, to be taken in conjunction with the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is an external view of an embodiment of the present invention;
Fig. 2 is a block diagram of a basic encoding circuit of said embodiment;
Fig. 3 is a schematic view of a mask level #1;
Fig. 4 is a schematic view of a mask level #2;
Fig. 5 is a schematic view of a mask level #3;
Fig. 6 is a schematic view of a mask level #4;
Fig. 7 is a schematic view of a mask level #5; and
Fig. 8 is a chart showing mask level assignment to significant and non-significant areas.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Now the present invention will be clarified in detail by preferred embodiments thereof shown in the attached drawings.
Fig. 1 is an external view of an embodiment of the present invention applied to a television telephone set, and Fig. 2 is a schematic block diagram of an encoding circuit thereof.
Fig. 1 illustrates a television telephone set 10, with a monitor image frame 12, which displays a received image in Fig. 1. The display can be switched between the input (transmitted) image and the received image by the operation of a keyboard 14. Naturally the input image and the received image may be displayed in different windows, by the use of a window display system.
In the keyboard, there is provided a television camera 17. In the present embodiment, within the display of the received image on the monitor 12, an area to be received with high image quality or low image quality is designated by a mouse 16. A cursor 18 moves in the displayed image, in linkage with the movement of the mouse 16, thereby designating an arbitrary area 20. For such designated area, the level of significance is designated from the keyboard 14. In order to regulate the image quality of encoding in the unit of each block wholly or partially contained in the designated area 20, the information on said area and significance is transmitted from the receiving side to the transmitting side. The transmission of the information is executed, through a communication channel to be explained later, in the course of image data transmission from the transmitting side to the receiving side.
Such information of designation is transmitted to the partner of communication, together with image information to be transmitted thereto. The terminal of the partner transmits the image information by encoding a significant area with high image quality but a non-significant area with low image quality, while maintaining a constant amount of codes in the entire image frame.
The mouse 16 may naturally be replaced by another pointing device, such as a track ball, a joy stick or a light pen.
The area designation may be achieved by designation of a significant area, designation of a non-significant area, or designation of level of significance to each designated area.
Fig. 2 shows an example of the encoding circuit, employing frame-to-frame difference encoding and DCT transformation, with a square block of 8 x 8 pixels as the unit of processing.
An input terminal 40 enters an image signal to be encoded, in the unit of the encoding block. A subtractor 42 subtracts estimated values from the pixel signals entered from the input terminal 40, and provides the predicted error in the frame-to-frame difference encoding.
A DCT circuit 44 effects discrete cosine transformation on the predicted error from the subtractor 42, thereby providing DCT coefficient data. A masking circuit 46 masks the DCT coefficient data from the DCT circuit 44 according to a mask level signal 47. The details of said masking circuit 46 will-be explained later. A quantization circuit 48 quantizes the output of the masking circuit 46 with predetermined quantizing steps. An output terminal 50 sends the output of the quantization circuit 48 to a succeeding transmission unit 60, composed for example of a variable length encoding circuit, an error correction encoding circuit, and a transmission buffer. The unit 60 is connected, through a communication control unit (CCU), to a totally doubled public communication channel 63 (for example a digital channel such as ISDN).
A transmission/reception unit 64 is provided, in addition to the aforementioned transmission unit 60, with a reception unit 62 which separates, from the received data, data indicating the significant area, for supply to a controller 65. The controller 65 releases the mask level signal 47 according to a procedure to be explained later, based on said data indicating the significance.
There are also provided an inverse quantization circuit 52 for inverse quantization of the output of the quantization circuit 48; an inverse DCT circuit 54 for inverse discrete cosine transformation on the output of the inverse quantization circuit 52; an adder 56 for adding a predicted value to the output of the inverse DCT circuit 54, thereby providing an output corresponding to a locally decoded value; a frame memory 58 for storing the output of the adder 56 for a frame period and providing pixel signals to be supplied, as the predicted value, to the subtractor 42 and the adder 56; a reception circuit 68 for decoding the received image data and converting the decoded data into a format suitable for display on a television monitor 69; and an adder 71 for superposing the data of the cursor 18 shown in Fig. 1 on thus converted image data.
The masking circuit 46 is provided with five mask levels #1 - #5, shown in Figs. 3 to 7, selectable by the mask level signal 47. Figs. 3 to 7 show the transmitted ones and masked ones, among 16 transformation coefficients released from the DCT circuit 44, in the block unit of 8 x 8 pixels. The coefficients in the hatched area are masked.
More specifically, in the mask level #1, all the outputs from the DCT circuit 44 are supplied to the quantization circuit 48, as shown in Fig. 3. In the mask level #2, as shown in Fig. 4, 3/4 of the frequency region are masked, whereby the resolving power of the reproduced image becomes 1/2 of that of the original image, but the final amount of encoded image data is reduced. In the mask level #3, as shown by the hatched area in Fig. 5, 3/4 of the encoding area of the mask level #2 is further masked, whereby the resolving power of the reproduced becomes 1/4 of that of the original image. In the mask level #4, as shown by the hatched area in Fig. 6, masking is executed except for the DC component. This corresponds to the transmission of the average value of the pixels in the block. In the mask level #5, all the outputs of the DCT circuit 44 are masked. In this situation, the encoding is executed in such a manner that the reproduced image is same as the preceding frame, equivalent to the freeze state.
In the following there will be explained the method of determining the mask level for each of the plural blocks constituting the image frame, in the controller 65, with reference to Fig. 8. In this embodiment, the image frame is classified into two levels of significant and non-significant. However, there are preferably employed three or more levels, or there may also be employed continuous levels. According to the data from the receiving side, indicating the area of significance, the controller 65 calculates the proportion of the blocks belonging to the significant area to all the blocks constituting the entire image frame, and determines the mask level of the blocks belonging to the significant area and that of the blocks belonging to the non-significant area, according to the chart shown in Fig. 8.
For example, if the proportion of the blocks belonging to the significant area is 40%, the mask level #1 is applied to the blocks belonging to the significant area, while the mask level #4 is applied to the blocks belonging to the non-significant area. In this case the resolving power of the reproduced image is same as that of the original image in the significant area, but becomes 1/8 of that of the original image in the non-significant area.
Now there will be explained the function of the entire circuits shown in Fig. 2. The intput terminal 40 receives the image signal to be encoded, in the unit of a block. The subtractor 42 subtracts the predicted value, supplied from the frame memory 58, from the image signal entered from the input terminal 40, and sends the predicted error to the DCT circuit 44.
The DCT circuit 44 effects discrete cosine transformation (DCT) on the predicted error from the subtractor, in the unit of a block, and supplies the masking circuit 46 with the DCT coefficient data. The masking circuit 46 masks the transformation coefficient data of each block to be supplied to the quantization circuit 48, according to the mask level signal 47, which is determined by the controller 65 as already explained in relation to Fig. 8, based on the data transmitted from the receiving side and separated by the reception unit 62. The quantization circuit 48 quantizes the DCT coefficient data from the masking circuit 46, with predetermined quantizing steps, and the output of the quantization circuit 48 is supplied from the output terminal 50 to the succeeding circuits.
The output of the quantization circuit 48 is also supplied to the inverse quantization circuit 52, which effects inverse quantization on the output with the same quantization steps as in the quantization circuit 48, and releases the representative value of the DCT coefficients. The inverse DCT circuit 54 effects inverse discrete cosine transformation on the output of the inverse quantization circuit 52, and the adder 56 adds the predicted value (output of the frame memory 58) to the output of the inverse DCT circuit 54. The output of the adder 56 is a locally decoded value, which is stored, as the predicted value for the next frame, in the frame memory 58. The image data read from said frame memory 58 are supplied, as the predicted value, to the subtractor 42 and the adder 56.
In the above-explained embodiment, the masking level for the DCT coefficients is regulated for the designated area, but similar effects can be obtained also by regulating the quantizing steps, or more specifically, adopting finer quantizing steps as the significance of the area increases. It is also possible to form a color image in the significant area and a monochromatic image in other areas, or to combine such operation mode with the above-mentioned modes.
The foregoing embodiment utilizes the discrete cosine transformation as encoding, but the present invention is applicable also to other exchange encoding methods or spatial area encoding methods.
Fig. 8 explains a case of dividing an image frame into a significant area and a non-significant area, but there may exist two or more significant areas and two or more non-significant areas in mixed manner. Also it can be naturally expanded to a case in which the level of significance is designated in each of the designated areas.
Also in the foregoing embodiment, the image quality of each area in the received image is designated in the image receiving side, but the system may also be so designed that the image transmitting side designates the image quality of each area in the transmitted image. This will provide an effect of intentionally blurring an image portion which the transmitting side wishes to hide.
As will be easily understood from the foregoing, in the above-explained embodiment, the receiving side designates the significant area and sends the data indicating such designated area to the image transmitting side. Consequently it is rendered possible to transmit a large number of images frames per second, sufficient for obtained a moving image, and to transmit the significant area with high image quality. Thus a smoothly moving image can be transmitted with sufficient image quality, even through a transmission channel of a low bit rate.
The encoding method employable in the present invention is not limited to that in the foregoing embodiment but can be modified in various manners.
Also the present invention is not limited by the foregoing embodiment but is subject to various modifications within the scope of the appended claims.

Claims

An image communication apparatus comprising:

receiving means for receiving image data encoded by encoding means (44, 46, 48), the encoding means comprising

transforming means (44) for orthogonally transforming the image data to be transmitted; and

quantization means (48) for quantizing said orthogonally transformed image data with predetermined quantizing coefficients, the apparatus further comprising

designation means (16) for designating a specified portion of image data received by said receiving means, the received image data being received through a communication line; and

transmission means (64) for transmitting information through the communication line relating to the specified portion designated by said designation means to the transmitting side,

and characterised in that said encoding means includes mask means (46) for masking the output of said transforming means and operable at a plurality of levels, and said information is adapted to vary the masking ratio of said mask means, by defining one of the levels in accordance with the ratio of the area of the designated portion of the image to the remainder of the image.
Apparatus according to claim 1 further comprising:

display means (12) for displaying received image data and the specified portion designated by said designation means.
Apparatus according to claim 2, wherein said display means is a television monitor.
Apparatus according to any preceding claim, wherein said designation means is a mouse.
Apparatus according to any preceding claim, wherein said information relating to said specified portion is transmitted to said transmitting side through a totally doubled channel.
An image communication apparatus comprising:

encoding means (44, 46, 48) for encoding image data to be transmitted over a communication line to a recipient apparatus, the encoding means comprising

transforming means (44) for orthogonally transforming the image data to be transmitted; and

quantization means (48) for quantizing said orthogonally transformed image data with predetermined quantizing coefficients; the apparatus further comprising

control means (64) for receiving from the recipient apparatus data indicating a specified portion of the total image data to be encoded and transmitted,

and characterised by mask means (46) adapted to mask the output of said transforming means; said masking means being operable at a plurality of mask levels, said control means being adapted to define one of the levels of said mask means in accordance with the ratio of the area of the designated portion of the image to the remainder of the image.
Apparatus according to claim 6, wherein said data indicating said specified portion are received through a totally doubled channel.
Apparatus according to either of claims 6 or 7, further comprising:

generation means (17) for generating image data to be transmitted.
Apparatus according to claim 8, wherein said generation means is a television camera.
An image transmission system comprising image communication apparatus as claimed in any one of claims 1 to 5 in combination with image communication apparatus as claimed in any one of claims 6 to 9.
A method of communicating an image comprising:

encoding image data by orthogonally transforming the image data to be transmitted and quantizing said orthogonally transformed image data with predetermined quantizing coefficients;

receiving the encoded image data, designating a specified portion of received image data received through a communication line; and

transmitting information through the communication line relating to the specified portion designated by said designation means to the transmitting side,

and characterised in that said transmitted information changes a masking level of the orthogonally transformed image data in accordance with the ratio of the area of the designated portion of the image to the remainder of the image.
A method according to claim 11, further comprising:

displaying the received image data and the specified portion designated by said designation means.
A method according to claim 12, wherein the received image data and the designated portion are displayed on a television monitor.
A method according to any one of claims 11 to 13, wherein said designation is carried out using a mouse.
A method according to any one of claims 11 to 13, wherein said information relating to said specified portion is transmitted to said transmitting side through a totally doubled channel.
A method according to any one of claims 11 to 15, wherein said image data is generated by a television camera.