JP2001313914A - Dynamic image encoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a dynamic image encoder which can reduce the amount of information to be transmitted when background brightness fluctuates, and can automatically display another background when the background is not desirable to show. SOLUTION: The dynamic image encoder is provided with a means 115 to encode a map for distinguishing a motion region and a background region about each picture of a dynamic image signal, a means 106 to encode the image of the motion region distinguished on the map, and a means 120 to output the map and the image. Furthermore, the dynamic image encoder bas a means 116 to read an another background image from a storage means in which the another background image instead of the true background is accumulated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture coding apparatus used for transmitting a moving picture such as a videophone.

[0002]

2. Description of the Related Art In the coding of a moving image signal in a video conference system or the like, a prediction adaptive to the characteristics of a region, such as using motion compensated inter-frame prediction for a moving region and background prediction for a background in recent years. Coding employing the scheme has been performed.

[0003] However, the determination of the area for performing these operations is as follows.
Conventionally, the input frame is locally examined and divided into a case where the luminance change is large and a case where the luminance change is small.
When it is small, a simple decision such as the background was made.In other words, rather than changing the prediction method adaptively between the moving area and the background, it is better to change the prediction method between the area where the luminance change is large and the area where the luminance change is small. correct. This means that, for example, if the luminance value of the background changes due to fluctuations in illumination even if the state of the background does not change, that area is treated as the background. For this reason, the conventional method does not differentiate between the true background and the moving region, and improvement of the coding efficiency cannot be expected without solving this point.

[0004] On the other hand, from the standpoint of using the apparatus, it is often thought that the differentiation based on the concept of the background and the moving region is more meaningful than the differentiation based on the price of the luminance change. For example, it is clear from the two points that (1) the background sometimes does not include information necessary for the receiving side, and (2) there is a case where the transmitting side does not want to show its background to the other party. . However, when trying to satisfy these requirements, if the conventional background determination rule based on the change in luminance is used, an area that is erroneously determined as the background appears even in the moving area, and the above-described problem occurs. become.

[0005]

As described above, in the conventional method, the amount of information increases when the background luminance fluctuates.
In addition, there has been a very inconvenient problem in an image transmission apparatus such as a videophone, in which an operator cannot perform an operation of hiding the background himself, so that the user does not want to see the background.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a moving picture coding apparatus capable of reducing the amount of information to be transmitted and automatically displaying another background when it is not desired to show the background to the other party. And

[0007]

According to the present invention, a first pixel value indicating a moving area and a second pixel value indicating a background area are determined for each screen of a moving image signal in order to distinguish between the moving area and the background area. A first encoding unit (115) for encoding the included map (113), a second encoding unit (106) for encoding an image of a moving region identified by the map, and the first code Means for outputting a map encoded by the encoding means and an image encoded by the second encoding means (1
19) A moving picture encoding apparatus comprising: (116) a means for reading out another background image from a storage means in which another background image as a substitute for a true background is stored. An encoding device is provided.

In addition, according to the present invention, a map (113) including a first pixel value indicating a moving area and a second pixel value indicating a background area for representing contour information of the moving area for each screen of the moving image signal is encoded. First encoding means for converting (115)
A second encoding unit (106) for encoding an image signal of a moving region distinguished by the map, and a map encoded by the first encoding unit and a code encoded by the second encoding unit. Means (11) for outputting a converted image signal
9) A moving picture encoding apparatus comprising: (11) a means for reading out another background image from a storage means in which another background image as a substitute for a true background is stored. An encoding device is provided.

[0009]

According to the present invention, the moving region and the other region (background) are separated from each other, and information about the background is discarded. The moving region is combined with another arbitrarily set background. On the transmitting side, there is no change in the lighting state, which does not cause an increase in the amount of information for transmission. Further, it is not necessary to transmit a true background that the user does not want to transmit, and instead, another background set arbitrarily on the transmitting side can be combined and displayed on the receiving side.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a basic block diagram of a transmission-side image transmission device according to the present invention.

An input image signal is sequentially stored in a frame memory 101 for each frame, then cut out into unit blocks for encoding in a block forming circuit 102, and sent to a significant block determining circuit 103. The signal of the frame memory 101 is also sent to a map creation circuit 113 and a background memory 114 which will be described in detail later. The background memory 114 is a background memory whose contents are updated by an input frame by detecting the outline of the moving area and treating the outside as the background.

The map creating circuit 113 separates the input frame into a background and a moving area by comparing the background stored in the background memory 114 with the newly input frame, and creates a map indicating separation information. I do. The map created here is used for motion detection, significant block determination, and conditional pixel replenishment, and this information is encoded by a contour encoding circuit 115, which outputs received data at the time of decoding on a receiving side (not shown). Or to output the contents of background memory. An encoding method using this map can also be used in the encoding circuit 106.

The significant block determination circuit 103, which has received the map information, determines whether the input block is outside or inside the outline or whether the block extends over the outline. The block is regarded as an invalid block, and the transmission to the next stage is stopped. In other cases, this block is regarded as a significant block and sent to the next difference circuit 104 and motion detection circuit 107. The motion detection circuit 107 to which the significant block and the map have been input stores the contents of the previous frame.
, And an optimum motion vector is detected. In this case, as shown in FIG. 5, after clearing both the pixel value of the pixel corresponding to the background portion of the input block to 0 in the input block and the search destination block, the error is reduced. Evaluate and perform matching. This can prevent the detected motion vector from being affected by the background.

The detected motion vector is supplied to a multiplexing circuit 11.
9 and to the variable delay circuit 108. The variable delay circuit 108 is a frame memory (1) 111
A block having a shift corresponding to a motion vector is sent to the difference circuit 104 and the block memory 122, and the difference circuit 104 calculates a prediction error between the input block and the motion compensation prediction block. Among the prediction errors, an error at a position corresponding to the moving area of the input block is evaluated by the conditional pixel replenishment determination circuit 105.

If this error is small, this block is considered to be subjected to conditional pixel replenishment, and a signal indicating this is sent to the multiplexing circuit 119 and the block memory 122, and the contents of the block memory are sent to the frame memory by this signal. (2) The process proceeds to step 112, and the transmission of the prediction error to the next stage is stopped.

If the error is large, the prediction error is sent to the encoding circuit 106 and encoded. The encoded data is sent to the multiplexing circuit 119 and the multiplexing circuit 109
And is stored locally in the frame memory (2) 112. The frame images stored in the frame memory (2) 112 are stored in the frame memory (1) 1 at the frame timing.
11 and is referred to at the time of motion compensation.

On the other hand, the background is transmitted only when communication starts or when there is a background transmission request from the transmitting side. Background memory 11
No. 4 selects only the background portion from the input frame as described above and continuously updates it. External storage device 116
Is a floppy (registered trademark) disk, cassette tape,
It consists of a compact and easily replaceable memory such as a rewritable storage device such as an IC card, a VTR and a DAT or a non-rewritable storage device such as an optical card and a CD-ROM, and a handler thereof, and serves as a true background Another background is stored. In the case of a rewritable memory such as a floppy disk, the user can easily compose the contents using a personal computer or a video device. By using this, it is also possible to use an image in which the person on the transmitting side is best captured or an image in which the background is best prepared as an alternative background.

The selection circuit 117 determines which of the background memory 114 and the external storage device 116 is to be transmitted as a background, according to the setting on the transmission side. The selected background information is encoded by the background encoding circuit 118. In this case, if the background information is encoded in advance, the background encoding circuit 118 is unnecessary.

The moving area information from the multiplexing circuit 119 and the background information from the background encoding circuit 118 are sent to the output switching circuit 120. The output switching circuit 120 switches to background output at the start of communication and at the time of a background transmission request on the transmission side,
At other times, the moving area output is in the connected state.

As an example of the usage of the background transmission request, the receiving side selects the background output from the external storage device 116 when receiving a call, switches the selection to the background output from the background memory 114 after confirming the other party, and issues the background transmission request. A kind of security function is possible, such as putting out and entering a normal call. In this case, it is possible to easily perform control such that the moving area is not transmitted from the time of the incoming call until the second background output request. Further, a control circuit capable of automatically performing the above operations can be provided.

Such an operation is effective when the caller does not want to show his or her background or the entire screen. In such a case, the transmission of the image should be stopped. However, in that case, the entire screen becomes black and the meaning of the TV phone is lost. Also, the person wants to send it out, but he can not respond to the request to hide only the background. In addition, when the user leaves the place after holding a conversation for a call or the like, there may be a request to display another image at a time to make the other party wait. In such a case, this embodiment can easily meet these demands.

The determination as to whether the signal is a moving area signal or a background signal can be made by a method such as writing the information in a frame header attached to each frame by the transmitting side. In this case, the output switch 120 starts reading the moving area signal when the background output request is output with reference to FIG. 6, and sets the moving area transmission selection bit to the background transmission selection bit when the next frame header comes. Rewriting, and then adding and sending background information and sending the frame memory (2) 11
2 Send write inhibit information. When the next frame header of the moving area signal comes again after the background transmission ends, the moving area transmission is restarted. In this example, the moving area information 2 and the moving area information 3 are not transmitted.

FIG. 7 is a block diagram showing the configuration of the background memory shown in FIG. Reference numeral 701 denotes a frame memory (1) 70 for sequentially storing two adjacent frames for calculating a difference between frames.
2 and a frame memory (2) 703, and an update control circuit 70 for appropriately setting the interval between sets of two frames.
Frame control circuits 702 and 703 for performing frame control in response to an operation end signal from the CPU 9; 702 and 703, respectively, a frame memory for storing a later input frame and a previously input frame image of the two frames;
Is a difference circuit for calculating the difference between the two frames, 7
05 is an absolute value circuit for obtaining the absolute value of the difference, 706 is a frame memory (3) for storing the absolute value of the inter-frame difference, and 707 is a contour detection for detecting a rough outline of a person or the like from the inter-frame difference image. A circuit 708 for generating a map indicating an update position as shown in FIG. 8 from the output of the contour detection circuit 707;
710, an update control circuit for controlling whether to update the contents of the background frame memory with the contents of the frame memory (2) 703 or to save the contents of the background frame memory 701 with reference to the created map; Is a frame memory for storing the background, and 711 is a delay circuit that performs a delay corresponding to the operation time and writes data into the background frame memory 710 pixel by pixel only when updating is permitted by a signal of the update control circuit. . The operating principle of this background memory will be briefly described with reference to FIGS.

FIG. 9 shows an example of the outline of a person obtained from the difference between frames. As shown in the figure, the outline includes a moving area (indicated by diagonal lines) of the current input frame and a background part which is newly visible as compared with the previous input frame. The background that was not updated cannot be updated, but the part that was not updated the first time may be updated from the second time onward due to the movement of the person,
As shown in FIG. 10, by updating a portion newly included outside the outline of the inter-frame difference, only the background can be accumulated in the background memory.

For this reason, as shown in FIG. 11, it is desirable that the two frames for calculating the difference between the frames be two frames that are temporally close to each other so that the area of the background area inside the outline is reduced. The interval between the sets of two frames can be arbitrarily set according to the desired update interval and calculation time. In FIG. 11, frames from t1 + # 1 to t2, frames from t1 to t1 + # 1, and frames from t2 to t2 + # 2 are generally dropped. However, # 1 and # 2 may be frame intervals. Frame (t1) and frame (t
1 + △ 1), an inter-frame difference image 1 is obtained from the difference between the frame (t2) and the frame (t2 + △ 2), and the outside of the outline is updated each time. You. In FIG. 10, the shaded portion rising to the right is (t)
The hatched portion falling to the right between (1, t1 + △ 1) is (t2, t2).
+ $ 2) respectively indicate the updated portions.
In the above description, all the frames or the frame memories have been described, but the fields or the field memories may be used.

FIG. 12 is a block diagram showing an example of the configuration of the contour detection circuit of the background memory shown in FIG. Reference numeral 801 denotes a frame memory in which a difference between frames is stored.
3 is a block circuit for extracting strips of L × H (H is the length of one side of a frame) as shown in FIG. 3 in the vertical and horizontal directions. Reference numeral 802 denotes a histogram creation circuit that takes a histogram in the short side direction of a strip. Reference numeral 803 denotes an end point detection circuit that searches the histogram while comparing the histogram with a certain threshold value from both ends of the strip, and outputs the coordinates of the point that first exceeds the threshold value as the end point of the contour.

Next, FIG. 2 shows a basic block diagram of the receiving side corresponding to the transmitting side shown in FIG. The signal from the transmission side is temporarily stored in the buffer memory 201, and is stored in the selection circuit 20.
In step 2, the output destination is selected according to whether the signal which has become a moving area signal or a background signal.

When the background side is selected, the signal is decoded by the background decoding circuit 212, stored in the background memory 213, and the contents of the background memory 213 are always output to the synthesizing circuit 214. On the other hand, when the moving area is selected, the signal is separated by the separating circuit 203 into contour information, coding information in block units, motion vector information, and conditional pixel replenishment determination information. 204, the variable delay circuit 210, and the write control circuit 211. The encoded information is decoded by the decoding circuit 204 for each block, and
Is output to On the other hand, the motion vector information is
10 and a block corresponding to the motion vector is selected from the frame memory (2) 209,
5 is output. The addition circuit 205 adds the two and outputs the result to the write control circuit 211.

After the contour information is decoded by the contour decoding circuit 206, it is sent to the map creating circuit 207, where the same map as that on the transmitting side is created. This map is sent to the write control circuit 211 and used for calculating the address of the significant block and sent to the synthesizing circuit 214. Here, the moving area information includes the moving area information from the adding circuit 205, and the background area includes the background information. Background information from the memory 213 is selected and a frame is synthesized. This composite image is output.

Write control circuit 211 Map creation circuit 20
7, the address of the significant block is obtained from the map input from the control unit 7, and the variable delay circuit 210 is used when the block is conditionally replenished with reference to the conditional pixel replenishment determination information.
The input from the adder 205 is transferred to the frame memory (2) 209 in synchronization with the frame if the input from the adder 205 is not supplemented.

In the above-described embodiment, the block configuration of the transmission side and the reception side which form a pair has been described. Next, another embodiment of the transmission side according to the present invention will be described with reference to FIG. The coding part is basically the same as that of the embodiment of FIG. In the present embodiment, there are two TV cameras, a TV camera (1) 301 and a TV camera (2) 302, which are installed so that they have the same horizontal coordinates and the optical axes are parallel. The input from these two TV cameras is a distance measurement circuit 305
, Corresponding points are matched, and for example, distance measurement of each point is performed by a three-dimensional measurement method as described in Chapter 17 of “Image Processing Handbook (Morio Onoe)”. Using this distance information, the map creation circuit 306 creates a map indicating information on the moving area and the background area. Further, in this embodiment, the background information is never transmitted, and a fixed pattern specified by the background code generation circuit 322 is displayed on the receiving side background.

FIG. 4 is a diagram showing an embodiment of the receiving side corresponding to the receiving side in FIG. The background code sent from the transmitting side is sent to the background pattern generation circuit 412, and the background pattern generation circuit 412 sets the background corresponding to the background code to RO.
A background pattern is selected from M413 to generate a synthesis circuit 4
14 is output. For example, as shown in FIG. 14, a background code is set to a plain gray background, a code 01 is set to a blue solid background, a code 10 is set to a black-and-white striped background, and a code 11 is set in advance as shown in FIG. In this case, the background of the predetermined landscape is selected.

Although the two embodiments (FIGS. 1 and 2 or FIGS. 3 and 4) differ in the method of creating the map and the method of displaying the background and designating the background, the combination is freely changed and implemented. it can.

For the encoding part, another encoding method can be used. For example, removing the motion compensation from the configuration in FIG. 1 does not impair the features of the present invention at all.

Further, an embodiment without the external storage device 16 in FIG. 1 and an embodiment in which a moving image is stored in the external storage device are possible. Further, in the third embodiment, as a method used for creating a map, an embodiment using a sensor such as an ultrasonic wave or an infrared ray instead of using a stereo image is also possible. To summarize, the means for separating the background and the same area are as follows: (1) A method of configuring a background memory that stores only the true background using the outline information of the moving area, and comparing the input frame with the contents of the background memory (2) ) Method using sensors such as ultrasonic waves and infrared rays (3) A method of performing distance measurement using stereo images is conceivable.

FIG. 15 shows another embodiment on the transmitting side.

In this embodiment, similarly to the embodiment of FIG. 1, an input frame fetched into a frame memory 901 is divided into blocks by a blocking circuit 902 and a background memory 90.
In step 4, the background is updated. By comparing the background with the input frame, the map creation circuit 903 creates a map indicating the separation information between the moving area and the background area. Apart from this, an external storage device 90 storing background information is stored in a frame memory 906.
From 5, a moving image is input for each frame. The external storage device 905 is an external storage device that provides an input of the same format as that of the input of the frame memory 901, and may be a storage device such as a VTR. This input is also a blocking circuit 90
Blocked at 7. The two blocked inputs and the map created by the map creation circuit 903 are sent to the background replacement circuit 908.

The background replacement control circuit 914 selects whether the transmitting side inputs the background from the camera, the external storage device 905, or the entire screen from the external storage device 905, and sends the control information to the background replacement 908. If this information specifies that the background is to be output from the external storage, the map is designated as the background among the input blocks from the blocking circuit 902 input to the background replacement circuit 908. The resulting pixel is replaced with the pixel at the same position in the input block from the blocking circuit 907 and output to the encoding circuit 909. The subsequent encoding operation is the same as the operation from the significant block determination circuit 103 in the embodiment of FIG. In this embodiment, there is no need for the receiving side to make any change to the conventional method.

As described above, in the above embodiment,
(1) Since the true background is not sent, no extra information is generated due to a change in the luminance of the background, etc. (2) If you do not want to show the true background to the other party, display another background instead of this background By displaying the true background while keeping the image natural, it is possible to hide the true background while keeping the image natural. (3) Confirm the point of time when the conversation is held and the other party The performance can be improved such that another image can be displayed at the same time.

[0040]

That is, according to the moving picture coding apparatus of the present invention, the amount of information to be transmitted can be reduced, and the background portion that the user does not want to transmit can be hidden or changed freely by the operator himself. is there.

[Brief description of the drawings]

FIG. 1 is a basic block diagram of a transmission side according to the present invention.

FIG. 2 is a basic block diagram of a receiving side according to the present invention.

FIG. 3 is a basic block diagram of another transmission side according to the present invention.

FIG. 4 is a basic block diagram of another receiving side according to the present invention.

FIG. 5 is a diagram showing that matching is performed after a background portion is cleared to 0 in motion vector detection.

FIG. 6 is a diagram showing timing after a background transmission request.

FIG. 7 is a diagram showing an embodiment of a background memory in FIG. 1;

FIG. 8 is a diagram showing an example of a map created by the map creation circuit in FIG. 7;

FIG. 9 is a diagram showing an outline of a difference between frames.

FIG. 10 is a diagram showing how a background is updated in the background memory shown in FIG. 7;

FIG. 11 is a diagram showing a temporal positional relationship of frames in the background memory shown in FIG. 7;

FIG. 12 is a diagram showing a configuration example of a contour detection circuit in FIG. 7;

FIG. 13 is a diagram showing the operation of the contour detection circuit.

FIG. 14 is a diagram showing an example of correspondence between a background code and a background.

FIG. 15 is a basic block diagram of still another transmitting side according to the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 101 Frame memory 102 Blocking circuit 103 Significant block determination circuit 104 Subtractor 105 Conditional pixel replenishment determination circuit 106 Coding circuit 107 Motion detection circuit 108 Variable delay circuit 109 Decoding circuit 110 Adder 111 Frame memory (1) 112 Frame memory (2) 113 map creation circuit 114 background memory 115 contour encoding circuit 116 external storage device 117 selection circuit 118 background encoding circuit 119 multiplexing circuit 120 output switching circuit 121 buffer 122 block memory

Claims (8)

[Claims] A first encoding unit configured to encode a map including a first pixel value indicating a moving area and a second pixel value indicating a background area for distinguishing a moving area from a background area for each screen of the moving image signal; Coding means, a second coding means for coding an image of a moving area distinguished by the map, a map coded by the first coding means, and a code by the second coding means. A moving image encoding apparatus comprising means for outputting a converted image, wherein the moving image encoding apparatus further comprises means for reading out another background image from a storage means in which another background image that substitutes for the true background is stored. Video encoding device. 2. The image processing apparatus according to claim 1, wherein the output unit is configured to output a map encoded by the first encoding unit, an image encoded by the second encoding unit, and another background image read from the reading unit. The moving picture coding apparatus according to claim 1, wherein the moving picture coding apparatus outputs. 3. The moving picture coding apparatus according to claim 1, wherein said storage means comprises a semiconductor memory card storing another background image. 4. The moving picture coding apparatus according to claim 1, wherein said storage means comprises a CD-ROM storing another background image. 5. A first code for encoding a map including a first pixel value indicating a moving area and a second pixel value indicating a background area for representing outline information of a moving area for each screen of a moving image signal. Encoding means, a second encoding means for encoding an image signal of a moving region distinguished by the map, and a map encoded by the first encoding means and encoding by the second encoding means. A moving image encoding apparatus comprising means for outputting a processed image signal, characterized in that the moving image encoding apparatus further comprises means for reading out another background image from storage means in which another background image as a substitute for the true background is stored. Video encoding device. 6. The output means outputs the map encoded by the first encoding means, the image encoded by the second encoding means, and another background image read from the reading means. The moving picture coding apparatus according to claim 5, wherein the moving picture coding apparatus outputs the moving picture. 7. The moving picture encoding apparatus according to claim 5, wherein said storage means comprises a semiconductor memory card storing another background image. 8. The moving picture encoding apparatus according to claim 5, wherein said storage means comprises a CD-ROM in which another background image is stored.