JPH11331851A - Low bit rate dynamic image coder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a dynamic image coder especially suitable for low bit rate coding by decreasing an arithmetic amount of coding while keeping high image quality. SOLUTION: This coder is provided with an input means 101 that receives a digital image, an image storage section 104 that stores the received image, a scene detection section 103, a mobile object detection section 105, a mobile object layer retrieval section 106, a background image discrimination compensation section 107, a coding control section 108, an encoder section 109 and an output means 110. The mobile object detection section detects a moving area in the unit of blocks at a head of a scene and outputs a 1st moving object layer. The moving object layer retrieval section generates a mobile object layer formed to be a rectangle of a succeeding frame based on a motion of a block in the layer in a current frame. The background image discrimination compensation section has an internal memory, which stores background information of the decoder side and analyzes a motion vector of a received background image, detects a motion of the background and provides an output of information to update the background newly.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to moving picture compression encoding. In particular, the present invention relates to a moving picture coding method for low bit rate coding effective on a transmission medium having a low transfer rate.

[0002]

2. Description of the Related Art In recent years, personal computers have been reduced in price, improved in performance, and easier to use, and their use in the general public has been accelerated. The era of one in each home has just arrived. Along with that, a communication infrastructure such as ISDN has been developed, and the opportunity to connect to the Internet and the like from a personal computer at home has increased. However, when a moving image is transmitted over a transmission medium having a relatively low transmission bit rate over a wide area such as the Internet unlike a LAN or the like, a moving image compression technique is indispensable. Then, in consideration of playing back while receiving moving picture data, in order to play back a smooth image, the coding apparatus must perform coding so as to satisfy a bit rate limited by a transmission medium.

For this reason, many low bit rate coding techniques have been devised. In a method of performing encoding without being concerned with the content of an image, basically, there is no choice but to reduce spatial information or time information. As in the former case, when the data compression ratio in the spatial direction is increased, the spatial resolution is degraded, and the reproduced image is blurred. When the number of transmission frames around the frame time is reduced as in the latter case, the reproduced image becomes jerky in motion.

Therefore, in order to solve this problem, encoding has been performed in which many bits are assigned to important places within one frame of an image in consideration of the contents of a moving image. At this time, a new problem arises as to which part of the image is regarded as important. However, a method called object coding in which a moving object is used as an object and bits are assigned to the object separately from the image (background) is generally used. . Object encoding is based on ISO-IEC / JTC
It is also adopted in MPEG4 which is a new video encoding system standardized by 1 / SC28 / WG11.

In object coding, a moving object is coded as one object separately from the background. At that time, as object information, the shape, color, and motion are encoded as parameters and transmitted to the decoder side. However,
There are still problems in realization, such as the time required to extract an object from an image.

As one of the prior art examples, background plane information consisting of a still image of a background image and a moving object plane moving on the background image are separately coded and transmitted, and a reproduced image is obtained by synthesizing at a decoder side. There is Japanese Patent Application Laid-Open No. 7-38858.
In this method, a preparation period is provided before photographing by a camera is started, and during that period, the signal processing unit separates the input image into a background plane and a moving object plane. FIG. 18 shows a simple configuration diagram of the method of Japanese Patent Application Laid-Open No. 7-38858. This conventional moving image encoding apparatus includes an input unit 1001 for receiving a digital input image from the outside, and a signal processing unit 1 for performing signal processing on input image data.
002, a background plane frame memory 1003 for storing the background plane image, a moving object plane frame memory 1004 for storing the moving object plane image, and a still image encoder 1005 for coding a still image in the background plane and the moving object plane. And a motion information encoder 1006 for encoding motion information from the output of the signal processing unit
And output means 1007 for outputting a low bit rate compressed image from the output of the still picture encoder 1005 and the output of the motion information encoder 1006.

The operation of the image coding apparatus having such a configuration will be described. First, a still image wider than the image frame is fetched into the background plane memory 1003. Then, the difference between the background image stored in the background plane memory 1003 and the input image is calculated, and for the pixels whose values are equal to or larger than the threshold value θ1, the pixels of the previous frame are further calculated. When the difference value is equal to or larger than the threshold value θ2, the pixel is regarded as a motion pixel. After that, scanning is performed from the upper left to the lower right, motion pixels are detected, and the eight neighboring pixels are integrated to finally obtain some moving objects. Then, a plurality of moving object planes are written in the moving object plane memory 1004.

The above processing is performed during a preparation period. The plane information is compressed at a low compression ratio (or uncompressed) by the still image encoder 1005 and output. Next, during the shooting period, using information written in the background plane memory 1003 and the moving object plane memory 1004,
The position of the background image and the motion information of the moving object are detected, and the information is also separately compressed and output.

[0009]

However, at this time, three problems occur. One is that it takes time to detect a moving object. In the method of JP-A-7-38858,
Although a moving object is detected with a preparation period, moving objects (moving pixels) are generally detected on a pixel-by-pixel basis, so it takes time to perform area integration (merging) after detecting moving pixels. Would.

[0010] The second problem is that it is difficult to encode the shape and motion of the moving object stably. The reason is that the motion and shape change information of the moving object in the next frame with respect to the moving object in the current frame must be encoded. In the method of JP-A-7-38858, when the shape does not change, a motion vector is obtained for a specific representative point or a representative block, and the obtained motion vector is used as a motion vector of a moving object.
The motion detection and the motion compensation are performed, and the error is used as the error information.

A third problem is a noise problem near a boundary when a moving object is superimposed on a background image. The reason is that the edges of the moving object are not perfectly aligned, so that even if the moving object is cut out in pixel units, the vicinity of the boundary looks dirty. In the end, the background is often mixed with the moving object in the extraction, and the boundary is blurred and synthesized.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and a first object of the present invention is to separately encode a background image consisting of a background still image and moving object information moving on the background image. An object of the present invention is to make it possible to detect a moving object at a high speed in a low bit rate moving image encoding apparatus.

A second object of the present invention is to provide a low bit rate moving picture coding apparatus which performs coding by dividing background information consisting of a background still picture and moving object information moving on the background picture. This is to prevent an error from occurring in detection and encoding of a moving object even after passing.

A third object of the present invention is to provide a low-bit-rate moving image encoding apparatus which performs encoding by dividing background information consisting of a background still image and moving object information moving on the background image. This is to eliminate noise near the boundary between the moving object and the background image without applying a filter.

[0015]

According to the present invention, in order to achieve the first object, first, a moving object region detecting section performs a motionless / non-moving determination at the beginning of a scene in units of blocks.
Subsequently, a moving area is detected by performing block-based area integration, and the first moving object layer is output. This makes it possible to quickly detect a moving object from the first frame of a scene.

Second, the moving object region detecting section determines whether or not there is a motion for each block, and then performs the region integration of each block using color information. As a result, it is possible to detect a moving object by performing block integration at high speed without requiring complicated processing such as checking the connectivity of images to determine whether or not the image belongs to the same area.

Next, in order to achieve the second object of the present invention, thirdly, the moving object layer searching section sets the shape of the moving object layer to a rectangle, and places the next frame layer in the current frame layer. Only the block including the outer edge of the included moving object is used for motion prediction, and a rectangle that includes the block obtained by the prediction is set as the next moving object layer. This makes it possible to reliably track and update the moving object.

Fourthly, the moving object layer search section selects a block including the outer peripheral edge of the moving object for searching for the layer of the next frame from the DCT coefficients of the blocks in the rectangular moving object layer. It is. This allows
The next moving object layer can be detected at a higher speed.

Fifthly, in order to achieve the third object of the present invention, fifthly, the background image decision compensator has a memory therein, stores background information held by the decoder side, and inputs the background information. The motion vector of the background image is analyzed to detect motion such as parallel movement, enlargement and reduction of the background, and to output only information necessary for updating the background. This makes it possible to greatly reduce noise near the boundary when the moving object layer including the background and the background image are superimposed.

[0020]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention according to claim 1 of the present invention provides an input means for inputting a digital image from an external device to a low bit rate moving image encoding apparatus, a frame memory for storing the input image, and a scene detecting section. And a moving object detecting unit, a moving object layer searching unit, a background image determination compensating unit, an encoding control unit, an encoder unit, and an output unit. This has the effect that the object can be detected at high speed.

According to a second aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the moving object region detecting section detects a moving region in block units at the beginning of the scene, The first moving object layer is output, and has an effect that a moving object can be detected at high speed from the first frame of a scene.

According to a third aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the moving object region detecting section determines whether or not there is a motion for each block, and further performs color coding. Region integration is performed using information, and does not require complicated processing such as checking the connectivity of images to determine whether or not they belong to the same region. Is performed.

According to a fourth aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the moving object layer searching section sets the moving object layer to a rectangular shape, and The layer is created from the motion of the blocks in the layer of the current frame, and has an effect that the moving object can be reliably tracked and updated.

According to a fifth aspect of the present invention, in the low bit-rate moving image encoding apparatus according to the first aspect, the moving object layer searching unit determines a DCT coefficient of a block in the rectangular moving object layer from the following. A block for searching for a layer of the next frame is selected, and has an effect that the next moving object layer can be detected at a higher speed.

According to a sixth aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the background image decision compensating section has a memory therein and has a decoder side. The background information is stored and only the information necessary to update the new background is output, and when the moving object layer including the background is superimposed on the background image, the noise near the boundary is greatly reduced. This has the effect that it becomes possible.

According to a seventh aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the background image decision compensator analyzes a motion vector of the background image,
It detects motions such as parallel translation, enlargement, and reduction of the background, and has the effect of greatly reducing noise near the boundary when superimposing a background image with a moving object layer including the background. Have.

According to an eighth aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the rectangular layer cut out by the moving object layer search unit includes background information. The background image determination / compensation unit detects and compensates for parallel movement and enlargement / reduction of the background so that noise at the boundary between the moving object layer and the compensated background image is not generated during reproduction. This has an effect that it is possible to greatly reduce noise near the boundary when superimposing the moving object layer and the background image.

According to a ninth aspect of the present invention, in the low bit rate moving image encoding apparatus according to the first aspect, the scene detecting section detects a scene change and encodes the scene in a picture group unit. The processing is performed.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

(Embodiment 1) A moving object region detection unit according to an embodiment of the first invention is a low bit rate moving picture encoding apparatus which converts a digital input image into a low bit rate image and outputs the image. Has been implemented. FIG. 1 is a block diagram showing a configuration of a low bit rate video encoding device according to the first embodiment of the present invention. A low bit rate moving image encoding apparatus according to this embodiment includes an input unit 101 that receives a digital input image from the outside, and a scene detection unit 102 that detects a transition of a scene of input image data.
And a frame memory 104 for storing images. Further, the frame memory 104 includes a moving object detecting unit 105 that detects a moving object in an image, a moving object layer searching unit 106 that searches for a moving object layer for each frame, and analyzes a motion of a background image for each frame. And a background image determination compensating unit 107 to be connected. 1
Reference numeral 08 denotes an encoding control unit that controls the encoding operation of image data. The encoding control unit 108 includes a scene detection unit 102, a moving object detection unit 105, a moving object layer search unit 106, a background image It is connected to a decision compensating unit 107 and an encoder unit 109 that performs image data encoding processing. The encoder unit 109 is further connected to an output unit 110 for outputting a low bit rate compressed image and the frame memory 104.

An outline of a low bit rate moving picture coding apparatus having such a configuration will be described. This device is a device for converting a digital input image into a low bit rate image and outputting the converted image. The moving object detection unit 105 detects a moving object included in the frame from the input image. A moving object is a continuous part having motion in a moving image sequence.
For example, a pitcher in a baseball broadcast corresponds to a moving object. Here, the background image and the moving object layer in one scene are separated. The moving object layer is a rectangular image containing the moving object.
The background image is a portion obtained by removing the moving object layer from the frame.

In the first frame of the scene, the encoder unit 109 encodes and transmits both the background image and the moving object layer as they are. After the second frame, the moving object layer search unit 106
Searches the moving object layer for each frame, and sends the position and size information of the moving object layer in the frame to the encoder unit 109. Further, the background image determination compensation unit 107 analyzes the motion of the background image for each frame, and sends motion information to the encoder unit 109. The encoder unit 109 reads a necessary image portion from the transmitted information from the frame memory 104, and encodes and transmits the read image portion. When the scene is switched, the moving object detection unit 105 detects a new moving object and the processing is repeated.

The low bit rate moving picture encoding apparatus according to the present embodiment separates background information and moving object information, and transmits the background only once at the beginning of a scene, and thereafter updates the background by background compensation. In addition, since the moving object layer is updated for each frame, compared to a device that uniformly encodes a frame without considering the content of the image, the obtained image has a high image quality even at the same bit rate, and the motion is smooth. It has the characteristic.

Next, the overall operation of the present apparatus will be described with reference to the flowchart of FIG. Step 101: First, the input means 101 sequentially inputs uncompressed digital images. Step 102: Write to the frame memory 104. Step 103: The scene detection unit 102 detects a frame in which the scene changes greatly from the continuously input image, and transmits the frame number to the encoding control unit 108. Step 104: The moving object detection unit 105 receives the number of the first frame at which the scene is switched from the encoding control unit 108, and reads the received number and the next numbered frame from the frame memory 104. Step 105: The moving object detection unit 105 determines an object to be encoded as a moving object in a series of scenes from the read frame, and sends rectangular object layer information including the object to the encoding control unit 108. Step 106: The moving object layer search unit 106 and the background image determination / compensation unit 107 receive the scene end signal from the encoding control unit 108. Upon receiving information about the body layer, frames belonging to a series of scenes are sequentially read from the frame memory 104, and a moving object layer is searched for each frame. Then, information on the moving object layer is sent to the encoding control unit 108. Step 108: The background image determination / compensation unit 107 receives from the encoding control unit 108 the same scene switching frame number (step 104) as the scene switching frame number received by the moving object detection unit 105 from the encoding control unit 108, and Frames belonging to a series of scenes are sequentially read from 104, the motion of the background image in the scene is determined, and the motion information of the background image is sent to the encoding control unit 108. Step 108: The encoding control unit 108 sends the layer information received from the moving object detecting unit 105 and the moving object layer searching unit 106 to the encoder unit 109. Step 110: The encoder unit 109 reads data from the frame memory 104 based on the layer information and encodes the moving object layer. Step 111: The encoding control unit 108 sends the motion information of the background image to the encoder unit 109. Step 112: The encoder 109 reads data from the frame memory 104 and encodes the data based on the background motion information. Step 113: The encoder unit 109 multiplexes the coding information of the moving object layer and the motion coding information of the background image into one stream. Step 114: The output unit 110 is the encoder unit 1
In step 09, the encoded data is output to the outside of the apparatus. Step 115: The process is repeated for the next scene until there is no more input data.

As described above, the low bit rate moving image coding apparatus generates a high quality low bit rate compressed image. Next, a method of detecting the moving object layer information in the moving object detecting unit 105 (Steps 104 and 105) will be described in detail with reference to FIG. The moving object detection unit 105 includes:
The number of the first frame at which the scene is switched is received from the encoding control unit 108, and the received number and the next numbered frame are read from the frame memory 104. The first frame of the scene is called frame 0, and the next frame is called frame 1. The moving object detection unit 105 obtains a difference between each pixel of the frame 0 and the frame 1. However,
Negative values are absolute values. Then, a certain threshold value is determined, and pixels exceeding the threshold value are counted for each block.
At this time, a block in which the number of pixels exceeding the threshold is equal to or more than a certain value is set as a moving block. Note that the size of a motion vector can also be used for detecting a moving block.

Next, the moving object detection unit 105 searches for a moving block in the frame, and when detecting the moving block X, if the moving block X has not yet obtained an area number, assigns an area number. Then, a block in the vicinity of the moving block X is checked, and if a neighboring block has already acquired an area number, the same block number as the neighboring block is re-assigned to the moving block. In this way, by detecting all moving blocks in order, neighboring moving blocks are merged and divided into several regions. However, this region division method is a heuristic method, and the result of region division changes depending on the order in which to search and the extent of the neighborhood.

Therefore, instead of the area integration (merging) method, the values of the pixels in the block are used as the feature values of the block.
A region division method of performing mapping by dividing into a space having an axis for each feature amount and performing division may be used. In FIG. 3, although the area is divided into three areas, the areas (b) and (c) are small and are not regarded as a moving object area. Here, the determination as to whether or not the area is a moving object area is made based on the number of moving blocks included in the area. Subsequently, the moving object detection unit 105 adds some blocks to the detected moving object region, and determines a rectangular moving region layer.

As described above, the moving object detecting unit 105 can cut out the moving object layer at a high speed by detecting the moving object in block units.

(Embodiment 2) A moving object region detection unit according to an embodiment of the second invention is a low bit rate moving image encoding apparatus which converts a digital input image into a low bit rate image and outputs the image. Has been implemented. Embodiment 1 of the low bit rate moving image encoding apparatus except for a moving object region detection unit
Is exactly the same as the low-bit-rate moving image coding apparatus of FIG.

A method of detecting the moving object layer information of the moving object detecting unit in this embodiment will be described. The moving object detection unit in the present embodiment also performs processing in block units until the region integration, similarly to the moving object detection unit 105 of the first embodiment. However, the moving object detection unit according to the present embodiment can reduce the detection error of the moving object due to excessive division of the area.

The moving object detecting section divides the area into several areas by area integration, and then examines the degree of correlation between the color information of each area and the color information of surrounding blocks to grow the area.
Each block has a luminance Y and color differences Cr and Cb. Therefore, a histogram is created for each of the luminance Y and the color differences Cr and Cb in each area. The width of the class is set to be relatively large, and the center value of the class with the highest frequency is set as the representative value of the block. From the luminance Y and the representative values of the color differences Cr and Cb, a representative color vector in the region can be obtained.

A representative color vector of a block that has not been recognized as a moving block can be obtained, and the moving object detection unit compares the representative color vector of each area with the surrounding blocks. The magnitude of the vector obtained by subtracting the block representative color vector from the region representative color vector is used as a comparison reference. If this value is less than or equal to the threshold, the block is taken into the region. As the color information, instead of the representative color vector,
For example, an average value of the color difference of each pixel in the block can be used.

Next, as a specific example, detection of a moving object in a scene in which a ball having a uniform color is flying to the right in a certain background will be described with reference to FIG. First, in order to detect a moving block, a difference image between frame 0 and frame 1 is obtained. The black dots in the difference image indicate pixels for which the difference between the pixel values of both frames is equal to or greater than a threshold. When noise, camera shake, and the like are sufficiently small, a pixel having a pixel value difference equal to or larger than a threshold appears at a position considered to be a stationary object. However, these pixels are not often included in a moving block because the number of pixels included in one block is small. As a result, four motion blocks are obtained.

The lower part of FIG. 4 is an enlarged view of the position of the moving block. By the region integration processing, two moving regions, a region a and a region b, are obtained. However, they are actually the same object, and a flat area inside the ball must also be added to the moving object. The moving regions a and b obtained by the region integration processing are portions including the edge of the ball and also include background information. However, the representative color vector based on the histogram is equal to that of the pixel in the ball portion. Conversely, a block including only the background is expected to have a different representative color vector. The moving object detection unit checks the correlation between the regions a and b and the representative color vectors of the surrounding blocks, and as a result, two regions sandwiched between the regions a and b having the representative color vectors similar to the two regions. The block is added to the area, and the two areas a and b are merged into one area.

As described above, by using the color information of the block when detecting the moving object area, the moving object detection unit 105 can eliminate the omission of detection of a flat area in the moving object.

(Embodiment 3) A moving object layer search unit according to an embodiment of the third invention is a moving image encoding apparatus for converting a digital input image into a low bit rate image and outputting the converted image. Has been implemented. The low bit rate video encoding device is exactly the same as the low bit rate video encoding device of the first embodiment.

The moving object layer searching method (step 107) in the moving object layer searching unit 106 will be described in detail with reference to the flowchart of FIG. Step 201: The moving object layer search unit 106 receives the number of the first frame at which a scene is switched from the encoding control unit 108, and reads out frames belonging to a series of scenes from the frame memory 104 in order. However, it is assumed that the first frame of the scene is frame 0 and the subsequent frames are sequentially numbered. Step 203: The moving object layer search unit 106 receives information on the moving object layer of frame 0 from the encoding control unit 108. Step 204: Divide the moving object layer of the frame n into blocks. However, the initial value of n is 0, which is the first frame of the scene. Step 205: The moving object layer searching unit 106 checks each block in the moving object layer of the frame n and the frame n + 1.
And block matching between frame n +
1 stores the matched position of the block. However, among the blocks in the moving object layer of frame n, those used for block matching are limited to blocks having a change in gray value. Step 206: The moving object layer search unit 106 creates a moving object layer of the frame n + 1 so as to include all positions matching the blocks in the frame n + 1. Step 207: Position and size of the moving object layer in the frame The encoding control unit 10 uses the
Send to 8. Step 208: The moving object layer searching section 106 repeats the processing until receiving the signal of the end of the scene from the encoding control section 108.

Next, among the blocks in the moving object layer of the frame n, the blocks used in the block matching and having a change in the gray value will be described. Even if block matching is performed on all the blocks in the moving object layer, the next moving object detection is not so effective. One reason is that one wants to know the outer frame of the moving object, and the other is that it is difficult to obtain an accurate motion vector because the inside of the moving object is flat. Therefore, it is effective to use a block including the edge of the outer frame of the moving object for motion prediction. A general DOG (Differen) is used for edge detection.
Tial of Gaussian) filter or the like is used. The convolution of the image and the DOG function results in zero crossings at the edges of the image. Therefore, a block having a zero crossing point is detected and used for block matching.

Next, as a specific example, a moving object layer detection in a scene in which a cloud moves while changing its shape in the background is performed with reference to FIG. The surrounding 10 blocks (a to j) from the moving object layer of frame 0 are used for block matching. As a result, a point that matches frame 1 is shown in the center of FIG. Here, the g block overlaps with the h block and has selected an incorrect point.
However, a rectangle that includes blocks matching the blocks a to j becomes the moving object layer of the frame 1.

As described above, the moving object layer search unit 106
Limits the shape of the moving object layer to a rectangle, uses the block containing the edge in the moving object layer for block matching, and searches for the next moving object layer, so it is stable even when the shape of the moving object changes. Detection can be performed. Further, by setting a rectangle that includes the matched block as the next moving object layer, a moving object layer can be stably obtained even with a slight error in matching.

(Embodiment 4) A moving object layer search unit according to an embodiment of the fourth invention is a moving image encoding apparatus for converting a digital input image into a low bit rate image and outputting the converted image. Has been implemented. However, the digital input image in the present embodiment is in the DV format. D
The V format is a standard for video cassette recorders established in 1886, "Specifica".
Tions of Consumer-Use Dig
ital VCRs (HD Digital VCR
Conference, 1886) ”, and DCT is performed in block units.

FIG. 7 is a block diagram showing the configuration of the moving picture conversion apparatus according to this embodiment. The configuration of this apparatus will be described with reference to FIG. In this moving image conversion apparatus, an input means 201 for taking in an image from the outside, a scene detection unit 202
And a DV decoding unit 203 for decoding the image, and a frame memory 204 for storing the image is connected to the DV decoding unit 203.
Moving object layer search unit 206, background image determination compensation unit 2
07 is connected. The encoding control unit 208 is connected to the scene detection unit 202, the moving object detection unit 205, the moving object layer search unit 206, the background image determination compensation unit 207, and the encoder unit 209. The encoder unit 209 is further connected to the output unit 210 and the frame memory 204. Further, the frame memory 211 stores the input means 201.
And the moving object layer search unit 206. The components other than the scene detection unit 202 and the moving object layer search unit 206 are the same as the modules of the first embodiment.

The scene detecting section 202 receives DV format data as input and can directly detect a scene change from the DV format data. Also, DV
DV decoder 203 for returning the DV format to uncompressed image data in order to input the format
And a frame memory 211 for holding the output of the input means 201 is newly required.

Next, the overall operation of the present apparatus will be described with reference to the flowchart of FIG. Step 501: First, the input unit 201 sequentially inputs an input image in the DV format, and Step 502: writes DCT coefficients in the frame memory 211 for each block. Step 503: The DV decoding unit 203 sequentially decodes the continuously input images, and
Write to 04. Step 504: The scene detection unit 202 detects a frame in which the scene changes greatly from the continuously input bv image, and transmits the frame number to the encoding control unit 208. Step 505: The moving object detection unit 205 receives the number of the first frame at which the scene is switched from the encoding control unit 208, and reads out the received number and the next numbered frame from the frame memory 204. Step 506: The moving object detecting unit 205 determines an object to be encoded as a moving object in a series of scenes from the read frame, and sends rectangular layer information including the object to the encoding control unit 208. Step 507: Until the moving object layer searching unit 206 and the background image determination compensating unit 207 receive the scene end signal from the encoding control unit 208, Step 508: The moving object layer searching unit 206 Receiving information about the body layer,
The frames belonging to a series of scenes are sequentially read from the frame memory 204, and a moving object layer is searched for each frame. At this time, DCT block data in the same area as the moving object layer is read from the frame memory 211 as well. Then, the obtained information about the moving object layer is sent to the encoding control unit 208. Step 509: Also, the background image determination compensation section 207
The same frame switching frame number (step 505) as the scene switching frame number received by the moving object detecting unit 205 is received from the encoding control unit 208,
The frames belonging to a series of scenes are sequentially read from the frame memory 204, the motion of the background image in the scene is determined, and the motion information of the background image is sent to the encoding control unit 208. Step 510: The encoding control unit 208 sends the layer information received from the moving object detection unit 205 and the moving object layer search unit 206 to the encoder unit 209. Step 511: The encoder unit 209 performs data conversion based on the layer information. From the frame memory 204, and encodes the moving object layer. Step 512: The encoding control unit 208 sends the motion information of the background image to the encoder unit 209. Step 513: The encoder unit reads out the data from the frame memory 204 and encodes the data based on the motion information of the background. . Step 514: The encoder unit 209 multiplexes the coding information of the moving object layer and the motion coding information of the background image into one stream. Step 515: The output unit 210 outputs the signal from the encoder unit 2
In step 09, the encoded data is output to the outside of the apparatus. Step 516: The process is repeated for the next scene until there is no more input data.

Next, a moving object layer searching method (step 508) in the moving object layer searching section 206 will be described. Basic Process of Moving Object Layer Search in Moving Object Layer Searching Section 206 (Steps 201 to 2
04 and steps 206 to 208) are the same as those of the moving object layer search unit 106 of the third embodiment.

Upon receiving the information on the moving object layer n from the encoding control unit, the moving object layer search unit 206 reads out an area having the same position and the same size as the moving object layer n from the frame memory 211. Then, the DCT block included in the region therein is used for detecting a block including an edge useful for block matching. The DCT block is obtained by decomposing an image into frequency components, and the component distribution differs between the block including the edge and the block not including the edge. By using this characteristic, the detection of the block including the edge in the moving object layer n is performed. Can do it.

FIG. 9 is an example of a block including a stepwise edge therein. When these blocks are subjected to DCT conversion, values concentrate on a specific area as shown in the lower part of FIG. In a natural image, values are usually concentrated on low frequency DCT components. Also in this example, the values are concentrated in the low-frequency portion in all the blocks, so that they are not suitable for edge detection. Therefore, the low-frequency components (2 × 2) are removed, and the absolute values of the remaining components are arranged in order, and the components from the smallest value to the 30th component are replaced with 0. Then, by seeing the arrangement of the pixels replaced with 0 in the block, it is detected whether or not an edge exists in the block. Specifically, whether or not the zero component region invades the component concentration region due to the edge in the figure is numerically determined. If the component-concentrated region due to the edge as shown in the figure remains even after the pixel is replaced with 0, the block includes the edge.

The moving object layer search unit 206 compares the block including the edge thus selected with the frame n + 1.
And block matching between frame n + 1
Find the matched position of the block within.

As described above, the moving object layer search unit 206
Can directly obtain, from the DCT image, a block including an edge in the moving object layer that is required when searching for the moving object layer. Therefore, when a DV format image is used for input, the moving object layer search unit 206
In this case, edge detection by a DOG filter or the like can be omitted, and the processing can be reduced in weight.

(Embodiment 5) A background image decision compensator according to a fifth embodiment of the present invention is a low bit rate moving picture coding apparatus for converting a digital input image into a low bit rate image and outputting the converted image. It has been implemented in. The low bit rate video encoding device is exactly the same as the low bit rate video encoding device of the first embodiment.

The motion judgment compensation of the background image (step 108) in the background image judgment compensation section 107 will be described in detail with reference to the drawings. The background image determination / compensation unit 107 determines the motion of the background in the frame using the model vector, and compensates for the background of the current frame using the background of the previous frame. The model vector is described below, but when parameters such as the object being photographed by the camera and the positional relationship between the camera and the object are known, the motion vector obtained when the camera is panned or zoomed is obtained. That is. That is, the motion of the background image can be approximately obtained from the degree of correlation with the model vector. The background image determination compensation unit 107 determines only a movement such as a parallel movement and an enlargement / reduction of the background. Also, the background image determination compensation unit 10
7, the background information newly appearing due to the movement of the area or the background image hidden by the moving object layer is compensated for.

Hereinafter, the reason why the motion of the background image is limitedly approximated to the parallel movement and the enlargement / reduction and the motion of the background image is compensated will be described. To determine (create) an image: 1) Object information (color, shape, size), 2) Positional relationship between object and camera, 3) Camera information (focal length, CCD cell size) Etc.) are required. Even when normal shooting is limited (excluding rotation in the direction of the optical axis of the camera), images are obtained from parameters such as the shape of the object (three-dimensional), camera position (three-dimensional), camera posture (two-dimensional), and camera position. The distance between the surfaces is required.

Assume that two continuous images are A and B, and when the image of A is used to create the image of B,
1), 2) when A is photographed, and the displacement of the camera for photographing B from A, 3). That is, assuming that all the objects to be photographed are stationary and the shape does not change, when the encoder 1 encodes A) 1), 2), 3) and the image A and transmits it to the decoder, the camera that photographs A The amount of displacement between the position and the camera position at which B was shot is required when the B image is created.

However, it is very rare that these parameters (1, 2, 3) are generally known when encoding an image. Therefore, these parameters are estimated only from the image sequence. Only images, except where the object being imaged is known and the relationship between the object and the camera is known (the object is a milk pack and the camera can only be at a certain distance from the milk pack) The amount of information is too small to estimate these parameters from

Unless all parameters are obtained, A
It is impossible to create an image of B accurately using the image of However, assuming that the image interval between A and B is small (about 1 or 2 frames), an approximate B image can be created using the A image.

For simplicity, consider a perspective image as shown in FIG. Let A be the camera position, and let this be the origin. The z axis is taken in the optical axis direction of the camera, and the x axis is taken in a direction perpendicular to the z axis. Let f be the distance to the image plane. The background object B is
It is assumed that the plane consists of a plane perpendicular to the z-axis. Assuming that the X coordinate on the image plane is x, the relationship X = f * x / z holds.

Here, it is assumed that the camera has moved slightly along the x-axis by Δx. The motion vector obtained at this time is f * Δx / z at a rate within the frame. As can be seen from this equation, the distance from the camera position of the background becomes a problem in the background parallel movement compensation. However, the position z of the background object is much larger than the camera movement Δx at minute frame intervals.

Therefore, when detecting a motion vector on an actual image plane, changes in the position and the depth direction of the background object can be neglected compared to the effects of image resolution and noise. Therefore, while the accuracy of the compensation of the background image depends on the image, the background is not parallel to the image plane where the background wall is inclined or a distant mountain is connected to the front and back. In this case too, this compensation, which limited the background motion to translation and scaling, should be sufficient to absorb.

The background image determination compensation unit 107 receives the number of the first frame at which a scene is switched from the encoding control unit 108, and sequentially reads out frames belonging to a series of scenes from the frame memory 104. First, a method of determining a motion of a background image will be described with reference to FIG. FIG. 11 is a diagram illustrating a method for detecting the movement of the background image in the background image determination compensation unit. In FIG. 11, it is assumed that the first frame of the scene is frame 0, and the subsequent frames are sequentially numbered.

The background image determination compensation unit 107 determines parallel movement and enlargement / reduction of the background. Several blocks are determined from frame 1 and matching is performed between those blocks and frame 0 to obtain a motion vector.
At this time, the block used for the determination is referred to as a determination motion vector detection block, and the obtained motion vector is referred to as a background image motion determination motion vector.

Next, a method of detecting enlargement / reduction will be described with reference to FIG. FIG. 12 is a diagram for explaining a method of determining the enlargement / reduction of the background image using the model vector in the background image determination compensation unit. The vector is placed at a position corresponding to the determination motion vector detection block, and the direction of the vector is the vector direction when the camera is zoomed at a magnification of 10%. However, the magnitude of the vector is the reciprocal of that when zooming at a magnification of 10%. Hereinafter, a processing procedure at the time of enlargement / reduction determination will be described. FIG. 13 is a flowchart for explaining a processing operation in which the background image determination compensating unit determines the enlargement / reduction of the background image using the model vector. Step 301: The background image determination compensation unit detects a motion determination vector. Step 302: The inner product of the detected motion determination vector and the scaled model vector is calculated. Step 303: A number of inner product values equal to the number of determination motion vector detection blocks are obtained, and it is determined whether or not these inner product values are dependent on each other. For the determination, a variance value may be calculated, and the determination may be made based on a threshold value.
If the values are not similar, it is not a scaling
End the scaling determination. Step 304: When it is determined that the values of the inner products are similar, the average value is further calculated. Step 305: According to the sign of the average value, Step 306: If the value is a positive value, the enlargement ratio is calculated. When the average value is 1, the enlargement ratio is 10%. In other cases, the enlargement ratio is obtained from an enlargement / reduction table obtained in advance by calculation. Step 307: Similarly, when the value is negative, the reduction ratio is calculated using the enlargement / reduction table. When the average value is -1, the reduction ratio is 10%.

In this way, an enlargement or reduction ratio can be obtained. Next, a method of detecting a parallel movement will be described with reference to FIG. FIG. 14 is a diagram for explaining a method of determining the parallel movement of the background image using the model vector in the background image determination compensation unit. The model vector is placed at the same position as the determination motion vector detection block, and the direction is horizontal right. However, the size is the unit length. Hereinafter, the processing procedure of the parallel movement determination will be described. FIG. 15 is a flowchart illustrating the processing operation of the background image determination compensation unit for determining the parallel movement of the background image using the model vector. Step 401: The background image determination compensation unit detects a motion determination vector. Step 402: By paying attention to the vertical component of the detected motion vector, it is determined whether or not there is a variation in the sign of the vertical component. A threshold may be provided for the determination, and the determination may be made based on whether or not a different code is included in a certain ratio or more. If it is determined that there is variation, it is determined that the movement is not parallel movement, and the parallel movement determination is completed. Step 403: If it is determined that there is no variation, a vector whose vertical component has the same (largest) code is extracted. In the subsequent processing, only the extracted vector is targeted. Step 404: Normalize each extracted vector to have a unit length. Step 405: Calculate the model vector and the inner product. Step 406: It is determined whether or not the obtained value of the inner product depends. The values are from -1 to 1 because they are inner products of vectors of unit length. For the determination, a variance value may be calculated, and the determination may be made based on a threshold value. If the values are not similar, it is determined that the movement is not parallel movement, and the parallel movement determination ends. Step 407: When the values are similar, since the directions of the vectors are aligned, the average value of the values of the inner product is calculated. Assuming that the average value is M, the unit vector in the translation direction is M in the horizontal direction and (1-M ＾ 2) ＾ (1/2) in the vertical direction. However, the code in the vertical direction is a code that occupies a large number of vertical components of the motion vector for motion determination of the background image (the code of the vertical component of the vector extracted in step 403). Step 408: Then, the vectors before normalization are orthogonally projected in the parallel movement direction to obtain the magnitude in the parallel movement direction. Step 409: It is determined whether or not the magnitude of the orthogonal projection vector is fluctuating. For the determination, a variance value may be calculated, and the determination may be made based on a threshold value. If the values are not similar, it is determined that the movement is not parallel movement, and the parallel movement determination ends. Step 410: If the values are similar, calculate an average value of the magnitudes. By matching the magnitude of the vector with the unit vector in the translation direction, a translation vector can be obtained.

Next, a method of compensating the background image in the background judgment compensation unit 107 based on the motion information such as the scaling ratio and the translation vector of the background image obtained as described above will be described with reference to FIG. As for the frame 0, the encoder 109 encodes all information, so the background image determination compensator 107 does nothing. Frame 1
It is necessary to transmit the difference between the subsequent frame and the previous frame.

For the sake of simplicity, FIG. 16 (a) shows a case where the background is stationary, FIG. 16 (b) shows a case where the background is moving in parallel, and FIG. The description will be made in each case. First, at the time of FIG. 16A, in the frame 1, the moving object layer is moved to the upper right. Therefore, in the transmission of frame 1, an L-shaped area indicated by an arrow must be sent. Next, in frame 2, the moving object layer is now moving to the lower right. However, since the area hidden by the moving object layer in frame 1 is the area exposed on the surface in frame 0, there is no need to send the area. Only the area hidden by the moving object layer of the frame 0 indicated by the arrow needs to be newly transmitted in the transmission of the frame 2. That is, the background image determination / compensation unit 107 transmits the necessary size when the background area that appears on the surface in frame 0 is always available and only the area hidden by the moving object layer is needed.

Next, consider the case of FIG. 16 (b).
It is assumed that the motion of the moving object layer is the same as that in FIG. Frame 1 is when the background moves to the upper left.
The background determination compensator 107 has to send the area indicated by the upper and left arrows of the frame 1. On the decoder side, the remaining area can be shifted from the frame 0. Frame 2 is the case where the background has moved to the lower left. At this time, the background determination compensator 107 has to send the area indicated by the lower and left arrows of the frame 2. However, at this time, it is not necessary to send the background area appearing on the front surface in frame 0.

Finally, consider the case of FIG. 16 (c). However, it is assumed that the moving object layer does not move for simplicity.
Frame 1 has a reduced background. At this time, the background image determination compensator must send the area indicated by the up, down, left, and right arrows of the frame 1. On the decoder side, the reduced area can be used by reducing the background area on the surface of frame 0. The frame 2 has an enlarged background. At this time, the background image determination / compensation unit 107 does not need to send any data from frame 2. On the decoding side, instead of enlarging the reproduced frame 1,
The background area on the surface of frame 0 is used as it is.

In this embodiment, the frame interval for background image compensation is one frame. However, any frame interval of 2 or 3 may be used.

Next, the fact that no noise is generated at the boundary when the moving object layer is superimposed on the compensated background image according to the background compensation method will be described using mathematical expressions. Let the frame be F (x, y, t). x and y (0 ≦ x ≦ M, 0 ≦ y ≦ N) are coordinates in the frame, and M and N are the vertical and horizontal sizes of the frame size. t indicates a frame number. Let the object layer be L (x, y, t). Regarding the x and y coordinates, the same coordinates as in the frame are used. Here, when the value of x and y is x ₁ (t) ≦ x ≦ x ₂ (t) and y ₁ (t) ≦ y ≦ y ₂ (t), the value is 1; Consider a window function W such that Here, 0 ≦ x ₁ (t) ≦ x ₂ (t) ≦ M 0 ≦ y ₁ (t) ≦ y ₂ (t) ≦ N.

That is, W (x, y, t) takes a value of 1 inside a rectangle that can be taken in a frame and takes a value of 0 outside the rectangle. L
Can be said to be obtained by covering this W with F. That is, L = W * F. However, actually outside the rectangle
L shall not be defined.

At the time of background image compensation, motion judgment and compensation are performed from consecutive frames. Further, since the moving object layer L is cut out from the frame F, the relationship of F (x, y, t) = L (x, y, t) always holds within the range where L is defined. Actually, the background judgment compensation unit 107
Then, it is impossible to detect the enlargement / reduction and translation at the same time. However, here, consider the case where the background is enlarged by the expansion rate k and translated by α in the x direction and β in the y direction.

At this time, the relationship between the background images is as follows: F (x, y, t) = F ((x−α) / k, (y−β) /
k, t + 1). Then, the pixel value at the boundary of the moving object layer is L
(X, y, t + 1). Where (x, y) is
It indicates the coordinates of the boundary. Next, the pixel value at the boundary of the compensated background image is F (k (x + α), k (y + β), t). Using the relationship of the background image, F (k (x + α), k (y + β), t) = F (x, y,
t + 1), and F (k (x + α), k (y + β), t) = L (x, y,
t + 1), and it can be seen that the background and the moving object layer have the same pixel value at the boundary and look smooth. Similarly, the compensation of the background image area hidden by the moving object layer can be described.
As described above, if the enlargement / reduction ratio or the translation vector can be accurately detected, the noise at the boundary can be suppressed to an inconspicuous level.

Eventually, the background image determination compensating section 107 stores the background area on the surface of the base frame 0 in the memory, and adds the area to the memory only when the frame read from the frame memory 104 changes. Then, it is assumed that the area which has been exposed on the surface can be effectively used until the end of the scene thereafter. Background image judgment compensation unit 1
07 sends the region to the encoding control unit 108,
This is to send not the image itself but area information such as which position and shape of the image to send in the image.
It is also assumed that this area information includes information such as which frame of which previous frame n can be used for reproduction.

Next, the background image compensation (step 112) based on the background motion information obtained by the background image judgment / compensation unit 107 in the encoder unit 109 will be described. The encoder unit 109 receives the motion / region information of the background still image for each frame from the encoding control unit 108. Based on the information, an image in an area that must be sent as background information from the frame memory is read and encoded. Also, region information related to reproduction is encoded together so that the background image can be reproduced on the decoder side.

Finally, as a specific example, processing when an image having a general background is input to the present apparatus will be described with reference to FIG. The input image is a scene 1 in which a car runs at the foot of the mountain, a scene 2 in which a person walks in the forest, and thereafter scene 3 and scene 4.

In scene 1, the car is running and is detected as a moving object. The camera is slowly panning left following this car. In the first frame of the scene, the background image and the moving object layer are compressed and sent. In the second frame, the background image, the moving object layer, and the position where the translational motion has been compensated are compressed and sent.

In scene 2, a tree stands before a human being, which is a moving object, and the camera performs zooming on the human. In the first frame of the scene, as in scene 1, the background image and the moving object layer are compressed and sent. In the second frame, the background enlargement ratio, the moving object layer, and the position are compressed and sent.

As described above, in the present apparatus, the scene detector 1
02, the usefulness of the background is ensured by detecting a scene change, and the background image determination compensation unit 107 determines and compensates for the movement of the background, so that the pixel values at the boundary with the moving object layer are equalized. It is possible to reduce the contamination of the boundary caused by the overlapping of the moving object layers. Further, the entire information of the background image needs to be sent only once per scene, so that the bit rate can be kept low.

[0088]

As described above, in the low bit rate moving image encoding apparatus according to the present invention, the moving object region detecting unit 105 determines whether or not there is a motion at the beginning of a scene in units of blocks, and continuously uses color information. By detecting a moving region by performing block-based region integration, a moving object in a frame can be detected at high speed.

The moving object layer searching unit 106 sets the shape of the moving object layer to a rectangle of variable size, and includes the moving object layer of the next frame including the outer peripheral edge of the moving object included in the moving object layer of the current frame. By using only a block for motion prediction and setting a rectangle that includes the block obtained by the prediction as the next moving object layer, the moving object can be reliably tracked. As a result, the moving object behaves exactly like uncompressed movement.

The moving object layer cut out by the moving object layer search unit 106 contains background image information, and generally, when the moving object layer is overlapped with the background or another layer, the boundary becomes conspicuous. However, the background image determination / compensation unit 107 of the present invention considers the pan and zoom of the camera, which is often performed at the time of shooting, and compensates for the parallel movement and enlargement / reduction of the background image by using a motion vector, thereby moving the moving object layer. It is possible to make the boundary at the time of superposition inconspicuous.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a moving image encoding device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the entire moving picture encoding apparatus according to the first embodiment of the present invention;

FIG. 3 is a diagram illustrating a moving object detection method in a moving object detection unit.

FIG. 4 is a diagram illustrating an example of moving object detection in a moving object region detection unit.

FIG. 5 is a flowchart showing a flow of a moving object layer search process in a moving object layer search unit.

FIG. 6 is a diagram showing an example of searching for a moving object layer in a moving object layer search unit.

FIG. 7 is a configuration diagram of a moving image encoding device according to a fourth embodiment of the present invention.

FIG. 8 is a flowchart showing an operation of the entire moving picture encoding apparatus according to the fourth embodiment of the present invention.

FIG. 9 is a view for explaining an edge detection method using DCT block components.

FIG. 10 is a diagram illustrating an example of a relationship between a camera movement and an image in a perspective image;

FIG. 11 is a view for explaining a method for detecting a motion of a background image in a background image determination compensation unit.

FIG. 12 is a diagram showing a method of detecting the enlargement / reduction of a background image in a background image determination compensation unit.

13 is a diagram showing a flow of processing based on a method of detecting the enlargement / reduction of a background image in the background image determination / compensation unit shown in FIG.

FIG. 14 is a diagram illustrating a method of detecting a parallel movement of a background image in a background image determination compensation unit.

15 is a diagram showing a flow of processing based on a method of detecting a parallel movement of a background image in the background image determination compensation unit shown in FIG.

FIG. 16 is a diagram illustrating a background compensation method in a background image determination compensation unit.

FIG. 17 is a diagram illustrating an example of processing for general image input;

FIG. 18 is a configuration diagram schematically showing a conventional image encoding apparatus.

[Explanation of symbols]

 101, 201, 1001 Input means 102, 202 Scene detecting unit 104, 204, 211 Frame memory 105, 205 Moving object detecting unit 106, 206 Moving object layer searching unit 107, 207 Background image determination compensating unit 108, 208 Encoding control unit 108, 208 Encoder unit 110, 210, 1007 Output means 203 DV decoding unit 100m Signal processing unit 1003 Background plane frame memory 1004 Moving object plane frame memory 1005 Still image encoder 1006 Motion information encoder

Claims (9)

[Claims] An input unit for receiving a digital image from outside, a frame memory for storing the input image, a scene detecting unit, a moving object detecting unit, a moving object layer searching unit,
A low-bit-rate moving image encoding apparatus, comprising: a background image determination compensation unit; an encoding control unit; an encoder unit; and an output unit. 2. The low bit rate moving image encoding according to claim 1, wherein the moving object region detection unit detects a moving region in block units at the beginning of the scene and outputs the first moving object layer. method. 3. The moving image coding method according to claim 1, wherein the moving object region detection unit determines whether or not there is a motion for each block, and further performs region integration using color information. 4. The moving object layer searching unit according to claim 1, wherein the moving object layer has a rectangular shape, and a next frame layer is created from a motion of a block in the current frame layer. Low bit rate video encoding device. 5. The moving object layer searching unit according to claim 1, wherein a block for searching for a layer of a next frame is selected from DCT coefficients of blocks in the rectangular moving object layer. Video encoding device. 6. The background image determination / compensation unit has a memory therein, stores background information held by a decoder side, and outputs only information necessary for updating a new background. The low bit rate video encoding device according to claim 1. 7. The moving image encoding apparatus according to claim 1, wherein the background image determination / compensation unit analyzes a motion vector of the background image and detects a motion of the background such as translation, enlargement, or reduction. 8. The rectangular layer cut out by the moving object layer searching unit includes background information, and the background image determination compensating unit detects and compensates for parallel movement and enlargement / reduction of the background so that it can be reproduced at the time of reproduction. 2. The moving picture coding apparatus according to claim 1, wherein the moving picture coding apparatus has a feature that does not generate noise at a boundary between a moving object layer and a compensated background image. 9. The low bit rate moving image encoding apparatus according to claim 1, wherein the scene detecting unit detects a transition of the scene and performs the encoding process on a per picture group basis.