JP2000004444A - Image encoding device and image decoding device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To attain satisfactory image acquisition by using a following low-order layer frame for the area of only a first component area and using a preceding low-order layer frame for the area of only a second component area when generating the non-encoded low-order layer frame. SOLUTION: A first area extracting part 101 extracts an area, which is the first area but is not the second area, from the first and second area information of low-order layers. A second area extracting part 102 extracts the area, which is the second area but is not the first area, from the first and second area information of low-order layers. When a concerned pixel position is the first area only, a controller 103 connects a switch 104 to a second decoding image side but when the concerned pixel position is the second area only, the switch 104 is connected to a first decoding side and in the other case, the switch 104 is connected to output from an interpolated image preparing part 105.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention belongs to the field of digital image processing, and more particularly to an image encoding apparatus for encoding image data with high efficiency and an image decoding apparatus for decoding encoded data created by the image encoding apparatus. It concerns the device.

[0002]

2. Description of the Related Art In image coding, a method of combining different moving image sequences has been studied. Document "Image Coding Using Hierarchical Representation and Multiple Templates" (IEICE Technical Report)
IE94-159, pp99-106 (1995)) combines a moving image sequence as a background and a moving image sequence of component moving images as a foreground (for example, a human image or a video of a fish cut out by chroma key technology) to create a new image. A technique for creating a simple sequence is described.

[0003] Further, a document "Temporal Scalability based on image conte
nt ", ISO / IEC / JTC1 / SC29 / WG11 MPEG95 / 211 (1995)) creates a new sequence by combining a low-frame-rate video sequence with a high-frame-rate component video sequence. The method is described.

In this method, as shown in FIG. 16, predictive coding is performed at a low frame rate in a lower layer,
In the upper layer, predictive coding is performed at a high frame rate only for the selected area (shaded area). However, the frame encoded in the lower layer is not encoded in the upper layer, and the decoded image of the lower layer is copied and used as it is. Further, it is assumed that a part that attracts the viewer's attention, such as a person part, is selected as the selection area.

FIG. 8 shows a block diagram of a conventional method. First, on the encoding side according to the conventional method, the input moving image is frame-thinned by the first frame dropping unit 801 and the second frame dropping unit 802, and the frame rate is set to be lower than the frame rate of the input image. Unit 803 and lower layer coding section 804. Here, the frame rate of the upper layer is equal to or higher than the frame rate of the lower layer.

[0006] The lower layer encoding section 804 encodes the entire input moving image. As the encoding method, for example, a moving image encoding international standardization method such as MPEG or H.261 is used. In the lower layer encoding unit 804, a decoded image of the lower layer is created and used for predictive encoding, and is input to the synthesizing unit 805 at the same time.

FIG. 9 is a block diagram showing a code amount control section in a conventional coding apparatus. In FIG. 9, an encoding unit 902 performs moving image encoding using motion compensation prediction, orthogonal transform, quantization, variable length encoding, and the like.

A quantization width calculation section 901 calculates a quantization width used in the encoding section 902 and generates a generated code amount calculation section 9.
03 calculates the accumulation of the encoded data. In general, when the generated code amount is large, the quantization width is controlled to suppress this, and conversely, when the generated code amount is small, the quantization width is controlled to be small.

In the upper layer coding section 803 of FIG. 8, only the selected area of the input moving picture is coded. Here, the moving picture coding international standardization method such as MPEG or H.261 is used, but only the selected area is coded based on the area information. However, frames encoded in the lower layer are not encoded in the upper layer.

The area information is information indicating a selected area such as a person, and is, for example, a binary image having a value of 1 at the position of the selected area and a value of 0 at other positions. Further, in the upper layer encoding unit 803, only the selected region of the moving image is decoded,
It is input to the synthesis unit 805.

[0011] In the area information encoding unit 806, the area information is encoded using an 8-way quantization code. As shown in FIG. 17, the eight-way quantization code indicates a direction to the next point by a numerical value, and is generally used when expressing a digital figure.

[0012] When the lower layer frame is encoded in the frame to be synthesized, the synthesizing unit 805 outputs a decoded image of the lower layer. When the lower layer frame is not encoded in the frame to be combined, a moving image is output using two encoded lower layer decoded images before and after the combined frame and one upper layer decoded image. .

Here, the frames of the two images of the lower layer are before and after the frame of the upper layer. Also,
The moving image created by the combining unit 805 is input to the upper layer encoding unit 803 and used for predictive encoding. Combiner 8
The image creation method in 05 is as follows.

First, two lower layer interpolation images are created. Let the decoded image of the lower layer at time t be B (x, y,
t) (where x and y are coordinates representing pixel positions in space), and the times of the two lower layers are t1, t2,
Assuming that the time of the upper layer is t3 (where t1 <t3 <t2), the interpolated image I (x, y, t3) at the time t3 is I (x, y, t3) = [(t2-t3) B (x, y, t1) + (t3-t1) B (x, y, t2)] / (t2-t1) (1)

Next, the decoded image E of the upper layer is synthesized with the interpolated image I obtained above. For this purpose, the area information M (x,
y, t) to create weight information W (x, y, t) for synthesis,
A composite image S is obtained by the following equation. S (x, y, t) = [1-W (x, y, t)] I (x, y, t) + E (x, y, t) W (x, y, t) (2) domain Information M (x, y, t) is 1 inside the selected area and 0 outside the selected area
The weight information W (x, y, t) can be obtained by applying a low-pass filter to the image a plurality of times.

That is, the weight information W (x, y, t) takes a value of 1 inside the selected area, 0 outside the selected area, and 0 to 1 at the boundary of the selected area. The above is the description of the image creating method in the combining unit 805.

The coded data coded by the lower layer coding section 804, the upper layer coding section 803, and the area information coding section 806 are integrated by a coded data integration section (not shown) and transmitted or stored.

Next, on the decoding side of the conventional method, the coded data is decomposed into coded data of a lower layer, coded data of an upper layer, and coded data of area information by a coded data decomposing unit (not shown). . These encoded data are decoded by a lower layer decoding unit 808, an upper layer decoding unit 807, and a region information decoding unit 809, as shown in FIG.

The combining unit 810 on the decoding side is composed of the same device as the combining unit 805 on the encoding side, uses the lower layer decoded image and the upper layer decoded image, and is the same as that described in the description on the encoding side. The images are combined by the method. The moving image synthesized here is displayed on the display,
It is input to the upper layer decoding unit 807 and used for prediction of the upper layer.

Here, the decoding device for decoding both the lower layer and the upper layer has been described. However, if the decoding device includes only the decoding unit for the lower layer, the upper layer encoding unit 807 and the combining unit 810 are unnecessary. Thus, a part of the encoded data can be reproduced with a small hardware scale.

[0021]

(1) In the prior art, when an output image is obtained from two lower layer decoded images and one upper layer decoded image as in the above equation (1), Since the interpolation of the two lower layers is performed, when the position of the selected area changes with time, a large distortion occurs around the selected area, and there is a problem that the image quality is largely deteriorated.

FIG. 18 illustrates this problem.
In FIG. 18A, images A and C are two decoded images of the lower layer, image B is a decoded image of the upper layer, and the display time order is A, B, C. However, the selected area is indicated by oblique lines. Also, since only the selected area is encoded in the upper layer, the area outside the selected area is indicated by a broken line.

Since the selected area is moving, images A and C
The interpolated image obtained from the above is an image in which the two selected regions overlap, as shown by the halftone dot portion in FIG. Further, when the image B is synthesized using the weight information, the output image is an image in which three selected regions overlap as shown in FIG.

In particular, around the selected area of the upper layer (outside)
Thus, the selected area of the lower layer appears as an afterimage, and the image quality is greatly deteriorated. As for the whole moving image, when only the lower layer is displayed, there is no such distortion, and when a composite image of the upper layer and the lower layer is displayed, the distortion appears. Then, the image quality is extremely deteriorated.

(2) In the conventional technique, an 8-way quantization code (FIG. 17) is used to encode region information.
When applied to a low bit rate or when the shape of an area becomes complicated, the ratio of the data amount of the area information to the entire coded data amount increases, which causes a problem of deteriorating the image quality.

(3) In the prior art, the weight information is obtained by applying a low-pass filter to the area information a plurality of times. However, since the filtering operation is performed a plurality of times, the processing amount increases. is there.

(4) In the prior art, predictive coding is used. However, even when there is a scene change in a lower layer, predictive coding may be used, resulting in large distortion. Since the distortion in the lower layer propagates to the upper layer, there is a problem that the distortion is maintained for a long time.

(5) In the prior art, since the moving picture coding international standardization scheme such as MPEG or H.261 is used in the lower layer, there is not much difference in image quality between the selected area and other areas. . On the other hand, in the upper layer, only the selected area is coded with high image quality, so that the image quality in the selected area changes with time, and this is detected as flicker-like distortion.

The object of the present invention is to solve these problems,
It is an object of the present invention to provide an image encoding device and an image decoding device that reduce the amount of data after encoding while not deteriorating the quality of a decoded image.

[0030]

According to the first aspect of the present invention, there is provided an image coding apparatus, comprising: a lower layer coding unit for coding one moving image sequence at a low frame rate; An upper layer encoding unit that encodes only one or a plurality of specific component regions at a high frame rate, and a moving image sequence of an upper layer is combined with a moving image sequence of a lower layer based on region information indicating the specific component region. A synthesizing unit, wherein when there is no lower layer frame corresponding to the position of the frame to be synthesized, the synthesizing unit includes a first layer in a lower layer temporally preceding the frame to be synthesized. Based on the component area information and the second component area information in the lower layer that is temporally later, the For a region including only the first component region, the lower layer frame that is temporally later is used, and for a region including only the second component region on the synthesis target frame, the temporally lower layer frame is used. The non-existent lower-layer frame is generated by using the lower-layer frame in (1).

According to a second aspect of the present invention, in the image encoding apparatus according to the first aspect, the synthesizing means does not include a lower layer frame corresponding to the position of the frame to be synthesized. In the case where the first component area and the second component area overlap on the synthesis target frame, the lower layer frames existing before and after the time are weighted and averaged to obtain the lower nonexistent lower layer frame. This is for generating a layer frame.

According to a third aspect of the present invention, in the image encoding apparatus according to the first aspect, the synthesizing means does not include a lower layer frame corresponding to a position of a frame to be synthesized. In the case, on the synthesis target frame, for a region where the first component region and the second component region overlap, by using the lower layer frame existing before or after the time, the lower layer frame that does not exist is used. Is generated.

According to a fourth aspect of the present invention, in the image encoding apparatus according to any one of the first to third aspects, the synthesizing means includes a lower level corresponding to a position of a frame to be synthesized. If the layer frame does not exist, the second component area is also
For an area that is not a component area, the non-existent lower layer frame is generated by performing a weighted average of the lower layer frames existing before and after the time.

According to a fifth aspect of the present invention, in the image encoding apparatus according to any one of the first to third aspects, the synthesizing means includes a lower level corresponding to a position of a frame to be synthesized. If the layer frame does not exist, the second component area is also
For an area that is not a component area, the non-existent lower-layer frame is generated by using one of the lower-layer frames existing before or after in time.

According to a sixth aspect of the present invention, in the image encoding apparatus according to any one of the first to fifth aspects, the synthesizing means includes a lower level corresponding to a position of a frame to be synthesized. A motion parameter estimating means for estimating a motion parameter between a lower layer image existing temporally before the frame to be synthesized and a lower layer image temporally existing when the layer frame does not exist; Depending on the parameter, a lower layer image existing before and after the frame to be synthesized temporally;
And a deforming means for deforming the first component region and the second component region by motion compensation based on the position of the compositing target frame.

An image decoding apparatus according to a seventh aspect of the present invention includes a lower layer decoding means for decoding coded data obtained by coding one moving image sequence at a low frame rate, and one or more of the moving image sequences. An upper layer decoding unit that decodes encoded data obtained by encoding only a plurality of specific component regions at a high frame rate; and a moving image sequence of an upper layer based on region information indicating the specific component region. A synthesizing unit for synthesizing the sequence into a sequence, wherein when there is no lower-layer frame corresponding to the position of the frame to be synthesized, the synthesizing unit may determine whether the lower layer exists temporally before the frame to be synthesized. First component area information;
On the basis of the second component area information in the lower layer that is temporally later, the lower layer frame that is temporally later is used for the area of only the first component area on the synthesis target frame, The non-existent lower-layer frame is generated by using the lower-layer frame that is temporally earlier in the area including only the second component area on the synthesis target frame.

An image decoding apparatus according to an eighth aspect of the present invention is the image decoding apparatus according to the seventh aspect, wherein:
When there is no lower layer frame corresponding to the position of the frame to be synthesized, the synthesizing unit determines whether the region where the first component region and the second component region overlap with each other on the synthesis target frame has the temporally earlier position. Then, the lower-layer frame that does not exist is generated by performing a weighted average of the lower-layer frame that exists later.

An image decoding apparatus according to a ninth aspect of the present invention is the image decoding apparatus according to the seventh aspect,
When there is no lower layer frame corresponding to the position of the frame to be synthesized, the synthesizing unit determines whether the region where the first component region and the second component region overlap with each other on the synthesis target frame has the temporally earlier position. Alternatively, the non-existent lower-layer frame is generated by using a lower-layer frame existing later.

According to a tenth aspect of the present invention, in the image decoding apparatus according to any one of the seventh to ninth aspects, the synthesizing means includes a lower layer frame corresponding to a position of a frame to be synthesized. If does not exist,
For the area that is neither the first component area nor the second component area on the compositing target frame, the lower layer frames existing before and after the time are weighted and averaged to obtain the lower layer frame that does not exist. This is to generate.

An image decoding apparatus according to an eleventh aspect of the present invention is the image decoding apparatus according to any one of the seventh to ninth aspects, wherein the synthesizing means includes a lower layer frame corresponding to a position of a frame to be synthesized. If does not exist,
For an area that is neither the first component area nor the second component area on the compositing target frame, the non-existent lower layer frame is obtained by using any of the lower layer frames existing before or after the time. Is generated.

According to a twelfth aspect of the present invention, in the image decoding apparatus according to any one of the seventh to eleventh aspects, the synthesizing means includes a lower layer frame corresponding to a position of a frame to be synthesized. Does not exist, a motion parameter estimating means for estimating a motion parameter between a lower layer image existing temporally before the frame to be synthesized and a lower layer image existing temporally after, and the motion parameter A lower-layer image temporally preceding and following the compositing target frame;
And a deforming means for deforming the first component region and the second component region by motion compensation based on the position of the compositing target frame.

[0042]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The first embodiment of the present invention solves the problem that occurs in the synthesizing unit 805 in FIG. That is, the present invention relates to an image synthesizing apparatus that does not generate distortion such as an afterimage around a selected region of an upper layer when an image is synthesized from two lower layer decoded images.
FIG. 1 is a block diagram showing an image synthesizing apparatus according to the first embodiment.

In FIG. 1, a first area extracting unit 101
Are the first area information of the lower layer and the second area information of the lower layer.
The region that is the first region and is not the second region is extracted from the region information. In FIG. 10A, if the first area information is represented by a dotted line (the inside of the dotted line has a value of 0 and the outside of the dotted line has a value of 1), and similarly, the second area information is represented by a broken line. The region extracted by the first region extraction unit 101 is a hatched portion in FIG.

The second area extracting unit 102 calculates the first area information of the lower layer and the second area information of the lower layer from the first area information.
An area that is the second area and is not the first area is extracted. In the case of FIG. 10A, a halftone dot portion is extracted.

The controller 103 controls the switch 104 based on the outputs of the first area extraction unit 101 and the second area extraction unit 102. That is, when the target pixel position is only in the first region, the switch 104 is connected to the second decoded image side, and when the target pixel position is only in the second region, the switch 104 is connected to the first decoded image. Side, and in other cases, the switch 104 is connected to the output from the interpolation image creation unit 105.

The interpolated image creating unit 105 calculates an interpolated image of the first decoded image of the lower layer and the second decoded image of the lower layer according to the above-mentioned conventional formula (1). Here, in equation (1), B (x, y, t1) is the first decoded image,
B (x, y, t2) is a second decoded image, I (x, y, t3) is an interpolated image, and t1, t2, and t3 are the first decoded image and the second
Are the times of the decoded image and the interpolated image.

As described above, an image is created.
For example, in the case of FIG. 10 (a), the second decoded image is used in the shaded area, so the background pixel outside the selected area appears, and in the halftone dot area, the first decoded image is used. , And an interpolated image of the first decoded image and the second decoded image appears in other portions.

Since the decoded image of the upper layer is superimposed on the image created in this way by the weighted averaging unit 106 shown in FIG. 1, the synthesized image is selected as shown in FIG. There is no afterimage around the shaded area) and an image with little distortion can be obtained. The weighted averaging unit 106 in FIG. 1 combines the above interpolated image and the decoded image of the upper layer by weighted averaging. Since the synthesizing method has been described in the above-mentioned prior art, the description is omitted here.

In the above-described first embodiment, the interpolation image creating unit 105 shown in FIG. 1 is provided.
Of the decoded image B (x, y, t1) and the second decoded image B (x, y, t2), the pixel value of the decoded image temporally closer to t3 which is the time of the upper layer is used. Is also good. In that case, using the frame number of each image, when t3-t1 <t1-t2, I (x, y, t3) = B (x, y, t1), otherwise, I (x, y, t3) = B (x, y, t2).

Here, t1, t2, and t3 are the times of the first decoded image, the second decoded image, and the decoded image of the upper layer, respectively.

Next, a second embodiment of the present invention will be described.
The present embodiment relates to an image synthesizing apparatus that more accurately synthesizes an image in consideration of motion information of a decoded image of a lower layer in the image synthesizing apparatus of the first embodiment. FIG.
FIG. 3 is a block diagram showing a device for estimating a motion parameter and transforming two decoded images and corresponding two pieces of area information.

In FIG. 2, a motion parameter estimating section 20
In step 1, motion information from the first decoded image to the second decoded image in the lower layer is estimated. For example, a motion vector is obtained for each block, or a motion (parallel movement, rotation, enlargement / reduction, etc.) of the entire image is obtained and used as a motion parameter.

In the deforming unit 202, the first decoded image, the second
, The first region information and the second region information are transformed based on the estimated motion parameters based on the temporal position of the synthesis target frame. For example, it is assumed that a motion vector (MVx, MVy) from the first decoded image to the second decoded image is obtained as a motion parameter. Where M
Vx is the horizontal component of the motion vector, and MVy is the vertical component of the motion vector.

At this time, a motion vector from the first decoded image to the interpolation image is calculated by (t3-t1) / (t2-t1) (MVx, MVy), and the first decoded image is converted to this motion vector. Shift. When rotation, enlargement / reduction, or the like is used as a motion parameter, deformation is involved instead of a simple shift.

In FIG. 2, the transformed data are represented by a, b, c and d, respectively, where a is the first data in FIG.
, B is the second decoded image in FIG. 1, and c is FIG.
Are input to the image synthesizing apparatus shown in FIG. 1 as the first area information in FIG. 1 and the second area information in FIG. 1 in d, and a synthesized image is created.

In the second embodiment, the motion parameters are estimated from two decoded images. However, at the time of predictive encoding, the motion vector of each block of the image is included in the encoded data. Since these are general, these motion vectors may be used.

For example, the average value of the decoded motion vectors is used as the motion vector of the entire image from the first decoded image to the second decoded image, or the frequency distribution of the decoded motion vector is obtained. A vector having a high frequency may be used as a motion vector of the entire image from the first decoded image to the second decoded image. The above processing is performed independently in the horizontal and vertical directions.

Next, a third embodiment of the present invention will be described.
The third embodiment relates to an area information encoding device that efficiently encodes area information. 3 and 4 are block diagrams of the present embodiment. FIG. 3 shows the encoding side, and FIG. 4 shows the decoding side.

The area information approximating unit 301 in FIG. 3 approximates the area information with a plurality of figures. FIG. 11 shows an example of approximation. In this example, a rectangle is used as a figure, and the area information (hatched portion) of a person is approximated by two rectangles. The rectangle 1 represents the head of the person, and the rectangle 2 represents the chest of the person.

The area approximation information encoding unit 302 encodes the approximated area information. As shown in FIG. 11, when approximated by rectangles, the upper left coordinate value of each rectangle and the size of the rectangle may be encoded with a fixed length.
Alternatively, when approximated by an ellipse, the coordinates of the center point of the ellipse, the length of the major axis, and the length of the minor axis may be encoded with a fixed length. The approximated area information and the encoded data are:
Sent to selection section 304.

The area information encoding unit 303 does not approximate the area information and encodes the area information using an 8-way quantization code, similarly to the area information encoding unit 806 described in the related art. The area information and the encoded data are sent to the selection unit 304.

The selection unit 304 is used to encode the region approximation information
02 or the output of the area information encoding unit 303. When the output of the region approximation information encoding unit 302 is selected, the coded data of the region approximation information is sent to a coded data integration unit (not shown) together with 1-bit selection information (for example, 0), and the region approximation information is shown. Not sent to the synthesis unit.

When the output of the area information encoding unit 303 is selected, the encoded data of the area information which is not approximated is set to 1
Along with the bit selection information (for example, 1), the information is sent to an unillustrated coded data integration unit, and non-approximate area information is sent to the image synthesis unit. The image synthesizing unit has been described in the first embodiment and the second embodiment of the present invention.

As a selection method in the selection unit 304,
For example, a method of selecting the smaller coded data amount, or the output of the region information coding unit 303 is selected when the coded data amount of the region information that is not approximated is within a certain threshold, and when the coded data amount exceeds the threshold, the region approximate information coding is performed. The output of the unit 302 is selected. By making such a selection, the amount of coded data can be reduced while suppressing the coding distortion of the area information.

Next, the decoding side (FIG. 4) of the third embodiment of the present invention will be described. Referring to FIG.
Selects whether the coded data is based on the area approximation information or based on the area information based on the 1-bit selection information included in the coded data.

The area approximation information decoding section 402 decodes the area approximation information. The area information decoding unit 403 decodes area information that is not approximated. The switch 404 is connected to the selection unit 40
1 and selects area approximation information or non-approximate area information as an output to the synthesis unit.

As described above, the region approximation information and the original region information which is not approximated are adaptively selected and encoded / decoded. Therefore, when the region information is complicated and has a huge data amount, The encoding of the area approximation information is selected, and the area information can be encoded with a small amount of information.

In the above example, the area information that is not approximated is 8
Although the encoding is performed using the direction quantization code, the encoding may be performed efficiently by further combining predictive encoding. The eight-way quantization code has a value of 0 to 7 as shown in FIG. 17, but if a difference is obtained by predictive coding, it becomes -7 to 7.

However, the difference value can be suppressed to -3 to 4 by adding 8 when the difference value is less than -4 and subtracting 8 when the difference value is larger than 4. At the time of decoding, the difference value is added to the previous value, and if the result is negative, 8 is added.
If it exceeds 7, the original 8-way quantized value can be obtained by subtracting 8.

An example is shown below. 8-way quantization value 1, 6, 2, 1, 3, ... Difference value 5, -4, -1, -2, ... Conversion value -3, 4, -1, 2, ... Decoding Value 1, 6, 2, 1, 3, ... For example, the difference between the value 6 and the previous value is 5,
Is subtracted to obtain -3. At the time of decoding, -2 is obtained by adding the previous value 1 to the decoded value -3. However, since the value is negative, 8 is added to this to obtain the decoded value 6. Such predictive coding utilizes the property that an 8-way quantization code is cyclic.

In the third embodiment, the coding of the approximated region information is performed independently for each image. However, since the moving image generally has a high correlation between frames, the previous coding result is Utilization may be used to increase the coding efficiency.

That is, when encoding of the approximated area information is continuous between frames, only the difference of the approximated area information is encoded. For example, the area is approximated by a rectangle, the rectangle of the previous frame is represented by the upper left point: (10, 20), the size: (100, 150), and the rectangle of the current frame is the upper left point: (13, 18) , Magnitude: (100, 152), the current frame encodes the upper left difference value: (3, 2) and the magnitude difference value: (0, 2).

When the change in the shape of the area is small, all the difference values are concentrated around 0. Therefore, if entropy coding such as Huffman coding is used, the code amount of the area information can be greatly reduced. Furthermore, when the rectangle does not change in many cases, 1-bit information in the current frame may be encoded as rectangular change information.

That is, when the rectangle does not change, only 1-bit information (for example, 0) representing this is coded, and when the rectangle changes, 1-bit information (for example, 1) and the above difference information are coded. I do.

Next, a fourth embodiment of the present invention will be described.
The fourth embodiment relates to a weight information creating device that creates multi-value weight information from region information. FIG. 5 is a block diagram of the present embodiment.

Referring to FIG. 5, a horizontal weight creating unit 501 is provided.
Scans the area information in the horizontal direction, finds a portion where the area information is 1, and finds a weighting function corresponding to it. Specifically, the coordinates x0 of the left end point of the area and the horizontal length N of the area
And a horizontal weighting function as shown in FIG. 12A is calculated.

The weight function may be created by combining straight lines, or may be created by combining straight lines and trigonometric functions. As an example of the latter, when the width of the trigonometric function part is W,
If N> 2W, when 0 ≦ x <W sin [(x + 1/2) π / (2W)] * sin [(x + 1/2) π / (2W)] W ≦ x <NW When 1 NW ≦ x <N sin [(x-N + 2W + 1/2) π / (2W)] * sin [(x-N + 2W + 1/2) π / (2W)] N ≦ If 2W, sin2 [(x + 1/2) π / N] * sin [(x + 1/2) π / N] can be used. However, the point x0 at the left end of the area is 0
And

The vertical weight generator 502 scans the area information in the vertical direction to find a portion where the area information is 1, and obtains a vertical weight function corresponding thereto. Specifically, the coordinates y0 of the upper end point of the region and the vertical length M of the region are obtained, and a vertical weighting function as shown in FIG. 12B is calculated.

The multiplier 503 includes a horizontal direction weight generator 50.
1 is multiplied by the output of the vertical weight generator 502 for each pixel position to generate weight information. By creating the weight information in this manner, the weight information that matches the shape of the area information can be obtained with a small amount of calculation.

Next, a fifth embodiment of the present invention will be described.
The fifth embodiment relates to a mode switching method for adaptively switching between intra-frame coding and inter-frame predictive coding in predictive coding of a lower layer or an upper layer. FIG. 6 is a block diagram of the present embodiment.

In FIG. 6, an average value calculation unit 601 receives an original image and area information, and calculates an average of pixel values for pixel values inside the area. The average value is input to the differentiator 603 and the storage unit 602. The differentiator 603 calculates a previous average value and an average value calculation unit 601 stored in the storage unit 602.
Calculates the difference from the current average value output from.

The determination section 604 compares the absolute value of the difference value calculated by the differentiator 603 with a predetermined threshold value, and outputs mode switching information. If the absolute value of the difference value is larger than the threshold, it is determined that there is a scene change in the selected area, and mode switching information is generated so that intra-frame encoding is always performed.

As described above, by performing the mode switching while judging the scene change of the selected area, a good coded image can be obtained even when, for example, a person appears from a shadow or the object is reversed. Obtainable.

This embodiment can be applied to a method of performing coding by separating a selected area and other areas in coding of a lower layer. In that case, the area information is input to the lower layer. Further, the present embodiment can be applied to encoding of only the selected region of the upper layer.

Next, a sixth embodiment of the present invention will be described.
The sixth embodiment relates to data amount control in a case where a selected area and another area are separately encoded in encoding of a lower layer. FIG. 7 is a block diagram of the present embodiment.

In FIG. 7, an encoding unit 703 encodes a selected area separately from the other areas. The region information is input to the region determining unit 701, and it is determined whether the region to be coded is within the selected region or outside the selected region. The generated code amount calculation unit 705 calculates the generated code amount in each area based on the determination result.

The target code amount distribution ratio calculation unit 704 determines the distribution ratio of the target code amount for each frame to be allocated to each area. The method for determining the distribution ratio will be described later. The quantization width calculation unit 702 determines the quantization width according to the target code amount, and the method for this determination is the same as the conventional method.

Here, a method of determining the distribution ratio in target code amount distribution ratio calculating section 704 will be described. First, the target code amount Bi of the frame is calculated using the following equation. Bi
= (Available code amount-used code amount up to the previous frame) / number of remaining frames This target code amount Bi is allocated to the selected area inside and outside the selected area at a certain ratio, but here an appropriate fixed ratio R0 and Using the previous frame complexity ratio Rp, the ratio is determined.

The previous frame complexity ratio Rp is determined by the following equation. Rp = (gen_bitF * avg_qF) / (gen_bitF * avg_qF + gen_bitB * avg
_qB) where gen_bitF: code amount generated in the previous frame selection area gen_bitB: code amount generated outside the previous frame selection area avg_qF: average quantization width within the selection area of the previous frame avg_qB: average quantization width outside the selection area of the previous frame is there.

To increase the quality of the selected area, the quantization width is controlled so that the average quantization width in the selected area is smaller than the average quantization width outside the selected area to some extent. It is desirable to follow the change of the image at the time.

In general, the distribution by the fixed ratio R0 is suitable for keeping the relation between the average quantization widths inside and outside the selected region almost constant, and the distribution by the previous frame complexity ratio Rp is the same as that in the moving image sequence. It is suitable to follow the change of the image at the time.

Therefore, in the present invention, the target code amount distribution ratio Ra
Is the average of the fixed ratio R0 and the previous frame complexity ratio Rp, and control having both advantages is performed. That is, Ra = (R0 + Rp) / 2.

For example, it is assumed that the fixed ratio R0 and the previous frame complexity ratio Rp in the selected area are as shown by a dotted line in FIG. 13 in the entire moving image sequence. At this time, the target code amount distribution ratio Ra is as shown by the solid line in FIG. 13, and it can be seen that the target code amount distribution ratio does not deviate much from the fixed ratio R0 and reflects a change in the image to some extent.

At this time, the fixed ratio outside the selected area is (1-R0),
Assuming that the previous frame complexity ratio is (1-Rp), the target code amount distribution ratio (1-Ra), which is the average of the two, is as shown by the solid line in FIG. The value obtained by adding the target code amount distribution ratio is 1.

In this way, the quantization width can be appropriately controlled. However, since the generated code amount may exceed the target code amount Bi depending on the frame, the bit rate of the entire sequence is reduced to a predetermined bit rate. Things that don't fit can happen. In such a case, the following method can be used.

As described above, the target code amount distribution ratio Ra in the selected area is an average of the fixed ratio R0 and the previous frame complexity ratio Rp. On the other hand, the target code amount distribution ratio outside the selected region is a fixed ratio outside the selected region (1-R0) and a previous frame complexity ratio (1-Rp).
With the minimum value Rm.

In this way, the change in the target code amount distribution ratio Rm outside the selected area is, for example, as shown by the solid line in FIG. At this time, since Ra + Rm ≦ 1, the target code amount of the frame can be reduced. That is, by suppressing the target code amount in the background area, the bit rate of the entire moving image sequence can be kept at a predetermined bit rate.

[0098]

According to the image encoding apparatus and the image decoding apparatus of the present invention, when an unencoded lower layer frame is generated from previous and subsequent lower layer frames, only the first component area is generated. Uses the lower layer frame that is temporally later, and uses the lower layer frame that is temporally earlier for the area of only the second component area, so that even if the target object is moving, the synthesized image is distorted. And a good image can be obtained.

Also, when the lower layer frame is generated, an area where the first component area and the second component area overlap, or an area that is neither the first component area nor the second component area,
By using a lower layer frame temporally closer to the lower layer frame to be combined, using only the first lower layer frame, or using only the second lower layer frame, the background area of the lower layer moves. In this case, the synthesized background image is not duplicated, and a good image can be obtained.

Further, when generating the lower layer frame,
By transforming the first lower layer image, the second lower layer image, the first component region, and the second component region by motion compensation based on the temporal position of the lower layer frame to be combined with the motion parameter, Even when the background area of the layer is moving, it is possible to generate a composite image that follows this movement, and it is possible to obtain a good image.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a second embodiment of the present invention.

FIG. 3 is a block diagram illustrating an encoding side according to a third embodiment of the present invention.

FIG. 4 is a block diagram illustrating a decoding side according to a third embodiment of the present invention.

FIG. 5 is a block diagram illustrating a fourth embodiment of the present invention.

FIG. 6 is a block diagram illustrating a fifth embodiment of the present invention.

FIG. 7 is a block diagram illustrating a sixth embodiment of the present invention.

FIG. 8 is a block diagram illustrating a conventional encoding device and decoding device.

FIG. 9 is a block diagram illustrating a conventional code amount control unit.

FIG. 10 is a diagram illustrating the effect of the first embodiment of the present invention.

FIG. 11 is an explanatory diagram showing an example of approximating region information according to a third embodiment of the present invention.

FIG. 12 is an explanatory diagram illustrating an example of a method for creating weight information according to a fourth embodiment of the present invention.

FIG. 13 is a diagram illustrating a target code amount ratio of a selected area according to a code amount control method according to a sixth embodiment of the present invention.

FIG. 14 is a diagram illustrating a target code amount ratio outside a selected area according to a code amount control method according to a sixth embodiment of the present invention.

FIG. 15 is a diagram illustrating a target code amount ratio outside a selected area according to another code amount control method according to the sixth embodiment of the present invention.

FIG. 16 is a diagram illustrating the concept of conventional video hierarchical coding.

FIG. 17 is a diagram illustrating an 8-way quantization code.

FIG. 18 is a diagram illustrating a problem of conventional video hierarchical coding.

[Explanation of symbols]

 Reference Signs List 101 first region extraction unit 102 second region extraction unit 103 controller 104 switch 105 interpolation image creation unit 106 weighted average unit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Norio Ito 22-22 Nagaikecho, Abeno-ku, Osaka-shi, Osaka Inside Sharp Corporation (72) Inventor Toshio Nomura 22-22 Nagaikecho, Abeno-ku, Osaka-shi, Osaka Inside the corporation

Claims (12)

[Claims] 1. A lower layer coding means for coding one moving image sequence at a low frame rate, and an upper layer coding means for coding only one or a plurality of specific component regions of the moving image sequence at a high frame rate. Means for synthesizing a moving image sequence of an upper layer with a moving image sequence of a lower layer based on region information indicating the specific component region. When there is no lower layer frame corresponding to the frame position, the first component region information in the lower layer temporally preceding the frame to be combined and the second component region information in the lower layer temporally subsequent to the compositing target frame On the compositing target frame, for the area of only the first component area, The non-existent lower-layer frame is generated by using a lower-layer frame that is temporally earlier in a region including only the second component region on the synthesis target frame while using a layer frame. An image encoding device, characterized in that: 2. The image encoding device according to claim 1, wherein the combining unit includes a first component area and a first component area on the combining target frame when there is no lower layer frame corresponding to the combining target frame position. An image coding method characterized in that, for a region where the second component region overlaps, the non-existing lower layer frame is generated by performing a weighted average of the lower layer frames existing before and after in time. apparatus. 3. The image encoding device according to claim 1, wherein the combining unit includes a first component area and a first component area on the synthesis target frame when there is no lower layer frame corresponding to the synthesis target frame position. An image coding apparatus, wherein a non-existent lower-layer frame is generated by using a lower-layer frame existing temporally before or after a region where a second component region overlaps. 4. The image encoding device according to claim 1, wherein the combining unit is configured to determine whether a lower layer frame corresponding to the position of the frame to be combined does not exist on the frame to be combined. For a region that is neither the first component region nor the second component region, the non-existent lower layer frame is generated by performing a weighted average of the lower layer frames existing before and after the time. Image encoding device. 5. The image encoding device according to claim 1, wherein the combining unit includes: when there is no lower layer frame corresponding to the position of the frame to be combined, on the frame to be combined; For a region that is neither the first component region nor the second component region, the non-existent lower layer frame is generated by using any of the temporally lower or higher-order lower layer frames. Image encoding device. 6. The image encoding device according to claim 1, wherein the combining unit is configured to determine a temporal position of the synthesis target frame when there is no lower layer frame corresponding to the synthesis target frame position. A motion parameter estimating means for estimating a motion parameter between a lower layer image existing before and a lower layer image existing after in time; and the motion parameter, before and after in time the synthesis target frame. An image coding apparatus, comprising: a deforming unit that deforms an existing lower layer image and a first component region and a second component region by motion compensation based on the synthesis target frame position. 7. A lower layer decoding means for decoding encoded data obtained by encoding one moving image sequence at a low frame rate, and encoding only one or a plurality of specific component areas of the moving image sequence at a high frame rate. Image decoding apparatus comprising: an upper layer decoding unit that decodes encoded data that has been encoded; and a combining unit that combines a moving image sequence of an upper layer with a moving image sequence of a lower layer based on region information indicating the specific component region. In the above, when the lower layer frame corresponding to the position of the frame to be synthesized does not exist, the synthesizing unit includes first component region information in a lower layer temporally preceding the frame to be synthesized and temporally following the lower layer. On the basis of the second component region information in the lower layer, only the first component region The lower layer frame that is temporally later is used for the area, and the lower layer frame that is temporally earlier is used for the area of only the second component area on the synthesis target frame. A non-existing lower layer frame. 8. The image decoding device according to claim 7, wherein, when there is no lower layer frame corresponding to the position of the frame to be combined, the combining unit includes a first component area and a first component area on the frame to be combined. An image decoding apparatus characterized in that, for an area where two component areas overlap, the lower layer frame that does not exist is generated by performing a weighted average of the lower layer frames that exist before and after the part. 9. The image decoding apparatus according to claim 7, wherein, when there is no lower layer frame corresponding to the position of the frame to be combined, the combining unit includes a first component area and a first component area on the frame to be combined. An image decoding apparatus characterized in that, for an area where two component areas overlap, the lower layer frame that does not exist is generated by using the lower layer frame that exists before or after the part in time. 10. The image decoding device according to claim 7, wherein the combining unit determines whether a lower layer frame corresponding to the position of the frame to be combined does not exist on the frame to be combined. For an area that is neither the one part area nor the second part area, the lower layer frame that does not exist is generated by performing a weighted average of the lower layer frames that exist before and after the time. Image decoding device. 11. The image decoding apparatus according to claim 7, wherein the combining unit determines whether a lower layer frame corresponding to the position of the frame to be combined does not exist on the frame to be combined. For an area that is neither a one component area nor a second component area, the generation of the non-existent lower layer frame is performed by using any of the lower layer frames existing before or after the time. Image decoding device. 12. The image decoding apparatus according to claim 7, wherein the combining unit determines a temporal position of the synthesis target frame when there is no lower layer frame corresponding to the synthesis target frame position. A motion parameter estimating means for estimating a motion parameter between a lower layer image existing before and a lower layer image existing later in time; An image decoding apparatus comprising: a lower layer image to be transformed, and first and second component areas, each of which is transformed by motion compensation based on the position of the compositing target frame.