JPH09307906A - Image coder and image decoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent distortion in a superimposed image even when an area shape is temporally changed by coding an compositing frames of a 1st area of a subordinate layer at a preceding time and a 2nd area of the subordinate layer at a succeeding time on a higher-order layer. SOLUTION: A controller 111 uses a switch 112 based on area information extracted by 1st and 2nd area extract sections 109, 110. That is, when a position of a pixel in question is in the 1st area, the switch 112 is thrown to a position of a decoder decoding an image at a frame time (t+a), when the position of a pixel in question is in the 2nd area, the switch 112 is thrown to a position of a decoder decoding an image at a frame time (t-a), and in other cases, the switch 112 is thrown to the position of an output of an interpolation image generating section 113. The changeover of the switch 112 allows composition of frames of subordinate layers and a weight mean section 114 applies weighted means to an interpolation image and a decoded image of a high-order layer to superimpose the images. Thus, no after-image is in existence around the selected area in the superimposed image and the image with less distortion is obtained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention belongs to the field of digital image processing, and is a moving picture coding method for coding image data with high efficiency and a moving picture decoding method for decoding coded data created by this moving picture coding apparatus. The present invention relates to a decoding device.

[0002]

2. Description of the Related Art In image coding, a method of superimposing different moving image sequences has been studied.

For example, the document "Image Coding Using Hierarchical Representation and Multiple Templates" (Technical Report IE94-159,
pp99-106 (1995)), a new moving image sequence is created by superimposing a moving image sequence that is a background and a moving image sequence of a component moving image that is a foreground (for example, a human image or a video of a fish cut out by chroma key technology). The method to do is described.

[0004] In addition, the document "Time hierarchical coding based on image contents"("TemporalScalability")
based on image content ”,
ISO / IEC / JTC1 / SC29 / WG11 M
PEG95 / 211 (1995)) describes a method of creating a new sequence by superimposing a moving image sequence of a component moving image having a high frame rate on a moving image sequence having a low frame rate.

In this system, as shown in FIG. 10, the predictive coding is performed at a low frame rate in the lower layer, and the predictive coding is performed at a high frame rate only in the selected region (hatched portion) in the upper layer. However, the frame encoded in the lower layer is not encoded in the upper layer, and the decoded image of the lower layer is directly copied and used. Also, it is assumed that a part that attracts the viewer's attention, such as a person part, is selected as the selected area.

FIG. 8 shows a block diagram of a conventional method. First, on the encoding side of the conventional method, the input moving image is frame-decimated by the first frame dropping unit 801 and the second frame dropping unit 802, and is set to be equal to or lower than the frame rate of the input image. And to the lower layer encoding unit 804. Here, it is assumed that the frame rate of the upper layer is equal to or higher than the frame rate of the lower layer.

The lower layer encoding unit 804 encodes the entire input moving image. Examples of the encoding method include MPEG and H.264. H.261 or the like is used. In the lower layer encoding unit 804,
The decoded image of the lower layer is created and used for predictive coding, and at the same time, input to the superimposing unit 805.

In the upper layer coding unit 803 of FIG. 8, only the selected area of the input moving image is coded. Again, MPEG and H.264. An international standardization method such as H.261 is used, but only selected areas are coded based on area shape information. However, frames encoded in the lower layer are not encoded in the upper layer. The area shape information is information indicating the shape of a selected area such as a human part, and takes a value of 1 at the position of the selected area and a value of 0 at other positions, for example.
It is a value image. The upper layer encoding unit 803 also decodes only the selected region of the moving image and inputs the decoded region to the superimposing unit 805.

In the area shape encoding unit 806, the area shape is 8
It is encoded using a direction quantization code. As shown in FIG. 11, the 8-direction quantized code is a numerical value indicating the direction to the next point, and is generally used when expressing a digital figure.

The superimposing section 805 outputs the decoded image of the lower layer at the frame position where the lower layer frame is encoded. At a frame position where the lower layer frame is not encoded, an image is created by using two encoded lower layer decoded images before and after the target frame and one upper layer decoded image at the same time as the target frame. Output.
The image created here is input to the upper layer encoding unit 803 and used for predictive encoding. The image creating method in the superimposing unit 805 is as follows.

First, two lower layer interpolated images are created. The decoded image of the lower layer at time t is B
(X, y, t) (where x and y are coordinates representing pixel positions in space), the times of the two lower layers are t1 and t2, respectively, and the times of the upper layers are t3 (where t is t).
1 <t3 <t2), the interpolated image I (x, y, t3) at time t3 is I (x, y, t3) = [(t2-t3) B (x, y, t1) + (t3 -t1) B (x, y, t2)] / (t2-t1) (1).

Next, the decoded image E of the upper layer is superimposed on the interpolated image I obtained above. For this reason, the area shape information M
Weight information W (x, y, t) for superimposing from (x, y, t)
t) is created, and the superimposed image S is obtained by the following equation.

S (x, y, t) = [1-W (x, y, t)] I (x, y, t) + E (x, y, t) W (x, y, t) (2 The area shape information M (x, y, t) is a binary image having a value of 1 inside the selected area and 0 outside the selected area, and the weight information W is obtained by applying a low pass filter to this image a plurality of times. (X,
y, t) can be obtained.

That is, the weight information W (x, y, t) is 1 inside the selected area, 0 outside the selected area, and 0 at the boundary of the selected area.
Takes a value of 1. The above is the description of the image creating method in the superimposing unit 805.

The coded data coded by the lower layer coding unit 804, the upper layer coding unit 803, and the area shape coding unit 806 are integrated by a coded data unifying unit (not shown) and transmitted or stored.

Next, on the decoding side of the conventional method, the coded data is decomposed into a coded data of a lower layer, a coded data of an upper layer and a coded data of a region shape by a coded data decomposing unit (not shown). These encoded data are decoded by the lower layer decoding unit 808, the upper layer decoding unit 807 and the area shape decoding unit 809 as shown in FIG.
The decoding-side superimposing unit 810 is composed of the same device as the encoding-side superimposing unit 805, and uses the lower layer decoded image and the upper layer decoded image to superimpose the images by the same method as described in the description of the encoding side. To be done. The moving image superimposed here is displayed on the display and the upper layer decoding unit 8
It is input to 07 and used for prediction of the upper layer.

Although the decoding device for decoding both the lower layer and the upper layer has been described here, if the decoding device includes only the decoding unit for the lower layer, the upper layer decoding unit 80.
7. The superimposing unit 810 is unnecessary, and a part of encoded data can be reproduced with a small hardware scale.

[0018]

In the prior art, when the output image is obtained from the two lower layer decoded images and the one upper layer decoded image as in the equation (1), the interpolation of the two lower layers is performed. Therefore, when the position of the selection area changes with time, a large distortion occurs around the selection area,
There is a problem that the image quality is greatly deteriorated.

FIG. 12 illustrates this problem.
In FIG. 12A, images A and C are two decoded images of the lower layer, image B is a decoded image of the upper layer, and the display time order is A, B, and C. However, the selected area is indicated by oblique lines. Further, in the upper layer, only the selected area is encoded, so that the area outside the selected area is indicated by a broken line. Since the selected area is moving, the interpolated image obtained from the image A and the image C has two selected areas that overlap, as in the halftone dot portion of FIG.

Further, when the image B is superposed using the weight information, the output image becomes an image in which three selected areas overlap as shown in FIG. 12 (c). In particular, the selected area of the lower layer appears around the selected area of the upper layer (outside) like an afterimage, and the image quality is greatly deteriorated. When only the lower layer is displayed as the whole moving image, the above distortion does not occur,
When the superimposed image of the upper layer and the lower layer is displayed, the above-mentioned distortion appears, so that flicker-like distortion occurs and the image quality deteriorates significantly.

An object of the present invention is to solve these problems and to provide an encoding device and a decoding device which reduce the amount of data after encoding and do not deteriorate the quality of a decoded image.

[0022]

In order to achieve the above object, the present invention provides (1) lower layer coding for coding a moving picture sequence at a low frame rate, and specifying one or more of the moving picture sequences. Performs upper layer encoding that encodes a region or the entire region at a high frame rate, decodes the lower layer by decoding only the lower layer with a lower frame rate, and decodes up to the upper layer with the lower layer and the higher frame rate. A hierarchical encoding device and a decoding device that decodes an upper layer and superimposes a lower layer and an upper layer, and exists before and after a lower layer frame corresponding to the same frame position as the upper layer does not exist at the time of decoding. The non-existing lower layer frame is synthesized by using the lower layer, and the lower layer which is present in time before the synthesis is performed. Encoding the first area shape of the ear and the second area shape of the lower layer that exists temporally later in the upper layer, and performing the combining using the first area shape and the second area shape, and (2) In the moving picture coding apparatus and the moving picture decoding apparatus according to (1), when a lower layer frame corresponding to the same frame position as the upper layer does not exist at the time of decoding, the lower layer that exists temporally earlier The first region shape of the layer and the second region shape of the lower layer that exists later in time are not encoded, and the first region shape and the second region shape are used as one or both of the lower layer and the upper layer. From the decoded data of (3), and further, (3) at the time of encoding the upper layer of (1) and (2), a first flag indicating whether or not to encode the pixel information of the upper layer is provided, In the decoding device, the first flag is used to judge whether only the area shape is coded by the ear, or when both the area shape and the pixel information are coded. Further, (4) and (2) In, when there is no lower layer frame corresponding to the same frame position as the upper layer, the decoded image of the lower layer is divided into regions when the region shape of the lower layer that exists temporally before and after is extracted. Is used to extract the region shape, and (5) In (2) above, when there is no lower layer frame corresponding to the same frame position as the upper layer, the lower layer that exists temporally before and after the lower layer frame When extracting the area shape, estimating and extracting the area shape by using the area shape obtained at the time of decoding the upper layer, and (6) above (1) to (5)
In the above, when there is no lower layer frame corresponding to the frame position of the upper layer at the time of decoding, it is indicated whether or not the lower layer existing before and after is combined with the lower layer frame which does not exist. Flag is used, and the lower layer frame existing before or after is used as the combined lower layer frame. Further, (7) In (6), when the lower layer frame is combined, it is preceded in time. A third flag indicating whether to encode the existing first layer shape of the lower layer and a fourth flag indicating whether to encode the second area shape of the lower layer that exists temporally later are set. If both the first area shape and the second area shape are not coded, the area shapes used in the previous composition are set as the area shapes used in the current composition, and In the case of encoding only the region shape, the second region shape used in the previous synthesis is set as the first region shape used in the current synthesis, and there is no case where only the first region shape is encoded. It is what

[0023]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIGS. 1, 14 and 15 are block diagrams showing a first embodiment. In FIG. 14, each unit other than the upper layer encoding unit 1403, the superimposing unit 1405, the upper layer decoding unit 1406, and the superimposing unit 1408 has the same function as in FIG.

The switch 101, the pixel data coding unit 102, the region shape coding unit 103, the pixel data decoding unit 104, the region information decoding unit 105 and the multiplexing unit 106 shown in FIG.
The upper layer coding unit 1403 of FIG. 14 is configured. In addition, the first delay unit 107, the second delay unit 108 of FIG.
First area extraction unit 109, second area extraction unit 110, controller 111, switch 112, interpolation image creation unit 1
13, the weighted average unit 114, the third delay unit 115, and the fourth
The delay unit 116 of the above constitutes the superimposing unit 1405 or the superimposing unit 1408 of FIG. The superimposing unit 1408 has the same function as the superimposing unit 1405.

The first embodiment will be described in detail below with reference to FIG.

The switch 101 switches between a mode in which the pixel values of the upper layer are encoded and a mode in which they are not encoded by a controller (not shown). That is, when the lower layer frame corresponding to the same frame position as the upper layer is encoded, the switch is controlled to be off, and the pixel value of the upper layer is not encoded. On the other hand, when the lower layer frame corresponding to the same frame position as the upper layer is not encoded, the switch is controlled to be turned on and the pixel value of the upper layer is encoded.

The pixel data encoding unit 102 includes a switch 10
When 1 is on, the pixel data of the upper layer is encoded. The encoding method is MPEG or H.264. An international standardization method such as 261 is used. However, at the time of encoding, the area shape is input from the area shape decoding unit 105 via a signal line (not shown), and only the pixel data in the area is encoded.

The area shape encoding unit 103 encodes the area shape in the same manner as the area shape encoding unit 806 of FIG. 8 described in [Prior Art].

The pixel data decoding unit 104 decodes the pixel data coded by the pixel data coding unit 102. Here again, the area shape is input from the area shape decoding unit 105 via a signal line (not shown) and used for decoding. The decoded pixel data is input to the third delay unit 115, fed back to the pixel data coding unit via a signal line (not shown), and used for prediction.

The area shape decoding unit 105 decodes the area shape data encoded by the area shape encoding unit 103, and outputs the decoded data to the first delay unit 107.

As described above, in the upper layer encoding unit of this embodiment, the switch 101 controls whether the upper layer is encoded or not. Next, the superposition unit of the present embodiment will be described.

The first delay unit 107 converts the area shape data into a
Delay only the frame. The delayed region shape data is input to the weighted average unit 114. The second delay unit 108 further delays the area shape data by b frames. The delayed area shape data is input to the first area extracting unit 109 and the second area extracting unit 110. In these circuits, the area shape data that has not been delayed is also input at the same time. Here, the symbols t + a, t, t-b on the signal line represent the time of each frame. Moreover, t, a, and b are integers.

The first area extraction unit 109 extracts an area which is the second area and is not the first area from the first area information and the second area information. In the case of FIG. 9A, the halftone dot portion is extracted.

The first area extracting section 110 extracts an area that is the first area and is not the second area from the first area information and the second area information. In the case of FIG. 9A, the shaded area is extracted.

The controller 111 is a part for controlling the switch 112 by the outputs of the first area extracting unit 109 and the second area extracting unit 110. That is, when the target pixel position is only the first area, the switch 112 is connected to the decoded image side at the frame time (t + a), and when the target pixel position is only the second area, the switch 112 is connected to the frame time ( tb) is connected to the decoding side, and in other cases, the switch 112 is connected to the output from the interpolation image creating unit 113.

The third delay unit 115 delays the decoded image data of the upper layer by a frame, and inputs the decoded image data at time t to the weighted average unit 114. Fourth delay unit 1
Reference numeral 16 delays the decoded image data of the lower layer by (a + b) frames and inputs the decoded image at time (t−b) to the interpolation image creating unit 113.

The interpolated image forming unit 113 interpolates the decoded image at the frame time (t-b) of the lower layer and the decoded image at the time (t + a) of the lower layer into the equation (1) described in [Prior Art]. ). However, in equation (1), B
(X, y, t1) is the first decoded image, B (x, y, t1)
2) is the second decoded image, I (x, y, t3) is the interpolated image, t1, t2 and t3 are the first decoded image, respectively.
It is the time of the second decoded image and the interpolated image. Therefore, FIG.
If the symbol is used, t1 = t−b, t2 = t + a, t3
= T.

As described above, since the lower layer composition is performed by using the switching of the switch 112, for example, in the case of FIG. 9A, the second decoded image (frame time t + a) is shown in the shaded area.
, The background pixels outside the selected area appear,
Since the first decoded image (frame time t-b) is used in the halftone dot portion, background pixels outside the selected area appear, and in other parts, the interpolated image of the first decoded image and the second decoded image is displayed. Appears. Since the decoded image of the upper layer is superimposed on the lower layer thus synthesized by the weighted average unit 114 of FIG. 1, the superimposed image is displayed around the selected area (hatched portion) as shown in FIG. 9B. An image with no afterimage and little distortion can be obtained. The weighted average unit 114 of FIG. 1 superimposes the above-described interpolated image and the decoded image of the upper layer by weighted average. The superposition method has been described in [Prior Art], and therefore the description thereof is omitted here.

FIG. 15 is a block diagram of the upper layer decoding unit 1406 in the decoding device of FIG. The diversion unit 1501 is a part that diverts the upper layer encoded data into the encoded data of the pixel data and the encoded data of the area shape. The switch 1504 is turned off at a frame position where the lower layer is encoded, and is turned on at a frame position where only the upper layer is encoded. The pixel data decoding unit 1502 decodes the pixel data of the upper layer, outputs the decoded pixels to the superimposing unit 1408, the area shape decoding unit 1503 decodes the area shape coded data, and the decoded pixels are superimposed on the superimposing unit 14.
It is a part to output to 08. The superposition unit 1408 is the superposition unit 1
By the same operation as 405, the upper layer is superimposed on the lower layer by the method of the present invention.

Next, a second embodiment of the present invention will be described. In this embodiment, the upper layer coding apparatus is provided with a mode in which the area shape is not coded to reduce the number of coding bits. When the area shape changes little or not at all with time, the number of bits can be significantly reduced by using the method of the present embodiment.

This embodiment is represented by FIGS. 2, 14 and 16. Although FIG. 14 is a diagram used for explaining the first embodiment, it is also used for explaining the second embodiment. However, upper layer encoding section 1403, upper layer decoding section 1406, superimposing section 1405 and superimposing section 1 in FIG.
The function of 408 is different from that of the first embodiment. These different parts will be described below.

FIG. 2 is a block diagram of the upper layer coding unit and the superimposing unit in the second embodiment. Switch 2
01, pixel data encoding unit 203, area shape encoding unit 2
04, pixel data decoding unit 205, region shape decoding unit 206
The description of the multiplexing unit 207 and the multiplexing unit 207 will be omitted because it has the same function as that described in FIG.

The switch 202 and the switch 208 are controlled by a control unit (not shown) so as to be turned on or off at the same time. When the lower layer frame corresponding to the same frame position as the upper layer is coded, the switch is controlled to be turned on and the region shape of the upper layer is coded. on the other hand,
When the lower layer frame corresponding to the same frame position as the upper layer is not encoded, the switch is controlled to be off, and the region shape of the upper layer is not encoded.

The first area shape extraction unit 209 extracts the first area shape based on the data obtained by the decoding device. Similarly, the second area shape extraction unit 210 extracts the second area shape based on the data obtained by the decoding device. As the data obtained by the decoding device, the decoded image of the lower layer and the region shape coding mode are turned on (switch 20
2. Area shape of the upper layer when the switch 208 is on). Although not shown in FIG. 2, these data are input to each area shape extraction unit and used for extraction of the area shape.

The first area extracting unit 211, the second area extracting unit 212, the controller 213, the switch 214, the interpolation image creating unit 215 and the weighted average unit 216 have the same functions as those already described in FIG. Therefore, the description is omitted.
In addition, the delay unit 217 functions similarly to the fourth delay unit 116 in FIG.

FIG. 16 is a block diagram of the upper layer decoding unit in the second embodiment. In this figure, 16
01 is a shunt unit, 1602 is a pixel data decoding unit, 1603
Is a region shape decoding unit, 1604 is a switch, and 1605 is a switch.

The difference between this figure and FIG. 15 is that a switch 1605 is provided in FIG. Switch 160
5 is turned off when there is no coded data of the lower layer frame corresponding to the same frame position at the time of decoding the upper layer. It is turned on at the frame position where the pixel data is encoded in the upper layer.

As described above, in the second embodiment, the switch 202 and the switch 208 are turned off when the lower layer frame corresponding to the same frame position as the upper layer is not encoded by the upper layer encoding unit. Control to
Since the area shape of the upper layer is not coded, the code amount can be reduced.

In the above-described first and second embodiments, the switch 101 or the switch 201 is used to switch on / off the encoding of the pixel data. This switching is performed by a control unit (not shown in FIGS. 1 and 2) by determining whether or not the lower layer frame corresponding to the same frame position as the upper layer is encoded. This determination can be made similarly by the encoding device and the decoding device. However, at least in the decoding device, it is possible to switch the switches without using the control unit for making such a determination. For that purpose, the first flag generator 301 and the first flag encoder 302 as shown in FIG. 3 may be provided in the encoder, and the switch in the upper layer encoder 303 may be switched according to the flag. However, FIG. 3 is a block diagram showing a part of the encoding device and the decoding device.

The flag generator 301 determines whether or not the lower layer frame corresponding to the same frame position as the upper layer is coded and generates a flag. The first flag encoding unit 302 encodes the first flag, and the encoded data is multiplexed with the encoded data by a multiplexing unit (not shown) and transmitted or stored. Fixed-length coding, variable-length coding, or the like is used as a flag coding method.

First flag decoding unit 30 in the decoding device
4 decodes the first flag from the encoded data and outputs it to the upper layer decoding unit 305. When the switch 1504 or the switch 1604 included in the upper layer decoding unit is switched, the switching is performed according to the decoded first flag without performing the above-described switching determination.

Next, the area shape extraction unit in the second embodiment will be described. Here, the second region shape is extracted based on the data obtained by the decoding device, but the decoded data of the lower layer, the decoding region shape of the upper layer, or the like may be used as the data obtained by the decoding device.

FIG. 4 is a block diagram of the superimposing unit in the case of extracting the area shape using the decoded data of the lower layer.
The first decoded image of the lower layer delayed by the delay unit 401 is input to the first area shape extraction unit 402, and the decoded image of the lower layer is input to the second area shape extraction unit 403 without delay. To be done. Each area shape extraction unit divides the input decoded image so as to extract the selected area, and extracts the area shape. As a method of region division, an edge detection method using a differential operation, morphological segmentation, or the like is used. Since the other parts of FIG. 4 have the same functions as those of FIG. 2, the description thereof is omitted here.

FIG. 5 is a block diagram when a decoding area shape of an upper layer is used as data obtained by the decoding device. In this figure, 501 is a delay unit and 502 is a first unit.
Region shape extraction unit, 503 is a second region shape extraction unit, 5
04 is a first area extraction unit, 505 is a second area extraction unit,
Reference numeral 506 is a controller, 507 is a switch, 508 is an interpolation image creating unit, and 509 is a weighted average unit.

In FIG. 5, when the region shape of the upper layer is encoded, the decoded data is the first region shape extraction unit 5
02 and the second area shape extraction unit 503. Each area shape extraction unit stores the decoded area shape,
The area shape corresponding to the lower layer frame is extracted. For example, as shown in FIG. 13, a method of extracting the region shape at the lower layer frame position by affine transformation expressing parallel movement, rotation, scaling, etc. from the decoding region shapes 1 and 2 of the upper layer before and after the lower layer frame. Can be considered.

For that purpose, first, the affine transformation from the area shape 1 to the area shape 2 is obtained. That is, the affine transformation parameter that approximates the area shape 2 is obtained by converting the area shape 1. Next, the affine transformation from the region shape 1 to the lower layer frame is obtained by linearly interpolating the transformation coefficient. The area shape on the lower layer frame can be obtained by using this affine transformation. Instead of the affine transformation, prediction from the area shape 1 to the area shape 2 may be performed by block matching, and the result may be linearly interpolated to obtain the area shape on the lower layer frame. Alternatively, the area shape 1 or 2 can be used as it is as the area shape on the lower layer frame.

In the second embodiment, the switch 202 of FIG. 2 is turned off in the frame in which the lower layer is not encoded, and the switch 202 is turned on in the frame position in which the pixel data of the upper layer is encoded, but this is different. You may control. For example, when the temporal change of the area shape is examined, the switch 202 is turned off when it hardly changes, and turned on otherwise, and as the decoded area shape data when it is off, a copy of the area shape data coded / decoded immediately before is made. May be used.

Next, a third embodiment will be described.
The present embodiment does not combine the lower layer frames described in the first and second embodiments when the lower layer encoded data corresponding to the frame position of the upper layer does not exist at the time of decoding the upper layer. The purpose is to provide a mode. For example, when the region shape does not change much over time, the problem described in [Problems to be Solved by the Invention] can be ignored, and therefore it is not necessary to combine lower layer frames. Even if the region shape changes significantly, it is possible to select a mode in which lower layer frame synthesis is not performed in order to prevent the encoding device and the decoding device from increasing the processing amount. For this purpose, a second flag generation unit 601 and a second flag coding unit 602 are provided in the coding device and a second flag decoding unit 604 is provided in the decoding device, as shown in FIG. However, FIG. 6 is a block diagram showing a part of the encoding device and the decoding device.

The second flag generator 601 in FIG. 6 generates a flag indicating whether or not to combine lower layer frames. The superimposing unit 603 switches between combining and not combining the lower layer frame according to the second flag. The second flag encoding unit 602 encodes the second flag, and the encoded data is multiplexed with the encoded data by a multiplexing unit (not shown) and transmitted or stored. Fixed-length coding, variable-length coding, or the like is used as a flag coding method.

Second flag decoding unit 60 in the decoding device
4 decodes the second flag from the encoded data and superimposes it on the superposition unit 60.
5 is output. The superimposing unit 605 switches whether or not to combine the lower layer according to the decoded second flag.

When the lower layer is not combined in the third embodiment, it is encoded in the lower layer,
Any one of the decoded lower layer frames before and after is used instead of the combined lower layer frame. The circuit configuration in this case is shown in FIG. In this figure, 701
Is a switch, 702 is a switch, 703 is a pixel data coding unit, 704 is a region shape coding unit, 705 is a pixel data decoding unit, 706 is a region shape decoding unit, 707 is a multiplexing unit, 708 is a switch, and 709 is a lower order. Layer composition section, 7
Reference numeral 10 is a switch, and 711 is a weighted average unit.

Next, the operation of the circuit of FIG. 7 will be described. First, the decoded image of the lower layer and the lower layer frame combined by the lower layer combining unit 709 are switched by the switch 71.
It is switched at 0 and input to the weighted average unit 711. The lower layer synthesizing unit 709 in FIG. 7 synthesizes the lower layer frames according to the methods described in the first and second embodiments. The switch 710 is switched to the lower side when the lower layer composition is on and to the upper side when the lower layer composition is off according to the second flag described in FIG.

The following method may be used as the area shape encoding method used for combining the lower layers in the third embodiment. That is, at the frame position of the upper layer where the lower layer composition is performed, the region shape in the lower layer before and after the frame position is encoded at the current frame position. An upper layer encoding unit and an upper layer decoding unit using this method are shown in FIG. 17 and FIG. 18, respectively.

In FIG. 17, reference numeral 1701 is a pixel data encoding unit, 1702 is a pixel data decoding unit, and 1703 is a first pixel data decoding unit.
Delay unit, 1704 second delay unit, 1705 switch, 1706 switch, 1707 region shape encoding unit, 1708 region shape decoding unit, 1709 third flag generating unit, 1710 fourth flag generating unit. 1711 is a third flag coding unit, 1712 is a fourth flag coding unit, 1713 is a controller, and 1714 is a multiplexing unit.

In FIG. 17, the pixel data coding unit 1701 and the pixel data decoding unit 1702 have the same functions as those described in the description of FIG. 1, and therefore their description will be omitted. In FIG. 17, the area shape is delayed by a frame by the first delay unit 1703 and further delayed by b frame by the second delay unit 1704.

The third flag generating section 1709 and the fourth flag generating section 1710 generate a third flag and a fourth flag, respectively. The third flag indicates whether to encode the area shape at the frame time t + a (hereinafter referred to as area shape 2), and the fourth flag indicates the area shape at the frame time t-b (hereinafter, area shape 1). Yes) is encoded or not. The controller 1713 controls the switch 1705 and the switch 1706 by inputting the third flag and the fourth flag.

That is, the switch 1705 is turned on when the third flag indicates that the area shape is to be encoded, and the switch 1705 is turned off otherwise. If the fourth flag indicates that the region shape is to be coded, the switch 1706 is turned on, and if not, the switch 1706 is turned off. The third flag coding section and the fourth flag coding section respectively code the third flag and the fourth flag. Fixed-length coding, variable-length coding, or the like is used as a flag coding method.

The area shape coding unit 1707 codes the area shape at the frame time when the area shape is input and outputs the coded data. The area shape decoding unit 1708 decodes the encoded data of the area shape, and sends the decoded area shape to the superimposing unit. Here, the device as shown in FIG. 1 is used as the superimposing unit, but the first delay unit 107 and the second delay unit 108 of FIG. 1 are not used. The decoded data of the area shape 1 is input to the first area extraction unit 109 and the second area extraction unit 110 of the superposition unit, and similarly, the decoded data of the area shape 2 is also the first area extraction unit 109 and the second area. It is input to the extraction unit 110. On the other hand, the decoded data of the area shape corresponding to the frame time t is input to the weighted average unit 114.

The on / off combinations of the switches 1705 and 1706 are controlled in the following three ways. That is, both switches are on, both switches are off, switch 1705 is on, and switch 1706 is off. When synthesizing the lower layer for the first time, both switches are both controlled to be turned on, the region shape at the front and rear frame positions, the toe region shape 1 and the region shape 2 are encoded / decoded, and the decoded region shape is the first Region extraction unit 109
And the second area extraction unit 110. Area shape 1
When the same area shape as that used at the previous lower layer composition is used as the area shape 2, both switches are controlled to be off. In this case, the area shape 1 and the area shape 2 used at the time of the previous lower layer composition are input to the first area extracting unit 109 and the second area extracting unit 110 from a memory (not shown).

When the area shape 2 used at the previous lower layer composition is used as the area shape 1 at the current lower layer composition, and the new area shape is used as the area shape 2 at the current composition, the switch 1705 is used. Is turned on and the switch 1706 is turned off. In this case, the area shape 2 used at the time of the previous lower layer composition is input as the area shape 1 at this time composition from the memory (not shown) to the first area extraction unit 109 and the second area extraction unit 110. Also,
The newly-encoded / decoded region shape 2 is input to the first region extraction unit 109 and the second region extraction unit 110.

The pixel data at the frame time t + a decoded by the pixel data decoding unit 1702 of FIG. 17 is delayed by a frame by the third delay unit 115 in the superposition unit of FIG.
It is input to the weighted average unit 114. The multiplexing unit 17 of FIG.
Reference numeral 08 multiplexes the pixel data, the region shape, the coded data of each of the third flag and the fourth flag, and outputs the coded data of the upper layer.

Next, the upper layer decoding unit for decoding the above-mentioned upper layer encoded data will be described with reference to FIG.
In this figure, 1801 is a shunting unit, 1802 is a pixel data decoding unit, 1803 is a region shape decoding unit, 1804 is a third flag decoding unit, 1805 is a fourth flag decoding unit, 1
806 is a first delay unit, 1807 is a second delay unit, 18
Reference numeral 08 is a switch, 1809 is a switch, and 1810 is a controller.

In FIG. 18, the diversion unit 1801 is a part for separating the upper layer encoded data into each encoded data of the pixel data, the region shape, the third flag and the fourth flag. Pixel data decoding unit 1802, area shape decoding unit 18
03, the first delay unit 1806, the second delay unit 1807,
The switches 1808 and 1809 have the same functions as those in FIG.

The third flag decoding unit and the fourth flag decoding unit in FIG. 18 decode the third flag and the fourth flag, respectively, and supply them to the controller 1810. The controller 1810 controls two switches similarly to the controller 1713 of FIG.
At 01, control for extracting the coded data of the area shape is also performed. That is, when the third flag indicates that the area shape 1 has been encoded, the area shape 1 data is separated from the upper layer encoded data, and otherwise the area shape 1 data does not exist. It is controlled not to be separated from the upper layer encoded data.

With respect to the fourth flag, similar control is performed on the flow dividing unit 1801. Two switches on
The combination of OFFs is 3 as in the upper layer encoding unit in FIG.
There are types. The operation in each combination is shown in FIG.
Is the same as that described in the explanation.

As described above, as the region shape encoding method used in the synthesis of the lower layer in the third embodiment,
At the frame position of the upper layer that performs lower layer composition,
It is possible to use a method of encoding the region shape in the lower layer of the frame positions before and after that at the current frame position.

In the above description of the present invention, the image in which the upper layer frame is superimposed on the lower layer frame is
In the encoding device, as shown in FIG. 14, it is fed back to the upper layer encoding unit and used for predictive encoding of the upper layer. In the decoding device, it is used for predictive coding of the upper layer and displayed on a display or the like. However, the superimposed image may be used only for display.

That is, the coding apparatus does not have the superposition unit of this embodiment, and the decoded image of the upper layer is directly fed back to the upper layer coding unit and used for predictive coding. In the decoding device, the decoded image of the upper layer is directly fed back to the upper layer decoding unit to be used for prediction and is input to the superimposing unit, and the output of the superimposing unit is displayed on a display or the like.

Further, in the description of the present invention, the coding of the area shape is described by using the method by the eight-direction quantization code, but it goes without saying that another shape coding method may be used.

[0080]

According to the moving picture coding apparatus and the moving picture decoding apparatus of the present invention, (1) when synthesizing a lower layer frame that has not been coded, a lower layer that is temporally preceding is added. By encoding the first area shape and the second area shape of the lower layer that exists temporally later in the upper layer and performing the synthesis using the first area shape and the second area shape, the area shape is Even in the case of temporal changes, it is possible to obtain a good image without distortion in the superimposed image of the lower layer and the upper layer. Further, (2) above (1)
When the lower layer frame combination is performed, the first region shape of the lower layer existing temporally before and the second region shape of the lower layer existing temporally later are not encoded, and the first region By extracting the shape and the second area shape from the decoded data of one or both of the lower layer and the upper layer, a mode in which the area shape is not encoded in the upper layer is introduced, and the number of bits can be reduced. .
Further, (3) at the time of encoding the upper layer of (1) and (2), a first flag indicating whether or not to encode the pixel information of the upper layer is provided, and only the area shape is encoded in the upper layer. The decoding device can easily determine the mode by determining, by the decoding device, whether or not both the region shape and the pixel information are coded. Further (4) In the above (2),
When there is no lower layer frame corresponding to the same frame position as the upper layer, the decoded image of the lower layer is divided into regions when the region shape of the lower layer that exists temporally before and after is extracted, and the region division result is used. Then, by extracting the area shape, the area shape can be accurately obtained without increasing the number of bits. Further, (5) in (2) above, when the region shape of a lower layer that is temporally before and after in time is extracted when a lower layer frame corresponding to the same frame position as the upper layer does not exist, when decoding the upper layer, By estimating and extracting the area shape using the obtained area shape, the area shape can be easily obtained without increasing the number of bits. Furthermore, (6) In the above (1) to (5), when there is no lower layer frame corresponding to the frame position of the upper layer at the time of decoding, the lower layer that exists before and after is used for the lower layer that does not exist. By providing a second flag indicating whether or not to perform layer frame composition and using a lower layer frame existing before or after as a combined lower layer frame, it is possible to reduce the amount of processing required for composition. Become. Furthermore, (7) since the region information is not encoded in the lower layer, many bits are not generated in the lower layer. Therefore, a good lower layer image can be transmitted or accumulated without causing a large distortion even in a transmission path or memory having a relatively low bit rate for transmitting or accumulating the lower layer.

[Brief description of drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a block diagram showing a second embodiment of the present invention.

FIG. 3 is a diagram illustrating a first flag of the present invention.

FIG. 4 is a block diagram when a region shape is extracted using decoded data of a lower layer according to the present invention.

FIG. 5 is a block diagram in the case of extracting a region shape by using a region shape of an upper layer according to the present invention.

FIG. 6 is a diagram illustrating a third embodiment of the present invention.

FIG. 7 is a diagram illustrating another example of the third embodiment of the present invention.

FIG. 8 is a block diagram illustrating a conventional encoding method and decoding method.

FIG. 9 is a diagram illustrating an example of effects of the present invention.

FIG. 10 is a diagram showing a concept of a conventional method.

FIG. 11 is a diagram illustrating an eight-direction quantized code.

FIG. 12 is a diagram illustrating a problem of the conventional method.

FIG. 13 is a diagram illustrating an example of extracting a region shape using a region shape of an upper layer according to the present invention.

FIG. 14 is a block diagram illustrating a first embodiment and a second embodiment of the present invention.

FIG. 15 is a block diagram illustrating a first embodiment of the present invention.

FIG. 16 is a block diagram illustrating a second embodiment of the present invention.

FIG. 17 is a block diagram showing an example of an upper layer encoding unit of the present invention.

FIG. 18 is a block diagram showing an example of an upper layer decoding unit of the present invention.

[Explanation of symbols]

 101 Switch 102 Pixel Data Encoding Unit 103 Region Shape Encoding Unit 104 Pixel Data Decoding Unit 105 Region Shape Decoding Unit 106 Multiplexing Unit 107 First Delay Unit 108 Second Delay Unit 109 First Region Extraction Unit 110 Second Area extraction unit 111 controller 112 switch 113 interpolation image creation unit 114 weighted average unit 115 third delay unit 116 fourth delay unit

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Hiroshi Kusao 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka Within Sharp Co., Ltd. Within the corporation

Claims (7)

[Claims] 1. A lower layer coding for coding a moving picture sequence at a low frame rate and an upper layer coding for coding one or more specific areas or all areas of the moving picture sequence at a high frame rate. , A hierarchical coding apparatus that decodes only a lower layer having a lower frame rate for decoding a lower layer, and decodes a lower layer and an upper layer having a higher frame rate for decoding up to an upper layer and superimposes the lower layer and the upper layer. And a decoding device, when there is no lower layer frame corresponding to the same frame position as the upper layer at the time of decoding, performs synthesis of the lower layer frame that does not exist using lower layers existing before and after, In synthesizing, the first region shape of the lower layer that exists before in time and the first region shape of the lower layer that exists after in time A moving picture coding apparatus and a moving picture decoding apparatus, wherein a second area shape is coded in an upper layer, and the combining is performed using the first area shape and the second area shape. 2. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 1, wherein when a lower layer frame corresponding to the same frame position as that of an upper layer does not exist at the time of decoding, it exists before in time. The first region shape of the lower layer and the second region shape of the lower layer existing temporally later are not encoded, and the first region shape and the second region shape are not encoded.
A moving picture coding apparatus and a moving picture decoding apparatus, wherein a region shape is extracted from decoded data of one or both of a lower layer and an upper layer. 3. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 1 or 2, further comprising a first flag indicating whether to code upper layer pixel information when coding the upper layer. A moving image code provided by a decoding device that determines whether only the area shape is coded in the upper layer and when both the area shape and the pixel information are coded by the decoding device. Apparatus and moving picture decoding apparatus. 4. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 2, wherein when there is no lower layer frame corresponding to the same frame position as that of the upper layer, an area of a lower layer that exists temporally before and after. A moving picture coding apparatus and a moving picture decoding apparatus, characterized in that when extracting a shape, a decoded image of a lower layer is divided into areas, and the area shape is extracted using a result of the area division. 5. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 2, wherein when there is no lower layer frame corresponding to the same frame position as that of the upper layer, a lower layer area existing temporally before and after. A moving picture coding apparatus and a moving picture decoding apparatus, wherein when extracting a shape, the area shape is estimated and extracted using the area shape obtained at the time of decoding an upper layer. 6. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 1, claim 2, claim 3, claim 4 or claim 5, which corresponds to a frame position of an upper layer at the time of decoding. When the lower layer frame does not exist, a second flag indicating whether or not the lower layer frames existing before and after the lower layer frame are combined is provided, and the lower layer frame is not combined. The moving picture coding apparatus and the moving picture decoding apparatus are characterized in that a lower layer frame existing before or after is used as a combined lower layer frame. 7. The moving picture coding apparatus and the moving picture decoding apparatus according to claim 6, wherein when synthesizing a lower layer frame, a first region shape of a lower layer existing temporally before is coded. A third flag indicating whether or not and a fourth flag indicating whether or not to encode a second region shape of a lower layer that exists temporally later are provided, and the first region shape and the second region are provided. When the shapes are not coded together, the region shape used in the previous synthesis is the region shape used in the current synthesis, and when only the second region shape is coded, the second region used in the previous synthesis A moving picture coding apparatus and a moving picture decoding apparatus, wherein the shape is a first area shape used for the present synthesis, and a case where only the first area shape is coded is not provided.