JPH01300688A - Decoder for highly efficient code - Google Patents

Abstract

PURPOSE:To improve decoded picture quality by using a picture element data adjacent timewise corresponding spatially to the picture element in the case of the picture element data during decoding at present being a picture element data of a block with frame omission processing and interpolating the data by an arithmetic mean applying weighting inversely proportional to a time difference with a time corresponding to the decoded picture element data. CONSTITUTION:In the case of the picture element data decoded at present being a picture element data of a block subject to frame omission processing, the data is interpolated by means of arithmetic mean using the picture element data corresponding spatially to said picture element data and adjacent timewise. For example, the time difference between the time of an original picture element data (b) and a time of a mean value 1/2(a+b) precedingly timewise is 1/2 frame because the mean value is generated at an intermediate time of the picture element data (a), (b), the time difference between the time of the original picture element data (b) and the time of the mean value 1/2(c+d) succeedingly timewise is 1.5 frame and a decoded picture element corresponding to the original picture element data (b) is obtained by the arithmetic mean inversely proportional to the time difference. The interpolation is repeated to apply the interpolation without any level change and offset of the original picture element data. thus, the decoded picture quality is improved.

Description

[Detailed Description of the Invention] [Industrial Application Field] This invention provides a three-dimensional block configuration when transmitting image data such as a digital television signal, and encodes each block. Regarding still blocks, the present invention relates to a high-efficiency code decoding device that compresses the amount of transmitted information by performing frame drop compression.

[Summary of the invention]

In this invention, on the transmitting side, for a three-dimensional block structure constituted by pixel data of regions that belong to each of a plurality of temporally consecutive frames and that correspond in position between the plurality of frames, In a high-efficiency code decoding device that performs motion-adaptive frame-drop processing based on stationary judgment to compress the amount of data and restores the transmitted signal to a three-dimensional block structure, the block in which the pixel data currently being decoded has been subjected to frame-drop processing. In the case of pixel data, at least pixel data that spatially corresponds to and temporally adjacent to this pixel is used, and interpolation is performed by weighted averaging in which weighting is inversely proportional to the time difference from the time corresponding to the decoded pixel data. , It is possible to prevent jerky and block distortion from occurring in the restored image, and the quality of the restored image can be improved.

[Conventional technology]

As a method for encoding image signals such as television signals, the applicant of the present application has proposed a high-efficiency encoding device that does not cause problems such as occurrence of aliasing distortion, propagation of errors, and occurrence of block distortion. For example, as described in Japanese Patent Application No. 60-232789, regarding a three-dimensional block formed from a plurality of SJI regions included in a plurality of frames, the dynamic range is the maximum value, the minimum value, and the difference between the two. A method has been proposed in which pixel data is encoded while adapting to the dynamic range.

Furthermore, in the specification of Japanese Patent Application No. 153330/1982, it is determined whether each three-dimensional block is a static block (a block with almost no image movement) or a moving block, and if it is a static block, the three-dimensional block is configured. A high-efficiency encoding device has been proposed that further compresses the amount of transmitted information by performing so-called frame dropping, which transmits average value information of n pixels at the same position.

This frame-drop compressed image data is interpolated to the original number of pixel data on the receiving side.

In order to prevent jerkiness from occurring as a result of this interpolation, as described in Japanese Patent Application No. 61-153328 or Japanese Patent Application No. 61-153329, the previous block in time is A decoding device for a high-efficiency code performs smoothing in which an average value is formed between data in the last area and data in the first area of the current block, and this average value is used as data in the first area of the current block. Proposed. Furthermore, when an object with a large area moves at high speed or during a scene change, blurring will occur if the above smoothing process is performed, so as described in Japanese Patent Application No. 189856/1980 , a technique has been proposed in which a stillness determination is performed pixel by pixel between blocks to be smoothed, and smoothing is performed only on still pixels.

Frame drop processing and smoothing processing will be described with reference to FIGS. 11 to 13. In FIG. 11, a −
f indicates original data, and these original data a to d are the first
As shown in Figure 2, spatially corresponding regions At, Bi, C1, DI included in each of temporally consecutive frames
This is data that occupies the same position within. These areas have a size of (4 lines x 4 pixels), and the three-dimensional block consists of two areas At and Bi (also C3 and DI).
Consisted of.

If the three-dimensional block is a stationary block, frame dropping processing is performed. That is, the average value ('A(a+b)) of data occupying the same position in the block.

y2(C+d), y2(e+f)) is transmitted instead of the original data. In this example, where one block consists of two areas, the amount of transmitted data is compressed to %. Furthermore, the frame-dropped data is subjected to dynamic range adaptive encoding (referred to as ADRC) processing, thereby further compressing the amount of transmitted data.

On the receiving side, after ADRC decoding is performed, the frame-dropped data is interpolated. This interpolation is a process of converting average value data for one region into data for two regions. Therefore, if we focus on one pixel, the data of this pixel will be (K (a+b)”A (c
+d)-'A(e+f)). Since jerkiness occurs at block boundaries, the data in the first area of the subsequent stationary block is the average value (
+/4(a+b+c+d)).

[Problem to be Solved by the Invention] Here, if the data of pixels at the same position has a level change that gradually increases with time as shown in FIG. When the frame is dropped in units, it is decoded as a repetition of the average value indicated by Δ. The above-described smoothing process forms decoded mouth data and prevents occurrence of jerkiness. As a result of smoothing,
In FIG. 13, as shown by the broken line, if the original data has a positive slope, a negative offset occurs.

This offset is barely noticeable visually.

However, it is assumed that smoothing is performed only on stationary blocks, and since the determination of stationary blocks and moving blocks involves threshold determination, even if the blocks are spatially adjacent, one A block may be treated as a stationary block and the other block may be treated as a moving block. For example, the above phenomenon may occur when an image containing a human face is vanned at low speed. Therefore, one block will have an offset because it is treated as a static block, and the other block will have no offset because it is treated as a motion block, e.g. adjacent areas of the face. Nevertheless, level differences occur, resulting in blocky distortion. For objects such as human faces, there was a problem in that vertical stripes appeared.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a high-efficiency code decoding device that prevents an offset from occurring between original data and original data when restoring a frame-dropped block.

[Means to solve the problem]

In this invention, on the transmitting side, for a three-dimensional block structure constituted by pixel data of regions that belong to each of a plurality of temporally consecutive frames and that correspond in position between the plurality of frames, In a high-efficiency code decoding device that performs motion-adaptive drop-off processing based on stationary determination to compress the amount of data and restore the transmitted signal to a three-dimensional block structure, a block in which pixel data currently being decoded has been subjected to frame-drop processing. In the case of pixel data, there is a small amount of pixel data spatially corresponding to this pixel and temporally adjacent to this pixel. will be done.

[Effect]

As a result of the frame dropping process, the average value between the two areas constituting one block is transmitted. When considering level changes in the original pixel data, this average value is considered to occur temporally between the two regions. When pixel data is interpolated by a weighted average that is inversely proportional to the time difference using decoded data of pixels that spatially correspond to and temporally precede and follow the pixel data to be interpolated, the time difference is calculated as described above. , the average value is determined as occurring midway between the two regions. This weighted average allows interpolation without level changes and offsets of the original data.

〔Example〕

An embodiment of the present invention will be described below.

This description is given in the following order.

a. Configuration of the transmitting side, interpolation method C0. Configuration and operation of the receiving side d. Modification a. Configuration of the transmitting side FIG.
In this case, the configuration of the recording side) is shown. For example, a digital video signal in which one sample is quantized to 8 bits is input to the input terminal indicated by (2).

This digital video signal is data with consecutive frames, similar to a television signal. This digital video signal is supplied to the blocking circuit 2.

A blocking circuit 2 converts the input digital video signal into a block order. In this example, the second
As shown in the figure, a three-dimensional block is formed from two-dimensional areas At and Bi belonging to two temporally consecutive frames, respectively. Each area has a size of, for example, (4 lines x 4 pixels).

Therefore, one block includes (4X4X2=32) pixels.

The output signal of the blocking circuit 2 is supplied to the motion determining circuit 3. The motion determination circuit 3 determines the presence or absence of motion for each block. The motion determination circuit 3 detects a difference between pixel data occupying the same position between two areas constituting a block, compares the maximum absolute value of this difference with a threshold value, and calculates the absolute value of the difference. When the absolute value of the difference is smaller than the threshold value, it is determined to be a moving block, and when the absolute value of the difference is smaller than the threshold value, it is determined to be a stationary block. In addition to this motion determination, there is also a method of comparing the integral value of the difference with a threshold value, a method of comparing the total number of differences larger than a certain level with a threshold value, a method of comparing the total number of differences larger than a certain level with a threshold value,
A method of comparing the sum of products with a threshold value, etc. can also be used. The motion determination circuit 3 generates a determination code SJ depending on the presence or absence of motion.

The determination code SJ is a signal that is "0" for moving blocks and "1" for stationary blocks.

The input digital video signal from the blocking circuit 2 is A
Supplied to ND gates 4 and 8. A judgment code SJ is supplied as another input to the AND gate 4, and the judgment code SJ is inverted and supplied as another input to the AND gate 8. One AND gate 4 separates the pixel data of the still block, and this pixel data is supplied to the interframe average value forming circuit 5. The inter-frame average value forming circuit 5 calculates the average value of 16 pixels located at the same position between two areas constituting a block, and
This is a circuit that outputs six average values instead of pixel data of a block. Therefore, the output signal of the inter-frame average value forming circuit 5 has a two-dimensional block configuration in which the average values are arranged in (4 lines x 4 pixels). The output signal of this inter-frame average value forming circuit 5 is supplied to a two-dimensional encoder 6.

The other AND gate 8 separates the data of the block in which the image has movement, and this pixel data is supplied to the three-dimensional encoder 9 . The two-dimensional encoder 6 and the three-dimensional encoder 9 perform encoding with a variable number of bits adapted to the dynamic range of each block. These encoders 6 and 9 generate a block dynamic range, a block minimum value, and a code signal of, for example, 0 to 5 bits.

The output signals of the two-dimensional encoder 6 and three-dimensional encoder 9 are supplied to framing circuits 7 and 10, respectively, and
The determination code SJ is supplied to the framing circuits 7 and 10 via the delay circuit 11.

Output signals of encoders 6 and 9 and judgment code SJ mentioned above
are converted into transmission data by the framing circuits 7 and 10. The output signals of the framing circuits 7 and 10 are O
R gate 12 and output terminal 1 of OR gate 12
3, the transmitted data is retrieved.

When focusing only on the code signal, the transmitted data consists of a set of data for two frames, as shown in Figure 3.
For example, each of two temporally consecutive frames is divided into areas A1 to An and areas 81 to Bn, and area A
A three-dimensional block is constructed by i and Bi (i=1 to n). As mentioned above, for stationary blocks,
Frame drop processing is performed. In the transmission data for the first two frames in FIG. 3, the block consisting of areas A2 and B2, the block consisting of areas A3 and B3, and the block consisting of areas A4 and B4 are determined to be stationary blocks, and are subjected to frame drop processing. ing.

b. Interpolation method The interpolation method of this embodiment will be explained below. Fourth
As shown in the figure, when original data a to f included in each continuous frame and located in spatially corresponding positions are included in a still block, the data amount is converted into a percentage and transmitted by frame drop processing. Ru. On the receiving side, in the case of a stationary block, interpolation is performed using average value data as data for two areas.

Pixel data included in a still block is interpolated by a weighted average using pixel data that spatially corresponds to this pixel data and is temporally adjacent to the pixel data. For example, the time difference between the time of original pixel data b and the time of the temporally previous average value %(a+b) is considered to occur at a time when the average value is between pixel data a and b. As can be seen from 2
The time difference between the time of the original pixel data b and the time of the temporally later average value 'A(C+d) is 1.5 frames. With a weighted average inversely proportional to this time difference,
A decoded pixel corresponding to the original pixel data b is obtained. That is, the level of this decoded pixel is 'A (3X+A(a+b
)+'A (c+d)). Similarly, the level of the decoded pixel corresponding to the original pixel data d is set to I/4 ('A (a + b) + 3X + A (c + d)). By repeating this interpolation, the level of the decoded pixel corresponding to the original pixel data d becomes Interpolation can be performed without level changes or offsets in pixel data.

Interpolation when moving blocks and stationary blocks coexist as shown in FIG. 6 will be explained. That is, the block containing the original pixel data a and b is a motion block,
This is a case where the block containing original pixel data C and d is a still block, and the block containing original pixel data e and f is a motion block. For motion blocks,
Frame drop processing is not performed.

As can be seen from FIG. 7, the time difference between the time of the original pixel data C and the temporally previous pixel data b is one frame, and the original pixel data C and the temporally subsequent average value 'A (c+d) There is a time difference of 2 frames.

A weighted average that is inversely proportional to this time difference is performed, and the value of the decoded pixel corresponding to the pixel data C is determined.

That is, the level of this decoded pixel is calculated as 1/3(b+2X+A(c+d)). Similarly, the level of the decoded pixel corresponding to the original pixel data d is calculated as 1/3 (2Xy2(c+d)+e). This interpolation allows interpolation without level changes and offsets of original pixel data even when blocks subjected to frame dropping processing and motion blocks coexist.

Configuration and Operation of C9 Receiving Side The receiving side configured to use the above-described weighted average interpolation method will be described with reference to FIGS. 8, 9, and 10.

In FIG. 8, received data from an input terminal indicated by 21 is supplied to a judgment code separation circuit 22, and the judgment code S
J is separated. Received data other than the judgment code is AND
Supplied to gates 23 and '26. AND gate 23
A determination code SJ is supplied as the other input of the AND gate 26, and an inverted signal of the determination code SJ is supplied as the other input of the AND gate 26.

The still block data is separated by the AND gate 23, and the still block data is supplied to the frame decomposition circuit 24. Furthermore, the data of the blocks with motion separated by the AND gate 26 is supplied to the frame decomposition circuit 27. The frame decomposition circuits 24 and 27 decompose the received data into an additional code (dynamic range and minimum value) for each block and a code signal for each pixel, and
Performs error correction processing. Frame decomposition circuits 24 and 2
The additional code and code signal from 7 are sent to the two-dimensional decoder 2.
5 and 3-dimensional decoders 28, respectively.

These decoders 25 and 28 perform processing opposite to that of encoders 6 and 9 on the transmitting side. That is, the data after the minimum value has been removed is restored as a representative level, and by adding this data and the minimum value, a restoration level corresponding to the original pixel data is obtained.

The output data of decoders 25 and 28 is stored in the buffer memory 2.
9. The buffer memory 29 includes a delay circuit 30
The frame-dropped data is interpolated with reference to the judgment code SJb passed through the frame, and the amount of data is made equal between the still block and the motion block.

The output data of the buffer memory 29 and the judgment code SJ are supplied to two-frame delay circuits 31 and 33 in order to perform interpolation regarding the stationary block.

2-frame delay circuit 3 for 2-frame delay circuit 31
2 is connected to the 2-frame delay circuit 33, and a 2-frame delay circuit 34 is connected to the 2-frame delay circuit 33.

- For example, data Sa shown in FIG. 9A is generated from the buffer memory 29, and determination code SJb shown in FIG. 9B is generated from the delay circuit 30. BLI, B in Figure 9
L2. BL3... each indicates a block period. Further, FIG. 9 shows the order of data in units of regions constituting a block, and decoded data of pixels is included in each region in a predetermined order.

Further, as shown in FIG. 9C, data Sc shifted by % blocks from data Sa is extracted from the middle of the 2-frame delay circuit 31, and data Sd shown in FIG. 9 is extracted from the 2-frame delay circuit 31. appear. The determination code 5Je (FIG. 9E) from the two-frame delay circuit 33 is synchronized with the data Sd. Further, from the two-frame delay circuit 32 to the ninth
Data Sf shown in FIG. F is generated. 2 frame delay circuit 3
4, a judgment code SJg (FIG. 9G) synchronized with data Sf appears. The 9th frame from the middle of the 2-frame delay circuit 32
As shown in FIG. H, data sh is generated which is shifted by two blocks from data Sf.

The above data Sc and data Sd are supplied to the weighted average circuit 35, and the data Sd and data sh are supplied to the weighted average circuit 3.
6. The weighted average circuits 35 and 36 have a configuration in which the weighted average operation can be switched by the determination codes SJb and SJg.
is “1” for a stationary block and “1” for a moving block.
The weighted average circuit 35 performs the following weighted average operation according to the determination code SJb.

When 5Jb="1": Va (Sc+3Sd) When SJb="Om: 1/3 (Sc+2Sd) The weighted average circuit 36 performs the following weighted average operation according to the determination code SJg.

% (Sc+2Sd) when SJg="1" 1/3 (S h + 2 S d) when SJg="0" The output signals of each of the weighted average circuits 35 and 36 are connected to one input terminal 38a of the switch circuit 37 and the other input terminal 3
8b. The switch circuit 37 is controlled by a control signal from a terminal 39 that is inverted every % block period. The output signal of the switch circuit 37 is supplied to one input terminal 41b of the switch circuit 40. switch circuit 40
The other input terminal 41a of the 2-frame delay circuit 31
The output data Sd of is supplied. The switch circuit 40 is
It is controlled by a switching signal Sl (FIG. 9I) from a switching signal generation circuit 42. An interpolated output signal Sj (FIG. 9J) is obtained from the switch circuit 40. This output signal is converted from the block order to the scanning order by the block decomposition circuit 43, and is taken out to the output terminal 44.

A switch circuit 40 is provided to switch between received data and an interpolated output signal. Therefore.

In the case of a motion block, the received data is selected, and in the case of a stationary block, the interpolation output is selected. Therefore, the switching signal generation circuit 42 is supplied with the determination code SJe.

Furthermore, even in the case of a still block, in the case of a scene change, etc., spatially corresponding data becomes completely different data, so it is necessary to prohibit selection of interpolation output. For this purpose, the output signals of comparison circuits 45 and 46 are supplied to switching signal generation circuit 42.

The comparison circuit 45 compares the output signal of the comparison circuit 48 with the threshold value from the terminal 47, and the comparison circuit 46 compares the output signal of the comparison circuit 48 with the threshold value from the terminal 47.
The output signal of 9 and the threshold value from terminal 47 are compared.

The comparison circuit 48 generates the absolute value of the level difference between corresponding pixels of data Sa and Sd, and the comparison circuit 49 generates the absolute value of the level difference between corresponding pixels of data Sd and Sf. When the absolute value of these level differences exceeds a threshold value, selection of interpolation output is prohibited.

In the timing chart of FIG. 9, block period B
The operation of one embodiment will be described in detail with attention to L2 with reference to FIG.

In the block period BL2, (SJb - "0"), the weighted average circuit 35 calculates 1/3 (S c + 2 S
d) generate the output signal. Therefore, the weighted average circuit 35
The output signal is as shown in FIG. 10A. Also,(
Since SJg="1"), the weighted average circuit 36 generates an output signal of (Sh+3Sd). Therefore, the output signal of the weighted average circuit 36 is as shown in FIG. 10B. Among the output signals of the weighted average circuit 35, the second half output signal effective as an interpolation output and among the output signals of the weighted average circuit 36, the first half output signal effective as an interpolation output is the first.
It is selected by a switch circuit 37 controlled by a control signal shown in FIG. Therefore, the output signal of the switch circuit 37 is as shown in Figure 10.

In the area C2 of the block period BL2, as shown in FIGS. 4 and 5, an interpolation operation is performed when the previous block is a stationary block, and in the area D2, the interpolation operation is performed as shown in FIGS. 7 and 8. An interpolation operation is performed when the block is a motion block. The timing charts in Figures 9 and 1O are as follows:
Although signal processing is shown in units of regions, weighted average processing is performed in units of pixel data. In addition, the switch circuit 40
The operation is also performed pixel by pixel.

d. Modified Example When weighted averaging is performed, in the above embodiment, the pixel data to be interpolated, one piece of data earlier in time, and one piece of data later in time are used. Two or more pieces of data may be used and a weighted average may be performed by attaching a weight inversely proportional to the time difference.

Moreover, a three-dimensional block may be formed by pixel data included in not only two frames but three or more frames.

〔Effect of the invention〕

According to this invention, it is possible to interpolate temporally continuous and spatially corresponding original pixel data without level changes and offsets, prevent jerkiness, and block It is possible to prevent distortion from occurring. Since the restored image quality can be improved in this way, the redundancy in the time direction can be greatly reduced on the transmitting side, that is, the ratio of deletion can be increased, and the compression ratio can be further increased.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing the configuration of the transmitting side of a high-efficiency encoding device to which the present invention can be applied, FIG. 2 is a route map showing an example of blocks, and FIG. 3 is a route map showing an example of transmission data.
4 and 5 are route maps used to explain an example of interpolation,
6 and 7 are route maps used to explain other examples of interpolation, FIG. 8 is a block diagram showing the configuration of the receiving side to which this invention is applied, and FIGS. 9 and 10 are the route diagrams used to explain other examples of interpolation. Timing charts used to explain the interpolation operation, and FIGS. 11, 12, and 13 are route maps used to explain the previously proposed motion adaptive smoothing process. Explanation of main symbols in the drawings 3: Motion determination circuit, 6: Inter-frame average value forming circuit, 25: 2-dimensional decoder, 28: 3-dimensional decoder, 31. 32, 33, 34: 2-frame delay circuit, 35 ．．
36: Weighted average circuit. An example of 棉° 4th VgJ 骪' r! I 1I71＠-/) Tsuku 1 No. 6 @ ＞Iil'Mn-a column) City's interval of misfortune's row no hite - Evening day shift customer hit one call H 17 Shiichimu off t-"zL/ 7 Season 5 Upper 5th Upper 11th vj!J Figure 12 buσtsufutQ Makij〈

Claims (1)

[Claims] On the transmitting side, a three-dimensional block structure constituted by pixel data of regions that belong to each of a plurality of temporally consecutive frames and that corresponds in position between the plurality of frames is determined based on a stillness determination for each block. The signal transmitted after applying motion adaptive frame drop processing and compressing the amount of data is
In a high-efficiency code decoding device that restores a dimensional block structure, if the pixel data currently being decoded is pixel data of a frame-dropped block, pixel data that spatially corresponds to this pixel and is temporally adjacent to this pixel is 1. A high-efficiency code decoding device, characterized in that interpolation is performed by a weighted average in which weighting is performed in inverse proportion to a time difference from a time corresponding to decoded pixel data.