JPH02234587A - Picture decoder - Google Patents

Abstract

PURPOSE:To obtain moving picture information without using a moving vector by storing picture information in the unit of frames to plural frame memories and reproducing it from a neutral network after picture information framed in the unit of frames is decoded by a decoder. CONSTITUTION:A memory 5 stores picture information by one frame, its picture signal is outputted to an output device 9 and a frame memory 6 and the output device 9 outputs the picture information by one frame externally, partial picture signals 72, 71 of the picture information by one frame stored in the frame memories 6, 5 are subjected to the processing method of a neural network 7 thereby reproducing a frame omitted and interpolated. Thus, the device outputs picture information from the frame memory 5 and the picture information from a frame memory 8 alternately to output the consecutive moving picture information externally without using the moving vector information.

Description

[Detailed description of the invention]

[Industrial Field of Application] The present invention relates to an image decoding device, and more particularly to an image decoding device that decodes a moving image from image information that has been chopped by an encoding device frame by frame. [Prior Art] Generally, in low bit rate video information communication used for videophones, etc., in order to reduce the amount of information to be transferred, the transmitting side transmits an encoded signal with frame-by-frame dropping, and the reception Frame interpolation technology is used to increase the frame rate by reproducing missing frames on the side. As a conventional frame interpolation technique, a technique has been proposed in which motion vectors between dropped frames are detected in units of pixel blocks and motion compensation coding is performed. In addition, as a document related to the frame interpolation technique, for example, Document I published by the Institute of Electronics, Information and Communication Engineers Image Engineering Study Group
E8g-42 (published in 1988), pages 39 to 4
An example of this is the article ``Motion compensated frame interpolation method for color moving image signals'' on page 6. [Problems to be Solved by the Invention] Since the frame interpolation technique using motion vector detection according to the prior art uses motion vectors, if motion vector information is not sent to the decoding side, the interpolation method cannot be performed. Furthermore, the motion vector estimation method does not always provide accurate estimation, leading to the problem of not being able to obtain visually good images. An object of the present invention is to solve the problems caused by the prior art, and to provide an image decoding device that can reproduce good interpolated frames without using moving image vector information. [Means for Solving Problem 31] In order to solve the above problems, an image decoding device according to the present invention includes a decoder that decodes image information that has been chopped in units of frames, and a decoder that decodes the image information that is A plurality of frame memories that store image information in units of frames, and a spatially weighted sum operation using the image information of successive frames stored in the plurality of frame memories, are used to calculate the segmented frames. A neural network that reproduces image information. A section was set up. [Operation] The image decoding device stores frame olfactory position image information in a plurality of frame memories after the decoder decodes the image information that has been chopped into frames, and stores the image information of the continuous frames in a plurality of frame memories. Based on this, the interpolated frames interpolated between these consecutive frames are reproduced by a neural network. Therefore, this device can reproduce frame-level image information that has been dropped during encoding to obtain moving image information without using motion vectors. [Embodiment] Hereinafter, an embodiment of an image decoding device according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a conceptual diagram for explaining the principle of the image decoding device according to this embodiment. An example will be described in which the i+jth frame 20 corresponding to the frame that has been truncated is played back. The principle of this reproduction is that the i-th frame 10 and the i+k-th frame
By extracting the image signals of partial images a and a° of frame 30 and inputting these partial images a and a' to a neural network C trained in advance, partial image b of the i+jth frame 20 to be interpolated is created. , to create this partial image b over the entire frame 20. The neural network C is modeled on the information processing method using a large number of neurons found in living organisms, and is based on the strength of the connections between elements when performing multiple parallel processing using relatively simple information processing elements (units). It corresponds to a neural computer that generates the values by learning and performs spatial weighted sum calculations. As the learning method, the so-called back propagation method is suitable, in which the strength of mutual coupling is modified so that the difference between the output corresponding to the given human power and the target output becomes smaller6. As is clear from the conceptual diagram in the figure, the image decoding device according to this embodiment uses the frames before and after the chopped image information to generate frames to be interpolated by the neural network C learned by the backpropagation method. It is something to be regenerated. Now, one embodiment of an image decoding device using the above principle will be described with reference to FIGS. 2 and 3. FIG. 2 is a diagram showing the overall configuration of the image decoding device according to the present embodiment, in which a decoder 4 receives an encoded signal 1 of a plurality of frames of a frame-dropped moving image and decodes the signal 1. , a frame memory 5 that stores one frame worth of image information output from the decoder 4, a frame memory 6 that stores the output of the memory 5, and the output signals of these memories 6 and 5, that is, the frame The above-mentioned partial image 73 of the interpolation frame is obtained by inputting the partial image signals 72 and 71 of the previous and subsequent frames.
a neural network 7 that outputs the output signal 73; a frame memory 8 that stores image information for one frame using the output signal 73 of the network 7; and a frame memory 8 that stores image information for one frame following the image information from the frame memory 5 It is composed of an output device 9 that outputs continuous moving image information to an external device by outputting a hat. As shown in FIG. 3, the neural network 7 includes an input layer 40 consisting of a plurality of units (indicated by circles in the figure) ix++ to Ll12n' which input partial image signals 7l and 72 in units of pixels, and a plurality of units similarly. 'Ll31~tz
311 and the partial image signal 7 of the frame 20 to be interpolated.
3, the input layer 40 and the output layer 6
0 and an intermediate layer 50 consisting of a plurality of units L121 to U2 located between L121 and U2. In this neural network 7, each unit is connected from the input layer 40 to the output layer 60, input codes are transmitted from the input layer 40 to the output layer 60, and learning is carried out at the output layer 60.
This is performed by the above-mentioned learning method in which the connections between each unit are changed from 0 to the input layer 40 so as to reduce the difference between the actual power value and the desired output value. Specifically, the input value of the unit of the input layer 40 is:! /+pq+middle layer 5
The output value of unit u2r (r:l~m) of 0 is y2r+
Output value y35 (sJ-
Q) *Unit LIIPQ of input layer 40 and intermediate layer 50
If the strength of the connection between the unit u2 of the intermediate layer 50 and the unit u3s of the output layer 60 is w2rs, then the output value y2r of the unit ugr of the intermediate layer 50 and the output value y2r of the unit ugr of the output layer 60 The output value y3g of unit u3s is obtained by the following equation 1.

[Formula 1] :! )/3S= f (y zs) tumor v 2S: Σ W2rgIly 2'+ W20S
r=1 :y 2r= f (vIr) +1=1 However, W@Q5 and WIOOr represent the coupling strength between each unit of the output layer 60 and the intermediate layer 50 and a constant value, and v
Ir and v2s represent temporary variables for assignment. Also V
The connection strength of i71Pgr and w2rs is set in advance by learning frames that have been dropped by the pack propagation as training data. That is, the partial image a of the i-th frame 10 and the i-th k-th frame 3
With partial pixel a' of 0 as input, i+j frame 20
The i-th image corresponding to the partial image b of
Using the partial image of the j frame as training data, learning is performed by varying the spatial position of the partial image within the frame and i, and the strength of the neural network connection is determined in advance. Regarding the neural network itself, please refer to "Using Neural Networks for Pattern Recognition and Knowledge Processing" on pages 115 to 124 of Nikkei Electronics (No. 427), dated August 1987. The article is described elsewhere. In the image decoding device configured in this way, a decoder 4 decodes an encoded signal 1 that has been chopped and encoded by an image encoding device (not shown) of a moving image. Output the output signal to frame memory 5. The memory 5 stores one frame's worth of image information, and outputs the image signal to an output device 9 and a frame memory 6, and the output device 9 outputs one frame's worth of image information to the outside.
Then, the image signal of the next frame is inputted to the frame memory 5, so that the decoded image information of successive frames is stored in the memories 5 and 6, respectively. The partial image signals 72 and 71 of one frame of image information stored in the frame memories 6 and 5 are reproduced into frames to be interpolated by dropping frames by a processing method according to the above formula of the neural network 7. That is, the neural network 7 processes the partial image signals 72 and 71 multiple times over the entire frame, thereby reproducing the frame to be interpolated. The image information of the frame to be interpolated is sent to the output device via the frame memory 8! 9 is output. Therefore, by alternately outputting the image information from the frame memory 5 and the image information from the frame memory 8, this device can output continuous moving image information to the outside without using motion vector information. Incidentally, in the above embodiment, an example was explained in which one interpolated frame is generated by using two frame memories 5 and 6, but the present invention is not limited to this, and it is possible to prepare a plurality of neural networks. You can also generate multiple interpolated frames by doing this. Also,
Three or more frame memories may be prepared before the neural network, and interpolation may be performed from the three or more frame memories, or each partial image signal may be input to the input layer. Furthermore, the neural network is not limited to the three layers described above, but may be configured with four or more layers. [Effects of the Invention] As described above, the image decoding device according to the present invention reproduces the dropped frames by performing spatial weighted sum calculation processing on the image information of a plurality of dropped frames using a neural network. be able to.

[Brief explanation of the drawing]

FIG. 1 is a diagram for explaining the concept of an image processing device according to the present invention, FIG. 2 is a diagram showing an image processing device according to an embodiment of the present invention, and FIG. 3 is a diagram for explaining a neural network. be. 1: encoded signal, 4: decoder, 5 and 6: frame memory, 7: neural network, 8: frame memory. 9: Output device. Patent applicant Graphics Communication Technologies Co., Ltd.

Claims (1)

[Claims] A decoding device that decodes moving image information by decoding frame-by-frame image information includes a decoder that decodes input image information, and a frame-by-frame frame that is decoded by the decoder. A plurality of frame memories storing image information and image information of successive frames stored in the plurality of frame memories are input and a spatially weighted sum operation is performed to reproduce the image information of the dropped frame. An image decoding device comprising: a neural network.