JP4829867B2 - Image encoding apparatus and image decoding apparatus - Google Patents

Description

  The present invention relates to a moving image compression encoding technique, and more particularly to an image encoding device and an image decoding device.

  The MPEG-2 system and H.264 currently used for video encoding of digital broadcasts. The H.264 / AVC format uses inter-frame prediction for image compression. When decoding an encoded image, it is necessary to start from the first intra frame of the GOP in the MPEG-2 system. In the H.264 / AVC system, it is necessary to start from an IDR intra slice.

  On the other hand, a moving image encoding method that does not use inter-frame prediction (for example, Motion JPEG is known) can start decoding from an arbitrary frame. However, since the Motion JPEG method has a lower compression performance than a method using inter-frame prediction such as the MPEG-2 method, a high bit rate is required to obtain an acceptable quality image. Not suitable for use in broadcasting or communication.

  By the way, attention has recently been paid to a technique for obtaining a high resolution image by transmitting a low resolution image. For example, on the transmission side, a motion vector or a correction vector is obtained from three low resolution frames before and after the processing target frame of the low resolution image (total 7 frames), and the low resolution image is transmitted together with the motion information. A method of obtaining a high-resolution image from the received motion information and low-resolution image (see, for example, Patent Document 1), an initial motion vector is obtained between thinned images, and an initial restored image is reconstructed. A technique (for example, refer to Patent Document 2) is disclosed that improves the accuracy of a motion vector and improves the image quality of a restored image by sequentially repeating the process of obtaining a new motion vector using a thinned image.

JP 2006-174415 A JP 2006-345446 A

  As described above, the MPEG-2 system and the H.264 standard. In the H.264 / AVC format, when an encoded image is decoded, in the case of the MPEG-2 format, H.264 In the case of the H.264 / AVC format, it is necessary to start from the IDR intra slice. However, these methods have a problem that even if the compression performance can be improved, decoding cannot be started from an arbitrary frame. That is, decoding cannot be performed until a frame for starting decoding arrives. Therefore, in general, a non-decoding time of 0.5 seconds (hereinafter also referred to as a waiting time for decoding) occurs in terrestrial digital broadcasting and satellite digital broadcasting and in 2 seg broadcasting.

  In order to improve this problem, even if the technique disclosed in Patent Document 1 is used, the amount of data can be reduced by transmitting a low-resolution image on the transmission side. Must be generated and transmitted, and a waiting time for processing at least a plurality of frames occurs, and there remains a problem from the viewpoint of achieving both the waiting time for decoding and high resolution using a plurality of frames.

  Moreover, even if the technique disclosed in Patent Document 2 is used, the amount of data to be transmitted is reduced by transmitting a low-resolution image not accompanied by a motion vector on the transmission side, and the waiting time for decoding is eliminated. Although decoding can be performed, for example, an improvement in the accuracy of the motion vector cannot be expected in an image portion of a still area in a moving image, so there is room for improvement in increasing the resolution at the time of decoding.

  Accordingly, an object of the present invention is to enable decoding from an arbitrary frame by completely eliminating the waiting time for decoding, and to use only a low-resolution image generation technique and a previous frame for realizing a high compression rate. Another object of the present invention is to provide an image encoding device and an image decoding device for effectively combining the inter-frame prediction techniques, and obtaining an improved coding rate and improved resolution at the time of decoding.

An image encoding apparatus according to the present invention is an image encoding apparatus that encodes a moving image composed of continuous frame images, and divides an input frame image into first blocks, which are included in the first block. All of the first blocks are subjected to sub-sampling processing for selecting one of the pixels to be selected in a predetermined selection order to generate low resolution images, and the first resolution when the low resolution images are generated A sub-sample unit for generating sample position information representing a selected pixel position in the block, and the input frame image is divided into second blocks, and each of the second blocks is input one frame before. performs motion estimation processing between the frame images, Ann cover territory before 1 frame from the input frame image is an image region that appears in the current frame does not exist And a motion estimation unit that generates motion information including information on motion parameters and uncovered areas of motion estimation processing, and for each frame, a moving image consisting of continuous frame images, the low resolution image, The sample position information and the motion information are encoded.

  The image encoding device of the present invention further includes an intra-frame encoder that performs compression encoding on the low-resolution image generated by the sub-sample unit using a compression encoding method that can be decoded for each frame. Features.

  The image encoding apparatus according to the present invention further includes a high-resolution information encoder that performs lossless compression of the sample position information and motion information generated by the motion estimation unit using a predetermined compression method. .

The image decoding apparatus of the present invention interpolates a moving image encoded by the image encoding apparatus of the present invention into an integrated image generated using the low resolution image, the sample position information, and the motion information for each frame. An image decoding apparatus that performs processing and decodes with high resolution, and generates an allocation image in which each pixel of the low resolution image is allocated so as to correspond to coordinates before sub-sample processing, using the sample position information In accordance with a motion parameter included in the motion information, the allocating unit that performs motion compensation on the integrated image of the previous frame, and using uncover information indicating whether or not the uncover area is included in the motion information, The motion compensation unit that erases the pixels designated as the uncovered area and generates a motion compensated image, the pixel of the layout image generated by the layout unit, and the motion compensation A low-resolution image comprising: a pixel integration unit that generates an integrated image obtained by integrating the pixels of the motion compensation image generated by the unit; and an interpolation unit that performs an interpolation process on the integrated image to generate a decoded image. The frame image is decoded for each frame using the sample position information and the motion information, and a moving image composed of continuous frame images is restored.

  The image decoding apparatus according to the present invention further includes an intra-frame decoder that compresses and decodes a low-resolution image compression-encoded by the intra-frame encoder using a decoding method corresponding to the compression encoding method. And

  The image decoding apparatus according to the present invention further includes a high-resolution information decoder that decodes the sample position information and motion information losslessly compressed by the high-resolution information encoder by a decoding method corresponding to the predetermined compression method. It is further provided with the feature.

  According to the present invention, a low-resolution image that can be decoded for each frame on the encoding side is generated for each frame, so that decoding from an arbitrary frame is possible. Furthermore, a low-resolution image is generated for each frame by selecting different pixel positions at a cycle determined by the number of pixels constituting a predetermined block on the encoding side, and any kind of low-resolution image including a still area is generated on the decoding side. However, since the integration process is performed with motion compensation, as the number of decoded frames increases, the number of pixels in the integrated image is increased to enable high-quality interpolation, thereby improving the image quality of the decoded image.

  Therefore, in an intra-frame encoding scheme such as the Motion JPEG scheme, the resolution of the original image is compression-encoded. According to the present invention, a higher compression rate than these intra-frame encoding schemes is used. Can be obtained.

  Furthermore, according to the present invention, since high resolution information that can reduce the amount of transmission data is used, even if there is a case where a frame of a low resolution image cannot be reconstructed due to a transmission error or the like, a high resolution processing by decoding a plurality of frames is performed. Can be continued. For this reason, it is particularly suitable for transmission and use of moving images when the bandwidth of a highly reliable line is insufficient.

  An image encoding device and an image decoding device according to an embodiment of the present invention will be described.

(Example)
First, an image encoding apparatus according to an embodiment of the present invention will be described with reference to FIG.

(Image coding device)
FIG. 1 shows an image encoding apparatus according to an embodiment of the present invention. The image encoding device 1 can be configured with only the image encoder 101. However, as will be apparent from the description below, in the present embodiment, the image encoding apparatus 1 will be described as further including an intra-frame encoder 105 and a high-resolution information encoder 106. Therefore, the intraframe encoder 105 and the high resolution information encoder 106 are not essential elements in the present invention. The image encoder 101 includes a subsample unit 102, a motion estimation unit 103, and an original image delay unit 104.

  The subsample unit 102 divides the input original image (that is, the frame image) by a predetermined block size, and performs subsample processing for selecting one of the pixels included in the block in a predetermined selection order. This is applied to the block to generate a low resolution image. Sub-sample processing refers to dividing an original image into blocks of a predetermined size (hereinafter referred to as SS blocks in the sense of sub-sample blocks), and selecting one pixel from each SS block. At this time, filtering by a low-pass filter may be performed before the sub-sampling process.

  In addition, the size of the SS block used in the sub-sample processing is determined in advance or described in units of frames in the stream output from the image encoding device 1. This is because the selected pixel position in the SS block can be identified on the decoding side. Although the compression rate can be improved by increasing the size of the SS block, the image quality is lowered. For this reason, it is preferable to select a suitable size according to a use. For example, when an SS block of 8 × 8 pixels is used, the image size of the low resolution image is 1/64 that of the original image.

  That is, when generating a low-resolution image for each frame, the subsample unit 102 changes the pixel position when selecting one pixel from the SS block within the SS block for each frame. For example, the sub-sample unit 102 generates a low-resolution image by selecting different pixel positions for each frame at a period determined by the number of pixels constituting the SS block.

  In addition, the sub-sample unit 102 outputs the low-resolution image generated in this way, and also generates sample position information representing the selected pixel position in the SS block when the low-resolution image is generated. Here, the sample position information can be represented by 64 identification numbers when the SS block is 8 × 8 pixels, for example. Therefore, the sub-sampling unit 102 only needs to transmit only one identification number as the sample position information for each frame, and the information on the pixel positions (sample position information) that are different for each frame with the order of the identification numbers as the selection order. (A detailed example will be described later with reference to FIGS. 3A and 3B).

  In order to predetermine the selection order of the pixel positions in each SS block, it is preferable to determine the image position in the SS block by evaluating the image quality when the interpolation process is performed, assuming that the image area in the SS block is a still image area. For example, the selection order of the pixels in each SS block can be determined using a method similar to the optimization of the dither pattern. Therefore, in the present embodiment, the frame image composed of the pixels selected in each SS block is a low-resolution image composed of pixels selected at equal pixel positions from the original image. If the low-pass filter is not used before the sub-sampling process, the pixel value of the pixel of the low resolution image is equivalent to the pixel value of the original image. In this case, the low resolution image is the original image. It becomes an image of thinning out at equal intervals, retaining the frequency component of.

  FIG. 3A shows an example of pixel positions to be selected when the SS block is 8 × 8 pixels. In this example, since the SS block is composed of 8 × 8 pixels, 64 identification numbers (0 to 63) are assigned to each pixel position. Therefore, the selected pixel position is completed in 64 frames, and is sequentially selected according to 64 identification numbers. For example, the sub-sample unit 102 selects the pixel at the identification number “0” in all SS blocks in the first frame, selects the position at the identification number “1” in the next frame, and thereafter for each frame. The position is selected in order, and the position of the identification number “0” is selected again at the 65th frame.

  Thus, it is preferable to select all the pixel positions in the SS block with the same number of frames as the number of pixels in the SS block (for example, 64 frames in the case of an 8 × 8 pixel SS block). For example, FIG. 3B shows an example of pixel positions to be selected when the SS block is 4 × 4 pixels. In this example, it is preferable to select all pixel positions in the SS block in 16 frames. As described above, the assignment of the identification number is preferably determined by image quality evaluation at the time of interpolation processing, but can be determined by an arbitrary method.

  The motion estimation unit 103 divides the input original image into blocks of a predetermined size (hereinafter referred to as ME blocks in order to distinguish them from SS blocks), and the original image one frame before each block of the original image. Motion estimation between the two and the uncovered area. The uncover area refers to an image area that does not exist one frame before and appears in the current frame, and is also referred to as an occlusion area. For example, the background area that appears from behind the moving foreground and the rightmost image area when the screen is scrolling to the left are examples of the uncover area.

  In such an uncovered area detection method, for example, a process of switching the frame order between the current frame image and the previous frame image is performed, and the previous frame image is used as a reference one frame before. There is a method of determining, as an uncovered area, an area that cannot be allocated when creating a motion prediction image with a frame image, or an area that cannot generate a motion prediction image (see, for example, Japanese Patent Laid-Open No. 7-336688). Alternatively, a method of detecting a motion vector bidirectionally between the current frame image and the previous frame image and specifying an area that cannot be assigned to the current frame image by the bidirectional motion vector as an uncovered region ( For example, see JP-A-9-214899.

  The original image delay unit 104 delays the original image by one frame in order for the motion estimation unit 103 to estimate the motion with the original image one frame before. The original image delay unit 104 can be realized preferably using a memory.

  Note that the motion estimation by the motion estimation unit 103 can use not only a motion vector representing the translation of the ME block of the original image but also various motion parameters such as a parameter representing deformation and a parameter representing rotation. Thereby, image compression with higher image quality can be realized. Note that an appropriate motion parameter can be arbitrarily used in consideration of the application and calculation load of the image encoding device 1 or the image decoding device 2. Here, in the image encoding device 1, the motion information (information indicating the motion parameter and the uncover area) of the frame from which encoding is started can be an arbitrary value. This is because, as will be apparent later, the motion information is not used (the motion compensated image is not used) in the first frame at which the image decoding apparatus 2 starts decoding. Therefore, motion information that minimizes the amount of information to be transmitted may be used, or motion information may not be transmitted.

  Further, the ME block used for motion estimation by the motion estimation unit 103 does not have to be the same as the SS block used in the sub-sampling process. For example, when the upper right pixel in the SS block is selected by the subsample unit 102 (FIG. 4A), the motion estimation unit performs motion estimation using the ME block centered on the pixel selected by the subsample unit 102. (FIG. 4B). Also, the size of the ME block may be different from that of the SS block. When a ME block different from the SS block size or position is used in the motion estimation by the motion estimation unit 103, the ME block is determined in advance, or the frame unit is included in the stream output from the image encoding device 1. To be described in This is to enable motion compensation using the ME block on the image decoding device 2 side. Note that the number of ME blocks per frame is preferably equal to the number of SS blocks.

  In addition, the motion estimation unit 103 determines whether or not a block having a predetermined size (hereinafter referred to as a UC block) including the pixel selected by the sub-sample unit 102 is an uncovered area, and uses the information of the determination result as a motion. Output in the information. The UC block may be the same as or different from the SS block or ME block. When a UC block having a size or position different from that of the SS block is used, it is determined in advance for the UC block or described in units of frames in the stream output from the image encoding device 1. This is because the uncovered area can be identified when the image decoding apparatus 2 generates the motion compensated image. It is preferable that the UC block has a size that allows the original image to be laid out so as not to overlap and overlap, except for the image edge of the frame image. Note that the number of UC blocks in the frame image is preferably equal to the number of SS blocks.

  In this way, the motion estimation unit 103 operates to output motion information that is a combination of the motion parameter estimated for each ME block and the uncover information indicating whether or not the uncover area is for each UC block.

  The intra-frame encoder 105 performs compression encoding of the low-resolution image output from the subsample unit 102 by a predetermined compression encoding method that can be decoded for each frame, such as the Motion JPEG method, Output as a stream. The compression encoding method may be lossless compression or lossy compression.

  The high resolution information encoder 106 performs lossless compression of the sample position information and the motion information by a predetermined encoding method, and outputs the result as a high resolution information stream. Note that the sample position information can be represented by one identification number for each frame, and since the data amount is small, it is not always necessary to compress the sample position information.

  The intra-frame encoder 105 and the high-resolution information encoder 106 are not essential for the present invention, but are preferably mounted in order to improve the compression rate. When the intra-frame encoder 105 and the high-resolution information encoder 106 are not used, the sub-sample unit 102 outputs a low-resolution image as a low-resolution image stream in a predetermined arbitrary format. The motion estimation unit 103 outputs the motion information as a high resolution information stream.

  Next, an image decoding apparatus according to an embodiment of the present invention will be described with reference to FIG.

(Image decoding device)
FIG. 2 shows an image decoding apparatus according to an embodiment of the present invention. When the image encoding device 1 is configured without the intra-frame encoder 105 and the high resolution information encoder 106, the image decoding device 2 can be configured with only the image decoder 201. However, in this embodiment, in order to describe the case where the image encoding device 1 includes the intra-frame encoder 105 and the high-resolution information encoder 106, the image decoding device 2 further includes an intra-frame decoder 207 and A description will be given assuming that the high-resolution information encoder 208 is provided. Therefore, the intra-frame decoder 207 and the high resolution information encoder 208 are not essential elements in the present invention. The image decoder 201 includes an allocation unit 202, a pixel integration unit 203, a motion compensation unit 204, a delay unit 205, and an interpolation unit 206.

  The intra-frame decoder 207 receives the low-resolution image stream transmitted from the intra-frame encoder 105 of the image encoding device 1 and decodes it in a decoding format corresponding to the encoding scheme by the intra-frame encoder 105. The low resolution image is output to the allocating unit 202.

  The high-resolution information decoder 208 receives the high-resolution information stream transmitted from the high-resolution information encoder 106 of the image encoding device 1, and uses the high-resolution information encoder 106 as an encoding method. Decoding is performed in a corresponding decoding format, and among the resolution enhancement information, sample position information is output to the allocation unit 202 and motion information is output to the motion compensation unit 204.

  Similarly, when the image encoding device 1 is configured without the intra-frame encoder 105 and the high-resolution information encoder 106, the allocating unit 202 receives the low-resolution image from the image encoding device 1 as well. The motion compensation unit 204 receives the motion information from the image encoding device 1. Whether or not the image encoding device 1 includes the intra-frame encoder 105 and the high-resolution information encoder 106 is determined in advance between the image encoding device 1 and the image decoding device 2. Alternatively, the image encoding device 1 may add the auxiliary information to the stream to be transmitted in units of frames and transmit the auxiliary information, and the decoding side may identify the stream.

  The allocation unit 202 uses the sample position information to generate and output an allocation image in which each pixel of the low resolution image is allocated so as to correspond to the coordinates before the sub-sample processing. In the generation of the allocation image, the coordinates other than the pixels allocated in the sub-sampling process (that is, the coordinates of the pixels not to be allocated) are generated so as to be identifiable as the coordinates of the pixel to be interpolated by the interpolation unit 206 described later. For example, a parity bit is provided in the pixel value of each pixel of the allocated image, “1” is set as the parity bit for the pixels allocated from the low resolution image, and “0” is set as the parity bit for the other pixels. Or you may make it identify whether the pixel of each pixel position is the pixel allocated from the low-resolution image using the coordinate map matched with an allocation image. Hereinafter, for convenience of description, a pixel used for interpolation by the interpolation unit 206 described later is referred to as an “effective pixel”, and a pixel at a coordinate to be interpolated is referred to as an “interpolated pixel”.

  Note that the resolution of the allocated image does not necessarily match the resolution of the original image, but it matches the accuracy of the motion information by the motion estimation unit 103 of the image encoding device 1. For example, when the motion estimation unit 103 uses a motion vector with ½ pixel accuracy horizontally and vertically as motion information, the allocated image is also an image with a resolution of ½ pixel accuracy. That is, in the case of this example, the number of pixels existing in the allocated image is twice as long as that of the original image. The pixel accuracy used for the motion estimation is determined in advance between the image encoding device 1 and the image decoding device 2, or is added as auxiliary information to a stream transmitted by the image encoding device 1 in units of frames. May be transmitted and identified on the decoding side.

  The pixel integration unit 203 generates an integrated image obtained by integrating the pixels of the allocation image acquired from the allocation unit 202 and the pixels of the motion compensation image generated by the motion compensation unit 204, and outputs the integrated image to the interpolation unit 206. As will be described later, in order to generate a motion compensated image by the motion compensation unit 204, an allocated image used for the integrated image of the previous frame is required. Therefore, the pixel integration unit 203 determines the first frame for starting decoding. , And operates to output the layout image acquired from the layout unit 202. Therefore, the pixel integration unit 203 is different from the operation in the frame at the start of decoding and the operation in the subsequent frames.

  Further, since the pixel integration unit 203 operates in units of frames, it is easy to control the first frame for starting decoding to be the same as the allocated image acquired at the time of starting decoding. In the subsequent frames, the pixel integration unit 203 combines the pixels of the motion compensated image generated by the motion compensation unit 204 and the effective pixels of the allocated image so as to be integrated as effective pixels used for interpolation on coordinates. Output as an image. When the pixel of the motion compensated image and the effective pixel of the allocated image have the same coordinates (or within a predetermined range), it is generated using the effective pixel of the allocated image. Therefore, the integrated image is an image in which effective pixels used for interpolation with motion compensation and interpolated pixels are mixed. Each time a frame period elapses from the start of decoding, the number of effective pixels taking into account motion compensation increases, and the image quality after interpolation by the interpolation unit 206 is improved.

  The interpolation unit 206 uses the effective pixels of the integrated image to interpolate the interpolated pixels and generate a decoded image. For example, the interpolated pixel can be interpolated by a simple weighted average regardless of the density of effective pixels used for interpolation in the integrated image. As an interpolation method used by the interpolation unit 206, a known super-resolution method can be used in addition to a method of interpolating from non-uniform effective pixel arrangement. Which method should be used may be appropriately determined according to the application in consideration of the calculation load and the like. Depending on the interpolation method, there are a method of changing the pixel value by processing the value of the effective pixel, and a method of changing the pixel value without processing the value of the effective pixel, either of which may be used. .

  The delay unit 205 delays the integrated image output from the pixel integration unit 203 by one frame so that the motion compensation unit 204 can generate a motion compensated image using the integrated image of the previous frame. Output.

  The motion compensation unit 204 generates a motion compensated image by performing motion compensation on the delay image output from the delay unit 205 according to the motion parameter included in the motion information acquired from the image encoding device 1. Note that the size of the block for which motion compensation is performed (corresponding to the ME block in the image encoding device 1) and the relative position with respect to the coordinates of the SS block are determined in advance, or a stream from the image encoding device 1 By being configured to acquire from the inside, the same device is used between the image encoding device 1 and the image decoding device 2.

  Also, the motion compensation unit 204 uses the uncover information included in the motion information, and the pixels present at the coordinates included in the block designated as the uncover area (corresponding to the UC block in the image encoding device 1). Are erased from the motion compensated image to be interpolated pixels. The size and position of this block are determined in advance, or are configured to be acquired from within the stream from the image encoding device 1, so that the same between the image encoding device 1 and the image decoding device 2. Use things.

  Next, operations of the image encoding device 1 and the image decoding device 2 will be described with reference to FIG.

  FIG. 5 is a data flow diagram for explaining the operation principle of the image coding apparatus and the image decoding apparatus according to the embodiment of the present invention. When the image encoding device 1 encodes a certain frame 606 of a moving image, the sub-sample unit 102 divides the frame into SS blocks, selects one pixel in the SS block, and generates a low-resolution image 612. Then, the low-resolution image stream is output by the intra-frame encoder 105. In addition, the image encoding device 1 performs motion estimation between the motion estimation unit 103 and the original image 601 one frame before for each ME block, and the obtained motion information is generated by the sub-sample unit 102. Along with the sample position information indicating which pixel position in the block has been selected, it is encoded by the high resolution information encoder 106 and output as a high resolution information stream. Furthermore, as described above, the motion information can include information indicating the uncovered area detected at the time of motion estimation in addition to the motion parameter.

  The image decoding apparatus 2 receives the low resolution image stream and the high resolution information stream from the image encoding apparatus 1 to generate an integrated image, and then interpolates and restores the high resolution image.

  First, a frame for starting decoding will be described. It is assumed that the data regarding the frame to start decoding is 602. Data 602 includes a low resolution image, sample position information, and motion information. The image decoding apparatus 2 assigns the pixels of the low resolution image to the coordinates corresponding to the original pixel positions in accordance with the sample position information by the assignment unit 202 via the intra-frame encoder 207 and the high resolution information encoder 208, An allocated image 603 composed of effective pixels used for interpolation and interpolated pixels to be interpolated is generated. In the frame where decoding is started, the pixel integration unit 203 uses the assigned image 603 as it is as the integrated image 604. In the integrated image 604, the coordinates of the interpolated pixel are present. These coordinates are interpolated by the interpolation unit 206 using effective pixels, and a decoded image 605 is generated.

  Next, after the start of decoding, the second and subsequent frames will be described. Data 607 relating to decoding of the second frame includes a low-resolution image, sample position information, and motion information. The image decoding apparatus 2 assigns the pixels of the low resolution image to the coordinates corresponding to the original pixel positions in accordance with the sample position information by the assignment unit 202 via the intra-frame encoder 207 and the high resolution information encoder 208, An allocated image 608 made up of effective pixels used for interpolation and interpolated pixels to be interpolated is created. The motion compensation unit 204 performs motion compensation on the integrated image 604 one frame before in accordance with the motion information, and generates a motion compensated image 609.

  At this time, for the region indicated as the uncovered region in the motion information, the motion compensation unit 204 deletes the pixels included in the uncovered region from the motion compensated image. Next, the pixel integration unit 203 integrates the layout image 608 and the motion compensation image 609 to create an integrated image 610 for the second frame. When the integrated image 610 is generated by the pixel integration unit 203, if the pixel of the allocated image and the pixel of the motion compensated image have the same coordinates (or within a predetermined range), the pixel of the allocated image 608 is given priority. The integrated image 610 has the coordinates of the interpolated pixel to be interpolated. Next, the interpolation unit 206 interpolates the coordinates of the interpolated pixel to obtain a decoded image 611. In the decoding of the third frame, the image decoding apparatus 2 processes and repeats the same processing as that for decoding the second frame.

  As described above, the image decoding device 1 transmits the low-resolution image and the high-resolution information in which the selected pixel position is shifted for each frame having an extremely small amount of information, and the image decoding device 2 immediately transmits the frame image from the start of decoding. It can be restored, and interpolation processing is performed on the integrated image that integrates the pixels of the motion compensated image from the previous frame and the pixels of the allocated image from the low resolution image of the current frame, so the image quality is greatly improved after multiple frames High resolution images can be obtained.

  In addition, according to the embodiment of the present invention, the image decoding device 2 could not receive a part or all of the low resolution image by acquiring only the high resolution information due to, for example, a transmission path failure. Even when there is an image, an allocation image can be generated using pixels with coordinates that could not be received by the allocation unit 202 as inserted pixels. Therefore, using the integrated image of the previous frame, the motion compensation unit 204 performs motion compensation in the same manner as in normal decoding, and the pixel integration unit 203 outputs the integrated image of the current frame, so the frame including the missing information is discarded. It can be restored without In addition, since the integrated image is always kept up-to-date by the motion compensation process, as long as the low-resolution image and the high-resolution information of the next (not just immediately after) frame can be obtained correctly, It is possible to continue the high resolution processing by decoding a plurality of frames from the time point.

  In addition, in an encoding method using inter-frame prediction such as MPEG-2, if a part of an intermediate frame (a frame used as a reference image of another frame such as an I or P picture) fails to be received. However, the subsequent frames (frames that use a frame that failed to be received as a reference image) cannot be correctly decoded, whereas according to the embodiment of the present invention, the image decoding apparatus 2 receives a low-resolution image. Even if it fails, as long as the high resolution information can be received correctly, it is possible to continue decoding correctly thereafter.

  Furthermore, even if there is no reliable line with sufficient bandwidth to transmit all the encoded streams, if the image is transmitted with low resolution, it is transmitted with the low reliability line and the high resolution information is sufficiently corrected. It is possible to transmit a moving image while avoiding a state in which transmission is impossible, including the addition of, transmission using a highly reliable line, and transmission using a plurality of lines. This is particularly useful for information transmission during disasters.

  In the image encoding device and the image decoding device of the above-described embodiments, a computer having a control unit and a storage unit can be preferably used. For example, a control unit for controlling the intra-frame encoder 105, the high-resolution information encoder 106, the sub-sampling unit 102, the motion estimation unit 103, and the original image delay unit 104 can be configured by a central processing unit (CPU). (Not shown). Similarly, a control unit for controlling the intra-frame decoder 207, the resolution enhancement information decoder 208, the allocation unit 202, the pixel integration unit 203, the motion compensation unit 204, the delay unit 205, and the interpolation unit 206 is a central processing unit. A device (CPU) can be used (not shown). You may comprise so that each component may be controlled by one control part according to a use. In addition, a computer that functions as an image encoding device or an image decoding device can include at least one memory so as to appropriately store an image necessary for causing each component to function in a storage unit (not shown). )

  Moreover, the function which each of these component has can be implement | achieved by making such a computer run a predetermined program by CPU. Furthermore, a program for realizing the functions of these components can be stored in a predetermined storage unit. Such a storage unit can be constituted by a RAM or the like inside the computer, or can be constituted by an external storage device (for example, a hard disk). Further, such a program can be constituted by a part of software on an OS used in a computer as an image encoding device or an image decoding device.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many variations and substitutions can be made within the spirit and scope of the invention. For example, in the above-described embodiment, the image encoding device 1 and the image decoding device 2 are described as separate devices, and the stream output from the image encoding device 1 is described as being received by the image decoding device 2. From the viewpoint of decoding, the image encoding device 1 and the image decoding device 2 can be configured by one device (for example, a computer). In this case, it is useful when it is desired to manage by reducing the data size of the moving image file. Accordingly, the invention should not be construed as limited by the embodiments described above, but only by the claims.

  The present invention is useful for all hardware or software applications that handle moving images.

It is a block diagram which shows the image coding apparatus of the Example by this invention. It is a block diagram which shows the image decoding apparatus of the Example by this invention. It is a figure explaining the subsample process of the Example by this invention. It is a figure explaining the motion estimation performed with the image coding apparatus of the Example by this invention. It is a data flow figure explaining the operation | movement principle of the image coding apparatus of the Example by this invention, and an image decoding apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image coding apparatus 2 Image decoding apparatus 101 Image encoder 102 Subsampler 103 Motion estimation part 104 Original picture delay part 105 Intra-frame encoder 106 High resolution information encoder 201 Image decoder 202 Allocation part 203 Pixel integration Unit 204 motion compensation unit 205 delay unit 206 interpolation unit 207 intra-frame decoder 208 high-resolution information decoder

Claims (6)

An image encoding device for encoding a moving image consisting of continuous frame images,
The input frame image is divided by the first block, and all the first blocks are subjected to sub-sampling processing for selecting one of the pixels included in the first block in a predetermined selection order. A sub-sample unit that generates a resolution image and generates sample position information representing a selected pixel position in the first block when the low-resolution image is generated;
The input frame image is divided into second blocks, and each of the second blocks is subjected to motion estimation processing with the frame image input one frame before, and the input frame A motion estimation unit that detects an uncover area that is an image area that does not exist one frame before and appears in the current frame, and generates motion information including motion parameters of the motion estimation process and information of the uncover area; Prepared,
An image encoding apparatus, wherein a moving image composed of continuous frame images is encoded into the low-resolution image, the sample position information, and the motion information for each frame.   The image according to claim 1, further comprising an intra-frame encoder that performs compression encoding on the low-resolution image generated by the sub-sample unit by a compression encoding method that can be decoded for each frame. Encoding device.   The image code according to claim 1, further comprising a high-resolution information encoder that performs lossless compression of the sample position information and the motion information generated by the motion estimation unit using a predetermined compression method. Device. The moving image encoded by the image encoding device according to any one of claims 1 to 3, for each frame, the low-resolution image, the sample position information and integrated image generated by using the motion information An image decoding apparatus that performs interpolation processing and decodes with high resolution,
Using the sample position information, an allocation unit that generates an allocation image in which each pixel of the low-resolution image is allocated so as to correspond to coordinates before sub-sample processing;
In accordance with the motion parameter included in the motion information, motion compensation is performed on the integrated image of the previous frame, and an uncovered area is designated using uncover information indicating whether the uncovered area is included in the motion information. A motion compensation unit that erases the generated pixels and generates a motion compensated image;
A pixel integration unit that generates an integrated image obtained by integrating the pixels of the allocation image generated by the allocation unit and the pixels of the motion compensation image generated by the motion compensation unit;
An interpolation unit that performs an interpolation process on the integrated image and generates a decoded image;
An image decoding apparatus, wherein the frame image is decoded for each frame using the low-resolution image, the sample position information, and the motion information, and a moving image composed of continuous frame images is restored. The image decoding apparatus according to claim 4, wherein the moving picture encoded by the image encoding apparatus according to claim 2 is decoded.
Wherein the low-resolution image compressed and encoded by intraframe encoder, the compression encoding method image decoding apparatus further comprising a frame decoder for compressing a decoding scheme corresponding to the. The image decoding device according to claim 4 or 5, wherein the moving image encoded by the image encoding device according to claim 3 is decoded.
The high resolution information encoder by a sample position information and motion information lossless compression, high resolution information decoder further images you characterized in that it comprises a to a decoding scheme corresponding to the predetermined compression method Decoding device.

Images