JP4667409B2 - Image decoding apparatus and image encoding apparatus - Google Patents

Description

  The present invention relates to image signal processing, and more particularly to an image decoding apparatus and image code for reducing a quantization error for synthesizing a decoded image of a signal obtained by encoding an image of the same content in a plurality of modes in compression encoding / decoding of an image. The present invention relates to a conversion device.

  Various services and businesses using the Internet are being developed as network technology advances. In addition, an environment in which images and moving images can be easily handled through a network has been constructed by image compression techniques such as MPEG and JPEG. In such an environment, with the spread of network services (such as YouTube) that distributes video files, the same video content is flooded with different encoding methods, image sizes, and compression rates. The situation can be seen.

  On the other hand, a technique called MDC (Multiple Description Coding) has been studied as one of encoding techniques for enhancing resistance to errors on a network (for example, Non-Patent Document 1). This is a technique in which a plurality of the same information sources are encoded and distributed using various network paths (see FIG. 9), and studies have been made on using this MDC for video signals.

  Since the MDC encodes a plurality of the same signal, the compression efficiency is inferior to the distribution by a single encoded signal. However, since it is distributed via a separate network path, it is a robust and robust technique.

V. K. Goyal, “Multiple Description Coding: Compression Meets the Network”, IEEE Signal Processing Magazine, July 2001, p. 74-93

  As described above, the MDC technique using a network is a technique whose main purpose is error tolerance. For this reason, although the quality of each content is not different from the case of single encoding, the total compression rate is low because a plurality of passes are used. On the other hand, many identical contents exist on the network regardless of the MDC framework.

  An object of the present invention is to provide an image decoding apparatus and an image encoding apparatus that reduce the quantization error due to compression encoding in order to improve the image quality of the same content by utilizing both of these situations.

  The present invention combines a plurality of compressed / decompressed images to reduce the quantization error of an image generated by a single encoding and to improve the image quality.

That is, an image decoding apparatus according to the present invention receives a plurality of compressed and encoded streams, decodes the plurality of encoded streams, and generates a plurality of decoded images; and the plurality of decoding A determination unit that determines whether another decoded image including the predetermined subject image of the plurality of decoded images is a similar image with respect to a decoded image including one predetermined subject image of the images And an alignment unit that aligns the position of the predetermined subject image for a plurality of decoded images determined as the similar images.

  In the image decoding device according to the present invention, the determination unit calculates a difference between pixel values of corresponding pixel positions among the plurality of decoded images, and an average value of the difference values is equal to or less than a predetermined threshold value. For example, it is determined that the image is similar.

  In the image decoding device according to the present invention, the determination unit calculates a difference between pixel values of corresponding pixel positions among the plurality of decoded images, and the number of the difference values that is equal to or less than a predetermined threshold is: If it is within a predetermined range, it is determined that the image is similar.

In the image decoding device according to the present invention, the plurality of images obtained by decoding the plurality of encoded streams are images having different compression ratios for the same image.

In the image decoding device according to the present invention, the plurality of images obtained by decoding the plurality of encoded streams are images having different cutout positions for the same image.

  Furthermore, an image encoding device according to the present invention is characterized in that a plurality of compressed encoded streams having different cutout positions for the same image are transmitted to the image decoding device according to the present invention.

  According to the present invention, it is possible to improve the image quality as compared with the case where a single encoded content is decoded by using a plurality of encoded identical contents and combining them. In addition, it is possible to efficiently use a large amount of content existing on the network and obtain a desired image with reduced quantization error.

  First, an example of a decoding system using the image decoding apparatus according to the first embodiment of the present invention will be described.

Example 1
1 is a configuration example of a decoding system including an image decoding apparatus according to a first embodiment of the present invention. The present decoding system example includes an image encoding device 1, an image decoding device 2, and a service server 3 that is used exclusively for providing a network service (for example, a network service site) of a moving image file or a still image. In the figure, the image decoding apparatus 2 can acquire a moving image file or a still image including the same content from the service server 3 via a network (communication network shown in the figure) (path ad), or alternatively, image encoding. A moving image file or a still image including the same content can be acquired from the device 1 via a network (a communication network shown in the figure) (path bd). Alternatively, the image decoding device 2 receives from the image encoding device 1 a moving image file including the same content multiplexed on a predetermined carrier wave (for example, including a broadcast carrier wave and a broadcast wave broadcasted) or By receiving a still image signal, it can be acquired wirelessly (path c).

  In other words, the image decoding device 2 can acquire a moving image file or a still image including the same content from any one of the path ad, the path bd, or the path c, and acquire such a moving image file or still image. The method itself can be realized by known means. Here, it is needless to say that a plurality of service servers 3 may be provided for each network service, and a plurality of such service servers 3 may be provided in the communication network of the present decoding system example. Moreover, the image encoding device 1, the image decoding device 2, and the service server 3 may be capable of mutual communication as necessary.

  Next, in the case of acquiring a number of still images or moving image compression-encoded streams (hereinafter referred to as encoded streams) that can be acquired from the network service site in the path ad, The image decoding device 2 will be described.

  FIG. 2 is a block diagram illustrating the image decoding apparatus according to the first embodiment of the present invention. The image decoding device 2 according to the first embodiment includes a plurality of decoding units (collectively described as the decoding unit 21), a determination unit 22, a positioning unit 23, and a synthesis unit 24.

  Here, in order to function as the image decoding device 2, a computer can be preferably used. Such a computer controls the decoding unit 21, the determination unit 22, the alignment unit 23, and the combining unit 24. Can be configured by a central processing unit (CPU) (not shown), and an image necessary for operating the encoded stream or decoding unit 21, the determination unit 22, the alignment unit 23, and the synthesis unit 24 A storage unit that stores data as appropriate can be configured by at least one memory (not shown).

  Further, by causing such a computer to execute a predetermined program by the CPU, it is possible to realize the functions (functions described later) of the decoding unit 21, the determination unit 22, the alignment unit 23, and the synthesis unit 24. Furthermore, a program for realizing the functions of the decoding unit 21, the determination unit 22, the alignment unit 23, and the synthesis unit 24 can be stored in a predetermined area of the storage unit (memory). Such a storage unit can be constituted by a RAM or the like inside the image decoding device, or can be constituted by an external storage device (for example, a hard disk). Further, such a program can be constituted by a part of software on an OS (stored in a ROM or an external storage device) used by a computer as an image decoding device.

  The decoding unit 21 includes a receiving unit (not shown), and decodes a stream that has been compression-encoded by a decoding method that conforms to each encoding method for each encoded stream that is received by the receiving unit and that conforms to each encoding method. -Compress / decompress, generate a plurality of decoded images, and output to the determination unit. As representatively shown in FIGS. 3A to 3E, the image decoded by the decoding unit 21 includes not only the same content but also different imaging times (illustration (a)), Those with different compression rates (illustration (b)), high-definition images (illustration (c)), those with different angle of view and image size (illustration (d)), those with completely different contents (illustration (e)), etc. Various images (a frame image in a moving image file) are included. Also, images with different resolutions or images with different encoding methods may be used. Furthermore, it is also possible to use another frame image in the same moving image for the moving image. Hereinafter, the terms “image”, “similar image”, “decoded image”, and “subject image” are used without distinction between a still image or a frame image in a moving image file. The term “pixel value” refers to a component that constitutes a pixel represented in RGB format, YUV format, or RGBD format with a depth value (D). Further, the difference in pixel value refers to a difference in pixel components between a plurality of images in the same format.

  The determination unit 22 inputs the plurality of images decoded from the decoding unit 21, selects an image (a predetermined image desired to be combined) that can be used by the combining unit 24 described later, and combines the images. A similar image including a desired predetermined subject image is determined and sent to the alignment unit 23. Similar image determination is performed, for example, when a plurality of arbitrarily selected decoded images have the same image size including a desired subject image (if the image size is different, the desired subject image or frame image size is used as a reference). The same image size), and the difference in the pixel value of the corresponding pixel position between the multiple decoded images (each image is divided into a plurality of blocks and the difference between the corresponding pixel blocks) If the average value of the difference values is equal to or less than a predetermined threshold, the image is determined as a similar image. Alternatively, the determination of a similar image is performed by calculating a difference in pixel values at corresponding pixel positions (including a difference between corresponding pixel blocks by dividing each image into a plurality of blocks) between a plurality of decoded images. If the difference value is equal to or smaller than a predetermined threshold value is within a predetermined range, it is determined as a similar image.

  Alternatively, when a plurality of similar images are selected from a single moving image file configured as a collection of N frame images, similar images are determined by determining whether adjacent similar frame images (for example, an Mth frame image and Frame images in the order of a predetermined range such as (M + m-th frame image) are determined as similar images (in this case, N, M, m are positive numbers excluding 0, and m <M, N ≦ M ≦ Nm). Alternatively, the determination may be made using an existing similar image search tool. A person who uses this image decoding apparatus may determine and set a similar range in advance.

Positioning unit 23 receives a plurality of decoded image determined as a similar image by determining unit 22, the scaling and the parallel move of which processing to generate a plurality of decoded image aligning the predetermined object image And sent to the synthesis unit 24. Here, the alignment of an object included in a plurality of recovery Goga image desired to be synthesized, for example, a matching pixel values in a two-dimensional plane between the images, performed while scaling, the difference value (i.e. , Error) is determined by determining an enlargement / reduction amount and a parallel movement amount. Image alignment may be performed based on scaling information accompanying enlargement / reduction in the determination unit 22. Alternatively, the accuracy can be further improved by performing the above-described matching after a three-dimensional conversion process such as affine transformation. In addition, a filtering operation can be included in enlargement or reduction, and examples of this filter include bilinear interpolation and trilinear interpolation. In general, images included in a moving image file can be relatively easily aligned with a subject image desired to be synthesized by adjusting the frame size. A person who uses this image decoding apparatus may use predetermined image processing software or the like to set and set the alignment in advance.

  The synthesizing unit 24 receives the image that has been registered by the alignment unit 23, performs a synthesizing process, and outputs a synthesized output image. In addition to combining the pixel values in a plurality of images as a simple average value between the plurality of images, the combining processing can be performed by weighting the pixel values of the plurality of decoded images. The output image includes generating a new moving image file from a frame image including a subject image desired to be synthesized. Weighting at the time of synthesis of a plurality of decoded image signals weights a strong weighting coefficient to the decoded image signal based on, for example, the height of the bit rate per unit pixel (or unit pixel block).

  Here, the principle that the image quality can be improved by combining a plurality of images with different quantization, that is, with different compression ratios will be described. FIG. 4 is an explanatory diagram of reduction of quantization error by synthesis. Here, the difference between the luminance level (Y component in the YUV format) of the original single-encoded image and the luminance level of the image after compression encoding is defined as a quantization error. The compression of the image signal is realized by quantization. That is, it is possible to reduce the amount of information and perform compression by replacing the finely defined levels (luminance and DCT coefficients) with rough representative points. Therefore, there is a quantization error in lossy compression encoding, and if this is large, image quality will be degraded. The first quantization and the second quantization in FIG. 4 indicate that the luminance level of a certain pixel has changed to another luminance level due to compression encoding.

  In the example of FIG. 4, the second quantization has a larger quantization error than the first quantization. The image decoding apparatus according to the first embodiment synthesizes such another compression-encoded (quantized) image including the same content collected via the network. The first quantization + second quantization shown in FIG. 4 is a combination of luminance levels of the first quantization and the second quantization (in this example, an average value is calculated). As in this example, if two quantized values exist above and below the original luminance level, the errors are averaged, and the average value often approaches the original luminance level. Further, if a plurality of quantized pixels are combined, the original luminance level value is statistically approached. As described above, by statistically synthesizing a large number of pixels, the image quality is improved as compared with a single compression coding result. An image synthesized by the synthesis unit 24 is obtained as a final output image (illustration (f) in FIG. 3).

  As a result, it is possible to improve the image quality as compared with the case where a plurality of identical encoded contents are used and the contents are combined to be single encoded. In addition, it is possible to efficiently use a large amount of content existing on the network and obtain a desired image with reduced quantization error.

  If you add a description as a specific application, if you have acquired multiple video files over the network in various compression modes (for example, video files that digitally saved and saved old movies that have become out of print) A moving image file having an image quality that cannot be restored by the prior art can be restored as a clearer moving image file.

  Next, an example of a decoding system using the image decoding apparatus according to the second embodiment of the present invention will be described.

(Example 2)
In FIG. 1 described above, when acquiring a plurality of still images or moving image compression-encoded streams (hereinafter referred to as encoded streams) on the path b-d or the path c, the image code of the second embodiment is used. The converting apparatus 1 will be described. It should be noted that the same constituent elements will be described with the same reference numerals. Note that the image decoding apparatus according to the second embodiment can be realized by the same components as the image decoding apparatus according to the first embodiment, and thus a detailed description thereof will be omitted, but the encoded stream from the image encoding apparatus 1 is received. Is different. In other words, also in the second embodiment, the image encoding device 1 can transmit images typically shown in FIGS. 3A to 3E to the image decoding device 2.

(Image coding device)
FIG. 5 is a block diagram showing an image coding apparatus according to the second embodiment of the present invention. The image encoding device 1 according to the second embodiment includes a block cutout unit 11 and a plurality of encoding processing units (collectively described as the encoding processing unit 12).

  Here, in order to function as the image decoding device 2, a computer can be preferably used, and such a computer has a control unit for controlling the block cutout unit 11 and the encoding processing unit 12 as a central processing unit. (CPU) can be configured (not shown), and a storage unit that appropriately stores an image necessary for operating the block cutout unit 11 and the encoding processing unit 12 can be configured with at least one memory. (Not shown).

  Further, by causing such a computer to execute a predetermined program by the CPU, it is possible to realize functions (functions described later) of the block cutout unit 11 and the encoding processing unit 12. Furthermore, a program for realizing the functions of the block cutout unit 11 and the encoding processing unit 12 can be stored in a predetermined area of the storage unit (memory). Such a storage unit can be constituted by a RAM or the like inside the image decoding device, or can be constituted by an external storage device (for example, a hard disk). Such a program can be constituted by a part of software (stored in a ROM or an external storage device) on an OS used in a computer as an image decoding device.

  The block cutout unit 11 cuts out an arbitrary input image into image blocks at a plurality of cutout positions, and sends each cut out image block to a plurality of encoding processing units 12. In the cutout process, the upper left coordinates of the image are set as reference coordinates, and an image block can be cut out at a plurality of cutout positions for an arbitrary input image.

  The encoding processing unit 12 performs respective compression encoding processing on each of the images cut out by the block cutout unit 11, and sends out an encoded stream of image blocks. A signal including the encoded stream is transmitted as a signal suitable for network communication or wireless communication. Note that the encoded stream of each image block may follow a different encoding method, and in this case, information on the encoding method is preferably transmitted as auxiliary information. Preferably, information on the cutout position by the block cutout unit 11 is transmitted as auxiliary information.

  Here, in order to facilitate understanding of the present image encoding apparatus, consider a case where an encoded stream of a plurality of image blocks is transmitted from a still image (input image). Furthermore, it is assumed that each encoding method of the encoding processing unit 12 uses, for example, a JPEG method in which an image is processed by being divided into rectangular image blocks. FIG. 6 shows an image of processing of the present image coding apparatus in this case. FIG. 6 shows an image in which a certain still image is cut out and encoded in two ways (image A and image B). Here, since JPEG is assumed as the encoding method, the DCT process is performed in units of 8 × 8 pixel blocks. Therefore, there are 8 × 8 = 64 ways to shift the cutout position. However, it should be noted that if the pixels are shifted by 8 pixels in the vertical and horizontal directions, they are the same as before the shift.

(Image decoding device)
Referring to FIG. 2, decoding unit 21 receives a plurality of encoded streams from image encoding device 1, performs appropriate decoding processing based on decoding information from image encoding device 1, and determines a determining unit. 22 to send.

  The determination unit 22 inputs a plurality of images decoded from the decoding unit 21, selects an image (a predetermined image desired to be combined) that can be used by a combining unit 24 described later between the plurality of images, A similar image including the subject image is determined and sent to the alignment unit 23. Here, when the image encoding device 1 transmits a plurality of encoded streams to the image decoding device 2, it is also possible to transmit auxiliary information indicating that the image is a similar image.

Positioning unit 23 receives a plurality of decoded image determined as a similar image by determining unit 22, the scaling and the parallel move of which processing to generate a plurality of decoded image aligning the predetermined object image And sent to the synthesis unit 24. Here, when image block cutout position information regarding a plurality of encoded streams from the image encoding device 1 is received, alignment is performed based on the cutout position information.

  The synthesizing unit 24 receives the image that has been registered by the alignment unit 23, performs synthesis processing, and outputs a synthesized output image (for example, the synthesized image shown in FIG. 6). A composite output image can be generated by averaging or weighting a plurality of decoded image signals. Weighting at the time of synthesis of a plurality of decoded image signals weights a strong weighting coefficient to the decoded image signal based on, for example, the height of the bit rate per unit pixel (or unit pixel block). Here, when the image encoding device 1 transmits a plurality of encoded streams to the image decoding device 2, auxiliary information indicating that the image is a high or low S / N image, a high bit rate image or a low image. Can also be sent.

  Next, the results of experiments using actual images will be described along the configuration example of this embodiment. In principle, the same effect as in the first embodiment can be obtained (see FIG. 4). However, in practice, a certain still image is cut out and synthesized from 1 (no synthesis) to 64 synthesis. did. FIG. 7 and FIG. 8 show the test image used and the image quality improvement effect associated with the increase in the number of images. FIG. 7 shows the result of using an image that hardly contains high-frequency components (for example, an image that can be used in a network service called “Airplane Landing”). FIG. 8 shows the result of using an image containing a lot of high-frequency components (for example, an image that can be used in a network service called “Whale Show”). In each graph, the horizontal axis indicates the number of combined images, and the vertical axis indicates PSNR. The PSNR is an error between the original image and the image that has been compressed and decoded in dB, and the larger the number, the higher the image quality (that is, the smaller the quantization error).

  7 and 8 show that the image quality is improved as the number of combined images increases. In particular, it was confirmed that a sufficient effect was obtained by combining two or three images and that a similar effect appeared. Note that there are discontinuous parts on the graph because clipping was performed with priority in the vertical direction, shifted by 8 pixels, and then shifted by 1 pixel in the horizontal direction.

  According to the second embodiment, it is possible to improve the image quality as compared with the case where a plurality of the same encoded contents are used and the contents are combined to be single encoded.

  If a description as a specific application is added, an image (input image) to be transmitted when a conventional image encoding device requires transmission of a moving image file or a still image with a low resolution or a high compression rate. In some cases, the image quality deteriorates and the image encoding device on the receiving side cannot accurately restore the image (for example, a portable terminal with a camera). By using the image encoding device and the image decoding device according to the second embodiment, even when the image encoding device is required to transmit a moving image file or a still image with a low resolution or a high compression rate, the image decoding is performed. The input image can be restored more accurately by the apparatus.

  As is apparent from the above-described embodiments, in order to facilitate image composition, a plurality of images transmitted in the plurality of encoded streams are images having different compression rates for the same image, or the same image. It is preferable that the images have different cut-out positions, different frame images for moving image files, or images having different resolutions for the same image.

  Note that a program for causing a computer that functions as an image decoding apparatus to function as means as each component of the present invention can be recorded on a computer-readable recording medium.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. For example, the weighting required for image composition may be performed by the determination unit 22 or the alignment unit 23, and the image alignment may be performed by the determination unit 22 or the composition unit 24. In addition, the present invention can be implemented in a mode in which the first embodiment and the second embodiment are combined. Accordingly, the invention should not be construed as limited by the above-described embodiments, but only by the claims.

  The image coding apparatus according to the present invention is useful in a system that acquires an image with a reduced quantization error.

It is an example of a structure of a decoding system provided with the image decoding apparatus of Example 1 by this invention. It is a block diagram which shows the image decoding apparatus of Example 1 by this invention. It is explanatory drawing in the case of using the several image obtained from a network service in a decoding system provided with the image decoding apparatus of Example 1 by this invention. It is explanatory drawing of reduction of the quantization error by the synthesis | combination of Example 1 by this invention. It is a block diagram which shows the image coding apparatus of Example 2 by this invention. It is a figure which shows the image of a process of the image coding apparatus of Example 2 by this invention. It is a figure which shows the experimental result showing the relationship between the composite number of Example 2 by this invention, and an image quality. It is a figure which shows the experimental result showing the relationship between the composite number of Example 2 by this invention, and an image quality. It is a block diagram of MDC (Multip1e Description Coding).

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image encoding apparatus 2 Image decoding apparatus 3 Service server 11 Block cut-out part 12 Encoding process part 21 Decoding part 22 Determination part 23 Positioning part 24 Composition part

Claims (6)

A decoding unit that receives a plurality of compressed and encoded streams, decodes the plurality of encoded streams, and generates a plurality of decoded images;
Whether or not another decoded image including the predetermined subject image among the plurality of decoded images is a similar image with respect to one decoded image including the predetermined subject image among the plurality of decoded images. A determination unit for determining;
An image decoding apparatus comprising: an alignment unit that aligns the position of the predetermined subject image with respect to a plurality of decoded images determined as the similar images.   The determination unit calculates a difference between pixel values of corresponding pixel positions among the plurality of decoded images, and determines that the image is a similar image if an average value of the difference values is equal to or less than a predetermined threshold value. The image decoding apparatus according to claim 1.   The determination unit calculates a difference between pixel values of corresponding pixel positions among the plurality of decoded images, and if the difference value is equal to or less than a predetermined threshold value within a predetermined range, The image decoding device according to claim 1, wherein the determination is performed. The image decoding apparatus according to claim 1, wherein the plurality of images obtained by decoding the plurality of encoded streams are images having different compression ratios for the same image. The image decoding apparatus according to claim 1, wherein the plurality of images obtained by decoding the plurality of encoded streams are images having different cutout positions for the same image.   An image encoding apparatus, comprising: a plurality of compressed encoded streams that are different from each other in cutout positions for the same image.

Images