WO2007034383A2

WO2007034383A2 - Image coding

Info

Publication number: WO2007034383A2
Application number: PCT/IB2006/053280
Authority: WO
Inventors: Peter-Andre Redert; Piotr Wilinski
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2005-09-19
Filing date: 2006-09-14
Publication date: 2007-03-29
Also published as: WO2007034383A3

Abstract

A method of encoding an image (OI) represented by pixel values comprises the steps of: - producing a partial image (PI) by selecting a number of pixel values, - outputting the partial image (PI), - producing a combined partial image using the current partial image and any preceding partial images, if available, - producing a reconstructed image (RI') using the combined partial image, determining a quality measure of the reconstructed image, and - repeating the above steps if the determined quality measure is smaller than a threshold value. Each partial image contains at least some pixels that have not been previously selected. The combined partial image contains fewer pixels than the original image. The step of producing a reconstructed image preferably involves a hole filling process. An encoding device (1) is arranged for carrying out these method steps and comprises a selection unit (12), an accumulation unit (15), a reconstruction unit (16) and a correspondence unit (17).

Description

Image coding

The present invention relates to image coding. More in particular, the present invention relates to a method and device for encoding an image, or a series of images, and a method and device for decoding an encoded image, or a series of encoded images, such as video.

Various image or video encoding methods are known. Many image encoding methods use a transformation, such as a digital cosine transform (DCT) or a digital Fourier transform (DFT) to obtain transform domain parameters. Such methods rely on the fact that some transform domain parameters can be truncated or even set to zero (that is, effectively deleted) without significantly reducing the quality of the image represented by those parameters.

International Patent Application WO 2004/080050 (Philips) discloses a video encoding apparatus which divides an image into a number of regions. For each region, an encoding parameter is determined. The encoded image is fed back and analyzed. This process is repeated until a desired encoding performance is achieved, for example a given quality level.

All image encoding methods essentially aim to produce the highest image quality level using the smallest amount of output image data. This may be done by reducing the amount of data while maintaining a certain image quality level, and/or by improving the quality level while maintaining the amount of data.

It is an object of the present invention to improve on the Prior Art and to provide a method and device for encoding an image which allow a iurther image data reduction while providing a good image quality.

Accordingly, the present invention provides a method of encoding an image represented by pixel values, the method comprising the steps of: producing a partial image by selecting a number of pixel values, producing a combined partial image using the current partial image and any preceding partial images, producing a reconstructed image using the combined partial image, determining a quality measure of the reconstructed image, - repeating the above steps if the determined quality measure is smaller than a threshold value, wherein each partial image contains pixels that have not been previously selected, and wherein the combined partial image contains fewer pixels than the original image.

By producing a partial image having a smaller number of pixel values than the original image, the number of pixel values to be transmitted or stored can be significantly reduced. Accordingly, the compression level may be significantly higher than in the Prior Art.

By reconstructing the image from the partial image, the relative quality of the reconstructed image can be assessed and one or more additional partial images can be produced by selecting additional pixel values. Alternatively, or additionally, the number of pixels of a partial image can be increased.

The quality measure of the reconstructed image may be determined in various ways, for example by determining the degree of correspondence between the reconstructed image and the original image. This degree of correspondence may be calculated using a distance measure (e.g. a Euclidean or other distance measure) applied to, for example, the gray values (pixel values) of corresponding pixels in the two images. The resulting distances of all pixels may be averaged or processed otherwise to yield the quality measure. Alternatively, only a limited number of distances may be used, for example maximum distances. The quality measure determined in this manner may be compared with a threshold value, which may be predetermined or adjustable. Instead of comparative quality measures, objective measures which involve no comparison with the original image may also be used.

The step of producing a partial image is preferably followed by a step of outputting the partial image. Accordingly, each partial image is produced and then output. It is, of course, also possible to collect a number of partial images and output these partial images in a single process step. The step of outputting a partial image may involve storing and/or transmitting the partial image.

The step of producing a combined partial image may involve combining the current partial image with a previous combined partial image to produce a current combined partial image. If no previous partial image derived from the same original image is available, the step of producing a combined partial image may be omitted.

The threshold value is preferably predetermined, but may be variable. In addition to a threshold value, a maximum number of partial images may be used or another criterion may be used.

The partial images are preferably non-overlapping. This ensures the highest encoding efficiency. However, in some embodiments the partial images pertaining to the same original image may share one or more pixels.

Preferably, the step of producing a reconstructed image involves a hole filling process. That is, a known hole filling process may be used to reconstruct an image from the (at least one) partial image. Such hole filling processes are well known in the art. An example of a hole filling process is disclosed in the paper "Texture Synthesis by Non-parametric Sampling" by Alexei A. Efros and Thomas K. Leung, University of California, Berkeley, USA, 1999, which paper is herewith incorporated in this document. The pixels whose values are used in a partial image may be selected at random, each time a partial image is to be created. However, it is preferred that the pixels values are selected according to at least one predetermined pattern. This eliminates the need to store or transmit pixel position information, should any reconstruction process require such information. The method may further comprise the step of outputting an indication if the determined quality measure is greater than the threshold value. This indication, which indicates that the required quality has been achieved and that the iterative process may be terminated, may be output as a separate data item, or may be embedded in the last partial image produced. The quality may for example be determined by comparing the reconstructed image and the original image, and determining some distance measure (e.g. Euclidian distance) between the pixel values of both images. However, any other suitable quality measure may also be used.

In advantageous embodiments, the step of outputting the partial image involves compressing the partial image. This compression, which typically takes place immediately prior to outputting the partial image, may involve a DCT or DFT transformation, run length coding, and/or any other suitable operation for reducing the amount of data to be transmitted or stored. The compression may be lossy but is preferably lossless so as to allow the highest quality reconstructed image. In a further advantageous embodiment, the step of producing a partial image involves making at least one hole in the original image. That is, a partial image may consist of selected pixels which surround groups of contiguous discarded pixels. These discarded pixels, which are not selected to be part of the partial image, correspond with holes in the original image. Although holes will typically consist of groups of contiguous pixels, single pixels may also constitute holes.

Although the method is described here with reference to a single image, the present invention is not so limited and multiple (consecutive or non-consecutive) images may be encoded and/or decoded in accordance with the present invention. The present invention therefore also applies to video sequences.

The present invention also provides a method of decoding an image encoded by the encoding method defined above, the decoding method comprising the steps of: receiving one or more partial images containing selected pixel values of a single original image, - producing a combined partial image using the received partial images, and producing a reconstructed image using the combined partial image.

In this way, the partial images produced with the encoding method defined above may be decoded. The quality of the reconstructed image is defined by the quality measure used in the encoding method and is therefore predictable. The received partial images may be compressed, and the decoding method may further comprise the step of decompressing the compressed partial images.

The present invention additionally provides a computer program product for carrying out the encoding method defined above and/or the decoding method defined above. The term computer program product is understood to refer to any physical realization, e.g. an article of manufacture, of a collection of commands enabling a (generic or special purpose) processor to execute any of the characteristic functions of an invention after a series of loading steps to load the commands into the processor. In particular the computer program product may be realized as program code, processor adapted code derived from this program code, or any intermediate translation of this program code, on a carrier such as e.g. a disk or other plug-in component, present in a memory, temporarily present on a (wired or wireless) network connection, or program code on paper. Apart from program code, invention characteristic data required for the program may also be embodied as a computer program product. The present invention still further provides an encoding device for carrying out the encoding method defined above, a decoding device for carrying out the decoding method defined above, and an image transmission system, comprising an encoding device as defined above and/or a decoding device as defined above. An image processing system may advantageously be provided with a decoding device of the present invention.

The present invention will further be explained below with reference to exemplary embodiments illustrated in the accompanying drawings, in which: Fig. 1 schematically shows an encoding device according to the present invention.

Fig. 2 schematically shows a decoding device according to the present invention.

Fig. 3 schematically shows a selection of pixels as may be used in the present invention.

Fig. 4 schematically shows an image system comprising a decoding device according to the present invention.

Fig. 5 schematically shows an encoding method according to the present invention. Fig. 6 schematically shows a decoding method according to the present invention.

The encoding device 1 shown merely by way of non- limiting example in Fig. 1 comprises a buffer (BUF) unit 11, a selection (SEL) unit 12, a compression (COM) unit 13, a decompression (DEC) unit 14, an accumulation (ACC) unit 15, a reconstruction (REC) unit 16, and a correspondence (COR) unit 17. The compression unit 13 and the decompression unit 14 are optional and may be omitted without departing from the scope of the present invention. Alternatively, the output of the selection unit 12 may be coupled to both the compression unit 13 and the accumulation unit 15, in which case the decompression unit 14 may be omitted. The buffer unit 11 may be incorporated in the selection unit 12.

The buffer unit 11 receives an original image OI and temporarily stores this image. This original image consists of a number of pixels (picture elements), for example 100 pixels, 10000 pixels or 1 million pixels, each having a pixel value representing the "gray level" of the pixel. This "gray level" may consist of a suitable number of bits, ranging from a single bit (black or white only) via 8 bits (white or 255 shades of gray) to 32 bits or more (it is noted that only black and white images are discussed here for the sake of clarity, however, those skilled in the art will easily be able to adapt the present invention to color images by assigning multiple pixel values to each pixel).

The selection unit 12 selects a number of pixels from the original image (OI) stored in the buffer unit 11 to produce a partial image (PI). This selection may be random, but is preferably carried out using predetermined selection patterns, as will later be discussed in more detail with reference to Fig. 3. The use of predetermined selection patterns may facilitate the reconstruction of the image, as the pixel positions of the selected pixels are known. When a random selection pattern is used, the number of selected pixels is preferably predetermined and will typically depend on the image size, that is, on the number of pixels of the original image. For example, each selection may produce 1/2O⁴¹¹ (5%) of the original number of pixels, that is, 500 pixels out of an original image consisting of 10000 pixels. Of course, other ratios than 1/2O* may also be used, for example 1/5O⁴¹¹ (2%), 1/100* (1%) or 1/10* (10%).

The partial image produced by the selection unit 12 is fed to the (optional) compression unit 13 for compression using a DFT, run length coding and/or any other suitable process. The compression process used by the compression unit 13 may be known per se. In some embodiments the compression unit 13 may be omitted.

The partial image (PI) produced by the selection unit 12 and (optionally) compressed by the compression unit 13 is output by the encoding unit 1. The same partial image is then decompressed by the decompression unit 14 and fed to the accumulation unit 15, where the current partial image is combined with any previous partial images associated with the same original image.

The accumulated or combined partial images are then reconstructed by the reconstruction unit 16 using a hole filling process or any other suitable image reconstruction process. Such processes may be known per se and are disclosed in, for example, the article by Efros and Leung referred to above. The selected pixels may serve as seeds from which the pixel values of the reconstructed pixels are derived.

The reconstructed image (RI') produced by the reconstruction unit 16 is fed to the correspondence unit 17 where the correspondence with the original image (OI) stored in the buffer 11 is evaluated. The correspondence unit 17 may receive a quality signal (QS) indicative of the desired correspondence level and hence of the quality of the reconstructed image. The correspondence level may be measured using a suitable process which may be known per se, for example a distance measure indicative of the (absolute or, alternatively, squared) distance in gray levels between each pixel of the original image

If the quality of the reconstructed image is high enough (that is, if the correspondence between the reconstructed image RF and the original image OI exceeds a threshold which may be determined by the quality signal QS), the contents of the buffer unit 11 and the accumulation unit 15 are deleted and a new original image may be loaded into the buffer unit 11.

If the quality of the reconstructed image is not high enough (that is, if the correspondence between the reconstructed image RI' and the original image OI fails to exceed the threshold referred to above), the selection unit 12 selects another set of pixels of the same original image, resulting in another partial image (PI) associated with this original image (OI). Accordingly, several partial images PI₁, PI₂, ..., PI_N pertaining to the same original image OI may be produced, until the desired reconstruction quality is achieved. In the embodiment shown in Fig. 1, the correspondence unit 17 produces a suitable repeat/flush (RTF) signal controlling the flushing of the buffer unit 11 and the accumulation unit 15, and also controlling the repeated selection of the selection unit 12. It will be understood that instead of a single control signal (R/F), multiple control signals may be used.

It can thus be seen that the present invention uses an iterative method in which several partial images may be produced until the combined partial images provide a satisfactory reconstruction of the original image. Each partial image contains a selected number of pixels of the original image. The partial images may be compressed if desired.

It is noted that the present invention utilizes relatively complex reconstruction processes, such as hole filling processes, to achieve very efficient image data encoding. The present invention benefits from the insight that recent advances in computer technology make the application of such complex processes feasible.

A decoding device according to the present invention is schematically illustrated in Fig. 2. The exemplary decoding device 2 of Fig. 2 comprises an (optional) decompression (DEC) unit 24, an accumulation (ACC) unit 25 and a reconstruction (REC) unit 26. The decompression unit 24 receives and decompresses or more partial images PI₁, PI₂, ..., PI_N preferably consecutively. The accumulation unit 25 combines the decompressed partial images into a single partial image which is then fed to the reconstruction unit 26. If the units 24-26 are functionally identical to their counterparts 14-16 in the encoding device 1, the resulting reconstructed image RI output by the reconstruction unit 26 will be identical to the reconstructed image RF output by the reconstruction unit 16 (assuming a lossless transmission from the encoding device 1 to the decoding device 2). As a result, the quality of this reconstructed image is known beforehand, as this quality has been assessed in the encoder 1 (correspondence unit 17). It is noted that the units 11-17 of the encoding device 1 and the units 24-26 of the decoding device 2 may be implemented in hardware or in software, or in any suitable combination of hardware and software.

Exemplary selection patterns are illustrated in Fig. 3. The first pattern 31 forms an edge of the image, leaving a "hole" in the center. In the example shown, the image has one hundred (10x10) pixels 30 of which thirty-six pixels are selected (36%). The selected pixels, designated with an X, form the outer rim of the image, leaving sixty- four pixels in the center which have not been selected.

The second pattern 32 has ten selected pixels located in the center. These selected pixels are spaced apart to optimize the information content of the partial image. Other selection patterns are also possible, including random patterns. For example, the number of selected pixels of pattern 31 may be reduced by omitting every other pixel, thus creating a space between the selected pixels. Advantageously, pattern 31 may be split up into two complimentary patterns of spaced selected pixels. Alternatively, or additionally, diagonal patterns may be used. It is preferred that the pixels of a selection pattern are approximately evenly distributed over the image, although this is not essential.

It is noted that the patterns 31 and 32 have no overlap. Although the absence of overlap increases the efficiency of the present invention, this is not essential and it is possible for patterns to share one or more pixels.

The patterns 31 and 32, and any other patterns (such as pattern 33), may be used consecutively. For example, pattern 31 may be used first to produce a first partial image PI₁. If this partial image does not allow a satisfactory reconstruction of the original image, pattern 32 is used to produce a second partial image PI₂. If the combination of partial images PI₁ and PI₂ allows a reconstruction having a sufficient quality, only forty-six out of one hundred pixel values have been transmitted, resulting in a data reduction of 54%. Of course the overall bit efficiency depends on both the original image and the selection patterns used, and additionally on the compression unit 13.

To ensure efficiency, each partial image contains at least one pixel that has not been selected by a previous partial image associated with the same original image. That is, each partial image adds at least one pixel to the accumulated (or combined) partial image. As mentioned above, it is preferred that all pixels of a partial image have not been selected before. In addition, it is preferred that the accumulated (or combined) partial image produced by the units 15 and 25 contains fewer pixels than the original image. Of course it would be possible for this combined partial image to have the same number of pixels as in the original image, but in that case no data reduction would have been achieved, other than any data reduction obtained by the compression unit 13.

The third pattern 33 also consists of pixels 30, some of which have been selected (pixels designated X) while the remaining pixels are discarded (blank pixels). This pattern has holes 39. These holes consist of discarded (that is, not selected) pixels which are at least partially surrounded by selected pixels. As can be seen in Fig. 3, the holes 39 may consist of several contiguous pixels, or of a single pixel. The holes 39 may extend to the edge of the pattern, or may be completely surrounded by selected pixels. By using pattern 33, effectively holes are made in the original image. Hole filling techniques are particularly suitable for reconstructing images which have been produced in this way. It is noted that pattern 31 can also be considered as a pattern defining a hole, as it has a central area consisting of discarded pixels, which are completely surrounded by selected pixels.

Two or more consecutive patterns displaying holes can be used to produce partial images of the same original image. It is preferred, however, that a pattern having holes, such as pattern 33 or 31, is followed by a pattern having scattered selected pixels, such as pattern 32. In this way, the amount of selected pixels can be kept at a minimum.

Each partial image could be transmitted or stored as a separate data packet. To indicate that no further partial images (pertaining to the same original image) are output, a "stop" data packet could be produced. Alternatively, a "stop" indication could be embedded in the last data packet containing a partial image. The partial images may also be transmitted or stored using a single data packet containing all partial images pertaining to a particular original image.

The merely exemplary image system 4 shown schematically in Fig. 4 comprises a decoding device 2 according to the present invention and an image display unit 41. The decoding device 2, which may be identical to the decoding device illustrated in Fig. 2, receives partial images PI and produces a reconstructed image RI using these partial images. This reconstructed image RI can be displayed by the image display unit 41, which includes a display screen. The image system 4 may be part of an image transmission system which further comprises an encoding device (1 in Fig. 1) and, optionally, an image source and a transmission channel. The image source may be a computer or other device having a suitable storage medium, such as a hard drive, a CD, a DVD, or a portable memory device. The transmission channel may include a cable (or cable network), an Internet connection, a wireless connection (using electromagnetic waves and/or infrared light), and/or a communication (e.g. telephone) network.

An image encoding method or procedure according to the present invention is schematically illustrated in Fig. 5. After start step 51, the procedure 5 commences with input step (IS) 52 in which an (original) image is received. A number of pixels is selected from this original image in selection step (SS) 53, the selected pixels constituting a partial image. This partial image is output (to be transmission or stored) in output step (OS) 54 and combined with any previous partial images in combination step (CS) 55. In reconstruction step 56, the combined or accumulated partial image is used to produce a reconstructed image. Hole filling techniques or any other suitable image reconstruction techniques may be used to produce a reconstructed image.

The quality of the reconstructed image is tested in test step (TS) 57. If the test result is positive (quality satisfactory), the procedure ends in stop step 59, else the procedure loops back to selection step (SS) 53 to select more pixels of the original image. It is noted that the steps of compressing and decompressing the partial images are not shown Fig. 5 as these steps are optional.

An image decoding method or procedure according to the present invention is illustrated in Fig. 6. After the start step 61, the procedure 6 commences with an input step (IS) 62 in which the one or more partial images are received. The partial images are combined in combination step (CS) 63, resulting in a combined (or accumulated) partial image which has fewer pixels than the original image.

It will be understood that combination step 63 may be omitted if only a single partial image is received in input step 62.

In reconstruction step (RS) 64, the combined partial image is used to produce a reconstructed image having the same number of pixels as the original image (and, typically, the same dimensions measured in pixels). This reconstructed image, which typically is a close approximation of the original image, is output in output step (OS) 65. The procedure ends with stop step 66. The present invention is based upon the insight that hole filling methods may advantageously be used for encoding images. The present invention benefits from the further insight that consecutive partial images may be used to convey just sufficient information to reconstruct the original image to a particular quality standard. It is noted that any terms used in this document should not be construed so as to limit the scope of the present invention. In particular, the words "comprise(s)" and "comprising" are not meant to exclude any elements not specifically stated. Single (circuit) elements may be substituted with multiple (circuit) elements or with their equivalents.

It will be understood by those skilled in the art that the present invention is not limited to the embodiments illustrated above and that many modifications and additions may be made without departing from the scope of the invention as defined in the appending claims.

Claims

CLAIMS:

1. A method of encoding an image represented by pixel values, the method comprising the steps of: producing a partial image by selecting a number of pixel values, producing a combined partial image using the current partial image and any preceding partial images, producing a reconstructed image using the combined partial image, determining a quality measure of the reconstructed image, repeating the above steps if the determined quality measure is smaller than a threshold value, wherein each partial image contains pixels that have not been previously selected, and wherein the combined partial image contains fewer pixels than the original image.

2. The method according to claim 1, wherein the partial images are non- overlapping.

3. The method according to claim 1, wherein the step of producing a reconstructed image involves a hole filling process.

4. The method according to claim 1, wherein the pixels values are selected according to at least one predetermined pattern (31, 32).

5. The method according to claim 1 , further comprising the step of outputting an indication if the determined quality measure is greater than the threshold value.

6. The method according to claim 1, further comprising the step of outputting each partial image.

7. The method according to claim 6, wherein the step of outputting the partial image involves compressing the partial image.

8. The method according to claim 1, wherein the step of producing a partial image involves making at least one hole in the original image.

9. A method of decoding an image encoded by the method according to claim 1 , the method comprising the steps of: receiving one or more partial images containing selected pixel values of a single original image, producing a combined partial image using the received partial images, and - producing a reconstructed image using the combined partial image.

10. The method according to claim 9, wherein the received partial images are compressed, further comprising the step of decompressing the compressed partial images.

11. A computer program product for carrying out the encoding method according to any of claims 1 - 8 and/or the decoding method according to any of claims 9 - 10.

12. An encoding device (1) for encoding an image represented by pixel values, the device comprising: - a selection unit (12) for selecting a number of pixel values to produce a partial image (PI), an accumulation unit (15) for producing a combined partial image using the current partial image and any preceding partial images, a reconstruction unit (16) for producing a reconstructed image (RI') using the combined partial image, a correspondence unit (17) for determining the correspondence of the reconstructed image and the original measure, wherein the selection unit (12) is arranged for selecting another number of pixel values to produce another partial image (PI) if said correspondence is insufficient, wherein each partial image contains pixels that have not been previously selected, and wherein the combined partial image contains fewer pixels than the original image.

13. The encoding device according to claim 12, further comprising a compression unit (13) for compressing a partial image.

14. The encoding device according to claim 13, further comprising a decompression unit (14) for decompressing a partial image prior to producing a combined partial image.

15. A decoding device (2) for decoding an image represented by partial images each containing a selected number of pixels of an original image, the device comprising: an accumulation unit (25) for producing a combined partial image using a current partial image and any preceding partial images, and - a reconstruction unit (26) for producing a reconstructed image using the combined partial image.

16. The device according to claim 15, further comprising a decompression unit (24) for decompressing any compressed partial images.

17. An image transmission system, comprising an encoding device (1) according to claim 12 and/or a decoding device (2) according to claim 15.