AU721187B2 - Image transmission - Google Patents

Description

WO 98/19273 PCT/AU97/00724 1 "IMAGE TRANSMISSION" Technical Field This invention relates to a system for and method of image transmission.

The invention has particular application to progressive image transmission systems.

Background of Invention Progressive image transmission systems are known and involve the transmission of image data in a way that the data received at the intermediate stages in the transmission can be used to reconstruct an approximation to the full image. In practice this is most efficiently achieved with an embedded image representation wherein an exact or nearly exact representation of the image is organised in such a way that any truncation of the bitstream resulting from the encoding can be used to reconstruct an approximation to the image which minimises the distortion for a given codelength.

Progressive image transmission has the advantage that the user can quite quickly see an approximation to the full image and can make decisions on that basis to terminate or react in some other way to the transmission at the earliest possible point.

In order to take advantage of the redundancy associated with local pixel correlation in most images, and for reasons of computational efficiency, image compression techniques typically rely on some form of spatial tiling. In the case of the JPEG standard this is an explicit tiling of the image into 8x8 pixel blocks as seen in FIG 1 which illustrates a standard JPEG image tiling. The assumption that the underlying image representation divides the image into spatial tiles is explicit in the case of JPEG or implicit in the case of most implementations of the discrete wavelet transform

(DWT).

It is not entirely obvious that the wavelet 1 WO 98/19273 PCT/AU97/00724 2 transform divides the image into spatial tiles and it is only possible to view the DWT in this way because in general, it is only useful from the point of view of compression to hierarchically decompose the image on the wavelet basis to the point that the scaling function (DC) components are spatially decorrelated. For most images a depth of 3-5 levels in the hierarchy is sufficient to achieve this. The coefficients in the wavelet domain can then be collected into groups of components for which the centres of the corresponding basis function lie within a given spatial region of the image.

As can be seen in FIG 2 which illustrates a Malletstyle hierarchical wavelet decomposition with the components which constitute a spatial tile, the collection of coefficients in the wavelet domain can be thought of as a spatial tile with one DC component (at the top left of FIG and an hierarchy of AC components in a scale hierarchy mimicking the structure of the image subbands. By reconstructing the components in each spatial tile it is possible to arrive at a set of orthogonal but overlapping image partitions which can be added together to reform the image.

FIG 3 illustrates a tiled wavelet coding in which the components of the standard Mallet style hierarchy are re-ordered to form spatial tiles whose arrangement follows that of the associated DC component in the standard Mallet hierarchy.

Each tile in a tiled representation of an image is represented as a stream of bits. One known manner of encoding a tile into a bit stream is that the resulting representation is embedded. An embedded coding is one in which the bits representing the image have been structured in a single stream which can be truncated at any point so that an approximation to the image can be generated from the information to that point and such that approximation has close to optimal distortion for the proportion of the information received.

Embedded encodings are well known. The simplest PCT/AU97/00724 Received 11 January 1999 3 embedded encoding re-orders the bit planes so that the most significant bits are sent first. Simple run length encoding is then able to achieve some compression while retaining the embedded character of the representation.

Such methods do not result in great compression.

More successful examples of embedded encodings are the Embedded Zerotree Wavelet (EZW) coding of Shapiro and the related Spatial Partitioning In Hierarchical Trees (SPIHT) encoding of Said and Pearlman. See the following:- A.Said, W.A.Pearlman, "A New Fast and Efficient Image Codec Based on Set Partitioning in Hierarchical Trees," Transactions on Circuits and Systems for Video Technology vol. JUN pp 243-250 (1996); J.M.Shapiro, "Embedded image coding using zerotrees of wavelet coefficients," IEEE, Trans. on SP, 41 (1993), pp 3445-3462; J.M.Shapiro, "An Embedded Hierarchical Image Coder using Zerotrees of Wavelet Coefficients," Proc. Data Compression Conference, J.Storer, M.Cohn Eds (1992), pp 214-223, and US Patents 5412741, 5321776 and 5315670 to Shapiro.

Summary of Invention The present invention aims to provide an alternative to known progressive image transmission systems and methods.

This invention in one aspect resides broadly in a method of progressively transmitting an image in which image compression techniques rely on spatial tiling of the image, the method including:allocating variable priority values to spatial regions within the image whereby a receiver of a transmitted image can interactively redefine the spatial prioritisation of the transmitted image data during transmission thereof to thereby control the order in 4A1 which the content of the transmitted image data is progressively received.

AMENDED SHEET IPEA/AU PCT/AU97/00724 Received 11 January 1999 4 It is preferred that variable priority values are allocated to independent spatially localised tiles within the image by encoding an embedded representation such that truncations of the representations for each tile can be used to generate valid approximations to the tiles.

In a preferred embodiment the compression techniques are based on embedded zero tree coding systems.

The method may further include:separating the bitstreams associated with each tile and weighting each bit.

In a preferred embodiment the method further includes:establishing an algorithm at each end of the transmission to determine the ordering of the information from the tiles.

Suitably the bits are weighted by assigning a priority index for each tile, the priority index comprising a first unsigned integer having a sufficient.

number of bits to facilitate fine increments of priority.

It is preferred that the priority index for each tile is an integer within a predetermined range and that the method further includes:transmitting a number of bits from the tile stream which equals the priority index.

The bits in the bitstring representation of each tile are preferably weighted in accordance with the length of each bitstring.

The method may also include:assigning a priority accumulator to each tile, the priority accumulator comprising a second unsigned integer value; scanning the tiles in a preset order and at each pass thereof incrementing the priority accumulator of each tile by the priority index, and transmitting a quantity of information each time the accumulator overflows.

The method of the present invention is suitable for Z implementation by appropriate software or purpose built AMENDED SHEET IPEA/AU PCT/AU97/00724 Received I January 1999 hardware.

Accordingly in another aspect this invention resides broadly in an image transmissiom system for progressively transmitting an image in which image compression techniques rely on spatial tiling of the image, the system including:prioritising means for allocating variable priority values to spatial regions within the image whereby a receiver of a transmitted image can interactively redefine the spatial prioritisation of the transmitted image data during transmission thereof to thereby control the order in which the content of the transmitted image data is progressively received.

In a preferred embodiment the system further includes:separating means for separating the bitstreams associated with each tile, and weighting means for weighting each bit.

The system may also include:means for establishing an algorithm at each end of the transmission to determine the ordering of the information from the tiles.

It is also preferred that the system includes:assigning means for assigning a priority accumulator to each tile, the priority accumulator comprising a second unsigned integer value, scanning means for scanning the tiles in a preset order and at each pass thereof incrementing the priority accumulator of each tile by the priority index, and transmitting means for transmitting a quantity of information each time the accumulator overflows.

Description of Drawings In order that this invention may be more easily understood and put into practical effect, reference will now be made to the accompanying drawings which illustrate a preferred embodiment of the invention, wherein:- AMENDED SHEET IPEA/AU PCT/AU97/00724 Received 11 January 1999 FIGS 1 to 3 illustrate known tiled image representation techniques as described above; FIG 4 illustrates the bitstrings resulting from the embedded coding of each spatial tile, and FIG 5 illustrates the priority variables in accordance with the present invention from a series of AMENDED SHEET IPEA/AU WO 98/19273 PCT/AU97/00724 6 scans of one tile.

Description of Preferred Embodiment of Invention The present invention utilises a transformation wherein the transformed representation of the image can be considered to be organised as spatial tiles. Although in the most general case the tiles could be single pixels or single coefficients of some transformation, this does not lead to efficient implementation and it is preferred that the nature of the spatial tiling implies no interdependence between the spatial tiles in the decoding. The encoding of the tile into a bitstream is also preferably done in such a way that the resulting representation is embedded.

The present invention generalises the concept of embedded encoding to some extent in that the method of the present invention embeds the representation of each tile in the usual sense. Thus if the priority given to each tile were to be equal and the bitstreams from each tile were to be interleaved so that bits of equal significance appear at similar points in the stream, then the resulting representation of the full image can be regarded as having been embedded in the usual sense.

However as will subsequently be described, in the present invention the separation of the streams associated with each tile facilities the incorporation of spatial weighting into the determination of the significance of each bit in the stream. This is referred to as spatial prioritisation.

In applications where some spatial regions of the image have priority over other spatial regions, bits from some blocks will be less important than bits from other blocks. The system of the present invention is implemented without significant bitstream overheads and tile index information is not transmitted. In order to facilitate the client/receiver being able to ascertain to which tile each incoming bit belongs, a simple algorithm is established at each end of the transmission to

T

WO 98/19273 PCT/AU97/00724 7 determine the ordering of the information from the tiles.

At the server end, the encoded image has the form of an array of bit strings, one associated with each spatial tile. The server scans through this array and at each tile it must decide how many bits to send for this scan for each tile. A number of options are available and require that when the server decides to send a given number of bits from a given tile string, the client/receiver can decide to receive the same number of bits and put them in the same tile.

For most purposes it is sufficiently general for the priority index for each tile to have the form of an unsigned integer with a sufficient number of bits to allow adequately fine increments of priority. The simplest arrangement is thus to use an integer in some range such as 0 32 as the tile priority and send a number of bits from the tile stream which equals the priority index.

FIG 4 illustrates the bitstrings formed by the embedded coding of each spatial tile.

Such an approach has a number of advantages.

Firstly the length of the string associated with each tile is related to the amount of information on the tile such that the longer the string the more information and vice versa. With a simple priority coding scheme, a situation can arise where two tiles of quite differing complexity are given the same priority and their bits will be transmitted in such a way that the less significant bits from the tile with the shorter bit string will be sent before more significant bits from the tile with the longer bitstring. This situation can be partially corrected by weighting the spatial priorities by the length of each bitstring. The weighting need not be exact and can be compressed thereby ensuring that full transmission of the image finds each tile completing at approximately the same time.

With many images however, the density of information for each tile will vary from bitplane to bitplane with WO 98/19273 PCT/AU97/00724 8 some tiles having no information in the upper bitplanes but a lot of low amplitude information, thereby producing a lot of bits for the final bitplanes. For the zero tree codings most tiles eventually have all their bits as refinement bits and the number of bits per tile will be the same for each tile in the lower bitplanes. This means that the bitplanes will not remain in step even with this weighting.

It is advantageous for the decoder to know when a bitplane is complete. If the unit of information transmitted from the server is a single tile bitplane, then equal spatial priority from all tiles will ensure that the bitplanes from all tiles will be delivered in step. This has the disadvantage that the embedding within the bitplane for each tile is no longer relevant.

This also means that the priority of each bitplane can be adjusted as part of the process. It is possible to breakdown each tile bitplane into smaller units however this complicates implementation and can lead to greater memory requirements in server and client.

The subdivision of the bitstring for the tile into sequential substrings corresponding to individual tile bitplanes (or smaller units) offers the possibility of finer control over the prioritisation as a function if bitplane depth while maintaining the strict ordering of the bitplanes within the tile.

It is also possible to depart from a simple integer scheme as described above for prioritising the transmission. However it is desirable to send no more than one bitplane from each tile to avoid sending less significant bits before more significant bits and thereby further eroding the embedded character of the transmission. Prioritisation must then be achieved by having a probability of less than 1 of sending the bitplane.

It is possible to achieve this with a pseudo random number generator at each end of the transmission with the same seed so that they produce the same set of random WO 98/19273 PCT/AU97/00724 9 probability which can then be weighted to make the decisions. This however can produce variable behaviour.

In the present invention the preferred approach is deterministic. As illustrated in FIG 5 which illustrates the priority variables from a series of scans of one tile, this is achieved by assigning another unsigned integer value to each tile which integer value is termed the priority accumulator.

The tiles are scanned in a preset order and at each pass the priority accumulator of each tile is incremented by the priority index. If the priority accumulator plus the priority index is greater than the priority limit, than an overflow of the accumulator will result.

The server then sends the next chunk of data and the accumulator loses the overflow bit. Careful choice of the range of the priority counter and the size of the priority index result in regular updates for each tile with a controlled probability.

A sequence of scans on a single tile is illustrated in FIG 5. At the end of each scan pass, the client/receiver and server communicate any changes in the priority step of the tiles. The priority accumulator is not modified. If the priority is a simple function of the location of the tile then this does not involve a large amount of data being communicated, thereby allowing the client/receiver to interactively control the transmission when a duplex line is available.

In use the method in accordance with the present invention can have numerous applications. For example a user with a low band-with access to the InterNet such as a satellite or digital mobile (GSM) connection ion the field may need to view images available on the web. The user dials in and runs Netscape, connects to the homepage holding the images which could be an image database and clicks on to view some images. The user may looking for images with certain characteristics but is unaware in advance which images are suitable, but will recognise them as such when seen.

WO 98/19273 PCT/AU97/00724 Standard image transfer within Netscape can be very slow via modem. However when transmitted in accordance with the present invention, a fuzzy version of the image quickly appears on the user's screen and quite soon it is possible to recognise the basic features of the image.

If one of these is of interest, the user clicks on to the portion of interest. The image continues to be downloaded but the image gets rapidly clearer in the nominated area and remains fuzzy elsewhere. After only a fraction of the image has been transmitted, the user may have seen all that is necessary and a decision can be made as to whether the image is required.

It will be appreciated that the system and method in accordance with the present invention have a number of advantages over known progressive image transmission systems and methods. These include:- An ability for a user to interactively prioritise regions of the image being transmitted.

A user can select a point or region in the image and have the server send data preferentially from that region.

The ability to provide interactive prioritisation while retaining the embedded nature of the underlying image representation and without significant compromising of compression.

The present invention allows an arbitrary image to be transmitted across a network such that at any point in the transmission the client/user/receiver requesting the image can prioritise regions of the image so that the information transmitted will preferentially enhance the image in a manner which varies across the image smoothly.

After transmission of as little as 1-2% of an image, the user has enough information to identify regions of potential interest. The user can then click on that area and define a smooth priority map which can be communicated to the server such that the image will appear to resolve smoothly and WO 98/19273 PCT/AU97/00724 11 progressively around the selected region.

The user can redefine the priority without the server having to reformat the image representation at the transmission end and without having to resend any information.

It will of course be realised that whilst the above has been given by way of an illustrative example of this invention, all such and other modifications and variations hereto, as would be apparent to persons skilled in the art, are deemed to fall within the broad scope and ambit of this invention as is herein set forth.

EDITORIAL NOTE NO: 46938/97 THIS SPECIFICATION DOES NOT CONTAIN A PAGE 12.

Claims (13)

1. 2. A method as claimed in claim 1, wherein variable priority values are allocated to independent spatially localised tiles within said image by encoding an embedded representation such that truncations of said representations for each tile can be used to generate- valid approximations to said tiles.
2. 3. A method as claimed in claim 2, wherein said compression techniques are based on embedded zerotree coding systems.
3. 4. A method as claimed in claim 2, and further including:- separating the bitstreams associated with each tile and weighting each bit.
4. 5. A method as claimed in claim 4, and further including:- establishing an algorithm at each end of the transmission to determine the ordering of the information from the tiles.
5. 6. A method as claimed in claim 4, wherein said bits are weighted by assigning a priority index for each tile, aid priority index comprising a first unsigned integer AMENDED SHEET IPEA/AU PCT/AU97/00724 Received 11 January1999 14 having a sufficient number of bits to facilitate fine increments of priority.
6. 7. A method as claimed in claim 6, wherein said priority index for each is an integer within a predetermined range; said method further including:- transmitting a number of bits from the tile stream which equals the priority index.
7. 8. A method as claimed in claim 4, wherein the bits in the bitstring representation of each tile are weighted in accordance with the length of each bitstring.
8. 9. A method as claimed in claim 6, and further including:- assigning a priority accumulator to each tile, said priority accumulator comprising a second unsigned integer value.
9. 10. A method as claimed in claim 9, and further including:- scanning said tiles in a preset order and at each pass thereof incrementing said priority accumulator of each tile by said priority index.
10. 11. A method as claimed in claim 10, and further including:- transmitting a quantity of information each time the accumulator overflows.
11. 12. An image transmissiom system for progressively transmitting an image in which image compression techniques rely on spatial tiling of the image, said system including:- prioritising means for allocating variable priority values to spatial regions within said image whereby a jreceiver of a transmitted image can interactively redefine the spatial prioritisation of the transmitted redefineAMENDED SHEET-EAU AMENDED SHEET LPEA/AU PCT/AU97/00724 Received 11 January 1999 image data during transmission thereof to thereby control the-order in which the content of the transmitted image data-is progressively received.
12. 13. A system as claimed in claim 12, and further including:- separating means for separating the bitstreams associated with each tile, and weighting means for weighting each bit.
13. 14. A system as claimed in claim 13, and further including:- means for establishing an algorithm at each end of the transmission to determine the ordering of the information from the tiles. A system as claimed in claim 14, and further including:- assigning means for assigning a priority accumulator to each tile, said priority accumulator comprising a second unsigned integer value, scanning means for scanning said tiles in a preset order and at each pass thereof incrementing said priority accumulator of each tile by said priority index, and transmitting means for transmitting a quantity of information each time the accumulator overflows. AMENDED SHEET IPEA/AU