CA2335022A1

CA2335022A1 - Method and apparatus in transmission of images

Info

Publication number: CA2335022A1
Application number: CA002335022A
Authority: CA
Inventors: Magnus Jandel; Mathias Larsson; Charilaos Christopoulos
Original assignee: Individual
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 1998-06-18
Filing date: 1999-06-10
Publication date: 2000-01-06
Also published as: EP1106014A1; AU753304B2; CN1305684A; AU4812099A; SE9802193L; WO2000001153A1; SE521021C2; SE9802193D0; CN1135848C; KR20010052710A; JP2002519953A

Abstract

An image (3) in digitized form shall be transmitted over a channel between a transmitter and a receiver. The channel has a limited bandwidth and the image has a less important background (R1) and also regions of particular importance, i.e. regions of interest (R2, Rn). The image is transformed into transform coefficients and compressed (21), and a mask corresponding to the regions (R1, R2, Rn) is defined in the transform domain (22). The transform coefficients are classified (23) and assigned to different segments (SG1, SG2, SGn) in accordance with the mask definition. These segments (24) are coded independently of one another to different degrees of accuracy, depending on the importance of corresponding regions (R1, R2, Rn) in the image (3). Coding results in sub-bit streams (25) which are linked together (26) with the image header (271, 272) to form a bit stream (27), which is sent to the receiver.
The receiver decodes the image header and the segment information and reconstructs the mask in the transform domain, including shapes and positions of the regions (R1, R2, Rn). The image is then recreated with the aid of the mask to desired degrees of accuracy in respective regions. It is possible to define several regions (R2, Rn) with different degrees of image quality, and only those parts of the image that are of interest need be decoded.

Description

' WO 00/01153 PCT/SE99/01024 Method and Apparatus in Transmission of Images.
FIELD OF INVENTION
'i The present invention relates to a method and to arrangement for coding and extracting regions of interest (ROI) in the transmission of still images and video images. The method and the arrangement are particularly well suited for transform-1() based coders, such as wavelets and DCT.
DESCRIPTION OF THE BACKGROUND ART
In transmission of digitized still images from a transmitter lii to a receiver, the image is usually coded in order to reduce the amount of bits required for transmitting the image.
The bit quantity is usually reduced, because the capacity of the channel used i~: limited. A digitized image, however, 21) consists of a very large number of bits. When transmitting an image that consists of a very large number of bits over a channel which has limited bandwidth, transmission times will be unacceptably long for the majority of applications if it is necessary to transmit every bit of the image.

2 !5 Consequently, in recent years research has been directed to coding methods and techniques for digitized images with the object of reducing the number of bits necessary to transmit the images.

These methods can be divided into two groups:

Lossless methods, i.e. methods exploiting the redundancy in -the image in such manner as to enable the image to be reconstructed by the receiver without loss of information.
Lossy methods, i.e. methods that exploit the fact that not all bits are equally as important to the receiver. Hence, the image received is not identical to the original but looks sufficiently like the original image to the human eye, for instance.
In some applications, certain parts of the transmitted image -y are of more interest than the remainder of the image, and better visual quality of these parts of the image is therefore desired. Such a part is usually called the region of interest (ROI). Applications in which this can be useful include, for example, medical databases or the transmission of satellite images. In some cases, it is also desired, or necessary, to transmit the region of interest loss-free, while the quality of the remainder of the image is of less importance-. There are also occasions when it is required to extract the regions of interest from the bit stream and decode these regions of interest without needing to decode the image as a whole.
2~i Swedish Patent Applications SE 9703690-9 and SE 9800088-8 both describe how a mask can be calculated for delimiting such a region of interest (ROI).
SUMMARY OF THE INVENTION
' The present invention addresses the aforesaid problem of defining and transmitting regions of interest and background regions of mutually different qualities in the transmission of images.
The basic concept of the invention in solving the problem is to transform the image and to define in said transform a mask that corresponds to the regions of interest and to the background regions. The region definition and the image transform are transmitted to a receiver capable of recreating the image with the quality desired in the predetermined 1.0 regions .
More specifically, t:he solution involves dividing the image into the desired regions. The image is then transformed to some type of transform coefficients. A mask corresponding to J.5 the separate regions in the image is deffined in the transform domain and the coefficients classified and assigned to different segments in accordance with the mask definition.
The segments thus belong to the corresponding regions in the image. The segments and the coefficients are transmitted in a ~!0 compressed state to a receiver that is capable of reproducing regions in the image on the one hand and of reproducing the actual image on the other hand with the desired image quality in the various regions.
:'5 One advantage afforded by the invention is that several different regions of interest can be defined.
Another advantage- is that different regions can have several different degrees of image quality.
:30 PC'T/SE99/01024 Still another advantage is that only those parts of the image that are of vital interest to the user need be decoded, while avoiding decoding of the whole of the image.
Yet another advantage is that the segments can be coded independently of each other.
The invention will now be described in more detail with reference to preferred embodiments thereof and also with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is a block schematic illustrating an inventive arrangement.
Figure 2 is a flow chart illustrating part of an inventive method.
:?0 Figure 3 is a flow chart illustrating a further part of an inventive method.
Figure 4 is a diagram illustrating classification of transform coefficients.
~! 5 Figure 5 is a diagram for interlinking image segments in a bit stream.
Figure 6 is a view of an image with object.
Figure 7 is a graphic representation of the topology in Figure 6.

.___ DESCRIPTION OF PREFERRED EMBODIMENTS
Figure 1 is an overview of an arrangement for coding and transmitting images. An image 3 of an object is stored in 5 digital form in a digital camera 1, and the image presented on a screen 4. The screen is connected to a computer 2 which is programmed to divide the image 3 into objects or regions, of which a background region R1 and regions of interest R1 and Rn are shown. An image coder 5 in the computer z wavelet-7.0 transforms the image, while simultaneously compressing the image, and generates a compressed bit stream PS1. An operator at the image screen 4 defines the regions of interest R2 and Rn. The image coder includes means for creating a mask PS2 in accordance with the regions and defines separate parts, 1.5 segments, of the bit streams with respect to the corresponding regions R1, R2 and Rn, with the aid of said mask. The definition also enables the regions R1, R2, Rn in the form of said separate segments in the bit stream PS1 to be coded to different degrees of accuracy. A transmitter 6 ~!0 sends the bit stream, including the definition of the positions and shapes of the regions R2 and Rn to a receiver 7 which is connected to a computer that includes an image decoder 8. The decoder decodes the bit stream PS1 and reproduces the mask definition PS2 and presents the image on ~'.5 an image display screen 9. The accuracy of the background R1 is relatively poor, whereas each of the regions R2 and Rn has respectively a higher degree of accuracy.
The following definitions are given in order to assist in :30 describing the inventive method:

- A segment is defined here as all of the coefficients in the transform domain that belong to a given object or the background in the image. The segment can then be divided further into subsets.
- A subset is defined here as a number of coefficients in a part of the transform domain (e.g. a subband in the case of the wavelet transform) which is required for the reconstruction and which belongs to a segment in the :LO digitized image, see Figure 4.
As before mentioned, the coefficients are classified and can be assigned to individual segments. When this classification is made, the segments are coded independently of one another 7.5 to different levels of accuracy, which yields a bit stream for each segment. These segments are then joined together.
The inventive encoding method will be described with reference to Figure 2. The digitized image 3 to be 20 transmitted presents the background R1 and the regions of interest R2 and Rn. The following procedural steps are carried out:
1. Perform a transformation of the image 3 according to 25 step 21. In the illustrated case, this transformation is performed with a wavelet transform or with a discrete cosine transform (DCT).
2. Create a mask according to step 2 with the aid of 30 information as to how the digitized image 3 shall be divided into the background R1 and the objects R2 and Rn. The techniques described in Swedish Patent Applications SE

9703690-9 and SE 9800088-8 can be used to this end. The mask _ is created in the transform domain and describes which coefficients are required to reconstruct the different objects or the background. Different segments SG1, SG2 and SGn correspond to the background R1 and the objects R2 and Rn.
3. Use the mask to classify the transform coefficients as belonging to the different segments SG1, SG2, SGn, according 7.0 to step 3.
4. Code the segments independently of one another, according to step 24. This gives the number of bits needed for each subset.
J. 5 5. Concatenate the subset streams together with necessary substream information and header information, according to step 26. This requires a bit stream description, given below.
:?0 6. Send the concatenated bit streams 27. This includes shape data 271, bit stream information 272, subband 0 referenced 273 and subband 1 referenced 274.
The method enables the receiver to have immediate access to 25 any parts of the image when so desired, as shown in Figure 3.
This is possible because the information as to where different parts are found in the bit stream is known.
One method of how the decoder may work is described below :30 with reference to Figure 3.

1. Receive the bit stream 27 and decode the header information required, according to step 31.
2. Find and decode the required segment information, step 32.
3. Create a mask in the transform domain, for instance with the aid of the technique described in said Patent Applications SE 9703690-9 and SE 9800088-8; step 33. The mask describes those coefficients that are required to reconstruct the desired objects or background.
4. Decode requisite segment data from the bit stream; step 34.
5. Reconstruct the requisite segments; step 35.
6. Decode and show the image; step 36.

~_...f A description will now be given of those components in the bit stream 27 that are required when applying the described technique.
c: 5 Data structures and pointers 30 A pointer is a set of symbols that defines the position of a bit or a byte in a bit stream or a file. Many ways of defining a pointer have been defined in computer science. Any WO 00/01153 PC'T/SE99/01024 one of these methods can be used here. A pointer can be defined implicitly by a specific bit stream composition rule.
A pointer can be defined relative to an explicitly or implicitly determined position. A simple way of defining a pointer is to determine the number of bits between the requested position and a known reference point, such as the first bit in the bit stream, for instance.
Topolocry descriptor The topology descriptor, TOP, is a set of symbols that defines the topological relationship between numbered objects and shapes. This is illustrated in Figure 6, in which four objects O1, 02, 03, 04 and four shapes S1, S2, S3 and S4 are shown. The topology of the image can be represented, e.g., as a tree graph as shown in Figure 7. The nodes and the edges of the tree graph can be coded in a data structure using well known methods. P TOP is a pointer to a topology descriptor.
Shape descriptor A shape descriptor, Si, defines the appearance of a closed boundary line of an object. The shape number, i, is given by a topology descriptor. Many different shape coding techniques can be used. Examples of such methods are chain coding and shape coding methods in MPEG-4. Shape descriptors can be decoded independently of one another once their respective positions in the bit stream is known. P-Si is a pointer to a shape descriptor.

A segment descriptor, Ti, is a compressed set of symbols that encode a segment as described above. The segment includes an 5 ordered set of subsets. The object number, i, is given by a topology descriptor. p_Ti is a pointer to a segment descriptor.
Subset d -~~r; r~t-~r G
_ ~e 7. 0 A subset descriptor, Bid, is an independently decodable subset, j, of a segment descriptor, Ti, which describes, e.g., the coefficients that belong to a given subband, j, as described above. p'B,L~ is a pointer to a subset descriptor.
M-ultlTJIPXP~ SPLfTIIPn+- ~pcnri r,i-r, a. i Several segment descriptors, {Ti, T~, Tx ~ ~ ~ }, can be multiplexed into a common data structure MT(.i,j,k). This is 2~D done normally for the purpose of simultaneous progressive transmission of a set of objects. The data structure, MT, is called a multiplexed segment descriptor. Several multiplexing methods can be used. p'MT is a pointer to a multiplexed segment descriptor. ....
Examples of multiplexing methods are shown in Figure 5. A
simple method is to interleave subsets 52 belonging to the component segments so that:
MT(i,j,k)={Bio. Boo. Bxm Bii. 8~1. Bxi. Bi2. Bj2~ Bxz~..

WO 00/01153 PC'T/SE99/O1024 _In this case, the order of the symbols corresponds to the order in the bit stream 51, with symbols on the left being sent first. Subsets in a multiplexed stream may be excluded !i if they are known by the decoder.
Bit stream storage format In order to obtain immediate access to any object whatsoever 1() in the image, the stored bit stream or file structureshould preferably include at least the following components:
In the image header, if required:
lei Topology descriptor TOP
Pointers to shape descriptors fp_S1, p-Sz...p-SN
Pointers to segment descriptors ~p To, p T1, . . .p~TN}
2 () Optional pointers to subset descriptors: for each k ~~ r NJ r {P Bk0 r ~ Bkl r . . . p BkNI
In the actual stored bit stream if needed:
2 'i Shape descriptors ~S1, SZ, . . .SN}
Segment descriptors f To, T1, . . . TN~
3(> A group of segment descriptors with index {k,l,m...} can optionally be replaced with a multiplexed segment descriptor MT(k,l,m...) _N is the number of stored objects. The background is the object with index O..
PROGRESSIVE TRANSMISSION WITH IMMEDIATE ACCESS TO OPTIONAL
OBJECTS
A server receives a request for sending image data to a client. The image i_s stored with the server in the format ~.: t :10 described in the preceding passage. Part of the stored data structures (topological data, shapes, segments and subsets) 'y may have already been sent to the receiving terminal. This section of the description describes a procedure for composing a bit stream with the server that handles the 7.5 request .
Example ReQUest from a e~r A simple request contains the following information:
Send objects with numbers k, 1, m ... with a respective accuracy of nk, nl, nt~ where the accuracy is the index for the highest subset that is sent for each index.
Several primitive requests may be sent. They will be served in the order in which they are received or in an otherwise specified order.

Send topological information if needed. TOP is sent in response to a first request for image information.
Send all shape descriptors that are necessary to describe the boundaries of the objects requested. It is not necessary to send shape descriptors that are already known to the decoder.
When using the topological tree structure in Figure 7, it is J.0 found that not all shape descriptors on the same branch as the object or on the same or lower hierarchical level need be sent. The server knows the state of the decoder and will send solely those shape descriptors that are unknown to the decoder.
J. 5 Send (multiplexed) subset descriptors that describe the objects requested to the defined accuracy. Subset descriptors that are already known to the decoder need not be sent. For instance, the user is aware of the subsets f Bko ~ Bxi r Bk2 ~ Bk3 ~!0 belonging _to segment k. Subset descriptors {BkS~ Bk6. Bx~} must be sent when object k is requested to accuracy 7.
EXAMPLES
25 In this section of the description, examples are given with respect to situations in which the proposed method can be applied.
Assume, according to Figure 5, that in the centre of the 30 image R51 there is an encircled region R52 whose quality must be better than the quality of the region R53 outside the circle, this latter region being referred to hereinafter as the background. However, both the background R53 and the region R52 shall be transmitted simultaneously. The following then takes place:
1. The original image is transformed with a wavelet transform.
2. A mask is then created in the transform domain. This mask describes the coefficients that are required in the transform domain in order to reconstruct the region R52 and the background R53. The created mask is then used to classify the coefficients in the transform domain in two segments, one segment for the region and one segment for the background.
The two segments are built up by a number of subsets. In the :L5 illustrated case, the number of subsets is the same as the number of subbands in the transform domain. The situation on hand is thus:
2.1 In respect of the region segment belonging to the region l 1r0,1 r r0, 2 r ~ ~ - r rp, l } r ~ ~ ~ {rno subbands, l r rno_subbands, 2 r ~ ~ ~ rno subbands, j ~~
where i,j are the number of coefficients in the different subsets.
2.2 In respect of the background segment belonging to the background R53:
b~.P~ r ' ~ ' r lbno subbands, 1 r bno subbands,2 r ~ ~ ~ bno-subbands,qJ~
where p,q are the number of coefficients in the different subsets.

WO 00/01153 PC'T/SE99/01024 3. The two subsets are then coded as follows:
3.1 In respect of the region segment:
5 A shape descriptor Tr={Br,O,Br,li ~ ~ ~ rBr,no subbands! arid a set Of subset pointers {p-Br,o, p-Br, l ~ ~ ~ ~ i P Br,no subbands!
3.2 In respect of the background segment:

10 A segment descriptor Tb={Bb, 0, Bb,1. ~ - ~ ~ Bbi no subbandsl and a set of subset pointers {p Bb,O,Bb,l~ ~ ~ ~ rp Bb,no subbandsJ' 4. The two segments are then combined into a single bit stream, bit stream 51, in the following manner:
<image header><TOP><Sr><{p Bb, 0, p-Br, 0 i p Bb, l ~ p Bb,no subbands ~
p Br, no subbands}~<MT ( b, r ) ={Bb, 0 , Br, 0 i Bb, 1 r Br, 1 i ~ ~ ~ r Bb, no subbands ~
Br,no subbandsl>
In this case, the subsets are combined in the manner shown in the upper part of Figure 5, with the sub-bit streams 52 of the region being taken alternately with the sub-bit streams of the background. It will be noted that the TOP field is not required when the receiver is aware of the order in which the various parts of the image are set. The first part of the array, from <image header> to ...p 8...}> is, in other words, a definition of where the different image regions are placed in the remainder_ of the compressed bit stream <MT(b,r)={...B...~>.
5. The combined bit stream is then sent to the receiver.

PC'T/SE99/01024 The following takes place on the decoder side:
6. The image header together with the topology, shape information and pointers are read.
7. The decoder is now able to create the same mask as that described above.
8. The decoder creates the segments with the underlying subsets.
J
9. The decoder commences with decoding the combined bit stream and filling in the transmitted transform coefficients :15 in the corresponding subsets.
10. An inverse transform is used.

11. The image is transmitted and reconstructed.
a! 0 The aforedescribed :is one way of using the proposed method. f Other methods may be to combine (mix) the bit streams in another way. For instance, as shown in the bottom part of Figure 5, the region R52 may be transmitted first, followed 25 by the background R53. Another example is one in which more than one region is found, as described with reference to Figure 6, wherewith these.regions are combined in a number of different ways.
30 In addition to the earlier mentioned advantages, the proposed method has the added advantage of enabling shape information to be sent only when needed.

Claims

17

1. A method of transmitting an image (3) between a transmitter (2, 5, 6) and a receiver (?, 8), comprising the steps of:
- dividing the image (3) into at least two image regions (R1, R2, Rn):
- coding the image regions (R1, R2, Rn) into a coded symbol stream (21), said coding utilising a symbolic representation and having predetermined accuracy levels in said image regions: and - compressing the coded symbol stream into a compressed bit stream (PS1, 27), characterised in that the method includes the further steps of:
- generating (22) a definition (PS2) of an outer boundary line (S i)of at least one of the image regions (R2,Rn);
- transmitting said definition (PS2) to the receiver (7);
- transmitting the compressed bit stream (PS1, 27) to the receiver (7, 8; and - decoding (33, 34) in the receiver with the aid of said definition.

2. The method of claim 1, characterised in that two different of the image regions (R2,Rn) are coded to have said predetermined accuracy levels independently of each other.

3. A method of transmitting an image (3) between a transmitter (2, 5, 6) and a receiver (7, 8), comprising the steps of:
- dividing the image (3) into at least two image regions (R1, R2, Rn):
- coding the image regions (R1, R2, Rn) into a coded symbol stream (21), said coding utilising a symbolic representation and having predetermined accuracy levels in said image regions; and - compressing the coded symbol stream into a compressed bit stream (PS1, 27), characterised in that the method includes the further steps of:
- generating (22) a definition (PS2) of a mask (PS2) for at least one of the image regions (R2, Rn), two different of the image regions (R2, Rn) being encoded to have said predetermined accuracy levels independently of each other;
- transmitting said definition (PS2) to the receiver (7);
- transmitting the compressed bit stream (PS1, 27) to the receiver (7, 8; and - decoding (33, 34) in the receiver with the aid of said definition.

4. The method of claim 1,2 or 3, characterised in that only predetermined parts of the compressed bit stream (PS1,27) are decoded.

5. The method of any of the claims 1, 2, 3 or 4, characterised by generating a topology description, defining the topological relationship between objects (O1, O2, O3, O4) and shapes (S1, S2, S3, S4) in the image.

6. The method of any of the claims 1, 2, 3 or 9, characterised by generating a shape description, defining the appearance of the closed boundary line (Si) of an object (O1, O2, O3, O4) in the image.

7. The method of any of the claims 1, 2, 3 or 4, characterised by generating a segment description, defining which transform coefficients that belong to respective segment.

8. The method of claim 7, characterised by generating a subset description, defining which transform coefficients that belong to an independently decodable part of a segment.

9. The method of any of the claims 5, 6, 7 or 8, characterised by generating of a pointer, defining a position in the bit stream (27) for the respective one of the above mentioned descriptions.

10. An arrangement for transmitting an image (3), comprising:
- a transmitter (2, 5, 6) and a receiver (7, 8);
- means (4, 5) for dividing the image (3) into at least two image regions (R1, R2, Rn);
- a coding device (5) for coding the image regions (R1, R2, Rn) into a coded symbol stream, said coding device utilising a symbolic representation and having predetermined accuracy levels in said regions;
- a compressing device for compressing the coded symbol stream into a compressed bit stream (PS1, 27); and - means in the transmitter (2, 5, 6) for transmitting said compressed bit stream (PS1, 27) to the receiver (7, 8), characterised in that the arrangement also includes:
- means (5) for generating (22) a definition (PS2) of an outer boundary line (S i) of at least one of the image regions (R2, Rn);
- means in the transmitter (2, 5, 6) for transmitting said definition (PS2) to the receiver (7, 8); and - a decoder (8) in the receiver for decoding (34, 35) of the compressed bit stream (PS1, 27) with the aid of said definition (PS2).

11. The arrangement of claim 10, characterised in that the coding device is arranged to encode (29) two different of the image regions (R2, Rn) to have the predetermined accurracy levels independently of each other.

12. An arrangement for transmitting an image (3), comprising:
- a transmitter (2, 5, 6) and a receiver (7, 8);
- means (4, 5) for dividing the image (3) into at least two image regions (R1, R2, Rn);
- a coding device (5) for coding the image regions (R1, R2, Rn) into a coded symbol stream, said coding device utilising a symbolic representation and having predetermined accuracy levels in said regions;
- a compressing device for compressing the coded symbol stream~into a compressed bit stream (PS1, 27); and - means in the transmitter (2, 5, 6) for transmitting said compressed bit stream (PS1, 27) to the receiver (7, 8), characterised in that the arrangement also includes:
- means (5) for generating (22) a definition (PS2) of a mask (PS2) for at least one of the image regions (R2,Rn), the coding device (5) being arranged to encode (24) two different of the image regions (R2,Rn) to have said predetermined accuracy levels independently of each other;
- means in the transmitter (2, 5, 6) for transmitting said definition (PS2) to the receiver (7, 8); and - a decoder (8) in the receiver for decoding (34, 35) of the compressed bit stream (PS1, 27) with the aid of said definition (PS2).

13. The arrangement of claim 10, 11 or 12, characterised in that the decoder (8) is arranged to decode only predetermined parts of the compressed bit stream (PS1,27).

14. The arrangement of claim 10, 11, 12 or 13, characterised in that the transmitter (2, 5, 6) has means for generating a topology description, defining the topological relationship between objects (O1, O2, O3, O4 ) and shapes (S1, S2, S3, S4) in the image.

15. The arrangement of claim 10, 11, 12 or 13, characterised in that the transmitter (2, 5, 6) has means for generating a shape description, defining the appearance of the closed boundary line (Si) of an object (O1, O2, O3, O4) in the image.

16. The arrangement of claim 10, 11, 12 or 13, characterised in that the transmitter (2, 5, 6) has means for generating a segment description, defining which transform coefficients that belong to respective segment.

17. The arrangement of claim 16, characterised in that the transmitter (2, 5, 6) has means for generating a subset description, defining which transform coefficients that belong to an independently decodable part of a segment.

18. The arrangement of claim 14, 15, 16 or 17, characterised in that the transmitter (2, 5, 6) has means for generating a pointer, defining a position in the bit stream (27) for the respective one of the above mentioned descriptions.