AU773574B2 - Method and apparatus for certain orthogonal transformations of compressed digital images - Google Patents

Description

S&FRef: 562611

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Actual Inventor(s): Address for Service: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan Tomasz Thomas Prokop Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Method and Apparatus for Certain Orthogonal Transformations of Compressed Digital Images Invention Title: ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PQ9215 [32] Application Date 04 Aug 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:aid N' L ~i~3 2I1~3~1 5815c -1- METHOD AND APPARATUS FOR CERTAIN ORTHOGONAL TRANSFORMATIONS OF COMPRESSED DIGITAL IMAGES Copyright Notice This patent specification contains material that is subject to copyright protection.

The copyright owner has no objection to the reproduction of this patent specification or related materials from associated patent office files for the purposes of review, but otherwise reserves all copyright whatsoever.

Technical Field of the Invention The current invention relates to orthogonal transformations performed during decompression of tiled images, and in particular to fast rotations and horizontal or vertical flipping of tiled compressed images. The present invention relates to a method and apparatus for orthogonal transformation of compressed digital images during decompression of the images. The invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for orthogonal transformation of compressed digital images.

Background Art .:i Fig. 1 shows how many computer based image processing systems 100 use °o.oo images 104 which are in tiled formats for the purpose of speed, more convenient handling or a need to conform to a standard. Such images 104 are made up of tiles eg 132. There 20 are also many methods and algorithms which can be used for compression 122 and decompression 124 of images, the methods offering lossless or lossy compression, !""""various compression ratios, algorithmic complexity and speed of processing.

Usually compression or decompression are only two of many elementary 0 operations 108 which an image processing system can perform. Such operations can include scaling 110, clipping 114, filtering 116, linear and non-linear transformations 112, 010800; 14:49 562611.doc 010800; 14:49 56261 Idoc -2and colour conversion 118, to name but a few. Bidirectional arrows 102, 126 depict that the image 104 can be operated on by one or more of the operations 108 in order to yield a processed image 128. Furthermore, application of associated inverse operations can reproduce, to a desired degree of accuracy, the original image 104 from the processed image 128. High processing speed is almost always an important system requirement, and thus much effort is spent on improvement of methods and algorithms for those operations.

Summary of the Invention According to a first aspect of the invention, there is provided a method for applying a pre-determined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said method comprising steps of: decompressing a tile of said compressed image to obtain a corresponding tile of pixels; 15 applying the transform to successive said pixels to form a decompressed transformed tile; and *i placing said decompressed transformed tile into the associated output image in a S.position dependent upon said transform.

According to another aspect of the invention, there is provided an apparatus for applying a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said apparatus comprising: means for decompressing a tile of said compressed image to obtain a corresponding tile ofpixels; means for applying the transform to successive said pixels to form a decompressed transformed tile; and 010800; 14:49 562611.doc 0180 44 661.o -3means for placing said decompressed transformed tile into the associated output image in a position dependent upon said transform.

According to another aspect of the invention, there is provided a computer program which is configured to make a computer execute a procedure to apply a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said program comprising: code for decompressing a tile of said compressed image to obtain a corresponding tile ofpixels; code for applying the transform to successive said pixels to form a decompressed transformed tile; and code for placing said decompressed transformed tile into the associated output image in a position dependent upon said transform.

According to another aspect of the invention, there is provided a computer program product including a computer readable medium having recorded thereon a 15 computer program which is configured to make a computer execute a procedure to apply a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said program comprising: code for decompressing a tile of said compressed image to obtain a corresponding tile ofpixels; code for applying the transform to successive said pixels to form a decompressed oooo transformed tile; and 9. 9.: code for placing said decompressed transformed tile into the associated output image in a position dependent upon said transform.

010800; 14:49 56261 I.doc 010800 144 66 .o -4- According to another aspect of the invention, there is provided an apparatus for applying a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said apparatus comprising: a memory for storing a program; a processor for executing the program including the steps of: decompressing a tile of said compressed image to obtain a corresponding tile of pixels; applying the transform to successive said pixels to form a decompressed transformed tile; and placing said decompressed transformed tile into the associated output image in a position dependent upon said transform.

Brief Description of the Drawings oo:. One or more embodiments of the present invention will now be described with reference to the drawings, in which: Fig. 1 shows a general image processing framework; Fig. 2 is a block diagram showing an arrangement of an operating environment for decompressing and transforming image data; Fig. 3 shows an exemplary transformation of a single tile of an image in o 20 accordance with a described arrangement; Fig. 4 illustrates an exemplary arrangement of data at the output of the "decompressor of Fig. 2; decompressor of Fig. 2; 010800; 14:49 562611.doc Figs. 5A and 5B show orientations of an output Minimal Coded Unit (MCU) for eight exemplary transformations; Fig. 6 depicts contents of half of a Pixel Reconstructor Buffer of Fig. 2 after a single MCU is written, for 4 colour component images; Fig. 7 shows an exemplary block diagram of the pixel reconstructor of Fig. 2; Fig. 8 illustrates processing results provided by the pixel reconstructor, and the address generator in a described arrangement; Fig. 9 depicts a flow chart of method steps for applying orthogonal transforms to tiles of a compressed image; and Fig. 10 is a schematic block diagram of a general purpose computer upon which described arrangements can be practiced.

Detailed Description including Best Mode Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

""In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have 20 corresponding meanings.

The following description presents an improved decompression method and apparatus for tiled colour images, whereby certain orthogonal transformations, from a class of orthogonal transformations, may be performed, on substantially a per-tile, or per- S.i fractional-tile basis, during decompression of the image without any negative performance impact upon the decompression operation. The aforementioned 010800; 14:49 56261 I.doc 010800; 14:49 562611 .doc -6transformations are performed by applying a desired transform from a class of applicable orthogonal transforms (also referred to as transformations) which includes, for example, flips, and rotation by integral multiples of ninety or minus-ninety degrees, of the image.

The term "flip" is defined in relation to Figs. 5A and 5B. The described arrangement allows overall faster processing in systems where both decompression and orthogonal transformations are required. Rotation which is not limited to integral multiples of ninety and minus-ninety degrees can also be performed, with some degradation in reconstructed image quality.

It has been found that embedding per-tile, or per-fractional-tile orthogonal image transformation processing within a pixel reconstruction block (see 206 in Fig. instead of after an image decompression module in which the entire image is decompressed, which is the normal arrangement, provides advantageous performance. In this manner, orthogonal transformations can be performed on a substantially per-tile, or per-fractionaltile basis, during decompression with no significant performance degradation or incremental hardware cost.

Fig. 2 shows how one arrangement is implemented as custom hardware functioning as part of a larger image processing system. The larger system 200 comprises a Joint Photographic Experts Group (JPEG) decompression module 204, a pixel reconstructor 206, and an address generator 208. In the arrangement, input data, which is S 20 depicted as an arrow 202, is a JPEG compressed image, which is input into the system 200. Output data, which is depicted as an arrow 210, along with associated address information which is depicted as an arrow 214, is a decompressed and transformed image which is output from the system 200, to be thereafter stored in an "i external memory, ie an image store (not shown).

°o 010800; 14:49 562611.doc 010800; 14:49 562611 .doc -7- Fig. 3 shows an example 800 of an image 808 having a tile 806 in which a letter (804) is displayed. The tile 806 is made up of pixels, of which one pixel 802 is explicitly shown. A transformation depicted by a wavy arrow 810 produces a transformed version 812 of the image 808, wherein the input image 808 has been rotated 90 degrees clockwise, and flipped vertically. The flip results in pixels, which were formerly on an upper edge 836 of the image 808 prior to the flip, now being on the bottom edge 822. The pixel 802 has thus been moved to a position 816, and the tile 806 has been moved to a position 814. The arrangement performs the aforementioned transformations (ie rotate and flip) on a per-pixel basis, in the context of the decompression operation as applied to an image tile, as described below.

Fig. 4 shows an exemplary arrangement of output data from the JPEG decompression module 204 (see Fig. this output data being the decompressed image in the form of a stream 216 of Minimal Coded Units (MCUs) 302 to 306, in accordance with a baseline subset of the JPEG standard. Fig. 4 depicts the image 104 (see Fig. 1) as being made up of MCUs, ie. MCU 0 (ie. 302) to MCU, (ie. 306). Each MCU consists of between 1 and 4 blocks of 8x8=64 bytes, depicted by reference numerals 308 to 312, the number of blocks being dependent upon the number of colour components in the image.

oooo One MCU is provided per colour channel. Each block of 64 bytes (ie. 308 to 312) is ooos provided to the pixel reconstructor 206 (see Fig. 1) as a sequence of 4-byte words 20 depicted by reference numerals 316 320 with bytes in column order as shown.

oo ••"The pixel reconstructor 206 (see Fig. 2) assembles entire MCUs from the JPEG decompression module 204, which are received, as depicted by the arrow 216, 4 bytes at a time in column order. Colour components are interleaved every 8x8(64) bytes. Pixels of the image are represented on a one byte per colour component basis. An entire MCU comprises 64 bytes per colour component, where the number of colour components can 010800; 14:49 562611.doc 01080; 1:49 6261 .do -8range from 1 to 4. Thus, the MCU can comprise up to 256 bytes. Once an entire MCU is collected in a Pixel Reconstructor Buffer (PRB) 614 (see Fig. the pixel reconstructor 206 outputs a data stream of interleaved colour components in row (ie scan) order, 1 byte at a time, as depicted by an arrow 218. The speed with which the output at 218 is produced is sufficient to keep up with the speed of the JPEG decompression module 204, which in the described arrangement produces one byte per clock cycle, when averaged over 64 clock cycles. Double buffering is provided for the data coming from the JPEG decompression module 204, since almost a complete MCU is required before the data can be extracted by the pixel reconstructor 206, in the required order. The provision of double buffering obviates the need for forced waiting by the pixel reconstructor 206, for the input data. The pixel reconstructor 206 implements the PRB 614 as a 128 x 32 byte Random Access Memory (RAM), as will be described in relation to in Fig. 6.

Advantageously, the order in which byte data is read out of the PRB 614 is made to depend upon the type of the transformation desired to be performed.

Figs. 5A and 5B shows 8 transformations 500 which can be performed by the pixel reconstructor 206. In Fig. 5A, a transformation 504 provides no rotation and no A flip is an operation in which bytes in each column of the image are vertically reordered. Thus, for example, in 516, having regard to 504, a pixel 502 is moved to a position 518, and a pixel 506 is moved to a position 514.

20 A transformation 508 provides 90 degrees of rotation and no flip. Turning to Fig. 5B, a transformation 510 provides 180 degrees of rotation and no flip. A transformation 512 provides 270 degrees of rotation and no flip. Returning to Fig. 5A, a

.*AA

transformation 516 provides no rotation and a flip. A transformation 520 provides degrees of rotation and a flip. Returning to Fig. 5B, a transformation 522 provides 180

*A

e to 010800; 14:49 562611.doc -9degrees of rotation and a flip. A transformation 524 provides 270 degrees of rotation and a flip.

The described arrangement performs the aforementioned transforms on a perpixel basis within the tile, and also, correctly places the tile within the output image.

Accordingly, the pixel 802 (see Fig. 3) is rotated and flipped to the position and orientation depicted by the reference numeral 816, and a "base offset" address is used in order to position the correspondingly processed tile 806 in the desired position 814 in the output image 820.

Fig. 6 shows the contents of half of the PRB 614, after a new MCU from the JPEG decompression module 204 has been written into the PRB, for the case in which four colour components are present in the image being considered. The half-PRB shown is four bytes wide, as depicted by a reference numeral 416. Row addresses depicted by the reference numeral 414 run from 0 for a bottom row 418, to 63 for a top row 420. In an alternate case, where only three colour components are present, the final 8x8 block 404 (having addresses 48 to 63) is unused, and in the case of single colour component images, the last three blocks 408, 406 and 404 (having respective addresses 16 to 31, 32 to 47 and Qoo• oooo .i'"48 to 63), are unused.

S. 9***o ~The pixel reconstructor 206 outputs each MCU of a tile (eg 504 in Fig. 5A) in scanline order, this corresponding to outputting the top left hand corner pixel first, 20 followed by the rest of the pixels of the top row 502, and then outputting pixels in all subsequent rows from top (502) to bottom (506), with pixels being output from left to I right within each row. For an output with no rotation and no flip, the order of output *055 pixels is the same as that shown in 504 in Fig. 5A. Figs. 5A and 5B show all orientations of elementary 8x8 tiles, and thus show the order of the pixels which are outputted by the *e 95 0o S S 010800; 14:49 562611.doc 010800; 14:49 562611 .doc pixel reconstructor 206, in relation to each of the 8 transformations which can be handled by the described arrangement.

Fig. 7 shows a block diagram arrangement of the pixel reconstructor 206. The PRB 614 is implemented as a 2 port RAM having one read port 684 (ie "dout"), and one write port 682 (ie This RAM 614 accommodates double buffering of MCUs.

Accordingly, while the PRB RAM 614 is being written to by the JPEG decompression module 204 at the write port 682 on the connection 216, data is being read out of the read port 684, on a line 600, byte by byte in the order determined by one of the 8 selected transformations shown in Figs. 5A and 5B. Functional elements shown in Fig. 7 are typically implemented in hardware, however these elements can, in many instances, be implemented in software, or can be implemented using a mix of hardware and software.

Finite state machines "Wordin FSM" 604 and "ByteOut FSM" 626 control the various registers and read write operations. A procedure "WordIn(word)", defined in a pseudo-code fragment discussed below, implements the "WordIn FSM" finite state machine 604. A procedure "ByteOuto", defined in the pseudo-code fragment in the Appendix, implements the "ByteOut FSM" finite state machine 626.

.An element "instore" 608 controls buffer swapping of data in the PRB RAM 614, thereby determining which half of the PRB RAM is active for writing, and which half of the PRB RAM is active for reading. The value of "in_store" 608 is inverted each MCU by a one-bit bit inverter 606. This implements double buffering, since "in_store" represents the Most Significant Bit (MSB) at the input write address port 678 (denoted by "wr_address") of the PRB RAM 614. An output of the element "in_store" is connected to an input of the bit inverter 606 by a line 636, and is also connected to the write address port 678 of the PRB RAM 614 by a line 638. An output of the bit inverter 606 is connected to an input of the "instore" element by a line 630, and is also connected to the 010800; 14:49 562611.doc 11 read address port 680 (denoted by "rd_address") of the PRB RAM 614 by a line 632. The element "in_store" represents the MSB of the output read address 680 of the PRB RAM 614.

An element "in addr" 612 determines a current word write address within the active buffer, thereby determining which image data is written from the decompression module 204 to the PRB RAM 614. The address for this write operation is incremented with each word that is written, in accordance with a six-bit counter 610. Each written word is four bytes wide, as depicted by a reference numeral 416 in Fig. 6. An output of the element "in_addr" is connected to an input of the counter 610 by a line 642, and is also connected to the write address port 678 of the PRB RAM 614 by a line 644. An output of the counter 610 is connected to an input of the "in_addr" element by a line 640.

An element "ch_cnt" is incremented each clock tick, after having read the byte from the PRB RAM 614. The element ch_cnt is incremented unconditionally by a 2-bit incrementer 620. However, ch_cnt is also checked to determine if its newly incremented o 15 value is equal to the number of colour planes in the processed image. The number of colour planes is defined by a parameter "channels" defined in the Appendix. The number of colour planes is constant for any given image. If the above condition is met, chcnt is reset to 00 and out_cnt counter 618 is incremented. An output of the element ch_cnt is connected to an input of the incrementer 620 by a line 676, and is also connected to the read address port 680 (denoted by "rd_address") of the PRB RAM 614 by a line 660. An •oo output of the counter 620 is connected to an input of the ch_cnt element by a line 622.

The 6-bit element outcnt counts a number of output pixels read from the PRB RAM 614, and is incremented each time ch_cnt is reset as described above. An output of the element out_cnt 618 is connected to an input of a 6-bit incrementer 616 by a line 648, 562611.doc -12and is also connected to an encoder 624 by a line 650. An output of the incrementer 616 is connected to an input of the "out_cnt" element by a line 646.

The encoder 624 generates a part of a read address for the PRB RAM 614, wherein the read address is communicated to the PRB RAM 614 on lines 654, 656 and 660. The encoder 624 also generates a read address for the 8x4:1 output byte selector 628 on lines 654 and 658, on the basis of a signal from the pixel counter out_cnt (ie 618) (the signal from out_cnt being provided on a line 650), and on the basis of the transformation code RFV 652, this being provided on a line 652.

A fragment of procedural pseudocode is provided in the Appendix, which describes operation of the pixel reconstructor 600. The pseudocode compliments the block diagram in Fig. 7. Some key procedures and variables are now briefly described in relation to the pseudocode.

"InitPRO" implements the initial (reset) state of elementary hardware components of the pixel reconstructor 206, these being shown in Fig. 7.

S 15 "Wordln(word)" implements the WordInFSM control state machine 604 which "•is depicted in Fig. 7, plus any relevant data-path components such as buses or registers, which handle writing of MCUs into the PRB RAM 614. This procedure operates on that number of blocks 314 (see Fig. 4) which is determined by the number of colour channels present in the particular image being considered.

"channels" (see also pseudo-code fragment is a variable defining the number of colour channels present in the image.

"ByteOuto" implements the ByteOutFSM control state machine 626 in Fig. 7, together with any relevant data-path components such as buses or registers, which handle reading of MCUs from the PRB buffer 614. ByteOut( relates to 010800; 14:49 562611.doc -13which word to read from the PRB RAM 614, and also relates to which bytes to read from the aforementioned word. ByteOut0 outputs a single byte per clock cycle.

"instore" (see also pseudo-code fragment is a variable which controls buffer swapping of data in the PRB RAM 614.

To illustrate how the pseudocode relates to Fig. 7, an exemplary case is described in which the selected signal RFV (on the line 652 in Fig. 7) is this indicating that the transformation to be performed is "no rotation, no flip".

In the described arrangement, the signal RFV can take on one of eight different values, these possible values representing the available transformations. The mapping between RFV values and corresponding transforms is shown in the following Table.

RFV Value Transformation Or no rotation, no flip 90 degrees rotation, no flip 180r 180 degrees rotation, no flip 270r 270 degrees rotation, no flip Orv No rotation, vertical flip 90rv 90 degrees rotation, vertical flip 180rv 180 degrees rotation, vertical flip 270rv 270 degrees rotation, vertical flip Although only "vertical flips" are explicitly included in the above Table, horizontal flips are effectively also provided for. This can be seen by reference to Figs.

and 5B and noting, for example, that "no rotation and horizontal flip" is identical to "180 degrees rotation, vertical flip".

010800; 14:49 562611.doc -14- The "InitPRO" procedure relates to the hardware counters used in the described arrangement, and their initial values at the beginning of an image processing cycle. The ByteOut( procedure, defined in the pseudocode fragment in the Appendix, has an initial set of instructions common to all types of transformations (RFV) requested for a particular image, these instructions being: if (out_addr 0) wait until start_out [1] start out false This means that the ByteOutFSM state machine 626 waits for a signal from.the WordInFSM state machine 604 on the connection 602, in order to commrorce operation.

This is explained further in the Wordln(word) pseudocodfrin 'te Appendix. The WordInFSM state machine 604 asserts a signal to the ByteOutFSM state machine 626 when one full MCU (tile) has been read into in the PRB RAM 614, the MCU thus being available to be read out.

The pixel reconstructor 206 reads out one byte of data from the PRB RAM 614 per clock cycle. A clock signal not shown explicitly in Fig. 7, however it is noted that the 20 pixel reconstructor 600 is synchronous to a single clock.

The following two code fragments and from the ByteOut( pseudocode in the Appendix define the value of the read address applied by the pixel reconstructor 206 to the PRB RAM 614 at any given clock cycle. The code fragments and also define the number of the byte selected out of the word read from the memory at any given clock cycle by the multiplexer 628. Similarly, two equations are provided in the pseudocode for each of the eight possible transformations handled.

010800; 14:49 562611.doc case RFV is when Or: [2] read_addr ch_cnt,out_cnt[2:0],out_cnt[5]; byte_out (store[!in_store][read_addr])>>(8*(out_cnt[4:3])) The above means that for any given clock cycle the read address applied to the active half of the PRB RAM is a concatenation of: ch cnt: 2-bit channel counter 3 least significant bits of the 6-bit out_cnt counter the most significant bit of the 6-bit out_cnt counter ch_cnt is initialised to 00 and out_cnt is initialised to 000000 at the beginning of processing of the whole image in the procedure InitPR0, and the active half of the PRB RAM 614 is determined by the value of in_store (this value being either 0 or 1, being initialised to 0 in InitPR().

By applying the particular calculated read address to the PRB RAM 614, a 4byte data word is obtained on the "dout" output 684 of the PRB RAM 614. The pixel reconstructor 206 needs to select one byte out of four. The selected byte is defined by the next pseudo-code fragment as follows: Sbyte_ out (store[!in store][read_addr])>>(8*(out_cnt[4:3])) [3] This means that for any given clock cycle, the selected byte comes from the word store[!in_store][read_addr], where "store" is the name of the whole PRB RAM 614.

The variable in_store (which can take on values of 0 or 1) defines the current half of the PRB RAM 614 from which the reading takes place. The PRB RAM 614 is, as previously 010800; 14:49 562611.doc -16described, double-buffered, and the variable "readaddr" is the current read location.

Once this word is available, it is shifted logically (serially) to the right by: 8*(out_cnt[4:3]), which equals 0 or 8 or 16 or 24, depending on the value of bits of the out cnt counter as follows: r4:3] shift value: 00 0 bits 01 8 bits 16 bits 11 24 bits After the shift, the 8 least significant bits of the word which has been read contain the byte read out by the pixel reconstructor 206, and these 8 least significant bits are passed on downstream in this particular clock cycle.

The following pseudocode fragment from the ByteOut( procedure (see the Appendix) defines the behaviour of the pixel reconstructor 206 for the last step of any given clock cycle: chcnt+= 1; 20 out_addr 1; //bytes if ch_cnt channels chcnt 0; out_cnt 1;//pixels if out_addr (channels<<6)) [4] 25 out_addr 0; outcnt 0; start_in true; start the input 010800; 14:49 562611.doc -17- This fragment has the effect that on each clock tick, after having read the byte from the PRB RAM 614, ch_cnt and outaddr counters are incremented unconditionally, after which ch_cnt is checked to determine whether its newly incremented value is equal to the number of colour planes in the processed image. The parameter "channels", which is constant for any given image, determines the number of colour channels present. If the above condition is met, chcnt is reset to 00 and outcnt counter is incremented.

Thereafter, the newly incremented out_addr is checked to determine whether its value is less than the number of colour planes in the image, multiplied by 64 (channels<<6). If this condition is not met, (ie if [out_addr>=(channels*64)]), then the variables out_addr and out_cnt counters are reset, and the start_in signal to the WordIn- FSM state machine (which dealt with by the Wordln(word) pseudocode routine) is sent to indicate that the current half of the PRB 614 has been emptied (read out), and thus may be used for filling in a new tile.

The Encoder block 624 in Fig. 7, which is defined by a pseudo-code fragment represents the fact that 4 bits of the read_addr (depicted by the reference numeral 656) and the byte_out selection denoted by the reference numeral 658 (ie. 2 bits needed to control the 4:1 byte output multiplexer 628) are calculated from the type of 4 transformation (ie. RFV, which is the case statement in the ByteOut( pseudocode procedure) and the out_cnt 6-bit counter 618: when Or: read_addr ch_cnt,out_cnt[2:0],outcnt[5]; byteout (store[!in store][read addr])>>(8*(out cnt[4:3])) 010800; 14:49 562611.doc -18- In the described arrangement the very last step of pixel reconstruction is assembly of the byte stream into 64 bit words for the Address Generator 208, the reconstruction being done in a packed way, little endian. This means that bytes read out of the PRB RAM 614 are placed, one by one, in 64-bit words starting from the least significant byte of the word, and the words are passed to the Address Generator 208 (see Fig. 2) as they fill up.

The Address Generator 208 is responsible for placing 64-bit words in memory external to the pixel reconstructor 206. That external memory holds the whole image in a decompressed format, and in the location(s) suitable for a particular application.

Irrespective of the storing scheme for a particular application, it is possible to determine base addresses of each and every single 8x8 block of the image.

In the described arrangement, a single 8x8 image block 806 (see Fig. 3) consumes from one to four 64-bit words, depending on the number of colour components or channels present. It is the role of the Address Generator 208 to place these words in the memory according to a number of factors. One such factor is the overall x-y size of the processed image depicted by the arrows 826 and 824 respectively. Another factor is the size of the elementary image block (8x8 bytes in the described arrangement) as depicted by arrow pairs 828/832 and 830/834. Further factors are the number of colour components in the image, as determined by the "channels" variable, and the selection from the eight transformations, performed on the decompressed image, as denoted by the variable RFV. A procedure, which can be implemented in hardware, which describes operation of the Address Generator 208 is given below.

TaskStartJPEG() current addr Destination-Address; rowaddr DestinationAddress; 010800; 14:49 562611.doc 19mcuaddr Destination_Address; case RFV is when Or or 180r or Orv or 180rv: [I current_row MCUHeight; iwidth MCUWidth; how many MCUs across image when 90r or 270r or 90rv or 270rv: currentrow MCUWidth; iwidth MCUHeight; II how many MCUs down image end case; 6 6 WordlnJPEG(word) burst_write(current_addr,wordbuf); //write buffer if noofbursts 0 last row of mcu (and pixel) noofbursts 7; restart mcu row count noofbursts if iwidth 0 {//last pixel in this row of MCUs if current_row 0 end of task exit; else new MCU row (or column if rotated) currentrow 1; advance MCU row count case RFV is when (no rotation, no flip) or (180 deg rotation and flip): current_addr row_addr (8*(MCUWidth+1)*Ch) row_addr row_addr (8*(MCUWidth+1)*Ch) mcuaddr =rowaddr+ (8*(MCUWidth+) *Ch) 010800; 14:49 562611.doc 010800; 14:49 562611 .doc iwidth MCUWidth; restart when (90 deg rotation) or deg rotation and flip): current addr =current_addr Ch; row addr row addr Ch; mcu addr mcu addr Ch; iwidth MCUHeight; restart when (180 deg rotation) or (no rotation and flip): current_addr rowaddr (8*(MCUWidth+1)*Ch) row_addr rowaddr (8*(MCUWidth+1)*Ch) mcu_addr row_addr (8*(MCUWidth+1)*Ch) iwidth MCUWidth; restart when (270 deg rotation) or (270 deg rotation and flip): current_addr current_addr Ch; row addr row addr Ch; mcuaddr mcuaddr Ch; iwidth MCUHeight; restart 20 end case; else next MCU in the same row (column) iwidth 1; advance iwidth case RFV is 25 when (no rotation) or no rotation and flip): currentaddr currentaddr Ch; mcu_addr mcu_addr Ch; 010800; 14:49 562611.doc 010800; 14:49 562611 .doc -21when (90 deg rotation) or (270 deg rotation with flip): current_addr current_addr +(8*(MCUWidth+1)*Ch); mcu_addr mcu_addr +(8*(MCUWidth+1)*Ch); when (180 deg rotation) or (180 deg rotation with flip): currentaddr currentaddr Ch; mcuaddr mcuaddr Ch; when (270 deg rotation) or (90 deg rotation and flip) current_addr current_addr +(8*(MCUWidth+1)*Ch); mcu_addr mcu_addr +(8*(MCUWidth+1)*Ch); end case; else //next scanline of current MCU 0. current_addr current_addr ((MCUWidth+1)*Ch); noofbursts 1; advance MCU row count The variable "Destination_Address" is not the address of the top left hand corer pixel of the decompressed image, but rather the address of the first byte of the decompressed image to be written to the memory. This is different for different selected transformations. The following process is used to to calculate the variable.

25 TL_address address of the first byte of the top left hand corner pixel of the decompressed image

I//

010800; 14:49 562611.doc 010800; 14:49 562611 .doc -22case RFV is when (no rotation,no flip) or (270 deg rotation,flip): Destination_Address TLaddress; when (90 deg rotation, no flip) or [7] (180 deg rotation, flip) the number of colour components equals the number of words needed to store one pixel row of an MCU (MCUWidth+1) Ch WordWidth; MCUWidth and MCUHeight are counted from 0 Ch number of colour components Destination_Address TL_address +(MCUWidth*Ch); S when 180r or Destination Address TL address (MCUHeight*(8*(MCUWidth+1 (MCUWidth*Ch); when 270r or Orv: Destination Address TLaddress+ (MCUHeight*(8*(MCUWidth+1)*Ch)); 25 end case; Fig. 8 illustrates processing results provided by the pixel reconstructor 206, and the address generator 208 in a described arrangement (see Fig. As will be recalled 010800; 14:49 562611.doc -23from Fig. 3, the tile in question 806 is rotated by ninety degrees and flipped vertically as represented by the relationship between the initial tile 806, and the transformed tile 912.

The tile 912 is then positioned at a lower right hand comer 814 of the reconstructed image 820.

The pixel reconstructor 206 reads pixels of the tile 806 from the PRB 614 in an order that reflects the transformation that has been performed. It is also noted, in relation to Figs. 5A and 5B, that the pixel reconstructor 206 outputs each transformed tile (eg 504 in Fig. 5A) in scanline order. Accordingly, the top left hand corner pixel 908 is output first, followed by the rest of the pixels of the top row, then all subsequent rows top to bottom, pixels left to right within a row.

The address generator 208 places the transformed tile 912 in the appropriate location 814 in the reconstructed output image 820. Once having determined the correct position for the tile 814, the address generator places the pixels which are output from the pixel reconstructor 206 in scanline order, as depicted by an arrow 922. The bidirectional 9**t 15 arrow 922 indicates that the order in which pixels are filled within the scanline can be either from left-to-right, or from right-to-left. Accordingly, pixels 908 and 910 of the o.

transformed tile 912, which correspond to pixels 906 and 904 respectively of the pretransformed tile 806, are output sequentially by the pixel reconstructor 206, and placed sequentially by the address generator 208 as depicted by pixels 916, 918 respectively.

It is noted that although the described arrangement uses a JPEG decompression module 204, decompression modules based upon other standards can also be used, provided images are processed in tiles (blocks). The tiles may be either square or S•rectangular.

Fig. 9 shows a flow chart of method steps for applying an orthogonal transform, selected from a class of such transforms, to a compressed image. In a first step 1002, the 010800; 14:49 562611.doc -24- 9process 1000 selects a transform from the set of transforms. Thereafter, in a step 1006, a tile from the compressed image is decompressed, after which, in a step 1010, the selected transform is applied to the decompressed tile. Thereafter, in a step 1014, the transformed tile is placed in the output image, in a manner dependent upon the transform which has been selected. Thereafter, a testing step 1018 determines if more tiles are to be decompressed from the compressed image. If there are such tiles, then the process 1000 is directed in accordance with a "yes" arrow back to the step 1010. If there are no more tiles to be processed in the compressed image, then the process 1000 is directed from the decision step 1018 in accordance with a "no" arrow to a step 1020 whereupon the process 1000 terminates.

The method for orthogonal transformations of compressed digital images can be practiced using a conventional general-purpose computer system 700, such as that shown in Fig. 10, wherein the processes depicted in Figs. 3-8 may be implemented as software, such as an application program executing within the computer system 700. In particular, Poo* the steps of the method for orthogonal transformations of compressed digital images can 6@S6 be effected by instructions in the software that are carried out by the computer. The 0@ software may be divided into two separate parts, one part for carrying out the orthogonal transformations of compressed digital images methods, and another part to manage the fops*: user interface between the latter and the user. The software may be stored in a computer 20 readable medium, including the storage devices described below, for example. The ~o• SS software is loaded into the computer from the computer readable medium, and then 6600 executed by the computer. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer preferably effects an advantageous apparatus 010800; 14:49 562611.doc 010800; 14:49 56261 Idoc for orthogonal transformations of compressed digital images in accordance with the described arrangements.

The computer system 700 comprises a computer module 701, input devices such as a keyboard 702 and mouse 703, output devices including a printer 715 and a display device 714. A Modulator-Demodulator (Modem) transceiver device 716 is used by the computer module 701 for communicating to and from a communications network 720, for example connectable via a telephone line721 or other functional medium. The modem 716 can be used to obtain access to the Intemrnet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN).

The computer module 701 typically includes at least one processor unit 705, a memory unit 706, for example formed from semiconductor random access memory (RAM) and read only memory (ROM), input/output interfaces including a video interface707, and an I/O interface713 for the keyboard702 and mouse703 and optionally a joystick (not illustrated), and an interface 708 for the modem 716. A storage device 709 is provided and typically includes a hard disk drive 710 and a floppy disk oooo drive 711. A magnetic tape drive (not illustrated) may also be used. A CD-ROM drive 712 is typically provided as a non-volatile source of data. The components 705 •oooo •to 713 of the computer module 701, typically communicate via an interconnected bus 704 and in a manner which results in a conventional mode of operation of the computer system 700 known to those in the relevant art. Examples of computers on which the arrangements can be practised include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved therefrom.

Typically, the application program of the described arrangement is resident on the hard disk drive 710 and read and controlled in its execution by the processor 705.

Intermediate storage of the program and any data fetched from the network 720 may be 010800; 14:49 56261 l.doc 26 accomplished using the semiconductor memory 706, possibly in concert with the hard disk drive 710. In some instances, the application program may be supplied to the user encoded on a CD-ROM or floppy disk and read via the corresponding drive 712 or 711, or alternatively may be read by the user from the network 720 via the modem device 716.

Still further, the software can also be loaded into the computer system 700 from other computer readable medium including magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red transmission channel between the computer module 701 and another device, a computer readable card such as a PCMCIA card, and the Internet and Intranets including email transmissions and information recorded on websites and the like. The foregoing is merely exemplary of relevant computer readable mediums. Other computer readable mediums may be practiced without departing from the scope and spirit of the invention.

The method for orthogonal transformations of compressed digital images is preferably implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions for orthogonal transformations of compressed digital images. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

.oooo Industrial Applicability It is apparent from the above that the arrangements described are applicable to ooo, the computer and data processing industries.

The foregoing describes only some embodiments of the present invention, and 9..9.

'modifications and/or changes can be made theretowithout departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

010800; 14:49 562611.doc 010800; 14:49 562611 .doc -27- Appendix InitPRO inaddr 0; outcnt 0; outaddr 0; instore 0; startout false; startin true; chcnt 0; current word write address counts out pixels read (6 bits) II counts out bytes read (8 bits) the half the write goes to /I stall the reading release the writing colour component count 2-bits Wordln(word) if (inaddr 0 !startin) wait until startin true; start_in false; not reset on first entry because of !start_in check, so first 2 MCU will be stored before we can stall store[in_store][in_addr] word; inaddr 1 channels number of colour components in the processed image if in_addr (channels<<4)) {//end of mcu inaddr 0 instore !instore; swap buffers 010800; 14:49 562611.doc -28 start-out true; I* start the output ByteOut() RFV is the code of transformation performed if (out addr 0) wait until start-out start-out false IIappend bits together to generate the read address Idepending on transformation case RFV is when Or: read-addr ch-cnt,out-cnt[2:0],out-cnt[5]; byte-out= (store[! in-store][readaddr]) (8*(out-cnt[4:3])) when read-addr ch-cnt out-cnt[5:3],!out-cnt[2]; byte-out= (store[! instore][readadd (8*(!out-cnt[1 when 180r: read-addr ch-cnt,!out Cnt[2:O],!out byte-out= *(store[! in-store][read-addr]) (8*(!out-cnt[4:3])) when 270r: read-addr ch-cnt_'out cnt[5:3],out cnt[2]; byte-out 010800; 14:495611.c 56261 I.doc -29- (store[! in storef[read add cnt[1 when On,: read-addr ch-cnt,out-cnt[2:Q], byte-out (store[! in-store][read-addr]) (8*(!out-cnt[4:3])) when 9Orv: read-addr ch-cnt,!out-cnt[5:3],!out-Cnt[2]; byte-out (store[! in-store][read-add (8*(!out-cnt[1 when l8Orv: read-addr ch-cnt_'out Cnt[2:O],out byte-out (store[! in store][read-addr]) (8*(out-cnt[4: when 270rv: read-addr oh-cnt out-cnt[5:3],out-cnt[2]; byte-out= ed(store[in-store] [read-addr]) (8*(out-cnt[: edcase; oh-cnt out-addr 1;I/bytes if (och ont channels) oh-cnt 0 out-cnt 1; //Pixels if out addr (channels«<6)){ out-addr 0; out-ont 0; start-in =true; start the input 0 10800; 14:49 56261 I.doc 010800; 14:49 562611 .doc 010800; 14:49 56261 I.doc 010800; 14:49 56261 Idoc

Claims (4)

1. A method for applying a pre-determined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said method comprising steps of: decompressing a tile of said compressed image to obtain a corresponding tile of pixels; applying the transform to successive said pixels to form a decompressed transformed tile; and placing said decompressed transformed tile into the associated output image in a position dependent upon said transform.

2. A method according to claim 1, wherein the applying step comprises sub-steps of: 15 writing said tile of pixels to a Pixel Reconstruction Buffer PRB; and reading pixels from the PRB in an order which is dependent upon the orthogonal transform.

3. A method in accordance to claim 1, wherein the orthogonal transform comprises 20 at least one of a rotation and a flip of the tile. *oee 0*oO*.

4. A method according to claim 2, wherein said reading step is performed in relation to successive colour channels of the tile. 010800; 14:49

562611.doc 010800; 14:49 56261 Idoc -32- A method according to claim 2, wherein the placing step comprises writing, by an address generator, said pixels in scanline order to an image memory store dependent upon the orthogonal transform, and in accordance with at least one of: an overall size of a decompressed image; S a size of the tile ofpixels; a shape of the tile ofpixels; an aspect ratio of the tile ofpixels; a number of colour channels in the decompressed image. 6. An apparatus for applying a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said apparatus comprising: means for decompressing a tile of said compressed image to obtain a corresponding tile of pixels; 15 means for applying the transform to successive said pixels to form a decompressed transformed tile; and *i means for placing said decompressed transformed tile into the associated output image in a position dependent upon said transform. S o 20 7. An apparatus according to claim 6, wherein the means for applying comprise: means for writing said tile of pixels to a Pixel Reconstruction Buffer PRB; and means for reading pixels from the PRB in an order which is dependent upon the orthogonal transform. 010800; 14:49 562611.doc -33- 8. An apparatus according to claim 7, wherein the means for placing comprise an address generator for writing said pixels in scanline order to an image memory store dependent upon the orthogonal transform, and in accordance with at least one of: an overall size of a decompressed image; a size of the tile of pixels; a shape of the tile of pixels; an aspect ratio of the tile ofpixels; a number of colour channels in the decompressed image. 9. A computer program which is configured to make a computer execute a procedure to apply a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said program comprising: code for decompressing a tile of said compressed image to obtain a 0**S S corresponding tile ofpixels; 15 code for applying the transform to successive said pixels to form a decompressed V. 0• transformed tile; and code for placing said decompressed transformed tile into the associated output 0 image in a position dependent upon said transform. 20 10. A computer program according to claim 9, wherein the code for the applying step comprises: code for writing said tile of pixels to a Pixel Reconstruction Buffer PRB; and code for reading pixels from the PRB in an order which is dependent upon the orthogonal transform. 010800; 14:49 562611.doc 010800; 14:49 56261 Idoc -34- 11. A computer program according to claim 10, wherein the code for the placing step comprises code for writing, by an address generator, said pixels in scanline order to an image memory store dependent upon the orthogonal transform, and in accordance with at least one of: an overall size of a decompressed image; a size of the tile of pixels; a shape of the tile of pixels; an aspect ratio of the tile of pixels; a number of colour channels in the decompressed image. 12. A computer program product including a computer readable medium having recorded thereon a computer program which is configured to make a computer execute a procedure to apply a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said program comprising: 15 code for decompressing a tile of said compressed image to obtain a corresponding tile of pixels; code for applying the transform to successive said pixels to form a decompressed transformed tile; and code for placing said decompressed transformed tile into the associated output image in a position dependent upon said transform. 13. A computer program product according to claim 12, wherein the code for the applying step comprises: code for writing said tile of pixels to a Pixel Reconstruction Buffer PRB; and 010800; 14:49 562611.doc 010800; 14:49 56261 idoc code for reading pixels from the PRB in an order which is dependent upon the orthogonal transform. 14. A computer program product according to claim 13, wherein the code for the placing step comprises code for writing, by an address generator, said pixels in scanline order to an image memory store dependent upon the orthogonal transform, and in accordance with at least one of: an overall size of a decompressed image; a size of the tile of pixels; a shape of the tile of pixels; an aspect ratio of the tile of pixels; a number of colour channels in the decompressed image. An apparatus for applying a predetermined orthogonal transform during decompression of a compressed image formed from a plurality of tiles, said apparatus comprising: a memory for storing a program; a processor for executing the program including the steps of: decompressing a tile of said compressed image to obtain a corresponding tile of pixels; applying the transform to successive said pixels to form a decompressed S• transformed tile; and placing said decompressed transformed tile into the associated output image in a position dependent upon said transform. 010800; 14:49 562611.doc 010800; 14:49 56261 idoc -36- 16. A method for applying an orthogonal transform substantially as described herein with reference to the accompanying drawings. 17. An apparatus for applying an orthogonal transform substantially as described herein with reference to the accompanying drawings. 18. A computer program which is configured to make a computer execute a procedure to apply an orthogonal transform substantially as described herein with reference to the accompanying drawings. 19. A computer program product including a computer readable medium having recorded thereon a computer program which is configured to make a computer execute a procedure to apply an orthogonal transform substantially as described herein with reference to the accompanying drawings. S DATED this nineteenth Day of March, 2004 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON oo• *o o *°oO o 010800; 14:49 562611.doc 010800; 14:49 5626 lIdoc