GB2360163A - Encoding and decoding image data - Google Patents

Abstract

Efficient encoding and decoding of image data according to a variety of non-standard formats is achieved by dividing different tasks between a modifiable software programmed processor stage (<B>14/24</B>) and an unmodifiable hardware (<B>17/27</B>) stage. Efficiency is ensured as the software stage is best suited to nontrivial peripheral tasks (<B>82, 83, 84, 85</B>), while the hardware (e.g. an ASIC) is best suited for central intensive computation tasks (<B>91, 92, 93, 94, 95</B>). The software may be upgraded or changed in the field by a user, for instance from a network e.g. the Web, to accommodate the variety of non-standard formats.

Description

1 2360163 1 SYSTEM FOR ENCODING AND DECODING IMAGE DATA

A closely related document is co-pending European patent application no. 00310644.0 filed on 30 November 2000, the disclosure of which is incorporated herein by reference.

This invention relates generally to image-related devices, for example economical devices for consumer use, such as incremental printers, small LCD or other video displays that are processor operated, scanners, digital cameras and the like; and to apparatus and methods that facilitate transfer of data to and from such devices, and use within 10 such devices preferably, by (1) encoding image data into files for transfer or storage, and (2) decoding such files for display, printing or other functions that call for bytewise processing.

The term "codec" is a combination of portions of the words "code" and "decode". According to some usages, a codec is a system that can perform both these functions. For purposes of this document and particularly the appended claims, however, a "codec" is hereby defined as a system that can perform either coding or decoding, not necessarily both. It is envisaged that the "codec" specified in the claims could perform coding and decoding if desired.

The following description of the prior art focuses on JPEG data files. This file format is the most commercially important one for still images, at present. However, it will be apparent to the reader that the invention is not limited to such data or file formats.

Currently, the JPEG format accounts for the great bulk of all image data transmitted by e-mail, and over the WorldWide Web; and also between apparatuses such as cameras and computers, computers and printers, printers and scanners, etc. The invention is not limited to JPEG files and rather has important application to TIFF and any other formats that share the general file- coding and -decoding characteristics introduced below.

(a) The problem of commercial data-encoding standards The JPEG ("Joint Photographic Experts Group") encoding has become an industry standard for image compression. Nevertheless different software and hardware vendors can use slightly different implementations or " flavors " of the encoding in 2 application programs that manipulate image data, and also in hardware such as cameras, scanners and printers that generate or use image data.

At the beginning 70 (Fig. 5) of a representative JPEG data stream is header or marker information, or both 71, that describes various aspects of the encoded image.

These include, for example, image dimension data, encoding/decoding tables and comments.

The header or marker information is followed by data "scans" 72 that contain compressed image data. This data may then be followed by more markers 73, 75, 77, each of which may be followed by more data 74, 76, 78 and so on, to the end 79 of the 10 data stream and of the file.

Thus, there may be several or many data scans 72, 74, 76, 78 separated by markers 73, 75, 77 throughout the entire data stream. Multiple images also may be contained within the same stream. It is the character and format of the headers, markers, tables and image separators, and the number of scans and images, that are particularly subject to the variations mentioned above.

The JPEG specification allows for great flexibility in terms of image compression parameters, which markers are used, and the number of markers, data scans and images contained within a JPEG data stream. All this makes handling JPEG files from various sources challenging for any single JPEG codec, especially if the code is implemented in 20 hardware.

(b) Hardware benefits and limitations The method of compressing or decompressing, or both, of actual image data contained within the scan segments, however, is well defined in the JPEG specification, and it is also the most computationally intensive portion of JPEG processing.

A hardware implementation has the advantages of being faster and less expensive than a firmware or software implementation. Therefore, since processing the image data makes up the bulk of the work and is intensive enough to be far more quickly completed by a hardware codec, a hardware implementation is greatly preferable for its speed and economy.

Dedicated hardware that is designed specifically for processing JPEG data can be optimized to provide the desired performance using the smallest possible silicon area on 3 an integrated circuit. It is this characteristic that is responsible for the favorable economics of the hardware approach; chip area is probably the greatest single determinant of chip cost.

Unfortunately hardware is far less flexible, and requires a much longer lead time to make any changes to the design. A hardware change involves waiting for new chips to be fabricated (which can take months), and any products built with the old chips have to be opened up to swap in the new chips (which is very costly for a moderate- to highvolume product).

(c) Software/firmware benefits and limitations A software or firmware implementation is much easier to change, since it requires only that new code be downloaded into each system. This can be performed easily either by a customer (after first downloading the code from, for example, the WorldWide Web) or at a manufacturing site. Such a downloading process is fairly quick and inexpensive, and is known in the art.

A pure software or firmware implementation, however, has the drawback of lower speed, taking longer to process data, and of being more expensive from a partscost (material list) perspective, relative to the hardware implementation. Software implementation is inherently slower and more costly. A general-purpose processor would have to be extremely powerful to execute instructions fast enough to keep up with a hardware implementation.

(d) Conclusion

Substantial cost and inconvenience arises in the processing of image-data files. Although the magnitude of this problem may not be great with respect to a typical individual file, the problem represents a major expense and obstacle for the processing of all such files considered in the aggregate.

An ideal JPEG codec would possess qualities of both implementations: the processing speed and economy of a hardware-based codec combined with the flexibility and upgradeability of the software approach. No such solution is known in the art.

This failing has continued to impede achievement of rapid but reliably accurate and economical encoding and decoding, adaptable to changing commercial implementations. Thus irnportant aspects of the technology used in the field of the invention remain

4 amenable to useful refffiement.

The present invention seeks to provide improved data processing.

According to an aspect of the present invention, there is provided a method of encoding or decoding an imagelike data set, comprising the steps of:

in firmware or software, performing peripheral housekeeping functions related to the encoding or decoding; and in hardware, performing central, computationally intensive encoding or decoding processes.

According to another aspect of the present invention, there is provided a task partitioned hybrid codec comprising:

modifiable processing means for handling data components that vary in format as among vendors, variants, versions or extensions; and relatively faster but substantially unmodifiable processing means for handling data components that substantially do not thus vary in format.

In its preferred embodiments, the present invention has several aspects or facets that can be used independently, although they are preferably employed together to optimize their benefits.

In preferred embodiments of a first of its facets or aspects, there is provided a method of encoding or decoding an image-like data set. The method includes the step of, in fir---mware or software, performing peripheral housekeeping functions related to the encoding or decoding. The method also includes the step of, in hardware, performing central, computationally intensive encoding or decoding processes.

For purposes of this document the "peripheral functions" performed in firmware or software are hereby defined as extensive operations calling for a significant amount of computation or data evaluation. In other words, they are not merely trivial operations but rather represent a meaningful sharing of the encoding or decoding labour as between hardware on one hand and firmware or software on the other.

To compare an extreme case, for instance, prior systems having hardware such as an application-specific integrated circuit ("ASIC") may possibly include a software or firmware operation that does nothing more than set a single bit which starts the ASIC Such operation and analogously inconsequential operations of slightly greater scope, do not represent any truly significant distribution of the overall encoding or decoding as between hardware and non-hardware operations.

Those skilled in the art will thus understand that the thrust of the preferred embodiments is to allocate a functionally significant fraction of the data processing to non-hardware operations. The skilled artisan will find this distinction even more plain based upon examples that follow.

The foregoing may constitute a description or of a first facet in its broadest or most general form. Even in this general form, however, it can be seen that this facet significantly nfitigates the difficulties left unresolved in the art.

In particular, through use of this feature, file coding and decoding can be accomplished in the most highly optimized manner possible. More specifically:

(1) core processing can be performed at highest speed and processing efficiency; while at the same time (2) supporting functions can be performed with best economy, easily and 15 inexpensively upgraded as required.

The first of these benefits stems from use of extremely fast, specialized hardware for those central processes, particularly steps that are most repetitive and most frequently required, and those that are most stable in their character. On the other hand, the second benefit arises from use of more accessible, more readily upgraded programs for the 20 supporting steps.

The latter may include, for instance, steps that are always performed just once for each image file as distinguished from once per pixel, or once per block or strip of pixels within an image. The peripheral or supporting procedures can also include, in particular, steps that vary with occasional revisions to the format specification, or that 25 vary as among equipment or software vendors.

Although this facet in its broad form thus represents a significant advance in the art, it is preferably practiced in conjunction with certain other features or characteristics that further enhance enjoyment of overall benefits. For example, it is preferred that the peripheral housekeeping functions include:

parsing and interpreting file headers, footers and some other threshold parameters to generate interpreted information; and 6 applying that interpreted information to configure the computations performed in hardware.

In common computer parlance the term "parsing" may be most typically understood to refer to a function of reading composite data from headers or footers, and sorting that data into its separate individual parts. For purposes of the present document, however, and particularly for the sake of simplicity in wording certain of the appended claims, "parsing" is hereby defined to encompass the inverse function as well, namely compiling individual data elements needed for headers or footers and forming those assembled elements into the headers or footers, or other threshold parameters.

It is also preferred that the central, intensive coding or decoding computations include at least one of these functions:

transformations between color spaces; segregating luminance data from chroma and hue data; Fourier transforms, discrete cosine transforms and other transforms; applying at least one quality-factor parameter selectively to particular components of transform-output data to effectuate one or more phases of a data-compression regimen; downsampling, upsampling, rotations, flips and other geometrical manipulations; and moving, storing and retrieving image pixel-content data.

It is still more strongly preferred that the two sets of preferences just articulated, for the peripheral housekeeping and the central intensive computations respectively, be implemented in conjunction together.

The method of the first facet may be particularly for use with a data set that is to be encoded or decoded by a protocol which is not wholly standardized. As set forth above, it is quite commonplace that a protocol is instead promulgated by plural vendors, or in plural variants or versions or extensions, or with combinations of some or all such irregularities.

In this common case, the peripheral housekeeping functions preferably include substantially all functions that vary with vendor, version, variant or extension of the encoding or decoding protocol. Such functions are intimately related to the intensive computations, but just a half-step removed from them.

7 These functions encompass reasonably extensive evaluation of information within the data, or computations based on such information. They enable the overall procedure to reach conclusions about the proper performance of the more-intensive data processing.

Accordingly, as represented earlier, the skilled person in this field can now appreciate from these preferences that the preferred embodiment assigns non-trivial operations for performance in software or firmware. Mere onloff switching and other relatively primitive relay-like tasks, for instance, are excluded.

Still other preferences relate to the character of the equipment used to perform the method or of the data processed. For example, preferably the hardware-performed step includes performing the computations in an ASIC. A parallel preference on the other side is that the firmware-or-software-performed step comprises performing the functions in either (1) a prograrnmed processor in a printer, or (2) a general-purpose computer or raster-image processor.

Yet another such preference is to use the system with a data set containing either (1) image-data signals from a camera, scanner or other image sensor; or (2) terrain or elevation data from a mapping system, or other image-like data in which precise data values for each location are not needed. In this case, a further preference is that the data set be received substantially directly from the sensor or system and include essentially no file headers. (This preference may be regarded as a species of the invention, since the contrary preference is also valid for data in other circumstances.) In preferred embodiments of a second facet, there is provided a method for encoding or decoding JPEG data. This method includes the step of, in firinware or software, performing peripheral housekeeping functions related to JPEG encoding or decoding.

The recitation "peripheral housekeeping functions" is to be understood as having the same meaning as discussed above for the first facet. Another preferred step of the method is, in hardware, performing central JPEG encoding or decoding processes. Central processes for JPEG files are well established and substantially consistent across many versions, variants and extensions promulgated by different vendors. Although the actual content of peripheral functions naturally varies even in the case of JPEG files, nevertheless it can be said that the general character of such peripheral processing and 8 the information to be processed are essentially uniform.

The uniformity with respect to JPEG files is even greater than for files in general as discussed above. More specifically these processes include, as suggested for the first facet, parsing and interpreting file headers and footers as well as some other threshold parameters.

The foregoing may constitute a description or definition of the second facet in its broadest or most general form. Even in this general form, however, it can be seen that this facet too significantly mitigates the difficulties left unresolved in the art.

In particular, the benefits of this second facet are substantially as described above for the first facet but more notable in view of the widespread use and popularity of the JPEG format. As suggested earlier, the problems with such files, although perhaps not severe when considered one file at a time, are economically troublesome in the aggregate.

Although this second facet in its broad form thus represents a significant advance in the art, it is preferably practiced in conjunction with certain other features or characteristics that further enhance enjoyment of overall benefits. The preferences applicable in this case closely parallel those for the first facet discussed earlier.

In preferred embodiments of a third facet, there is provided apparatus for encoding an image-like data set. The apparatus includes at least part of an application specific integrated circuit (ASIC) for discriminating fine detail of perceptual chromatic information in the data set.

The apparatus may include at least part of a firmware or software stage for parsing and interpreting file headers or markers to generate interpreted information, and for applying the interpreted information to control the circuit. The foregoing may represent a description or definition of the third facet in its broadest or most general form.

Even as couched in these broad terms, however, it can be seen that this facet importantly advances the art. In particular, proper design of an ASIC for selectively editing chromatic information of varying detail provides adequate computing power for this demanding work.

At the same time, it yet remains necessary to sort out field demarcations, file

9 types and other threshold parameters to properly set the ASIC for each image. By assigning firmware or software to perform these commercially more volatile preliminaries, the preferred embodiment facilitates their completion in accordance with their own more frequently shifting bookkeeping standards. Although the third facet thus significantly advances the art, nevertheless to optimize enjoyment of its benefits preferably the system is practiced in conjunction with certain additional features or characteristics. In particular, preferably the circuit is configured to perform basic JPEG decoding functions. In this case it is still more distinctly preferred that the circuit perform at least one of these functions: transformations between color spaces; segregating luminance data from chroma and hue data; Fourier transforms, discrete cosine transforms and other transforms; applying at least one qualityfactor parameter selectively to particular components of transform-output data to effectuate one or more phases of a data-compression regimen; downsampling, upsampling, rotations, flips and other geometrical manipulations; and direct memory access operations.

Another preference is that the firmware or software stage also include at least part of a programmed processor.

In preferred embodiments of a fourth facet, there is provided a taskpartitioned hybrid codec. It includes some means for handling data components that vary in format. Such variance may be as among vendors, variants, versions or extensions. These first data-handling means are modifiable. For purposes of generality and breadth in discussing the invention, these means will be called simply the "modifiable processing means".

Also included in the codec is means for handling data components that substantially do not thus vary in format; they are faster than the modifiable processing means, but substantially not modifiable. These means will be called, also for breadth and generality, the "substantially unmodifiable processing means" or simply the 30 "unmodifiable processing means".

Another preference is that the codec be a JPEG codec, and the data components JPEG components. It is further preferred that the modifiable processing means pass compressed image data while the substantially unmodifiable means pass uncompressed image data.

Still another preference is that the modifiable means do not comprise a direct memory access controller, while the substantially unmodifiable means comprise at least part of a direct memory access controller. Yet another is that the modifiable means comprise at least part of a programmed processor in a printer, or in a general-purpose computer or raster-image processor.

It is further preferred that the substantially unmodifiable means include at least part of an ASIC. Preferably also the modifiable means are capable of being modified in the field by an end user, by downloading code from a computer network, ideally from a WorldWide-Web site, into the apparatus.

It is additionally preferable that the modifiable means include some means for separating data for different images, if such different-image data are incorporated into a single data stream. This is a particularly useful feature because it can greatly simplify procedures for an operator.

An embodiment of the present invention is described below, by way of example only, with reference to the accompanying drawings, in which:

Fig. 1 is a block-diagrammatic representation of an embodiment of decoding system; Fig. 2 is a like representation of an embodiment of encoding system; Fig. 3 is a like diagram of a Fig. 1 or 2 codec following the notation of Figs.

and 2 but incorporating additional subblocks to focus upon certain of the system features and preferences discussed above; Fig. 4 is a flow chart illustrating certain of the method features and preferences discussed above; and Fig. 5 is a diagram of a prior-art data structure associated with most JPEG files, including those used in the described embodiments.

THE HYBRID CODEC Preferred embodiments of this invention use a software or firmware module 14, 11 24 (Figs. 1 and 2) in combination with a hardware module 17, 27. The two modules receive a data stream 12, 13, 28 from a source 11, 29 of encoded or decoded data respectively and proceed to share the work in a particularly effective and efficient way.

The software/firmware subsystem 14, 24 parses (as previously defined) all header information 12 and extracts or generates encoding parameters 16, 26. Based on this information it then configures the hardware subsystem 17, 27, which in turn performs the computationally intensive processing of the image data.

This hybrid approach provides the flexibility of a nonhardware codec where needed for the handling 82 (Fig. 3) of variations in the format of headers, markers and parameters. Yet it also achieves the power and low cost of a hardware codec for the handling 91-95 of time-consuming, complex data processing which is well defined and somewhat invariant across all implementations.

The result is a fast, inexpensive hybrid codec that is well suited for low-cost imaging applications. Despite its extraordinary combination of favorable cost and speed, the hybrid codec can tackle virtually any file, substantially independent of originating vendor, within the general file-format family (e. g. JPEG or TIFF) for which it is designed.

To keep the system updated for new developments in the field, the programming in the software/firmware stage 14124 can be replaced by information downloaded from the WorldWide Web or alternatively, via an associated computer, downloaded from a bulletinboard service or read from a floppy disc or CD-ROM. In the case of a Web source an end-user in the field can readily find and install the updated code through an interface 81 supplied as part of the system, simply by activating an update procedure.

For example, if the codec is part of an incremental printer or other itnage-related device, this download capability 81 can be designed directly into that device. In some cases, for example, the interface 81 can automatically manage the entire transaction, including establishment of telephone contact with a toll-free Internet access node. (Implementation of any or all of these approaches, based on the general teachings provided in this document, is well within the state of the art for both equipment and coding.) In such cases the user naturally may have to connect the system to a phone line, and supply any necessary outside-line prefix etc. In other cases the firmware or software 12 part of the codec may be updated through dial-up networking and WorldWide- Web access features of a general-purpose computer associated with the device.

DATA MOVEMENT In the most highly preferred embodiments, uncompressed data 18, 28 are passed using direct memory access (DMA) within the hardware module 17, 27. Compressed data 13, 23, however, are passed under firmware control 14, 24. If the compression system in use gives a very high compression factor, as can be made the case with JPEG and TIFF formats, the compressed data are much smaller than the uncompressed data. Consequently the compressed data do not require as much data-bus bandwidth.

Typically the compressed-data transfer rate can be between one-fourth and onetenth, or somewhat less, of that required to transfer the uncompressed image data (for medium to highquality images). This being so, compressed data 13, 23 passed at lower firmware transfer rates will be able to keep up with the high-speed DMA transfers of the uncompressed image data 18, 28.

Using this approach, only one DMA controller is required, thereby saving cost and design time. Handling of the compressed data remains simple and flexible under firmware control.

A second DMA controller could be used for passing compressed data as well, but in a sense the higher bandwidth would be wasted. On the other hand, since DMA operations in hardware are more compact and therefore less costly in terms of silicon 44real estate", the overall economy may nevertheless be better.

STRUCTURE AND CONTROL FLOW File formats are advantageously the same as in the prior-art formats introduced earlier (Fig. 5). Program flow, however, is importantly different as the system distributes process steps in a novel way.

Reception 50 (Fig. 4) of input data may occur through any of a wide variety of devices and procedures. For definiteness the illustration focuses upon JPEG files, and JPEG-like files, although as mentioned earlier the invention is not thus limited.

13 The procedures can range from a local-disc or remote-file-transfer acquisition 5 1, or through reception 52 of raw, never-compressed data directly from a camera or scanner for coding. Acquisition can also be the converse, namely already-encoded (compressed) data for decoding.

As mentioned earlier, the acquisition may also be reception 53 of information that is not strictly an image but shares key characteristics such as JPEG-like compressibility.

A primary requirement for JPEG-like processing is satisfaction of the JPEG-protocol assumptions of insensitivity to imprecision, particularly for high spatial frequencies in perceptual-chromatic or analogous data.

Data once acquired proceed either directly or indirectly to processing 55 by housekeeping tasks 56 that are peripheral to JPEG codec operation. These tasks 56 are subject to variation as among vendors, versions, variants or extensions and accordingly are performed in firmware or software which may take place in a programmed processor within a printer, or as a driver in a general-purpose computer or raster image processor 15 (RIP), or in combinations of these facilities.

These peripheral tasks 56 most typically include parsing (as previously defined) and interpretation 57 of information from (or about) headers, footers and other parameters to generate "interpreted information" therefrom. The next step is application 59 of the interpreted information to configure the hardware for the more intensive computations 62-66 that follow. Performance 61 of these latter steps is in hardware, mostpreferably in an ASIC.

Some of the illustrated features of Figs. 3 and 4 are set forth inexactly in the sense that they are labeled in terms of either decoding or coding specifically, whereas a more precise labeling in some cases could refer as well to an inverse function or to both functions in the coding and decoding set. Based upon the teachings in this document, the applicable inverse functions will be evident to people skilled in this field.

The disclosures in United States patent application no. 091492,906, from which this application claims priority, and in the abstract accompanying this application are incorporated herein by reference.

14

Claims (17)

1.

of:

2.

A method of encoding or decoding an image-like data set, comprising the steps in firmware or software, performing peripheral housekeeping functions related to the encoding or decoding; and in hardware, performing central, computationally intensive encoding or decoding processes.

A method as in claim 1, wherein the central, intensive coding or decoding computations include at least one function selected from the group consisting of: transformations between color spaces; segregating luminance data from chroma and hue data; Fourier transforms, discrete cosine transforms and other transforms; applying at least one quality- factor parameter selectively to particular components of transform-output data to effectuate one or more phases of a data-compression regimen; downsampling, upsampling, rotations, flips and other geometrical manipulations; and comprise:

moving, storing and retrieving image pixel-content data.

A method as in claim 1 or 2, wherein the peripheral housekeeping functions parsing and interpreting file headers, footers and some other threshold parameters to generate interpreted information; and applying the interpreted information to configure the computations performed in hardware.

4. A method as in any preceding clahn wherein the peripheral housekeeping functions comprise substantially all functions that vary with vendor, version, variant or 30 extension of the encoding or decoding protocol.

5. A method as in any preceding claim, wherein the hardware-performed step comprises performing the computations in an application-specific integrated circuit.

6. A method as in any preceding claim, wherein the firmware-or-softwareperformed step comprises performing the functions in: a programmed processor in a printer, or a general-purpose computer or raster-image processor.

7. A method as in any preceding claim, wherein the data set is of at least one data 10 type selected from the group consisting of: JPEG data; image-data signals from a camera, scanner or other image sensor; and terrain or elevation data from a mapping system, or other image-like data, wherein precise data values for each location are not required.

8. A method as in claim 7, wherein: the data set is received substantially directly from a sensor or system and comprises substantially no file headers.

9. A task-partitioned hybrid codec comprising:

modifiable processing means for handling data components that vary in format as among vendors, variants, versions or extensions; and relatively faster but substantially unmodifiable processing means for handling data components that substantially do not thus vary in format.

10. Apparatus as in claim 9, wherein: the modifiable means is operable to pass compressed image data; and the substantially unmodifiable means is operable to pass uncompressed image data.

16 Apparatus as in claim 10, wherein: the modifiable means does not comprise a direct memory access controller; and the substantially unmodifiable means comprises at least part of a direct memory access controller.

12. Apparatus of claim 9, 10 or 11, wherein: the modifiable means comprises at least part of a programmed firmware or software stage in a printer or in a general-purpose computer or raster-image processor, for parsing and interpreting file headers or markers to generate interpreted information and for applying the interpreted information to control the circuit.

13. Apparatus as in any of claims 9 to 12, wherein the substantially unmodifiable means comprises at least part of an application-specific integrated circuit for discriminating fine detail of perceptual chromatic information in the data set.

14. Apparatus as in claim 13, wherein the modifiable means is capable of being modified in the field by an end user, by downloading code from a computer network into the apparatus.

15. Apparatus as in claim 14, wherein the modifiable means comprises means for separating data for different images if such different-image data is incorporated into a single data stream.

16. A method of encoding and decoding an image like data set substantially as hereinbefore described with reference to and as illustrated in the accompanying drawings.

17. A task-partitioned codec substantially as hereinbefore described with reference to and as illustrated in the accompanying drawings.