WO2002045017A2

WO2002045017A2 - Method for image compression with optimization parameters

Info

Publication number: WO2002045017A2
Application number: PCT/US2001/046739
Authority: WO
Inventors: Michael D. Murphy; Randall L. Dougherty; Jonathan N. Bradley; Vance Faber
Original assignee: Lizardtech, Inc.
Priority date: 2000-12-01
Filing date: 2001-12-03
Publication date: 2002-06-06
Also published as: AU2002232501A1; WO2002045017A3

Abstract

In image compression, certain less significant data may be discarded. When multiple sets of image data are involved, an issue may arise as to which sets of data contain the most significant data. The invention is directed to finding efficient computational solutions for the corresponding rate-distortion problem. In a case where there are end blocks which comprise an image, for each block a separate bit stream may be generated with a property that it can be truncated to a variety of discrete lengths. Associated with each length is a measure of distortion that the image would incur if the given block was truncated to a given length. Given a user-specified size for the image, it is desired to find a set of truncation lengths for each block such that the overall distortion to the image is minimized. Alternatively, a user may select a desired quality for the image. The solution provided by the invention is a heuristic based on priority cues to approximate the optimal truncation point so as to meet the desired criteria.

Description

METHOD FOR IMAGE COMPRESSION WITH OPTIMIZATION PARAMETERS

CROSS-REFERENCE(S) TO RELATED APPLICATION(S) This application claims the benefit of provisional U.S. Application Serial No. 60/250,825, filed December 1, 2000, under 35 U.S.C. § 119.

FIELD OF THE INVENTION This invention relates generally to methods for image processing and compression, and more particularly, to methods for image processing and compression where optimization parameters are utilized to achieve a desired file size or image quality.

BACKGROUND OF THE INVENTION

In image processing and compression, it is sometimes desirable to discard certain less-significant data. When multiple sets of image data are involved, an issue may arise as to which sets of data contain the most significant data. Certain prior art methods have been directed to finding efficient computational solutions for the corresponding rate- distortion problem.

One prior art method that is directed to rate-distortion optimization is described in the JPEG Standards document, JPEG 2000, part I, section J.14.3, entitled "Rate-distortion Optimization". A rate distortion algorithm is described that relates the distortion to the truncation points for selected code blocks. It is presumed that the distortion of each code block can be summed to provide an overall distortion quantity, assuming certain criteria are met. Once a constraint is defined (e.g., image size), the optimization problem is to find the set of truncation points for the code blocks which minimizes the distortion subject to the defined constraint. It is stated that the solution to this constrained optimization problem by the method of Lagrange multipliers is well known. Specifically, the problem is stated to be equivalent to minimizing the following equation:

where D is the distortion, n_z- is a truncation point for a code block B_t, R denotes the number of code bytes associated with some layer in the bit-stream (and all preceding layers), and where the value of λ must be adjusted until the rate yielded by the truncation points which minimize Equation J.14 satisfies R = R_max. As stated, there is no simple algorithm which can yield a globally optimal set of truncation points in general. However, any set of truncation points, n_t, which minimizes Equation J.14 for some λ is guaranteed to be optimal in the sense that the minimum distortion is achieved at the corresponding bit-rate.

As further stated, for the problem of minimizing the expression in equation J.14, it is a separate optimization problem for each individual code block. Specifically, for each code block, the truncation point needs to be found which minimizes the equation J.14. A simple algorithm for accomplishing this is can be provided.

The present invention is directed to finding improved computational solutions for the rate-distortion problem. More specifically, the present invention is directed to methods where optimization parameters are utilized to achieve a desired file size or image quality.

SUMMARY OF THE INVENTION

In accordance with one aspect of the invention, a method is provided for processing a digital image which has been or is being encoded into a format in which the information is divided up into pieces called components, which are further broken up into parts called planes or bitplanes. The processing is done in such a way that one can produce an approximation to the original image by using only some of the planes in each component. In general, the planes in each component are ordered in such a way that the later planes are typically not considered useful unless one also has the earlier planes. The method involves the solution to an optimization problem in order to select best (or nearly best) planes according to a constraint and a goal, either trying to attain the greatest image quality subject to a resource bound or trying to attain a specified image quality by using minimal resources. The optimization generally involves criteria, weights, or subimages which need not be specified or known when the image is originally encoded, so that the planes cannot simply be ranked in advance.

In one embodiment, the method comprises the following steps, some or all of which may be performed simultaneously or in combination: computing or retrieving (if it was computed previously) a distortion for each plane, which is a number measuring or estimating the improvement in quality or fidelity of the processed image that would result from using the data in this plane as opposed to discarding said data; if desired, modifying these distortions to reflect user-selected criteria; running an optimization algorithm in order to select the best (or nearly best) collection of planes satisfying the given constraint; and collecting or retrieving the data for the selected planes for storage, transmission, or use.

In accordance with another aspect of the invention, a method is provided for computing relevance coefficients which measures or estimates the extent to which the values in a component of an encoded form of an image are relevant to a particular subimage of the image. The encoded form generally involves a wavelet transformation or related linear transformation from pixel values to component values. In general, the encoded image contains one or more components, and each component contains one or more values, so that each pixel value from the image can be expressed (at least approximately) as a linear combination of the component values with linear coefficients a(ij) where i indexes the value in the component and / indexes the pixel. In one embodiment, the method comprises the following steps: computing the sum oϊa(ij)² over all / for the component and ally for the subimage; computing the sum oϊa(i,j)² over all i for the component and all / for the entire image; and dividing the first sum by the second sum.

In accordance with another aspect of the invention, a method is provided for sending selected pieces of data (called planes) from a server computer storing a set of data to a client computer, where each plane has an associated benefit and an associated cost. The transmissions involved in the method include: message(s) from the client to the server requesting the benefit and cost information for the pieces of data which might be relevant, said message(s) specifying for which planes this information is needed; message(s) from the server to the client sending the requested benefit and cost information (if any), after which the client will perform an optimization algorithm to determine which planes to request; message(s) from the client to the server requesting specific planes; and message(s) from the server to the client sending the requested planes.

Alternatively, the optimization algorithm can be performed on the server, in which case the only transmissions (other than network protocol messages) are the request(s) from the client to the server (specifying the part of the data of interest to the client, possibly a resource restriction, and possibly a list of which planes the client already has) and the reply or replies sending planes from the server to the client. In accordance with another aspect of the invention, a method is provided for sending selected pieces of data (called planes) from one or more server computers storing a set of data to a client computer, where each plane has an associated benefit and_, an associated cost. The transmissions involved in the method include: message(s) from the client to a server, or from a receiving server to a sending server, requesting the benefit and cost information for the pieces of data which might be relevant, said message(s) specifying for which planes this information is needed; message(s) from a server to the client, or from a sending server to a receiving server, sending the requested benefit and cost information (if any), after which the client or receiving server may perform an optimization algorithm to determine which planes to request; message(s) from the client to a server, or from a receiving server to a sending server, requesting specific planes; message(s) from the sending server to the client or receiving server sending the requested planes; and repetition of some or all of the preceding steps as desired.

Alternatively, the optimization algorithm can be performed on the sending server, in which case the only transmissions (other than network protocol messages) between these two machines are the request(s) from the client or receiving server to the sending server (specifying the part of the data of interest to the client or receiving server, possibly a resource restriction, and possibly a list of which planes the client or receiving server already has) and the reply or replies sending planes from the sending server to the client or receiving server.

In accordance with another aspect of the invention, a method is provided for selecting initial segments of strings from a given set of strings, with each initial segment having an associated cost as well as an associated benefit. The method generally involves the solution to an optimization problem in order to select best (or nearly best) planes according to a constraint and a goal, either trying to attain the greatest total benefit subject to a bound on total cost or trying to attain a specified total benefit at minimal total cost.

In one embodiment, the method comprises the following steps: computing an incremental cost and incremental benefit for each plane; finding for each component an initial "conglomeration" (substring with benefit/cost ratio no lower than that of any proper initial segment of the substring) with maximal benefit/cost ratio; setting up a data structure (such as a priority queue) containing the information about these conglomerations and their benefit/cost ratios; repeating an evaluation subroutine; and terminating when the queue is empty, or optionally in other circumstances, such as when the desired total benefit is reached, or when a conglomeration which exceeds the remaining available cost is reached (if it is not very important to use up the available cost as completely as possible). In accordance with another aspect of the invention, an evaluation subroutine comprises: taking from the queue or data structure a conglomeration with greatest priority or benefit/cost ratio; if this conglomeration can be selected (without exceeding the specified total cost bound, or if the benefit bound has not yet been reached), then select the conglomeration (i.e., extend the selected substring for this component to include the conglomeration), and, if this conglomeration does not reach all the way to the end of the component, determine the next conglomeration from the so-far-unselected planes in the component and add it to the queue or data structure; and if the conglomeration cannot be selected, but it consists of more than one plane, then determine the proper sub conglomeration of the conglomeration with maximal benefit/cost ratio and add it to the queue or data structure.

Instead of working directly with conglomerations, one may simply assign to each plane a modified benefit/cost ratio equal to that of the conglomeration it is assigned to, and then work with planes rather than conglomerations throughout the algorithm; also, in the case of a benefit goal, one may take only part of the final conglomeration if that will achieve the desired benefit.

In one embodiment, the method comprises the following steps: setting up a finite list of benefit/cost ratio values (each actual benefit/cost ratio will be converted to a value in this list), and initializing one or more numerical tables indexed by this list; doing the following for each component: find the cost and benefit information for the component, and partition the component into conglomerations, each of which has a benefit/cost ratio which is converted as above; and using this information to update the table values to include this component; using the table(s) to compute a threshold benefit/cost ratio such that all conglomerations with converted benefit/cost ratio greater than the threshold will be selected, and all conglomerations with converted benefit/cost ratio less than the threshold will not be selected; and going through all components again, selecting all conglomerations with converted benefit/cost ratio greater than the threshold, and as many of the conglomerations with converted benefit/cost ratio equal to the threshold as can be chosen within the cost bound, or as many as are needed to meet the benefit goal.

In one embodiment, the tables used comprise: if there is a benefit goal, a table storing, for each value in the finite list, the total benefit of all "conglomerations" examined so far with converted benefit/cost ratio equal to that value; and if there is a cost bound, a table storing, for each value in the finite list, the total cost of all conglomerations examined so far with converted benefit/cost ratio equal to that value.

BRIEF DESCRIPTION OF THE DRAWINGS The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:

FIGURE 1 is a flow diagram illustrating a method for solving an optimization problem in order to select desired planes according to a selected constraint and a selected goal;

FIGURE 2 is a flow diagram illustrating a method for computing relevance coefficients which measure or estimate the extent to which the values in a component of an encoded form of an image are relevant to a particular sub-image of the image;

FIGURE 3 is a flow diagram of a method for sending selected pieces of data from a server computer to a client computer;

FIGURE 4 is a flow diagram of an alternative method for sending selected pieces of data from one or more server computers to a client computer; FIGURE 5 is a flow diagram of a method for selecting initial segments of strings from a given set of strings; and

FIGURE 6 is a flow diagram of an alternative method for selecting initial segments of strings from a given set of strings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIGURE 1 is a flow diagram of a method 100 for processing a digital image which has been or is being encoded into a format in which the (transformed) information is divided up into pieces called components, which are further broken up into parts called planes or bitplanes, in such a way that one can produce an approximation to the original image by using only some of the planes in each component. The planes in each

- component are generally ordered in such a way that the later planes are not very useful unless one also has the earlier planes. The method involves the solution to an optimization problem in order to select best (or nearly best) planes according to a constraint and a goal, either trying to attain the greatest image quality subject to a resource bound or trying to attain a specified image quality by using minimal resources.

The optimization may involve criteria, weights, or subimages which need not be specified or known when the image is originally encoded, so that the planes cannot simply be ranked in advance.

As shown in FIGURE 1, at a block 102, the method computes (or retrieves, if it was computed previously) a distortion for each plane, which is a number measuring or estimating the improvement in quality or fidelity of the processed image that would result from using the data in this plane as opposed to discarding said data. At a block 104, the method modifies these distortions to reflect user-selected criteria. At a block 106, the method runs an optimization algorithm such as (but not limited to) the ones described below with reference to FIGURES 5 and 6. This is done in order to select the best (or nearly best) collection of planes satisfying the given constraint.

At a block 108, the method collects or retrieves the data for the selected planes for storage, transmission, or use.

FIGURE 2 is a flow diagram of a method 120 for computing relevance coefficients. The method measures or estimates the extent to which the values in a component of an encoded form of an image are relevant to a particular subimage of the image. The encoded form involves a wavelet transformation or related linear transformation from pixel values to component values. The encoded image contains one or more components, and each component contains one or more values. Thus, each pixel value from the image can be expressed (at least approximately) as a linear combination of the component values with linear coefficients a(i ) where i indexes the value in the component and indexes the pixel.

As shown in FIGURE 2, at a block 122, the method computes (possibly through efficient formulas based on specific properties of the wavelet or linear transformation) the sum of a(ij) over all for the component and all for the subimage. At a block 124, the method computes (possibly through efficient formulas based on specific properties of the wavelet or linear transformation) the sum of a(ij)² over all / for the component and all / for the entire image. At a block 126, the method divides the first sum by the second sum.

FIGURE 3 is a flow diagram of a method 140 for sending selected pieces of data

(called planes) from a server computer storing a set of data to a client computer, where each plane has an associated benefit and an associated cost. FIGURE 3 shows the transmissions involved in the method (other than network protocol messages such as positive or negative acknowledgments or retransmissions). At a block 142, message(s) are sent from the client to the server requesting the benefit and cost information for the pieces of data which might be relevant. The message(s) specify for which planes this information is needed (this step may be skipped if the client already has sufficient benefit and cost information). At a block 144, message(s) are sent from the server to the client sending the requested benefit and cost information (if any), after which the client will perform an optimization algorithm such as (but not limited to) those described below with respect to FIGURES 5 and 6 to determine which planes to request.

At a block 146, message(s) are sent from the client to the server requesting specific planes. At a block 148, message(s) are sent from the server to the client sending the requested planes. At a block 150, the method performs repetition of some or all of the preceding steps as desired.

As an alternative to the steps shown in FIGURE 3, the optimization algorithm can be performed on the server, in which case the only transmissions (other than network protocol messages) are the request(s) from the client to the server (specifying the part of the data of interest to the client, possibly a resource restriction, and possibly a list of which planes the client already has) and the reply or replies sending planes from the server to the client. FIGURE 4 is a flow diagram of a method 160 for sending selected pieces of data (called planes) from one or more server computers storing a set of data to a client computer, where each plane has an associated benefit and an associated cost. FIGURE 4 shows the transmissions involved in the method (other than network protocol messages such as positive or negative acknowledgments or retransmissions). At a block 162, message(s) are sent from the client to a server, or from a receiving server to a sending server, requesting the benefit and cost information for the pieces of data which might be relevant. The message(s) specify for which planes this information is needed (this step may be skipped if the client or receiving server already has sufficient benefit and cost information). At a block 164, message(s) are sent from a server to the client, or from a sending server to a receiving server, sending the requested benefit and cost information (if any), after which the client or receiving server may perform an optimization algorithm such as (but not limited to) those described with regard to FIGURES 5 and 6, to determine which planes to request. At a block 166, message(s) are sent from the client to a server, or from a receiving server to a sending server, requesting specific planes. At a block 168, message(s) are sent from the sending server to the client or receiving server sending the requested planes. At a block 170, the method performs repetition of some or all of the preceding steps as desired.

Alternatively to the steps shown in FIGURE 4, the optimization algorithm can be performed on the sending server, in which case the only transmissions (other than network protocol messages) between these two machines are the request(s) from the client or receiving server to the sending server (specifying the part of the data of interest to the client or receiving server, possibly a resource restriction, and possibly a list of which planes the client or receiving server already has) and the reply or replies sending planes from the sending server to the client or receiving server.

FIGURE 5A is a flow diagram of a method 180 for selecting initial segments of strings from a given set of strings. Each of the given strings is sometimes referred to as a "component", and the objects making up the strings are sometimes referred to as "planes." Each initial segment has an associated cost as well as an associated benefit. The method involves the solution to an optimization problem in order to select best (or nearly best) planes according to a constraint and a goal, either trying to attain the greatest total benefit subject to a bound on total cost or trying to attain a specified total benefit at minimal total cost. As shown in FIGURE 5A, at a block 182, the method computes an incremental cost and incremental benefit for each plane. In an alternate embodiment, the cost and benefit information may already be presented in that form. At a block 184, the method finds for each component an initial "conglomeration" (substring with benefit/cost ratio no lower than that of any proper initial segment of the substring) with maximal benefit/cost ratio. At a block 186, the method sets up a data structure (such as a priority queue) containing the information about the conglomerations and their benefit/cost ratios. At a block 188, the method performs a conglomeration evaluation subroutine, as will be described in more detail below with regard to FIGURE 5B. At a block 190, the method terminates when the queue is empty, or optionally in other circumstances, such as when the desired total benefit is reached, or when a conglomeration which exceeds the remaining available cost is reached (if it is not very important to use up the available cost as completely as possible).

FIGURE 5B is a flow diagram of a conglomeration evaluation subroutine 200 which is repeated as many times as required. At a block 202, the subroutine takes from the queue or data structure a conglomeration with greatest priority or benefit/cost ratio. At a block 204, if the conglomeration can be selected (without exceeding the specified total cost bound, or if the benefit bound has not yet been reached), then the routine selects the conglomeration (i.e., extend the selected substring for this component to include the conglomeration), and, if this conglomeration does not reach all the way to the end of the component, the routine determines the next conglomeration from the so-far-unselected planes in the component and add it to the queue or data structure. At a block 206, if the conglomeration cannot be selected, but it consists of more than one plane, then the routine determines the proper subconglomeration of the conglomeration with maximal benefit/cost ratio and add it to the queue or data structure. This final step, at block 206, is optional.

With regard to the methods of FIGURES 5 A and 5B, instead of working directly with conglomerations, one may simply assign to each plane a modified benefit/cost ratio equal to that of the conglomeration it is assigned to, and then work with planes rather than conglomerations throughout the algorithm. Also, in the case of a benefit goal, one may take only part of the final conglomeration if that will achieve the desired benefit.

FIGURE 6A is a flow diagram of a method 220 for selecting initial segments of strings from a given set of strings, with each initial segment having an associated cost as well as an associated benefit. The method involves the solution to an optimization problem in order to select best (or nearly best) planes according to a constraint and a goal, either trying to attain the greatest total benefit subject to a bound on total cost or trying to attain a specified total benefit at minimal total cost. As shown in FIGURE 6A, at a block 222, the method sets up a finite list of benefit/cost ratio values (each actual benefit/cost ratio will be converted to a value in this list), and initializing one or more numerical tables indexed by the list.

At a decision block 224, if there is a benefit goal, the method proceeds to a block 226. At block 226, a table stores, for each value in the finite list, the total benefit of all "conglomerations" examined so far with converted benefit/cost ratio equal to the determined value. The conglomeration may be defined as discussed above with respect to FIGURES 5A and 5B.

At a decision block 228, if there is a cost bound, the method proceeds to a block 230. A table stores, for each value in the finite list, the total cost of all conglomerations examined so far with converted benefit/cost ratio equal to the determined value.

At a block 232, a table updating routine is performed, as will be described in more detail below with regard to FIGURE 6B. At a block 234, the method uses the table(s) to compute a threshold benefit/cost ratio such that all conglomerations with converted benefit/cost ratio greater than the threshold will be selected, and all conglomerations with converted benefit/cost ratio less than the threshold will not be selected. At a block 236, the method goes through all components again (recomputing the benefit/cost and/or conglomeration information if it was not saved from the previous pass), selecting all conglomerations with converted benefit/cost ratio greater than the threshold, and as many of the conglomerations with converted benefit/cost ratio equal to the threshold as can be chosen within the cost bound, or as many as are needed to meet the benefit goal.

FIGURE 6B is a flow diagram of a table updating routine 250. At a block 252, the routine finds the cost and benefit information for the component, and partitions the component into conglomerations, each of which has a benefit/cost ratio which is converted as above. At a block 254, the routine uses this information to update the table values to include the component.

With regard to FIGURES 6 A and 6B, instead of working directly with conglomerations, one may simply assign to each plane a modified benefit/cost ratio equal to that of the conglomeration it is assigned to, and then work with planes rather than conglomerations throughout the algorithm. Also, in the case of a benefit goal, one may take only part of the final conglomeration if that will achieve the desired benefit. SPECIFIC EXAMPLE OF A COMPRESS-TO-QUALITY OPTION

The quality option Q is a user option that controls the quality of the image (0 <= Q <= 1). Compressing to a target quality is an alternative to compressing to a target file size. Setting Q=l results in a lossless image, whereas >setting Q=0 results in "no quality", i.e., a black image resulting from no planes of image data.

The quality option is defined as follows. The mean squared errors (MSEs) for the bit planes obtained by the MG3 encoding process are scaled based on the user frequency balance and luminance weighting settings. The Q setting controls how many of these planes are kept to form the lossy compressed file. In particular, the ratio of the sum of scaled MSEs for the kept planes to the sum of scaled MSEs for all the planes of the original image is required to be no less than Q. The code chooses planes to keep based on assigning an importance value to each plane, sorting the planes on this key, and keeping the planes with highest importance. To apply the Q parameter, enough planes are kept (based on the importance rank) so that this MSE condition is just satisfied, and all others are discarded.

We are concerned with the problem of finding "efficient" computational solutions for the Rate-Distortion problem as follows (using slightly different notation.) Suppose we have a set of n blocks {B_\, . . . , B_n} which comprise an image. For each block, B_{, a separate bit-stream is generated with the property that it can be truncated to a variety of discrete lengths, j RJ, R_r- ,... . Associated with each length is a measure of distortion D_j that the image would incur if R_z- was truncated to length R . Given a user-specified size for the image, R_maz, we wish to find the set of truncation lengths, denoted {x\,..., x_n} (one for each block) such that:

is minimized subject to the constraint that:

x -_t. ≤ <- R ι_„ i=\ The problem can be seen as a variant of the "Knapsack Problem" a famous NP-Hard problem. The MG3 optimizer is a heuristic based on priority queues to approximate the optimal truncation point which works very well on the Rate-Distortion problem instances generated by certain image processing methods. An "off-line" variant can be used to compute the truncation points when the size of the R and B arrays are too large to fit in main memory using only two passes through these arrays.

The following sections provide a detailed description of selected optimizer methods in accordance with the present invention. These methods are also described in part in provisional U.S. Application Serial No. 60/250,825, filed December 1, 2000, which is commonly assigned and hereby incorporated by reference in its entirety. GOALS OF THE OPTIMIZER IN ABSTRACT FORM.

The discussion may begin by assuming that there is a collection of objects called SBCs, each of which is composed of parts called planes. Let «g be the number of planes in SBC B. For each SBC, there is a choice of taking all, some, or none of its planes. Each plane has a certain cost (in length or storage space), and provides a certain amount of benefit (reduction of distortion, or change in MSE). The planes in each SBC come in an ordered list - a particular plane cannot be taken unless all preceding planes in the list are taken. So the only choice for each SBC is how many of its planes to take. Let c(B,k) be the cost of taking (the first) k planes from SBC B,. and let p(B,k) be the benefit of taking these k planes.

A first goal would be to obtain the maximum benefit given an upper limit M on the cost. That is, choose a A^. for each B so that Σβp(B,k_B) is maximized subject to Σ_Bc(B,kβ) < M. A second goal would be to obtain a specified benefit P at minimal cost. That is, choose a k_β for each B so that _Hβc(B_kβ) is minimized subject to Σβp(B,kβ) ≥ P. These are dual versions of the same goal.

In designing methods to accomplish these goals, the following assumptions are made. Costs and benefits are additive over the different SBCs. Each plane has a positive cost. (A plane with zero cost and zero benefit can basically be ignored by the algorithm, although one has to take care in the programming to handle it.) For a given SBC B, taking all of the planes gives the maximum benefit: p(B,nβ) is the largest of the numbers p(B, k). (If this is not the case, then the algorithm will never want to take the plane(s) at the end, and it can simply discard them and reduce .) Note that there is no assumption that each plane has a positive benefit; it may be that taking a particular plane is harmful, but it may be desirable to take it in order to get a large benefit from the following plane.

With the above assumptions, there are certain general equations for accomplishing the above goals. For goal l, an equation is Σβc(B,0) < M. For goal 2, an equation is

Σβp(B,nβ) ≥ P. A perfect solution is generally not required. As will be described in more detail below, is a solution comes within one SBC or so of the optimal benefit or cost, that will suffice for many applications. Finding a perfect solution is difficult; it is easy to reduce the knapsack problem to this one. KNOWN ALGORITHM - "LAGRANGE MULTIPLIERS".

There is a known algorithm which has some resemblance to the calculus method of Lagrange multipliers. The idea is to choose a parameter λ > 0 (the desired benefit-to- cost ratio) and select planes so as to maximize the benefit minus λ times the cost.

Since this combination is linear, the maximization can be done for each SBC separately: it is desirable to choose kβ so that p(B,kβ) - λc(B,kβ) is maximized. This can be done in the obvious way, or by the following algorithm: set kβ to 0; for k from 1 to n_B do: if

p(B,k) -p(B,k_B) _{≥ ι} c(B,k)-c(B,k_B)

then set kg to k.

Let C(λ) and P(λ) be the total cost and benefit resulting from this maximization.

Then any selection with total cost < C(λ) must have total benefit < P(X), so this solves goal 1 if M happens to be C(λ); and any selection with total benefit > R(λ) must have total cost > C(λ), so this solves goal 2 if P happens to be P(λ).

Note that decreasing λ may cause more planes to be selected for a given SBC, but it cannot cause fewer planes to be selected. (If kβ planes were selected for SBC B, then p(B, k) - λc(B, k) ≤ p(B, kβ) - λc(B, kβ) for all k < kβ; since c(B, k) < c(B, kβ), and decreasing λ to λ' will give p(B, k) - λ'c(B, k) < p(B, kβ) - Vc(B, kβ), so the method cannot select k < kβ planes for λ'.) So the method must have C(λ') > C(λ) for λ' < λ, and it follows that P(V) ≥ P(λ) (additional planes will not be selected if they do not produce any incremental benefit).

Now the above maximization is performed for various values of λ, using a binary search approach to find a λ as small as possible such that C(λ) < M (for goal 1) or P(λ) > P (for goal 2). The resulting selection will be optimal for a slightly different bound, and it should be close to optimal for the actual bound.

OPTIMAL ORDERING.

The goal is to obtain the greatest benefit for a given cost, or to pay the least cost to obtain a given benefit. In either case, the best planes to keep will be the ones with the highest ratio of benefit to the cost.

If we could simply select the planes in decreasing order of benefit-to-cost ratio until the goal was reached, we would get a solution to the problem quite easily.

However, this does not quite work, because of the restriction that planes in a SBC must be taken in order. It might be desirable to take a plane with low benefit-to-cost ratio in order to be able to take the next plane, having a very high benefit-to-cost ratio.

In a typical case, the points (c(B, k), p(B, k)) will lie on a curve which is concave downward: the first plane provides the greatest benefit, the second one provides a smaller benefit, and so on (perhaps decreasing exponentially), while the costs of the planes might not vary much, so the benefit-to-cost ratio decreases for later planes in the list. The cases where the benefit-to-cost ratio of a later plane is greater than the benefit-to-cost ratio of an earlier plane are less typical, but they do occur and must be dealt with.

To handle these exceptions, we allow the possibility of treating several consecutive planes as a single conglomeration. This conglomeration would have a combined cost, a combined benefit, and a combined benefit-to-cost ratio. To see which planes should be grouped into conglomerations,, consider a graph of the points (c(B, k),. p(B, k)) and select out those points which are on the upper convex hull (so the curve connecting the selected points is concave downward, and the unselected points line on or below this curve). If the point (c(B, k),p(B, k)) is not on this convex hull, then we do not want to select exactly k planes from this SBC (unless we have to); we are taking the i plane in order to get to a more desirable plane, so the / th plane should be in the same conglomeration as this later plane.

Given a SBC B, to find the values k_t of /c such that (c(B, k),p(B, k)) is on the upper convex hull of the cost-benefit curve, proceed as follows. First let k₀ = 0. Given k_{ < n , k_i+ι is that k > k_t such that (p(B, k) -p(B, k)) l(c(B, k) - c(B, k_j)) is maximized. (If there is a tie, it means that some cost-benefit point is at an intermediate location on a straight line segment in the convex hull. It does not matter much whether such a point is selected as a convex hull point; probably not selecting it will save some work and rounding problems. So k_t should be chosen to be the largest such k in case of a tie.)

Now, for each /, planes numbered / _z- +1 through k_i+ι (assuming the planes are numbered starting at 1) are to be treated as a conglomeration. This conglomeration has the combined benefit-to-cost ratio (p(B, k_i+ ) -p(B, k_t)) I (c(B, k_i+_) - c(B, k;)); for each from kf+1 to k_i+γ, define the modified benefit-to-cost ratio λ_Bj- to be this combined benefit-to-cost ratio. Conceptually, this just moves all of the cost-benefit points up to the convex hull curve. In the typical case where plane number / is in a conglomeration by itself, λβ is just the beneflt-to-cost ratio of this plane. - We should now have λβ ≥ λβ ^ ≥ — ≥ AB,n_B- If this fails because of the vagaries of floating-point arithmetic, then the values should be adjusted, because it is important for the rest of the algorithms that these inequalities hold. These values λ_Bj- can be treated as measures of importance for the planes. One can show that the planes selected by the Lagrange multipliers method for the value λ are precisely those planes (BJ) for which λ_B ,• > λ.

Suppose that the planes are presented in an order so that the planes in each SBC occur consecutively in the proper order. If we now sort these planes in decreasing order of λ_B , using a stable sort so that planes in a single conglomeration are kept together in the proper order, then in the sorted file the planes of a SBC will still occur in the proper order. Now each of the Lagrange multipliers solutions will just be an initial segment of this sorted file. Furthermore, if we truncate the sorted file after any conglomeration, we will get a file which is a type of optimal solution as described below.

. If we let S be a set of planes obtained by taking an initial segment of the above sorted list of conglomerations, and if we let C and P be the total cost and total benefit of S, then it can be stated that any set S of planes with total cost < C will have total benefit <

P. To see the proof of this statement, let λ be the value of λ_B ,- for the last plane

(conglomeration) in the chosen initial segment. We get from S to the alternate set S" by deleting some planes from S and then adding some other planes.

If a conglomeration U is broken into two parts U_\ and I , with U_\ coming before L 2 in the ordering of the SBC, then the benefit-to-cost ratio for U_ must be no greater than the benefit-to-cost ratio of U as a whole. (If the benefit-to-cost ratio of U_\ were greater, then U would not be a conglomeration at all.) And since the benefit-to-cost ratio of U is a weighted average of the benefit-to-cost ratio of U\ and the benefit-to-cost ratio of U2, as demonstrated by the formula

Pl +Pl _^ ^cl Pi , ^c2 Pi q +c₂ q +c₂ q q +c₂ c₂ '

the benefit-to-cost ratio of U2 must be as least as great as the benefit-to-cost ratio of U. The deleted planes all have modified benefit-to-cost ratio at least λ, and since the deletion must be from the end of any conglomeration that is partially deleted, the actual benefit-to-cost ratio of each deleted conglomeration or partial conglomeration is at least λ. The added planes all have modified benefit-to-cost ratio at most λ, and since the addition must be from the start of any conglomeration that is partially added, the actual benefit-to-cost ratio of each added conglomeration or partial conglomeration is at most λ. And the total cost of the additions is no more than the total cost of the deletions. Putting this together, we find that the total benefit of the additions is no more than the total benefit of the deletions, so the total benefit of S¹ is no greater than the total benefit of S.

This allows more truncation points than the Lagrange multipliers method (if there are multiple conglomerations with the same benefit-to-cost ratio), so we can come closer to the exact cost or benefit bound for the problem. We still will probably not hit the bound precisely, though. For goal 1, this means that we take as many conglomerations as we can (in sorted order) within the cost bound; there will be a little space left over, but not enough to take the entire next conglomeration. Since most conglomerations will be single planes, we probably have come within a single plane of the cost bound, with optimal benefit. This is where we assume that a perfect solution is not required; a difference of one plane or a few planes from optimality (and these are not the most important planes, but planes less important than the ones already accepted) is not vital.

The following describes how the remaining allowable cost may be used. If the first conglomeration not taken contains multiple planes, then we can take part of the conglomeration if this will yield some additional benefit (we should not do so if the resulting partial conglomeration has negative benefit). Or one can look farther in the sorted list to try to find blocks to fill up the remaining cost.

If one only uses a truncation of the sorted file, then one will get the following idempotence property: optimizing for a cost bound of M and then cutting down further to a cost bound M < M yields the same result as just applying the algorithm with cost bound M in the first place. This will no longer hold if one looks later in the sorted file to use up the last bit of available cost (skipping over some planes that are too expensive).

For goal 2 one should just truncate the sorted file when the desired benefit is reached; it will not matter if this occurs in the middle of a conglomeration. While this may not be perfectly optimal, the expense of making it perfect may be more than is desirable. (This will yield the idempotence property.) PRIORITY QUEUE METHOD.

An alternative to simply computing the value λ_B ,• for each plane and sorting based on this value is to set up a priority queue which will produce the conglomerations of planes in the proper order. Each entry in the queue will specify: a SBC B; the current value k of k_B (in some embodiments, there is no need to store this in the queue entry); a proposed new value k > t, and the benefit-to-cost ratio (p(B, K) -p(B, k))/(c(B, k) - c(B, k)). This last number will be the priority. Abbreviate the above data as "B(k -> k)"; this entry indicates that planes k + 1 through k should be added to the list of kept planes.

To initialize, first set all values k_B to 0; then perform the following step for each

SBC R: λ <- 0 for k = k_B + 1 to n_B: (*) if (p(B, K - p(B, k_B))/(c(B, K) - c(B, k_B)) ≥ λ then: λ <- (p(B, V) -p(B, k_B))/(c(B, k) - c(B, k_B)) k* ^ k if λ > 0 then put B(k_B -» k*) on queue. This procedure determines the conglomeration starting at kβ + 1 (in this case, the first conglomeration in the SBC) and puts it on the queue. Note that conglomerations which produce no benefit are never put on the queue.

Now, to solve goal 1, iterate the following: Take the front entry B(k — » k) from the queue. If c(B, k) - c(B, k) ≤ M ^' - Σ _B c(B , k_B ) (i.e., we can afford the new planes), then set kβ to k and apply (*) to block B. One could terminate the iteration at the first time that an entry fails the cost test. This will yield exactly the same results as sorting the planes according to λ_B , and truncating as soon as one reaches a conglomeration of planes that will not fit. In particular, it will give the idempotence property. However, one can do a better job of using the last remaining cost by continuing the iteration until the queue is empty. (During this last 'endgame' phase, one must add an extra test in the loop in (*) to stop when c(B, k) - c(B, k_B) exceeds the remaining cost available. Also, when the newly obtained entry B(k -_> K) from the queue fails the cost test and k > k + 1 , one should immediately apply (*) to B again, thus putting on the queue a partial conglomeration that might be useful.) This will generally do a better job than searching through the later parts of the sorted list for planes to fill up the cost remaining. In general, it will not be idempotent. (For goal 2, different methods may be used; such as taking the conglomeration which completes the desired benefit total, or just part of that conglomeration if that is all that is needed, thus getting the same result as from sorting by ^λBJ-)

In summary, the Lagrange multipliers method, sorting method, and priority queue method have some similarities. The primary differences between the results from them are: (1) the Lagrange multiplier method cannot separate planes with the same modified benefit-to-cost ratio (using some of them but not others; when there is not room for all of them, the Lagrange multipliers method will keep none of them); (2) the priority queue method is better at filling in the remaining space at the end (particularly if allowed to run until the queue is empty). As for algorithm speed, it will depend on how the priority queue is implemented. The Lagrange multiplier method may be particularly slower depending on how many iterations it uses.

TWO-PASS EXTERNAL METHOD ASSUMING LIMITED BENEFIT-TO-COST RATIOS.

One may be in a situation where the number of planes is so large that one cannot keep a single piece of data for each of them in main memory at once. In this case, the sorting method cannot be applied directly without an external sorting algorithm (which may require a large number of passes through the data). It is possible (if the number of

SBCs is much smaller than the number of planes) that the priority queue will fit in main memory, but the method will still be infeasible because one will need random access to the numbers p(B, k) and c(B, k) in order to update the queue. There is another algorithm which will work well on such huge files if we assume that the number N of possible modified benefit-to-cost ratio values λβ is limited, so that we can store one or two tables of length N in main memory. This is not an unreasonable assumption, because the computation of the benefit values p(B, k) could be uncertain enough that we might as well keep a limited number of bits of precision in computing them and the corresponding values λβ . Or it may be that slight differences in λβ are not important enough to matter, and we can therefore group these values into a reasonable number of ranges and treat the planes in each range as having the same λβ . Basically, if this assumption holds, then we can use a bin sorting method to sort the planes into optimal order. However, we will not actually perform this sorting operation (which would still require a large number of passes through the file). Assume that the file is arranged so that planes for a particular SBC occur consecutively in the correct order. Create a table T where T [λ] is the sum of the costs of those planes (B, j) for which λβ = λ. This takes one pass through the file of planes. Now a second pass can be used to solve the problem.

For goal 1 : Find the critical value λ such that

.

Let x = M - _λ> T[λ] . Now, in the second pass through the file, when plane (B,j) is reached: keep it if λβ j > λ ; discard it if λβ < λ ; if λβj = λ , keep the plane if its cost is < x (and, in this case, subtract this cost from x), discard it otherwise. One may want to take care to avoid breaking a conglomeration of planes having the critical value λβ = λ ; or one can go ahead and include part of this conglomeration to use up the remaining small cost available. (This is the same decision one must make for the internal sorting algorithm. Again one should avoid taking a partial conglomeration of negative , benefit. Note that this means that the processing must be done an SBC at a time, rather than a plane at a time.)

For goal 2, the algorithm is similar. Now T[λ] is a sum of benefits rather than a sum of costs, and λ is chosen so that

∑ T[λ] < P ≤ ∑ T[λ] . λ>λ λ≥λ

Let x = P - T

. In the second pass, keep any planes with λβj > λ and discard any planes with λg ,• < λ ; if λβ = λ , then keep the plane if x > 0 (and subtract the benefit of the plane from x), discard it otherwise. There is a third task which one may also want to perform: number the planes according to where they would occur if the planes were actually stably sorted in decreasing order of λβj. (One will not actually sort the planes.) To do this, in the first pass, compute a table Tγ where Jifλ] is simply the number of planes (B, j) with λβ = λ. Then, from T_λ , get a table Ti so that

λ >λ

(It is possible to do this computation in place, using the same storage for T₂ as for T\, which is no longer needed.) Now, on the second pass through the file, when plane (B, j) is reached, it is assigned the rank number T₂_λβj], and this table entry T₂[λβ _j_ is then incremented by 1. (The resulting rank numbers will start with 0.) Or one can keep track of both T and T₂ if one wants to solve goal 1 (or 2) and compute rank numbers at the same time.

ALGORITHMS WITH EXPENSIVE COST COMPUTATIONS.

One may be in a situation where computing the costs c(B, k) is quite expensive, but the benefits P(B, k) are easier to compute. The external algorithm from. the above two-pass external method requires computing the cost (and benefit) of every plane in the file twice. There may be ways to reduce this cost computation.

It is obvious that one can simply save the computed costs in a temporary file for reuse on the next pass. If such a temporary file is undesirable, at least some of the cost recomputation can be saved by using the following method. Along with the table T, create a table T_ where one stores for each a the smallest total benefit (for this plane and all following planes in the SBC) for a plane with benefit-to-cost ratio equal to λ. Then, once the critical value λ is determined, compute the minimum of T _λ] over all λ > λ and call it w. Now, on the second pass, when a plane is reached, again compute the total benefit for this plane and all following planes in the SBC; if the resulting value is less than w, then the plane could not possibly have benefit-to-cost ratio as large as λ , so it can be discarded immediately, with no need to compute its cost again.

But this still requires computing the cost of every plane once. It would be highly desirable to configure the method so that some planes can be discarded without the need to go through the complete cost computation at all for them. This could be of benefit for both the internal and external algorithms.

For the internal version, we can use the priority queue method with 'on-demand' cost evaluation. For each SBC, we will be putting its first conglomeration into the queue immediately; at this time, we only need the costs of the planes in that conglomeration. When a conglomeration is taken out of the queue, some other conglomeration from that SBC will be put into the queue (if there are any untaken planes remaining in the SBC); the cost of the new conglomeration can be computed at this time. If we proceed along these lines, then only planes which have actually been put into the queue should require cost computations; this would mean that costs would be computed only for the kept planes and one additional conglomeration (most commonly just one plane) per SBC.

One issue with this is how to determine what the first or next conglomeration in a SBC is without computing the costs of all planes in the SBC. One certainly will need to know the cost of at least the next plane beyond the conglomeration in order to determine that it is not actually in the conglomeration.

In actual practice, it is often possible to determine that a conglomeration has ended without computing the costs of all of the later planes in the SBC. This can be done because we know the benefits of all of the planes in the SBC, and we may be able to determine that later planes cannot be in the conglomeration no matter how low their costs are. Specifically, suppose we are trying to determine the conglomeration in SBC B that starts after plane k , and we have seen that taking planes k + 1 through k gives the best benefit-to-cost so far. If the number k > k is such that

p{B,k) -p(B,k) p(B,k) -p{B,k)

(WP) -=X- < c(R,/c') -c(R,/c) c{B,k)-c(B,k)

then we know that the desired conglomeration does not end exactly at plane k. But if we have the stronger property

p(B,n_B)-p(B,k) _< p(B,k)-p(B,k) c(B,k')-c(B,k) ^< c(B,k)-c(B,k) then we know that the weaker property (WP) will hold for all / " > K, because p(B, k") - p(B, k ) < p(B, n_B) - p(B, k ) and c(B, k") - c(B, k ) ≥ c(B, k¹) - c(B, k ). So the conglomeration must have ended before plane k, and since k was the best location found up to that point, plane k must be the end of the conglomeration. Because the benefits of planes tend to drop exponentially in the applications of interest, it is quite common for this strong condition to occur after only one plane beyond the conglomeration has been examined. Hence, the on-demand priority queue algorithm will require that costs be computed for the planes actually kept, plus one conglomeration and a small number (typically 1) of additional planes in each SBC; this still means that many cost computations will be avoided. (Of course, if possible, any costs computed while determining a conglomeration should be saved for when the next conglomeration in this SBC is needed.)

It is harder to apply the on-demand method in the external algorithm, because one is not allowed to go back to an earlier SBC and compute costs of additional planes later. One is limited to the case where one can deduce solely from the information already obtained that it is not necessary to compute the costs of some planes. Nonetheless, one can save a substantial number of these computations.

To do this, one can keep track during the first pass of the provisional critical benefit-to-cost ratio value λ , as well as the current sum s = T ^>J^[^] • For goal 1, we want λ to be as large as possible so that s ≤ M; for goal 2, we want λ to be as large as possible so that s < P. Initially, λ = -∞ (or some other impossibly low value) and s = 0. When the next new conglomeration is processed, with benefit-to-cost ratio λ, the table T and these new values are updated as follows, for goal 1 [goal 2]: add the cost [benefit] of the new conglomeration to T[λ] ifλ > I then: add the cost [benefit] of the new conglomeration to -? while ,s > [s > R] do: increase λ to the next possible value subtract T_λ _ from s Now, if one reaches a conglomeration with benefit-to-cost ratio less than the current value of λ , then one can stop processing the current SBC and move on to the next one.

One can analyze how much of the total cost computation is saved by this. If we assume that the SBCs are independently and uniformly varying (a questionable assumption in practice, but it should be close enough for a rough estimate), and the bound M or P is such that we will end up keeping about a fraction 1/r of the planes, then the fraction of planes that we will compute costs for and consider keeping (because their benefit-to-cost ratio values are at least as great as the current λ ) is about

1 + lnr

To see this, note that for the first 1/r of the file we will be (provisionally) keeping everything, and once we have processed a fraction xlr of the file (x > 1) we expect to keep about 1/x of the current SBC, so the total fraction of provisionally kept planes is about

However, this underestimates the number of cost computations, because, as mentioned for the internal algorithm, we have to compute enough costs to find the first conglomeration in the SBC that we will not keep (or at least enough to determine that this conglomeration must have. benefit-to-cost ratio less than λ ; this will often require the cost of one additional plane, not the entire next conglomeration). Let c be the expected number of extra planes for which we will need to compute costs, divided by the expected total number of planes in the SBC. Then c is the expected fraction of excess cost computation, and the total fraction of the cost computation needing to be performed is

1+lnr + c . r

Actually, this overstates the total fraction, because it counts the fraction c even for those early SBCs for which we performed the entire cost computation (or more than 1 - 1/c of it); a more precise computation gives the total fraction

l+ln(r(l -c)) ^ ' + c . . The above was the fraction of cost computations in the first pass. If we assume that nothing is saved from the first pass except T (and the derived values λ and s), then the fraction of planes for which we must compute costs on the second pass is approximately (1/r) + c. At the other extreme, it is conceivable that we have room in memory to store all of the costs and associated data for the planes which might possibly be kept (those for which the benefit-to-cost ratio is at least equal to the current provisional λ ). These should be placed in a priority queue set up so that planes with low benefit-to-cost ratio have high priority; this will make it easy to discard the correct planes when λ is increased. (A calculation involving s and the current T[λ _ will tell us whether we need to keep the data for a new plane with benefit-to-cost ratio exactly equal to the current λ ) Once the first pass is done, the remaining planes can be sorted back into the proper order and sent to the output file. This means that no second pass will be needed.

An intermediate case is where we have enough memory to store the benefit-to- cost ratio values of the currently kept planes, but not enough to store the entire set of data associated with the planes. Iii this case, a second pass will be needed, but only the planes we are actually keeping (a fraction 1/r of the total number of planes) will need recomputation.

A different approach may be available, if one has an inexpensive way to estimate the cost of a plane. In this case, one can use a three-pass external algorithm. In the first pass, one computes a table T_e which is just like the table T from the ordinary external algorithm, except that estimated costs are used instead of ordinary costs to compute the benefit-to-cost ratio values. Using this table and an adjusted cost limit M_e, one gets a threshold λ _e on estimated benefit-to-cost ratio. The value M_e should be substantially larger than the actual cost limit M (for example, five times as large); the choice of M_e depends on the accuracy of the cost estimation algorithm and the tradeoff between running time for the algorithm and nonoptimality of the results.

On the second pass, each SBC is examined again, and those planes with estimated benefit-to-cost ratio less than λ _e are skipped. The planes of estimated benefit-to-cost ratio exactly equal to λ _e are skipped for now.) For the remaining planes, the actual costs are computed, and these are used to construct the table T from the ordinary external algorithm. Then T and the actual cost bound M are used to get a new threshold λ . Finally, on the third pass, those planes with estimated benefit-to-cost ratio less than λ g are skipped again; for the remaining planes, the actual benefit-to-cost ratio is computed, and the plane is kept if this is greater than λ (and some planes with actual benefit-to-cost ratio equal to λ are kept, depending on the remaining cost available). This algorithm only requires full cost evaluations for a fraction of the planes in the file; this fraction depends on the choice of M_e (and hence on the quality of the cost estimator). As given above, these cost evaluations are done twice, and they could be saved in a temporary file during pass 2 for reuse during pass 3 if desired.

Of course, requiring three passes through the file may be a drawback in some implementations even if not every plane is fully processed. Some time can be saved by terminating the first pass prematurely; after, say, a third of the planes have been examined, it may be possible to extrapolate from the tables produced so far (e.g., by simply multiplying all the totals by 3) to get a table _e from which λ _e can be approximated. Since M_e is an inflated version of M, an approximate value for λ _e may be sufficient to make the second pass work. However, it is often desirable to compute the table T_e again on the second pass, and get the real λ _e. If this turns out to be so much lower than the approximated λ _e that not enough planes were examined on the second pass, it will be necessary to redo the second pass. The trade-off in this method is that it is sometimes required to perform 3-1/3 passes in certain cases in order to only require 2-1/3 passes in more typical cases. The on-demand and estimation methods may also be combined for reducing cost computations in the external algorithm.

ALLOWING ZERO COST OR INFINITE BENEFIT.

Recall an earlier assumption that every plane has positive cost. The method may also address the possibility of planes of zero cost, if it is checked for when forming conglomerations and computing benefit-to-cost ratio ratios. Once the values λβ_j have been computed, the optimization algorithms should work without difficulty even if some of the values are infinite (perhaps stored as the largest possible floating-point number, perhaps as an IEEE infinity).

The place where care is needed is in algorithm (*) from the priority queue method, where the next conglomeration after plane kβ is determined. Suppose that, after plane kβ, the next m planes have zero cost (but then comes either a nonzero-cost plane or the end of the SBC). If there is i ≤ m such that ρ(B, kβ + /) >p(B, kβ), then find such an i so t aip(B, kβ + i) is as large as possible; the next i planes form a conglomeration with infinite benefit-to-cost ratio. (This should happen only if kβ = 0; if there were previous planes, then these i planes would have been included in the preceding conglomeration.) Now mark these i planes as having λβj = ∞, and put B(kβ — kβ + i) on the queue if the priority queue method is being used. If there is no such /, then the algorithm (*) may be run with the loop starting at kβ + m + 1 instead of kβ + 1 to find the next conglomeration.

A slightly different situation is where we have planes that have been assigned infinite benefit, instead of (or in addition to) planes with zero cost. This should not be taken too literally; if we had an infinite-benefit plane and a plane of finite but positive benefit p, then there would be no reason to take the latter plane, because we get the same benefit without it (∞ + p = ∞). Instead, "infinite benefit" should be thought of as a flag indicating that this plane absolutely must be kept. All of the algorithms for goal 1 can handle this situation as well if configured correctly.

As a side note, one does not really need to handle this case in the algorithms themselves; one can take the infinite-benefit planes and their predecessors "off the top," reduce the available cost accordingly, and proceed with the algorithm for the finite- benefit planes. But this is an extra step, possibly requiring an additional pass in the external algorithm, and it is easier to handle this case in the algorithms instead.

First, one cannot work with the cumulative benefits p(B, k), since one cannot subtract ∞ from ∞. Instead, one will have to directly compute λβj = ∞ for any plane of infinite benefit and for any plane which comes before a plane of infinite benefit in a SBC. Once this is done, no further changes should be needed for the sorting algorithm or the external algorithm. For the priority queue algorithm, one just needs to make the appropriate change in (*): once K reaches a plane of infinite benefit, put B(k — » k) on the queue with priority ∞ and stop. Of course, the program will produce an error message if the total cost of the infinite-priority planes alone exceeds M.

Goal 2 does not make sense with infinite-benefit planes. Instead the method pulls out the required planes in advance, and goals are set as to how much benefit one wants from the remaining planes, and proceeds from there. ADJUSTMENTS FOR SLIGHT NONADDITIVITY OF COST. It may be that the costs associated to planes are not quite additive — the cost of taking two planes is not exactly the sum of the separate costs of the planes. For the planes within a single SBC, this causes no problem; for the purposes of these algorithms, we consider the 'cost' of the second plane to be the total cost of taking the first two planes minus the cost of taking only the first plane. It is when costs of planes in separate SBCs are honadditive that difficulties occur.

However, it may be that the nonadditivity is slight enough that the priority ordering is not significantly affected. (In particular, we will assume that the actual incremental cost of taking an additional plane is always positive.) In this case, the algorithms for goal 2 will not be affected at all; the only difference is that the final total cost of the chosen planes will be slightly different from the sum of the individual costs.

Goal 1 is a little different. If the target cost Mis a 'soft' bound (it is acceptable for the total cost to be roughly that much, and a slight overrun is also acceptable), then again the algorithms can be run without change and see what the accurate total cost is later. But if M- is a 'hard' bound which must not be exceeded, more precise methods must be utilized.

As long as we can maintain a running count of the current total cost, the sorting algorithm and the priority queue algorithm can be run without requiring modification. In the sorting case, one simply needs to use accurate cost computations when deciding where to truncate the sorted file. In the priority queue algorithm, we need to keep track of the current actual cost and what the new actual cost would be if a conglomeration of planes were added, and use that when deciding whether to keep the conglomeration which is next on the queue. The external algorithm requires additional consideration, since there will be no way to determine the actual incremental cost of a plane or conglomeration of planes at the time it is processed. One way to modify the algorithm is as follows. Instead of using the second pass to determine completely which planes are kept and which planes are deleted, one can determine that some planes will definitely be kept, some will definitely be deleted, and some are optional planes that will be kept or deleted depending on how the total cost actually works out. The number of such optional planes needed will depend on how accurately the actual total costs can be estimated. For each optional plane, one will have to keep track of enough information to compute the actual incremental cost of adding that plane later. After the second pass is complete, one can look at the actual cost of the definitely kept planes and then go through the list of optional planes (in priority order) to see which of them can actually be kept. (The number of optional planes should be small enough that this step can be performed in main memory.) Now a third pass will produce the actual output file. (If the optional planes themselves, rather than just information about them, can be kept in main memory, then one may be able to avoid a third pass through the huge input file, instead just combining the appropriate optional planes with the kept planes output in the second pass.)

If the estimates turn out to be too far off, so that even keeping all of the optional planes does not use up the cost available (or there is a cost overrun even if none of the optional planes are kept), then one may need to rerun the second pass with adjusted bounds.

Note that the amount of information needed for each optional plane may depend on a number of considerations, such as whether the SBCs in the output file are to be stored in the same order as they were in the input file. (It is quite likely that the input file is so large that rearranging it in a convenient order is infeasible, while the output file will be small enough to be reordered, but such reordering will slightly change its cost; the optimizer will need to know the output order in order to know how many planes can fit in.) MORE GENERAL SBC STRUCTURE.

Up until now we have been assuming that there is a linear ordering of planes within each SBC and that we cannot take later planes without having taken the earlier planes in the SBC. In other words, the set of options from which we have to choose for each SBC is linearly ordered (and normally rather small). It turns out that we can consider the more general case where each SBC has an arbitrary (but reasonable-sized) list of options, and reduce it to the linearly-ordered case. Suppose that SBC B has a list of options k, each with an associated cost c(B, k) and benefit p(B, k). We may make various assumptions, such as that there is a do-nothing option with cost and benefit both 0 and all other options have positive cost, but these assumptions are not really needed.

Some of the options can be eliminated immediately. If we have two options k and k such that c(B, k) > c(B, k) and p(B, k) ≤ p(B, k), then we will never prefer option k over option k, so we may as well discard option K. (If two options have the same cost and the same benefit, we can choose arbitrarily which one to discard.) Once this is done, the remaining options can be listed in increasing order of cost, which will be the same as increasing order of benefit. Now the algorithms of the preceding sections can be applied. The only real limitation here is that the entire list of options must be produced and examined (and sorted into increasing order of cost; once this is done, it is easy to find and discard the unwanted options). So this would not be feasible if, say, the SBC consisted of 64 planes and any subsets of the planes could be chosen.

One case where this more general situation is applicable is the following. It was stated above in the section related to "Adjustments for Slight Nonadditivity for Cost" that, if costs are highly nonadditive, then the situation is beyond the scope of the algorithm.

But suppose the SBCs come in pairs {B_\, B₂} such that choices in Ri affect the costs in

R2 a d vice versa, but SBCs not in the same pair do not affect each other (i.e., their costs add together as usual). Then we can consider each such pair to be a super-SBC; each option for a super-SBC is a pair of numbers, the number of planes to keep in B\ and the number of planes to keep in R2- So the number of options for the super-SBC is (n_B. +1)

(n_B2 +1); this should be manageable if the numbers n_Bl and w^ ^are reasonably small.

Hence, this limited-coupling system can be handled by our algorithms. (For the external algorithm, coupled SBCs should be adjacent in the input file.) The same might work if the SBCs that affect each other come in triples or maybe quadruples. However, we cannot go much beyond this before the number of options per super-SBC becomes very high. As an extreme case, if there is no clean separation at all between the SBCs, so that they all affect each other directly or indirectly, then all of the options get put together into one huge list and there is nothing left for the optimizer to do (except choose the single option of greatest benefit within the cost bound). ADDITIONAL BENEFIT-TO-COST RATIO CONSIDERATIONS.

In certain embodiments, the benefit is a reduction in total squared error (MSE) for the image, as computed from the differences between the correct wavelet coefficients and the approximate coefficients resulting from removal of bit planes. (This computation relies on the assumption that the wavelet coefficients are orthogonal linear combinations of the original pixel values; which produces reasonable results.) This MSE can be adjusted in various ways to change the relative priorities of the planes; as will be described in more detail below in the section "MSE Adjustments."

The cost of a plane is the length of its compressed form, including associated data (length field, MSE field, etc.). This measure is slightly inaccurate when skip markers are used to indicate the gaps between included planes. One can partially correct for this by adding 1 to the cost of the first plane in each SBC. The reason is that, when this plane is being added, it normally lies between two planes that have not yet been added: the next plane in the current SBC, which certainly has not been added yet, and the last plane from the previous SBC, which probably has not been added yet if we haven't even started adding planes from the current SBC. In fact, one can guarantee this if the plane number of this first plane is strictly between the minimum and maximum possible values. So instead of one skip marker including this plane, there will now be two skip markers, one before the plane and one after, for a net increase of one skip marker, which is commonly one byte long.

If, for some reason, there is a gap in the planes in an SBC (some are present, then some are absent, then some more are present), then this gap will entail a skip marker, so one should add 1 to the cost of the plane immediately after the gap. (This skip marker should definitely be small enough to fit in one byte.)

If one knew that the SBC after the current one had the maximum possible number of planes, then one would want to subtract 1 from the cost of the last plane in the current SBC, because adding this plane would cause a skip marker to disappear. But since SBCs rarely have the maximum number of planes, this correction is usually inappropriate. (In fact, in certain formats, it is impossible for SBCs to have the nominal maximum of 32 planes. This is because in certain formats the plane number 0 represents the sign bits of the numbers being coded, but the signs are in fact not coded in a separate plane; the sign of each number is put in the plane containing the first significant bit of that number.) This cost computation will still not be completely accurate (because of skip markers that are actually longer than one byte, and other factors). The section above on "Adjustments for Slight Nonadditivity of Cost" and the section below on "Exact Cost Computations" provide more discussion on how these types of problems may be addressed. REGIONS OF INTEREST.

The use of weighted benefits allows an easy way to handle region-of-interest (ROI) coding for goal 1 : fix a factor r > 1 and multiply the benefit of each plane in the ROI by r before computing the benefit-to-cost ratio values and applying the above algorithms. (Goal 2 would need to be restated in order to make sense with ROI.) It should be noted that this is quite different from the JPEG 2000 method which shifts bit planes relative to each other.

For planes which consist of coefficients of which some belong to the ROI and others do not, one can apply the factor r to only the coefficients belonging to the ROI before adding the pixel MSE reductions together to get the MSE reduction for the plane. So a plane which is partially in the ROI will get a partial promotion; this will lead to a smooth drop in image quality at the edge of the ROI (like anti-aliasing).

Further refinements may also be implemented, as a coefficient in a higher-level subband might represent or be associated with several pixels in the original image, only some of which are in the ROI. In this case, one could prorate the factor r (e.g., arithmetically or geometrically), although in some configurations it is better to give the full magnification factor to all such entries, thus giving extra consideration to the entire ROI. A precise method for deciding how much of a coefficient should be deemed to apply to a particular region is described in more detail below in the section "Relevance Coefficients for the Local Optimizer," but simpler allocation methods (such as seeing which pixels are related to the coefficient and counting what fraction of them are in the ROI), will often suffice here.

An extreme case of the above is when we want to make sure that there is no loss of quality at all in the region of interest; this can be handled by allowing the factor r to be infinite. The resulting infinite benefit values can be handled by the methods of the preceding section "Allowing Zero Cost or Infinite Benefit."

It is also possible to have regions of disinterest (ROD), where accurate rendering is less important or perhaps completely irrelevant. (For instance, the pixels in holes left by certain masking techniques.) This can be handled by the same approach as above, using a multiplier r which is less than 1 (perhaps even 0). The only difference is that, in this case, if a coefficient applies only partially to the ROD, the factor r should not be applied in its entirety to the coefficient; either the factor should not be applied to that pixel at all, or one of the more precise proration methods should be used. MORE ON EXPENSIVE COST COMPUTATIONS.

In the preceding section "Algorithms With Expensive Cost Computations," an algorithm was described which used estimated costs instead of actual costs on the first pass through the data, and only computed the actual costs for those planes with a high enough estimated benefit-to-cost ratio. One possible way to estimate the cost (i.e., the length after arithmetic coding) would be an entropy computation. If all bits were being coded together, then this computation would be quite simple: if the plane contained a 1 bits and b 0 bits, then the estimated coded length would be (a + b)log ( + b) - alog₂a - blog b.

However, since the bits are actually divided into contexts with separate probability tables, one must count the 1 and 0 bits in each context, apply the above formula, and add the results. Unfortunately, separating the bits into contexts is almost half of the computation required for the exact cost evaluation, so this estimation method does not generally save much computation time.

Another possible approach is to use statistical modeling. One could examine a wide variety of images, and for each plane in these images, compute four quantities: the fraction (from 0 to 1) of pixels in the subblock which become newly significant in that plane, the fraction of pixels which became significant one plane ago, the fraction which became significant two or more planes ago, and the exact cost of the plane (divided by the number of pixels in the subblock). In some implementations, it is probable that there will be a strong enough correlation between these four numbers that a suitable linear combination of the first three numbers will serve as a good estimator for the fourth. MSE ADJUSTMENTS.

As mentioned earlier, the MSE values initially computed for the planes can be adjusted in various ways. This will affect the relative priorities assigned to the planes, and hence will cause the optimizer to make different choices. Such adjustments can improve the resulting image quality if they produce an error measure which is closer to human-perceived image difference (or to the difference considered to be important by the user) than raw MSE is.

One simple example of this is an adjustment which multiplies all MSEs for planes in a certain color band by a constant. If this constant is greater than 1, then this color band will be emphasized; if the constant is less than 1, the color band will be considered less important. For instance, if the color bands are Y, I, Q (luminance and two chrominance bands used by NTSC), then the Y band MSEs in most instances should be scaled up, because luminance differences are more easily perceived than chrominance differences. (It may be useful to give I more weight than Q as well. If the two chrominance bands are weighted equally, then there is little or no difference between the various alternative chrominance spaces such as YIQ, YUV, YCbCr, and so on.)

Or one can adjust the MSEs to alter the balance between low-frequency (high- level) and high-frequency (low-level) subbands. Giving more emphasis to low-frequency subbands causes more accuracy to overall color levels in large parts of an image, while giving more emphasis to high-frequency subbands will cause rapid variations or boundaries in the image to be rendered more accurately.

This last point is worth examining in more detail, because it demonstrates one way in which MSE is not the best measure of perceptual difference. As mentioned above, the high-frequency subbands are important in rendering both rapidly varying image areas (e.g., textures) and sharp image boundaries. Now, if we have a subblock which contains a sharp image boundary between two relatively constant colors, then the only high-frequency coefficients in the subblock which will be large are the ones near the boundary. Hence, if the subblock is an m X m array of coefficients, then only about cm (for some constant c) of the m² coefficients will have large values, and the MSE for planes of this subblock will be relatively small.

On the other hand, in a subblock with rapidly varying color values throughout (e.g., due to noise, texture, etc.), most or all of the high-frequency coefficients will have large values, and the planes will have very high MSE values. This does not reflect the relative perceptual importance of these subblocks; inaccuracies in rendering a sharp image boundary will be highly visible, while much greater errors in rendering the noisy or textured area are more likely to go unnoticed.

Modelling perceptual differences quantitatively is a difficult task. But the above anomaly can be reduced by a simple MSE adjustment: instead of using the actual MSE reduction computed for the plane, use a 'nominal' MSE value computed by assuming that the coefficients in the subblock are random (and hence not depending on the actual image data at all). So all coefficients in a subband will get equal weight, and the undesired preference for textured subblocks over sharp-boundary subblocks will disappear; the main effect will be to improve rendering of boundaries in "less-busy" areas of the image, at the expense of slight reduction in quality for rapidly- varying or "busy" areas.

This nominal MSE value is easy to compute. For any particular coefficient, knowledge of bit number k from the bottom of this coefficients reduces the interval of possible values of this coefficient from one of length 2^k+l to one of length 2 If the estimated value of the coefficient without this bit is the center of the longer interval, the estimated value with the bit is the center of the shorter interval, and the actual value is at some random location in the shorter interval, then the expected improvement in squared error from keeping the bit is

So the nominal MSE for plane number k from the bottom is 2² 4 times the number of coefficients in the subblock times the subband multiplier (high-level subband coefficients are associated to large regions of image pixels, and hence get greater MSE multipliers). Note that, although corresponding planes in subblocks in the same subband will get the same nominal MSE, this does not mean that they will get the same priority; they will usually have different costs (coded lengths) and hence different benefit-to-cost ratio values.

Still, one may not want to ignore the actual MSEs altogether. Instead, one can take a weighted average of the nominal MSE and the actual MSE to use in the optimization. The weight w_n to be used for the nominal MSE can take any value from 0 to 1 ; w_n - 0 gives the actual MSEs, while w_n = l gives fully nominal MSEs. Note that it might be desirable to use a value of w_n which is slightly less than 1 rather than exactly equal to 1, because this allows the actual MSEs to be used as tiebreakers in the optimization.

It may appear that using the nominal MSE as computed above (assuming random noise at all pixels) will give excessively high values at all planes, because, as noted before, if the image really does have such high variation, then errors are much harder to perceive. It might be desirable to scale all of these nominal MSE values down by a constant factor. Of course, this will have no effect on the results if w_n = 1, because the relative values of the nominal MSEs will not be changed by such a global scaling. But even for intermediate values of w_n, such scaling does not have any effect except to remap the scale of w_n values. Using weight w_n with a global scale factor of C on the nominal

MSEs generally gives the same results as using a weight w„C/(l - w_n + w_nC) with no scale factor, because

w_n (Cx)A-(l-w_n)y = (l-w_n

The effects of varying w_n are difficult to predict with precision. This is in part because the effect depends on the nature of the image. This nominalization is intended to reallocate the priority between busy and less-busy areas of an image. Hence, if the entire image is fairly busy, then it will take a large change in w_n to produce any noticeable difference; but if there are large non-busy areas separated by sharp boundaries, then smaller changes in w_n will have larger effects. However, there are heuristic reasons to believe that the sensitivity of the optimization to the parameter w_n will be much greater near the endpoints of the interval

[0,1] than in the middle of the interval. If we have some planes whose nominal benefit- to-cost ratio values are quite different but whose actual benefit-to-cost ratio values are about the same, then w_n will only need to be slightly larger than 0 for the ordering of the planes to come out in the way given by the nominal values. Similarly, for planes with close nominal benefit-to-cost ratio values but quite different actual benefit-to-cost ratio values, a value of w_n which is even slightly less than 1 will cause the resulting plane priorities to come out in the order given by the actual values.

So instead of using a user-input value w from the interval [0,1] directly as w_n, it might be better to let w_n =f(w), where /is chosen to map its argument more toward the sensitive areas near 0 and 1, and thus cause variation of w to produce effects more uniformly. For instance, we can takey(x) = x² (3 - 2x), which has (0) = 0, f \) = 1, / strictly increasing from 0 to 1, and/(0) = (l) = 0. Or one could use a higher-degree polynomial to get even more derivatives to be 0 at the ends. RELEVANCE COEFFICIENTS FOR THE LOCAL OPTIMIZER.

When a region or scene within an image is being decoded, some SBCs provide data entirely within the region, some provide data partially within the region, and some are disjoint from the. region. If we want to run the optimizer specifically for that region, then the benefit values for the planes should be weighted by a factor or coefficient which tells how relevant the plane is to the region.

Another application for these relevance coefficients is to editing of images that are encoded in certain selected formats. If one wants to replace the pixels in a small region of an image which is already in a compressed selected format, then one should be able to do so by replacing only the SBCs containing modified data. But then one has to assign modified MSEs to planes which are computed from a combination of both old pixels and new pixels. One has an old MSE for this plane and a new MSE which is correct as regards the replaced pixels but is not as reliable with respect to the old data, because the exact old pixel values can no longer be reconstructed. Possibly the best way to come up with a combined MSE in this situation is to use a weighted average of the old and new MSEs, with the weight for the new MSE being the relevance coefficient for this plane and the modified region.

We can measure this relevance as follows. Suppose that each coefficient in a particular SBC (coefficients indexed by / ) decodes to a particular linear combination of pixel values, with coefficients a(i , j ) (where j indexes the pixels). Then the coefficient of relevance for a plane in SBC B to a region R is

(As usual, this equation generally relies on the assumption that the wavelet coefficients are orthogonal, even though this assumption is somewhat inaccurate.) This looks like a very large computation, but it can be made much smaller in the case of a wavelet which is simply the composition of a wavelet in the x-direction and a wavelet in the -direction. Both SBCs and image regions (as well as the full image) are assumed to be rectangular. In these circumstances, each value a(i,j ) can be expressed in the form a_x(i_xJ_x)a_y(i_y, j_y), and the entire expression (tt) can be expressed as the product of an x part and a y part:

So the relevance coefficient is the product of an x-relevance coefficient and a _y-relevance coefficient, each of which is computed in terms of a one-dimensional wavelet, SBC, and region. One still has to compute sums of the form

In most cases, a(i, j) will have the form f J - 2^Lϊ), where L is the level number (0 for the original image, 1 for the coefficients, after one wavelet iteration, etc.) of the SBC and w is the subband number (identifying both the level number and whether this is a low- frequency or high-frequency subband at that level). The exceptions occur when i is near the left or right boundary for the subband, so that boundary conditions for the transform become relevant; this happens for only a small fixed number of /'s at either end of the subband.

For a fixed w and i, if we already have a table of the cumulative sums of a(i, j)², then ∑ . α(z, /) can be computed in constant time as the difference of two entries in this table. Suppose this has been done for the boundary /'s; then it remains to compute

Let \p, q] be the support off_w. We can try evaluating the above sums, using stored tables of ∑ . f_w and ∑ . jf_w , using a discrete version of the formula

l fa l fc f(x-y) ' dy dx = l\a-d l(τna__;(c, 'a-z ^Z) f(z) ' dy dz

=j (min ( , b - z) - max (c,α -z))/(z) cfe ,

but this will need step functions like jl2^L_ to handle the 2^L step size. A different version which works better here is: to get

first trim down the interval [α, b] to ensure that c - 2^L α ≤ q and d - 2^Lb ≥p (the remaining values of i contribute 0 to the sum), and compute

b d

Σ ∑ Σ∑ _w(y -2^Z )² =(b-« + l)S_w-g_w(c-l-2 ^: _G) + g_w(c-l-2^I (b + l)) i=αj=c

where S_w = ∑_j ^q _=pf_w(j) ,

(so g_w(j) = 0 for/ < p), and

(so h_w(j) = 0 for / > q). The values S_m g_w and h_w can be precomputed (one number and two arrays of length q - p).

There may be large cancellations (and resulting inaccuracies) here if the scene R is small compared to the interval [p, q_. However, the interval [p, q_ is usually at most a small multiple of 2^L in length. So in

most t's give either nothing or the entire S_w; only a couple of exceptional z's give the partial sum, and these can be handled individually (using a precomputed table of ∑ ). One still needs to precompute^ for each subband; this can be done by straightforward polynomial multiplications (with due care for boundary cases). If L = 0, then the wavelet coefficients are just the pixel values themselves, so f_w(j) is 1 for j = 0 and 0 for y^' ≠ 0. Now, let > (z) and q__(z) be the generating functions for thej^'s of the low-frequency and high-frequency subbands at level L, respectively. If the polyphase matrix for the inverse wavelet transform is

^f «a({zz)) bb((zz))^ c(z) d z) then we have

P_l (z) = a z²]A-zc z²],

and, forE ≥ 1,

PL+I (^Z) = PL (^Z2)PI (^Z)>

In the special case of the standard 5-3 wavelet, we have a(z) - 1 and c(z) = (l+z^~ l)/2, sop\(z) = 1 + (z + zr^l)/2. It turns out that doing one inverse wavelet with all 0's in the high subband (an alternative to interpolation of the low subband) for this wavelet is just linear interpolation of the low subband. So Pι(z) and q_L(z) are easily obtained by linear interpolation of p_(z) and q_\(z) by the factor 2^L . So, for all w, f_w can be easily computed rather than stored, and it is piecewise linear, so partial sums of f are just simple piecewise cubic functions. Even the boundary condition cases are relatively simple (reflection through an L point does nothing extra; reflection through an H point on the right just extends the coefficients by repeating the rightmost value). So one can write a constant-time routine to compute the relevance coefficients with no extra tables at all. If one must work with a nonrectangular scene, one can decompose it into a disjoint union of rectangles, compute the relevance coefficient for each rectangle, and add these coefficients up to get the total relevance coefficient for the scene. Actually, one can also use set differences in expressing the scene in terms of rectangles: if the scene R is a subset of the scene Rj, then the relevance coefficient for R\\R is the relevance coefficient for R\ minus the relevance coefficient for R . However, such subtractions might cause a slight loss of numerical accuracy. EXACT COST COMPUTATIONS.

As mentioned in the preceding sections "Adjustments for Slight Nonadditivity of

Cost" and "Additional Benefits-to-Cost Ratio Considerations," meeting a 'hard' total cost bound presents difficulties, because the exact cost of a plane is not independent of planes in other SBCs; one must account for the costs of skip markers before and after the plane in question, and these costs depend on the number of planes being skipped. It turns out that both the internal and external algorithms can be modified to account for these costs, thus allowing us to come as close as possible to the target cost without exceeding it.

The internal (priority queue) algorithm. When no planes are selected, the total cost is the cost of a single skip marker for the total number of planes. When a new plane or conglomeration of planes is selected, we find the skip marker which previously included that conglomeration, and replace it with the new conglomeration, a possible skip marker before (this will not be needed if the immediately preceding plane in the SBC was already selected), and a possible skip marker after. To compute the sizes of these skip markers, we simply have to find the previous and next SBCs for which we have already selected one or more planes. (For convenience, put fictional SBCs before and after the list of real SBCs, and mark the planes in the fictional SBCs as all, selected.) If the current SBC already has selected planes in it, then we do not need the previous SBC, only the next one. The only difficulty here is finding the previous and next 'touched' SBCs (SBCs with selected planes); we do not want to have to do a linear search through the SBC list each time a new SBC is touched. There is a usable approach for this which involves attaching only a single link to each SBC in the list (touched or not). For a touched SBC, the link will point forward to the next touched SBC, since that is all that is needed in order to compute the cost of adding planes to an already-touched SBC.

For an untouched SBC, we want a link that will take us backward to the previous touched SBC. (We can then follow the forward link from that touched SBC to get to the touched SBC following our original untouched SBC.) We do not want to require this to be a direct link, though, because then too many such links would need to be updated when a new SBC was touched (namely the links for all untouched SBCs between the newly touched SBC and the following touched SBC).

Instead, suppose that SBC number n is untouched, and let 2^k be the largest power of 2 dividing n. Then the link for SBC number n will point backward to the nearest touched SBC or to SBC number n - 2^k, whichever is nearer. (If SBC number 0 is untouched, then its link points to the fictitious touched SBC number -1.)

Now, if SBC number n is untouched, then the previous touched SBC can be found by following the chain of links starting at n until a touched SBC is reached, and one further link leads to the following touched SBC. This chain is of length at most 1 + log n (more precisely, 1 plus the number of 1 bits in the binary representation of ). And the link updates required when SBC number n is touched can also be performed in logarithmic time. If the previous and next already-touched SBCs are numbers mγ and m₂ respectively, then the link update algorithm is:

link[m_] <- n i r- n + \ while i < m do: link[i] — n i r- i +J\i)

Here/Ϊ) is defined to be the largest power of 2 dividing . One can easily compute + i) as (i I (/ - 1)) + 1, where I is the bitwise OR operation. (Similarly, i -fli) is i & (i - 1) where & is bitwise AND. This is useful when initializing the links.)

The external (two-pass) algorithm. For this algorithm, we will want to change the computation of the table T so that.J[λ] will be the exact incremental cost of taking the planes with modified benefit-to-cost ratio equal to λ (assuming we had already taken those with benefit-to-cost ratio greater than λ). To do this, we maintain during the first pass a list indicating which plane would have been the most recent kept plane, for all possible threshold values λ . This is maintained as a list of pairs

(λ n_λ), (λ₂, nj , ... , (λ£, njr),

where λ_j > λ₂ > . . . and n\ < n < . . . ; if the threshold were set so that all planes (B, j) with λβj > λ were kept, and λ_z- > λ ≥ λ_/+j (or i - k and λ^ > λ ), then the most recent kept plane would be plane number «_z-. The number λ^ will always be the highest possible benefit-to-cost ratio value. The list is initialized so that k = 1 and n\ - -1. In the first pass, we work with this list as follows. When a new SBC is processed, let μ be the benefit-to-cost ratio of its first conglomeration. Then we must add to the relevant table entries T λ_ the cost of a skip marker before this first new conglomeration.

Let m be the plane number (in the list of all planes, not just in this SBC) of the first plane in this new conglomeration. For each i such that λ < μ, if the threshold is set to a value λ such that λ < λ and (if / < k - 1) λ > λ +ι, then there will be a skip marker for m - nf - 1 planes before the new conglomeration. (If m - « - 1 = 0, then there is no skip marker.) The same applies if λ_z- > μ > λ and (if < k - 1) λ ≥ λ +j.

Let to be the largest i < k such that λ_z- > μ. The cost of a skip marker for m - n_iQ - 1 planes is added to T[μ\. And, for each / such that Ϊ_Q ≤ i < k - 1, if the cost of a skip marker for m - n_i+\ -1 planes is less than the cost of a skip marker for m - n_z- -1 planes, then the difference between these costs is subtracted from T[λ_i+_].

Next, the list is updated. All of the entries (λ;, «_r-) with λ,- < μ are deleted, and in their place (at the end of the list) are put entries (μø, mo), (μi, m_), . . . , where μ,- and m,- are the benefit-to-cost ratio and last plane number of conglomeration number j in the SBC. The costs of these conglomerations are added to the table entries J μ_z-] as usual.

Once all SBCs are processed, the cost of the terminating skip marker must be added in. The procedure for doing this is the same as if we were adding a fictional new conglomeration of highest possible benefit-to-cost ratio with plane number one more than that of the last real plane. Now the first pass and the table T are complete, and we can determine the threshold benefit-to-cost ratio value λ as usual.

Theoretically the length k of the new list could grow to as large as the number of possible values of λ. In practice, though, it should remain much smaller than this. After a new SBC is processed, the length of the list will be the number of conglomerations in the new SBC plus the number of list entries left over from previous SBCs because their first coordinates λ_z- are greater than the benefit-to-cost ratio of the first conglomeration in the new SBC. If we assume that the first conglomerations of the various SBCs tend to have much higher benefit-to-cost ratio than the remaining conglomerations (which is commonly true; the benefit-to-cost ratio values tend to decrease exponentially within a SBC), then the leftover list entries will usually be from first conglomerations of particular previous SBCs. Those particular SBCs are the ones for which, if all of the previous SBCs were examined in reverse order, the first conglomeration would have a record-setting benefit-to-cost ratio value (one greater than that for any previously examined SBC). The standard theory of records shows that, if the SBCs are assumed to occur in random order, then the number of record-setting SBCs is expected to grow no more than logarithmically with respect to the total number of SBCs. Even if the order is only partially random, the number of record-setting SBCs will tend to remain quite small. So the time needed to process the new list in the first pass will not be excessive. (The space needed for the list ^■ would not be a problem even if the list did grow to its maximum length, assuming tables of that length are already being maintained.)

In the second pass, all planes with benefit-to-cost ratio greater than λ will be kept. To see which of the planes with benefit-to-cost ratio equal to λ can be kept, we keep track of the following data:

• n, the plane number of the most recent preceding plane with benefit-to- cost ratio greater than λ ;

• b, a flag indicating whether a plane with benefit-to-cost ratio equal to λ has been seen more recently than plane n; • m, the plane number of the most recent such plane, if b is true;

• x, the amount of cost remaining available for planes with benefit-to- cost ratio equal to λ .

Initially n

• •

These values are updated during the second pass as follows. When a new conglomeration with benefit-to-cost ratio equal to λ is reached: compute the cost of the preceding skip marker

(back to n or, if b is true, back to m) let c be this cost plus the cost of the conglomeration if c ≤ then: x <r- x - c keep this conglomeration set b to true and m to the last plane number of the conglomeration When a new conglomeration with benefit-to-cost ratio greater than λ is reached: ifb is true then:

_Qtp be the plane number of the first plane in the conglomeration compute the cost of a skip marker from m top compute the cost of a skip marker from n top if these costs differ, add the difference to x set b to false keep this conglomeration set n to the last plane number of the conglomeration Following this procedure will ensure that a conglomeration with benefit-to-cost ratio equal to λ will be kept whenever it is possible to do so and still meet the cost bound (given the conglomerations one has previously decided to keep), except in one rare case. Let C_Q be the cost of the conglomeration. Let c_m be the cost of the skip marker that will be present at this location if we do not keep the conglomeration; let c_m' and c^. be the costs of the skip markers that will be present before and after the conglomeration if we do keep it. Then the net cost of keeping the conglomeration is c₀ + c_m' + c^. - c_m.

However, the algorithm above will keep the conglomeration only when the amount of cost remaining available is at least C_Q + c_m' . Hence, if c_m" < c_m, then the algorithm may reject the conglomeration even though it would in fact have fit. In order to avoid this, one would have to read ahead to the next conglomeration one might keep before making a decision on the current conglomeration, because c_m and c^- cannot be computed without knowing where the next conglomeration will be. The considerable extra complications involved in doing this (especially if the next conglomeration also has benefit-to-cost ratio exactly λ ) may not be considered worthwhile in some applications.

Note that this can lead to the following situation: we run the optimizer on an input file using a cost bound M, and get an output file of actual cost M where M is slightly less than M; but if we run the optimizer on the same input file with cost bound M , then the resulting new output file is not the same as the first output' file, but is slightly smaller. This will probably happen only for large input files (which are likely to have multiple conglomerations with the same value of λ). Note, though, that if we run the optimizer on the first output file using cost bound M, we will get that file back unchanged. More generally, if we run the optimizer on any input file using a cost bound which the input file already meets, then the output will be the entire input file. This is because the problem in the previous paragraph only applies ,when some but not all of the conglomerations with benefit-to-cost ratio equal to λ will (actually) fit within the cost bound. Of course, the above example computations can be improved significantly with exact cost computation algorithms.

While the preferred embodiment of the invention has been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the invention.

Claims

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method for processing a digital image which is encoded into a format in which the information is divided up into pieces called components, which are further broken up into parts called planes, in such a way that one can produce an approximation to the original image by using only some of the planes in each component; the method selecting planes according to a constraint and a goal, said optimization involving criteria such that the planes cannot be ranked in advance; said method comprising: computing a distortion for each plane, the distortion estimating the improvement in the processed image that would result from using the data in this plane as opposed to discarding said data; running an optimization algorithm in order to select a collection of planes satisfying the given constraint; and collecting the data for the selected planes.

2. The method of claim 1, wherein the distortions of the planes are modified in order to give additional preference to those planes affecting a region of interest in the image.

3. The method of claim 2, wherein relevance factors are used to compute the modified distortions for those planes which only partially affect the region of interest.

4. The method of claim 1, wherein only a part of the image is desired and those planes which are not relevant to the selected part of the image are discarded.

5. The method of claim 4, wherein those planes which are only partially relevant to the selected part of the image have relevance factors applied to their distortions.

6. The method of claim 1, wherein the distortions of the planes are modified in order to adjust the relative weights of the bands in the image or the bands in the encoded form of the image.

7. The method of claim 1, wherein the distortions of the planes are modified in order to adjust the balance between low-frequency and high-frequency wavelet subbands.

8. The method of claim 1, wherein the distortions of the planes are modified in order to make the relative weights of planes reflect visual criteria other than the actual computed distortion reduction that would be achieved by keeping the planes.

9. The method of claim 1, wherein the constraint is that the total cost of the selected planes must be at or below a given bound, and the goal is to maximize the total modified distortion reduction, of the selected planes.

10. The method of claim 1, wherein the constraint is that the total modified distortion reduction for the selected planes must be at least a given fraction of the total modified distortion reduction of all planes, and the goal is to minimize the total cost of the selected planes.

11. The method of claim 1 , wherein the cost of a set of planes is defined to be the size of an image file in a given file format containing just those planes.

12. The method of claim 1, wherein the cost of a set of planes is defined to be the size of a network transmission packet containing just those planes.

13. The method of claim 1, wherein it is permitted for a plane to have infinite distortion value or zero cost.

14. The method of claim 1, wherein it is permitted for a plane to have zero or negative distortion value.

15. The method of claim 1, wherein some planes are assumed to have already been obtained and therefore need not be included in the total cost.

16. The method of claim 1, wherein the planes in each component do not have a linear precedence ordering, wherein the problem is transformed to a modified form in which the alternative choices available in each component do have a linear precedence ordering.

17. The method of claim 1, wherein the storage or transmission format for the image is the proposed JPEG2000 image coding system or a related format.

18. The method of claim 1, wherein all relevant planes are selected, so that the pixels of the desired image can be reconstructed exactly from the encoded form.

19. A method for computing relevance coefficients which estimates the extent to which the values in a component of an encoded form of an image are relevant to a particular subimage of the image, said encoded form involving a transformation from pixel values to component values such that each pixel value from the image can be at least approximately expressed as a linear combination of the component values with linear coefficients a(i,j) where i indexes the value in the component and j indexes the pixel; said method comprising: computing the sum of a(i j) over all i for the component and all j for the subimage; computing the sum of a(ij)² over all i for the component and all j for the entire image; and dividing the first sum by the second sum.

20. The method of claim 19, wherein the transformation is equivalent to a composition of one-dimensional wavelets or linear transformations applied in the separate dimensions of the image, and the components and subimages are rectangles of length 1 or more in each dimension, or Boolean combinations of rectangles, so the computation can be reduced to a combination of one-dimensional computations.

21. The method of claim 20, wherein the one-dimensional wavelets are the standard 9-7 wavelet used in the proposed JPEG2000 image coding system.

22. The method of claim 20, wherein the one-dimensional wavelets in question are the standard 5-3 wavelet used in the proposed JPEG2000 image coding system, and special formulas applicable only to this wavelet are used to compute the relevance coefficients.

23. A method for sending selected planes from a server computer storing a set of data to a client computer, wherein each plane has an associated benefit and an associated cost; the transmissions involved in the method comprising: one or more messages from the client to the server requesting the benefit and cost information; one or more messages from the server to the client sending the requested benefit and cost information, after which the client will perform an optimization algorithm to determine which planes to request; one or more messages from the client to the server requesting specific planes; and one or more messages from the server to the client sending the requested planes.

24. The method of claim 23, wherein the cost constraint used when deciding which planes to transmit is computed in terms of network characteristics and/or limitations of the client and/or server.

25. The method of claim 23, wherein the client performs the computational work involved in selecting the planes to be transmitted and decoding them to usable form.

26. The method of claim 23, wherein the client maintains one or more caches of already-received data so that some or all previously-transmitted planes will not need to be transmitted again.

27. The method of claim 23, wherein data is transmitted progressively, with the client first obtaining an initial amount of data from the server and then, zero or more times, requesting additional data from the server, until all relevant planes have been sent or until the client ceases requesting additional data.

28. The method of claim 26, wherein an optimization method is applied to the plane cache in order to decide which planes to evict, with cost being computed in terms of storage space in the cache and perhaps with an adjusted distortion metric possibly involving the number of bits on cache entries.

29. The method of claim 23, wherein only a part of the total data is currently desired and those planes which are not relevant to this part of the data are neither requested nor transmitted.

30. The method of claim 29, wherein those planes which are only partially relevant to this part of the data have relevance factors applied to their benefit values for the purpose of selecting the best planes to transmit.

31. A method for sending planes from one or more server computers storing a set of data to a client computer, wherein each plane has an associated benefit and an associated cost; the transmissions involved in the method comprising: one or more messages from the client to the server, or from a receiving server to a sending server, requesting benefit and cost information; one or more messages from a server to the client, or from a receiving server to a sending server, sending requested benefit and cost information, after which the client performs an optimization algorithm to determine which planes to request; one or more messages from the client to a server, or from a receiving server to a sending server, requesting specific planes; and one or more messages from the sending server to the client or receiving server sending the requested planes.

32. The method of claim 31, wherein the cost constraint used when deciding which planes to transmit is computed in terms of network characteristics.

33. The method of claim 31, wherein the client maintains one or more caches of already-received data so that some or all previously-transmitted planes will not need to be transmitted again.

34. The method of claim 31, wherein a server which does not store the entire file may maintain one or more caches of already-received planes and/or benefit and cost information so that some or all previously-transmitted planes will not need to be transmitted to that server again.

35. The method of claim 33, wherein an optimization method is applied to the plane cache in order to decide which planes to evict, with cost being computed in terms of storage space in the cache and with an adjusted distortion metric involving the number of hits on cache entries.

36. The method of claim 31, wherein only a part of the total data is currently desired at the client and those planes which are not relevant to this part of the data are neither requested by nor transmitted to the client.

37. The method of claim 36, wherein those planes which are only partially relevant to this part of the data have relevance factors applied to their benefit values for the purpose of selecting the best planes to transmit.

38. A method for selecting initial segments of strings from a given set of strings, each string being a component, the object making up the strings being planes, with each initial segment having an associated cost as well as an associated benefit; said method involving the solution to an optimization problem in order to select planes according to a constraint and a goal, said method comprising: computing an incremental cost and incremental benefit for each plane; finding for each component an initial conglomeration with maximal benefit/cost ratio; setting up a data structure containing the information about the conglomerations and their benefit/cost ratios; repeating the following steps: take from the queue a conglomeration with greatest benefit/cost ratio; and if this conglomeration can be selected, then select the conglomeration, and, if this conglomeration does not reach all the way to the end of the component, determine the next conglomeration from the so-far-unselected planes in the component and add it to the queue or data structure.

39; The method of claim 38, wherein the constraint is that the total cost of the selected planes must be at or below a given bound, and the goal is to maximize the total benefit of the selected planes.

40. The method of claim 38, wherein the constraint is that the total benefit of the selected planes must be at or above a certain bound, and the goal is to minimize the total cost of the selected planes.

41. The method of claim 38, wherein costs of planes are computed or requested only on an as-needed basis, so that it will not be necessary to compute the costs of planes which are never considered for selection.

42. The method of claim 38, wherein it is permitted for a plane to have infinite benefit or zero cost.

43. The method of claim 38, wherein it is permitted for a plane to have zero or negative benefit.

44. The method of claim 38, wherein some planes are assumed to have already been obtained and therefore need not be included in the total cost.

45. The method of claim 38, wherein the total cost of a set of planes may differ from the sum of the costs of the planes taken individually, and adjustments are made to the costs and/or the benefit/cost ratios in order to partially compensate for this nonadditivity.

46. The method of claim 39, wherein the total cost of a set of planes may differ from the sum of the costs of the planes taken individually, and such deviations are carefully accounted for during the computation so as to ensure that the cost constraint is strictly obeyed.

47. The method of claim 38, wherein the priority assigned to a plane or conglomeration of planes is the ratio of benefit to cost for that plane or conglomeration.

48. The method of claim 38, wherein the priority assigned to a plane is the total benefit of all planes following that plane in the component (either including or not including the plane in question).

49. A method for selecting initial segments of strings from a given set of strings, where each of the. strings is a component, and each of the objects making up the strings are planes, with each initial segment having an associated cost as well as an associated benefit; said method involving the solution to an optimization problem in order to select planes according to a constraint and a goal, said method comprising: setting up a finite list of benefit/cost ratio values, and initializing one or more numerical tables indexed by this list: if there is a benefit goal, a table storing, for each value in the finite list, the total benefit of all conglomerations examined so far with converted benefit/cost ratio equal to that value; if there is a cost bound, a table storing, for each value in the finite list, the total cost of all conglomerations examined so far with converted benefit/cost ratio equal to that value; doing the following for each component: find the cost and benefit information for the component, and partition the component into conglomerations, each of which has a benefit/cost ratio which is converted as above; use this information to update the table values to include this component; using the table(s) to compute a threshold benefit/cost ratio such that all conglomerations with converted benefit/cost ratio greater than the threshold will be selected, and all conglomerations with converted benefit/cost ratio less than the threshold will not be selected; going through all components again, selecting all conglomerations with converted benefit/cost ratio greater than the threshold, and as many of the conglomerations with converted benefit/cost ratio equal to the threshold as can be chosen within the cost bound, or as many as are needed to meet the benefit goal.

50. The method of claim 49, wherein the constraint is that the total cost of the selected planes must be at or below a given bound, and the goal is to maximize the total benefit of the selected planes.

51. The method of claim 49, wherein the constraint is that the total benefit of the selected planes must be at or above a certain bound, and the goal is to minimize the total cost of the selected planes.

52. The method of claim 49, wherein costs of planes are computed or requested only on an as-needed basis, so that it will not be necessary to compute the costs of planes which one can guarantee will not be selected.

53. The method of claim 49, wherein it is permitted for a plane to have infinite benefit (meaning that this plane must be selected) or zero cost.

54. The method of claim 49, wherein it is permitted for a plane to have zero or negative benefit.

55. The method of claim 49, wherein some planes are assumed to have already been obtained and therefore need not be included in the total cost.

56. The method of claim 49, wherein the total cost of a set of planes may differ from the sum of the costs of the planes taken individually, and, adjustments are made to the costs and/or the benefit/cost ratios in order to partially compensate for this nonadditivity.

57. The method of claim 50, wherein the total cost of a set of planes may differ from the sum of the costs of the planes taken individually, and such deviations are carefully accounted for during the computation so as to ensure that the cost constraint is strictly obeyed.

58. The method of claim 49, wherein the amount of primary memory used by the algorithm is bounded by a fixed amount independent of the size of the input, and no secondary memory is. needed except for the input and output files.

59. The method of claim 49, wherein the finite list of values for the benefit/cost ratio is obtained by starting with the ratio in a floating-point format such as that given by the IEEE 754 standard and truncating or abbreviating the representation to a limited number of bits.