US5761341A  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform  Google Patents
Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform Download PDFInfo
 Publication number
 US5761341A US5761341A US08548943 US54894395A US5761341A US 5761341 A US5761341 A US 5761341A US 08548943 US08548943 US 08548943 US 54894395 A US54894395 A US 54894395A US 5761341 A US5761341 A US 5761341A
 Authority
 US
 Grant status
 Grant
 Patent type
 Prior art keywords
 image
 edge
 motion
 images
 embodiment
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Expired  Lifetime
Links
Images
Classifications

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
 H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
 H04N19/51—Motion estimation or motion compensation
 H04N19/537—Motion estimation other than blockbased

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
 H04N19/63—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using subband based transform, e.g. wavelets

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
 H04N19/39—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data
Abstract
Description
This invention relates to a compressive image encoding and decoding method using edge synthesis and the inverse wavelet transform, and to digital image encoding and decoding devices employing this method.
Compression is essential for efficient storage and transmission of digitized images. Compression methods have been described by the Joint Photographic Experts Group (JPEG) for still images, and the Motion Picture Experts Group (MPEG) for moving images. The JPEG method involves a discrete cosine transform (DCT), followed by quantization and variablelength encoding. The MPEG method involves detecting motion vectors. Both methods require substantial computation, the detection of motion vectors being particularly demanding.
Recently there has been much interest in the wavelet transform as a means of obtaining high compression ratios with relatively modest amounts of computation. This transform employs a family of wavelets related by dilation and translation; that is, the family consists of occurrences of the same basic wavelet at different locations and on different scales. If the scales form a progressively doubling sequence, and if the basic wavelet is zero everywhere except in a limited domain, wavelet transforms and inverse wavelet transforms can be carried out with efficient computational algorithms.
A wavelet transform can be described as a filtering process executed at, each wavelet scale. A digitized image, for example, is transformed by filtering with the basic wavelet, then with the basic wavelet dilated by a factor of two, then with the basic wavelet dilated by a factor of four, and so on.
One priorart wavelet encoding scheme employs a complementary pair of wavelets to divide an image into a highfrequency component and a lowfrequency component. These components contain information about variations on scales respectively less than and greater than a certain cutoff scale. This process is iterated on the lowfrequency component with a doubling of the wavelet scale, obtaining new lowfrequency and highfrequency components, then iterated again on the new lowfrequency component, and so on. After a certain number of iterations, the components are encoded by an encoding scheme that works from lowtoward highfrequency information. This scheme enables accurate image reconstruction, but retains too much highfrequency information to achieve high compression ratios.
Another priorart wavelet encoding scheme employs a basic wavelet that is the first derivative of a smoothing filter (that is, the first derivative of a lowpass filtering function). This type of wavelet acts as a highpass filter. Highfrequency information is obtained by detecting local peaks (local maxima of absolute values) in the result of the wavelet transform, which correspond to edges in the original image. The size and location of the peak values at a selected scale are encoded, along with a lowfrequency image obtained by smoothing at the largest scale of the wavelet transform. Fairly high compression ratios can be obtained in this way.
To reconstruct the original image from the encoded data, this priorart method employs an algorithm derived from a mathematical procedure involving iterated projections in Hilbert space. Under ideal conditions, the projections converge toward a unique set of data that (i) have the required local peak values and (ii) are within the range of the wavelet transform operator. An inverse wavelet transform is then carried out on the converged data to obtain the original image.
It has yet to be shown, however, that the projections always converge, or that data satisfying conditions (i) and (ii) are unique. In practice, there is difficulty in knowing when to stop iterating. For some images, it seems that data satisfying (i) and (ii) are not unique, and instead of converging, the iteration wanders endlessly through Hilbert space, first approaching the desired image transform, then moving away again.
A further problem of edgebased encoding schemes in general is that encoding efficiency tends to be impaired by edge fragmentation and irregularity.
It is accordingly an object of the present invention to encode digitized images efficiently, with a high compression ratio but without requiring extensive computation.
Another object of the invention is to encode moving images efficiently.
Still another object is to reduce irregularity of edges.
Yet another object is to avoid fragmentation of edges.
Still another object is to reconstruct digitized images from encoded information efficiently and reliably.
Yet another object is to reconstruct digitized moving images efficiently and reliably.
The invented method of encoding a digitized image comprises the steps of:
detecting edges in the digitized image;
encoding the position and sharpness of the detected edges to generate edge image information;
filtering and downsampling the digitized image to generate a reduced image; and
encoding the reduced image to generate reduced image information.
Detecting edges in the digitized image may comprise the further steps of thickening the detected edges, then trimming each edge to a uniform thickness.
The invented method of reconstructing a digitized image encoded by the invented encoding method comprises the steps of:
generating a set of decoded edge images from the above edge image information;
filtering these decoded edge images to generate a series of sets of decoded edge images that are increasingly smooth, or increasingly small in size;
synthesizing a series of sets of highfrequency images by filtering the decoded edge images with respective edge synthesis filters;
decoding the reduced image information to obtain a decoded reduced image; and
performing an inverse wavelet transform on the decoded reduced image and the above sets of highfrequency images.
FIG. 1 is a block diagram of the digital image encoder in a first embodiment of the invention.
FIG. 2 is a block diagram of the digital image decoder in the first embodiment.
FIG. 3 is a graph illustrating pixel values at an edge.
FIG. 4 is a block diagram of the digital image encoder in a second embodiment of the invention.
FIG. 5 is flow diagram illustrating the smoothing and downsampling process in the encoder of the second embodiment.
FIG. 6 is a block diagram of the digital image decoder in the second embodiment.
FIG. 7 is a flow diagram illustrating the synthesis of highfrequency images in the second embodiment.
FIG. 8 is a flow diagram illustrating the inverse wavelet transform in the second embodiment.
FIG. 9 is a block diagram of the digital image encoder in a third embodiment of the invention.
FIG. 10 is a flow diagram illustrating onedimensional filtering and downsampling in the third embodiment.
FIG. 11 is a block diagram of the digital image decoder in the third embodiment.
FIG. 12 is a flow diagram illustrating the synthesis of highfrequency images in the third embodiment.
FIG. 13 is a flow diagram illustrating the inverse wavelet transform in the third embodiment.
FIG. 14 is a diagram of an edge, illustrating irregular shape and fragmentation.
FIG. 15 illustrates thickening of the edge in FIG. 14 according to a fourth embodiment of the invention.
FIG. 16 illustrates trimming of the edge in FIG. 15 according to the fourth embodiment.
FIG. 17 is a block diagram of the digital image encoder in a fifth embodiment, for encoding moving images.
FIG. 18 is a block diagram of the digital image decoder in the fifth embodiment.
FIG. 19 is a block diagram of the digital image encoder in a sixth embodiment of the invention.
FIG. 20 is a block diagram of the digital image encoder in a seventh embodiment.
FIG. 21 is a block diagram of the digital image decoder in the seventh embodiment.
FIG. 22 is a block diagram of the digital image encoder in an eighth embodiment.
FIG. 23 is a block diagram of the digital image encoder in a ninth embodiment.
FIG. 24 is a block diagram of the digital image decoder in the ninth embodiment.
FIG. 25 is a block diagram of the digital image encoder in a tenth embodiment.
FIG. 26 is a block diagram of the digital image decoder in the tenth embodiment.
FIG. 27 is a block diagram of a digital image encoder/decoder in an eleventh embodiment of the invention.
FIG. 28 is a block diagram of a digital image encoder/decoder in a twelfth embodiment.
Embodiments of the invention will be described with reference to the attached illustrative drawings. The following terminology will be used.
A (twodimensional) digitized image is an array of pixels having values x(i, j), where the horizontal coordinate i and vertical coordinate j range over sets of integers. The range of these coordinates is the size of the image, i.e., the number of pixels in the horizontal and vertical directions. The pixel values represent, for example, intensity gradations.
A onedimensional image is a horizontal or vertical line in a twodimensional digitized image, i.e., a set of pixels with values x(i) indexed by a single coordinate.
A standard edge is a onedimensional image having just one sharp variation point. An example would be an image with pixel values that change first at one constant rate, then at a different constant rate, such as the following:
. . . 2.0 1.5 1.0 0.5 0 0.5 1.0 1.5 2.0 . . .
In this standard edge the pixel values first decrease at a rate of 0.5 per pixel, then increase at a rate of +0.5 per pixel. A standard edge serves as a basic model for all the edges occurring in a digitized image.
Downsampling a digitized image means reducing its size by a process generally analogous to photographic reduction: by selecting every nth pixel, for example, where n is a suitable integer. Upsampling means increasing the size of an image by interpolating new pixels between the existing pixels.
A (onedimensional) filter is a set of coefficients f(k), where k ranges over the integers. The nonzero coefficients f(k) are the taps of the filter.
The Fourier transform F of a filter f is defined as follows, where summation is over all taps, e is the natural logarithm base, and j is the square root of minus one. ##EQU1##
The conjugate f* of a filter f is obtained by reversing the order of coefficients:
f*(k)=f(k)
The Fourier transform of f* is therefore the complex conjugate of the Fourier transform of f.
A filter has even symmetry around i_{0} if it has equal values on both sides of i_{0} : that is, for all values of k:
f(i.sub.0 k)=f(i.sub.0 +k)
In particular, a filter has even symmetry around zero if it is equal to its own conjugate.
A filter has odd symmetry around i_{0} if it has opposite values on both sides of i_{0}, that is:
f(i.sub.0 k)=f(i.sub.0 +k)
A filter satisfies the exact reconstruction condition if its Fourier transform F satisfies the following condition for all values of ω:
F(ω).sup.2 +F(ω+π).sup.2 =1
Two fitters with Fourier transforms G and H satisfy the exact reconstruction condition if:
G(ω).sup.2 +H(ω).sup.2 =1
Filters satisfying these exact reconstruction conditions, also referred to as perfect reconstruction conditions, arc well known in the wavelet transform art. A pair of filters satisfying the second exact reconstruction condition above is referred to as a complementary pair.
To filter a onedimensional image x by a filter f means to obtain a new onedimensional image y with the following pixel intensities: ##EQU2##
This operation is equivalent to the conventional convolution operation using the conjugate filter f*. ##EQU3## The difference between convolution and filtering, as the term is used herein, is purely one of notation. The meaning of the following description and claims would be unchanged if the terms "convolution" and "convolving" were used throughout, instead of "filtering."
To filter a digitized image x horizontally by a filter f means to obtain a new image y as follows: ##EQU4##
Similarly, filtering an image x vertically by f means: ##EQU5##
Filtering can be done twodimensionally by, for example, filtering first in the horizontal direction, then in the vertical direction, or vice versa.
A highpass filter is a filter that retains smallscale variations and rejects largescale variations. A lowpass filter rejects smallscale variations and retains largescale variations. Smoothing filter is a synonym for lowpass filter.
The cutoff frequency of a highpass or lowpass filter indicates the scale that divides the retained variations from the rejected variations. A lower cutoff frequency corresponds to a larger scale of variations. Frequency actually refers to the variable ω in the Fourier transform F(ω) of the filter.
Generally speaking, the Fourier transform of a highpass filter satisfies F(0)=0, while for a lowpass filter, F(0)>0. If F(0)=1, a lowpass filter can retain largescale variations without changing their size. Incidentally, F(0) is the sum of the filter coefficients f(k)
First Embodiment
Referring to FIG. 1, the digital image encoder in the first embodiment comprises an input terminal 10, a twodimensional smoothing and downsampling section 20, and an edge detector 21. The twodimensional smoothing and downsampling section 20 comprises a cascaded series of twodimensional smoothing filters 22m (m=1 to n, where n is an integer greater than one), and a down sampler 23. The digital image encoder also comprises an edge image encoder 24, a reduced image encoder 25, and a multiplexer 26, which together constitute an encoding section 27.
A digitized image X_{0} is input from the input terminal 10 to the edge detector 21 and first smoothing filter 221. Smoothing filter 221 outputs a filtered image X_{1} to the next smoothing filter 222, and this process continues, each smoothing filter 2m filtering the image X_{m1} received from the preceding smoothing filter and furnishing a new filtered image X_{m} to the next smoothing filter, until the last smoothing filter 22n is reached. The last smoothing filter 22n outputs a lowfrequency image X_{n} to the down sampler 23, which outputs a reduced image R to the reduced image encoder 25. The reduced image encoder 25 encodes the reduced image R to obtain reduced image information Cr.
The edge detector 21 detects edge points in the input image X_{0} and outputs horizontal and vertical edge images Sh and Sv to the edge image encoder 24 in the encoding section 22. The edge image encoder 24 encodes these edge images to obtain edge image information Cs. The multiplexer 26 combines the reduced image information Cr and edge image information Cs into an encoded image C, which is output to a first input/output device 28 such as a communication channel, semiconductor memory, or magnetic disk storage device.
Referring to FIG. 2, the digital image decoder has a decoding section 29 comprising a demultiplexer 30, an edge image decoder 31, and a reduced image decoder 32. The digital image decoder also has a cascaded series of twodimensional smoothing filters 33m (m=1 to n1), a series of edge synthesizers 34m (m=1 to n), an up sampler 35, and an inverse wavelet transform processor 36. The inverse wavelet transform processor 36 comprises a cascaded series of inverse wavelet processors 36m (m=1 to n).
The demultiplexer 30 receives the encoded image C from the first input/output device 28 and separates it into edge image information Cs, which it supplies to the edge image decoder 3l, and reduced image information Cr, which it supplies to the reduced image decoder 32. The edge image decoder 31 decodes Cs to obtain a pair of horizontal and vertical edge images Sh_{1} and Sv_{1}. These decoded edge images are filtered in turn by the smoothing filters 33m (m=1 to n1), producing a series of pairs of edge images Sh_{m} and Sv_{m} (m=1 to n) in which the unfiltered edge images Sh_{1} and Sv_{1} are the first pair.
Each edge synthesizer 34m receives the corresponding pair of edge images Sh_{m} and Sv_{m}, and synthesizes a pair of horizontal and vertical highfrequency images Yh_{m} and Yv_{m} (m=1 to n). The reduced image decoder 32 decodes Cr to obtain a decoded reduced image R', which the up sampler 35 enlarges to obtain a decoded lowfrequency image X'_{n}.
In the inverse wavelet transform processor 36, each inverse wavelet processor 36m (m=1 to n) receives the corresponding highfrequency images Yh_{m} and Yv_{m} and a partially reconstructed image X'_{m} from the preceding inverse wavelet processor 36(m+1) in the cascade, and outputs a partially reconstructed image X'_{m} 1 to the next inverse wavelet processor 36(m1). The first inverse wavelet processor 36n in the cascade receives the decoded lowfrequency image X'_{n} from the up sampler 35. The last inverse wavelet processor 361 outputs the fully reconstructed image X'_{0} to a second input/output device 37.
Each of the elements in the FIGS. 1 and 2 comprises a memory for storing data and wellknown circuits for performing arithmetic and logic operations. Descriptions of the circuit configurations will be omitted to avoid obscuring the invention with irrelevant detail. The entire digital image encoder can be fabricated as a single semiconductor integrated circuit, or can be built from a small number of such circuits; likewise the digital image decoder. The invention can also be practiced by providing a generalpurpose processor such as a microprocessor or digital signal processor with memory for storing data and programs for executing the functions of the individual elements in FIG. 1 or FIG. 2.
Next the operation will be described starting with the operation of the encoder in FIG. 1.
The input image X_{0} is a digitized image with pixel values X_{0} (i, j). The edge detector 21 detects edges by detecting variations in the rate of change of pixel values from pixel to pixel. More specifically, the edge detector 21 takes differences between adjacent pixel values, then takes differences between these differences, and finally performs a thresholding operation to reduce small differences to zero. This process is carried out separately in the horizontal and vertical directions.
In the horizontal direction, for each pixel, the edge detector 21 computes the following pixel difference Pdh (i , j):
Pdh(i, j)=X.sub.0 (i, j)X.sub.0 (i1, j)
Next it computes differences Sh(i, j) of these pixel differences Pdh(i, j) as follows: ##EQU6##
Equivalent edge detection could be performed by filtering the image X_{0} with a secondderivative filter having coefficients (1, 2, 1), but taking differences of differences is preferable because it requires less computation.
The detected value Sh(i, j) is the edge sharpness at pixel (i, j), also referred to as the edge size. Edge sharpness values with absolute values equal to or less than a certain threshold value T are quantized to zero. Values exceeding T are also quantized, to reduce the number of bits in the edgeimage data; then the quantized sharpness values Sh(i, j) at all pixels are output as the horizontal edge image Sh. Nonzero pixels in the edge image, i.e. pixels at which Sh(i, j)>T, are referred to as edge points.
The meaning of the edge sharpness computation can be seen in FIG. 3. The horizontal axis represents, for example, the horizontal direction in the image, and the vertical axis represents the pixel value. The difference between pixel values 38 and 39 is A, the difference between pixel values 40 and 38 is B, and the difference between these differences is B (A)=A+B. If A+B>T, then pixel 38 is an edge point with sharpness A+B. The other pixels in FIG. 3 have zero sharpness. FIG. 3 thus illustrates a standard edge with just one sharp variation.
The vertical edge image is generated in the same way, by taking differences in the vertical direction, then taking differences of these differences. The edge sharpness Sv in the vertical direction is:
Sv(i, j)=X.sub.0 (i, j+1)2X.sub.0 (i, j)+X.sub.0 (i, j1)
The vertical edge image is quantized as was the horizontal edge image, points for which Sv(i, j)≦T being set to zero. Quantization can be performed by dividing the edge sharpness values by a power of two, for example, or by using a quantization table. Incidentally, the same notation Sh and Sv will be employed herein to denote edge sharpness values both before and after quantization.
Smoothing filter 221 employs a lowpass filter h_{1} with even symmetry and a certain number of taps. In the following description it will be assumed that this filter either has 2N+1 taps and is symmetric around zero, or has 2N taps and is symmetric around 1/2, where N is an appropriate positive integer. The even symmetry condition in the first case is:
h.sub.1 (k)=h.sub.1 (k) (k=1, 2, . . . , N)
The even symmetry condition in the second case is:
h.sub.1 (1k)=h.sub.1 (k) (k=1, 2, . . . , N)
The lowpass filter h_{1} satisfies the exact reconstruction condition, and the sum of its tap coefficients is plus or minus one. That is, the Fourier transform H_{1} of h_{1} satisfies the following conditions: ##EQU7##
The smoothing filter 221 performs al twodimensional lowpass filtering operation by filtering the input image X_{0} with h_{1} horizontally and vertically, obtaining: ##EQU8##
The other smoothing filters 22m (m>1) also employ lowpass filters h_{m} with even symmetry, and with tap coefficients summing to plus or minus one. The cutoff frequency of each smoothing filter 22m (m>1) is onehalf the cutoff frequency of the preceding filter 22(m1) in the cascade. Filters with these cutoff frequencies can be obtained by doubling the tap spacing from one filter to the next, and inserting zeros to fill the vacated spaces. If the number of taps is odd, for example, then h_{m} can be obtained from h_{m1} as follows: ##EQU9##
In smoothing filter 22m, image X_{m1} is filtered horizontally and vertically by filter h_{m} to obtain image X_{m}. As the image passes through the cascade, variations on progressively larger scales are smoothed out, and resolution of fine detail decreases correspondingly.
Incidentally, if the lowpass filters h_{m} (m>1) are obtained by doubling the tap spacing as described above, they will pass highfrequency variations in certain regions above their cutoff frequencies. This is allowable because those highfrequency variations will already have been removed by previous filters in the cascade.
The down sampler 23 downsamples the lowfrequency image X_{n} output by the smoothing filter 22n, thereby reducing the amount of image data. One simple method of downsampling is to divide the image into blocks or M×M pixels each, and replace each block by a representative pixel value such as the mean or median value in the block. This reduces the size of the image by a factor of M in both the horizontal and vertical directions, and reduces the amount of image data by a factor of M^{2}. (M is an appropriate integer, such as M=2^{n}.) The reduced image R output by the down sampler 23 resembles a photographic reduction of the original image X_{0} .
The edge image encoder 24 encodes the quantized horizontal and vertical edge images Sh and Sv by using, for example, runlength encoding to encode the positions of edge points, and differential encoding to encode the sharpness values at these points. Alternatively, chain encoding can be used to encode the edge positions, taking advantage of the fact that edge points tend to be organized into chains of mutually contiguous points. Chain encoding encodes only the displacement from one position to the next in such a chain.
The reduced image encoder 25 encodes the reduced image R by, for example, performing a discrete cosine transform, quantizing the resulting DCT coefficients, and encoding them in zigzag order by a variablelength encoding method. Alternatively, the reduced image R can be encoded by a predictive encoding method similar to the differential pulsecode modulation method (DPCM) commonly employed for audio signals, or by any other suitable method. If the above value of M is sufficiently large, high coding efficiency is not critically important, because the amount of data to be encoded has already been greatly reduced by downsampling.
The encoded image C, consisting of the edge image information Cs and lowfrequency information Cr as multiplexed by the multiplexer 26, is output to the first input/output device 28 for transmission or storage. The amount of encoded data is generally reduced to a small fraction of the amount in the original image X_{0} , so the encoded image C can be efficiently transmitted or stored.
Next the decoding operations will be described. These operations employ further filters e_{m}, f_{m}, and g_{m}.
Filter g_{1} is a highpass filter related to the smoothing filter h_{1} as follows:
g.sub.1 (k)=(1).sup.k h.sub.1 (k)
This relation implies that g_{1} has even symmetry if h_{1} was symmetric around zero, or odd symmetry if h_{1} was symmetric around 1/2. It further implies that the Fourier transform G_{1} of g_{1} is the complex conjugate of H_{1} (ω+π), so in particular, h_{1} and g_{1} satisfy the exact reconstruction condition and form a complementary pair. Furthermore, G_{1} (ω+π) is the complex conjugate of H(ω) , so g_{1} also satisfies an exact reconstruction condition by itself: ##EQU10##
Since H_{1} (0)=1, it follows that G_{1} (0)=0, so the sum of the tap coefficients g_{1} (k) is zero.
The other filters g_{m} (m=2 to n) are also highpass filters, and are related to the corresponding h_{m} by a similar equation:
H.sub.m (ω).sup.2 +G.sub.m (ω).sup.2 =1
These filters g_{m} can be obtained from g_{1} by successively doubling the tap spacing, in the same way that the filters h_{m} were obtained from h_{1}.
Filter f_{m} is a completion filter, defined in terms of its Fourier transform F_{m} as follows. ##EQU11##
Filter f_{m} is a lowpass filter, referred to as a completion filter, obtained by executing an inverse Fourier transform on F_{m}. Once f_{1} has been obtained, the other f_{m} (m=2 to n) can be derived from it by successively doubling the tap spacing. The above definition of F_{m}, incidentally, is well known in the wavelet transform art and comes from the following equation:
H(ω).sup.4 +2·F(ω)·G(ω).sup.2 =1
Filter e_{m} is an edge synthesis filter derived by filtering a standard edge of unit sharpness by the corresponding highpass filter g_{m}. When g_{m} has even symmetry, all standard edges with unit sharpness lead to the same e_{m}, which has the following tap values e_{m} (t). L is the halflength of g_{m}, i.e. the distance from the central tap to the outermost tap. If g_{m} is derived as described above, then L=2.sup.(m1) N). ##EQU12##
When g_{m} has odd symmetry, the situation is a little more complicated and will be described later.
Referring again to FIG. 2, after the demultiplexer 30 separates the input encoded image C into edge image information Cs and reduced image information Cr, decoders 31 and 32 carry out processes that are reverse to the processes performed by encoders 24 and 25 in FIG. 1. For example, if the reduced image encoder 25 in FIG. 1 performed a discrete cosine transform followed by quantization and variablelength encoding, the reduced image decoder 32 in FIG. 2 performs variablelength decoding, dequantization, then an inverse discrete cosine transform to obtain the decoded reduced image R'.
If the edge images were encoded by runlength or chain encoding of the positions of the edge points and differential encoding of their sharpness, the edge image decoder 31 performs runlength or chain decoding to recover the edge positions and an accumulation process to obtain the sharpness values. The edge image decoder 31 also dequantizes the sharpness values.
The up sampler 35 obtains the decoded lowfrequency image X'_{n} by restoring the decoded reduced image R' to the original image size (the size of the edge images and highfrequency images). If, for example, each pixel value in the reduced image R represented an M×M block in the lowfrequency image X_{n}, the tip sampler 35 can simply copy this value to all pixels in the corresponding M×M block of the decoded lowfrequency image X'_{1}. More sophisticated upsampling methods involving interpolation chain also be employed, but smooth interpolation is not required, because the image will be smoothed during the inverse wavelet transform.
The cascaded smoothing filters 33m (m=1 to n1) carry out successive twodimensional filtering operations on the pairs of edge images Sh_{m} and Sv_{m} in the same way that the cascaded smoothing filters 22m carried out successive filtering operations in the encoder. Specifically, Sh_{m} is filtered by h_{m} horizontally and vertically to obtain Sh_{m+1}, and Sv_{m} is filtered by h_{m} horizontally and vertically to obtain Sv_{m+1}.
Each edge synthesizer 34m filters the corresponding horizontal edge image Sh_{m} by the edge synthesis filter e_{m} in the horizontal direction, and the vertical edge image Sv_{m} by e_{m} in the vertical direction. When the smoothing filters h_{m} have an odd number of taps and the highpass filters g_{m} accordingly have even symmetry, the resulting horizontal and vertical highfrequency images Yh_{m} and Yv_{m} can be described by the following equations: ##EQU13##
Conceptually, for each edge point, the edge synthesizer 34 synthesizes the result of filtering a standard edge of the corresponding sharpness by the highfrequency filter g_{m} . It then adds these results together for all edge points to synthesize a highfrequency image.
When the smoothing filters h_{m} have an even number of taps and the highpass filters g_{m} accordingly have odd symmetry, it is advantageous to carry out the edge synthesis indirectly. One reason is that in this case, the results of filtering different standard edges by the same g_{m} need not be the same. Another reason is that the resulting edge synthesis filters e_{m} can have an unlimited number of nonzero taps.
If the differences de_{m} (t) between adjacent coefficient values in e_{m} are taken, however, these differences have the desirable properties of depending only on the edge sharpness, and being zero except in a limited domain. These differences are given by the following equation, in which L has the same meaning as before (L=2^{m1} N). They represent the result of filtering a standard edge of unit sharpness with the highpass filter g_{m}, then taking differences between the resulting values at adjacent pixels. ##EQU14##
The horizontal edge image Sh_{m} is first filtered horizontally by de_{m} (t) to obtain a horizontal differential highfrequency image Δ Yh, and the vertical edge image Sv_{m} is filtered vertically with de(t) to obtain a vertical differential highfrequency image Δ Yv. These differential highfrequency images are described by the following equations: ##EQU15##
The horizontal and vertical differential highfrequency images are then cumulatively summed to obtain the horizontal and vertical highfrequency images Yh_{m} and Yv_{m}. If the horizontal and vertical coordinates i and j both start at zero, these accumulation operations can be described as follows:
Yh.sub.m (0, j)=Δ Yh.sub.m (0, j)
Yh.sub.m (i, j)=Δ Yh.sub.m (i , j)+Yh.sub.m (i1, j) (i>0)
Yv.sub.m (i, 0)=Δ Yv.sub.m (i, 0)
Yv.sub.m (i, j)=Δ Yv.sub.m (i, j)+Yv.sub.m (i, j1) (j>0)
In the inverse wavelet transform processor 36, each inverse wavelet processor 36m performs three twodimensional filtering operations, using the conjugate filters f*_{m}, g*_{m}, and h*_{m} of f_{m}, g_{m}, and h_{m}. The image X'_{m} is filtered by h*_{m} horizontally and vertically, the horizontal highfrequency image Yh*_{m} is filtered by g*_{m} horizontally and f*_{m} vertically, and the vertical highfrenquency image Yv_{m} is filtered by g*_{m} vertically and f*_{m} horizontally. The three results are added to obtain the partially reconstructed image X'_{m1}. This inverse wavelet transform is well known in the art, and is described by the following equation: ##EQU16##
The reconstructed image X'_{0} is output to the second input/output device 37 for display or printout. Alternatively, the reconstructed image can be output to a computer for further image processing.
Synthesizing highfrequency image information from edge images as described above enables high compression ratios to be obtained. Highfrequency information corresponding to edges below the sharpness threshold T is lost, so there is some loss of lowlevel, highfrequency shading variations, but the reconstructed image is not distorted, and sharp edges are faithfully preserved.
One feature of the invented encoder is that edge detection is performed only at the original image scale, which permits edge detection to proceed concurrently with the smoothing filtering and downsampling operations in the encoder, to speed up the encoding process. Another feature is that in the decoder, highfrequency images are synthesized in a single step at each wavelet scale, without recourse to iterative procedures. This reduces the computational load in the decoder, speeds up the decoding process, and eliminates the priorart problem of deciding when to stop iterating.
Second Embodiment
The second embodiment is similar to the first, but incorporates a downsampling process into each of the smoothing filters in the encoder and decoder, and an upsampling process into each of the inverse wavelet processors in the decoder.
Referring to FIG. 4, the digital image encoder in the third embodiment has the same edge detector 21 and encoding section 27 as in the first embodiment, but its twodimensional smoothing and downsampling section 41 employs smoothing and downsampling filters 42m (m=1 to n). In addition to lowpass filtering, these cascaded filters 42m downsample the filtered data by discarding every other pixel in both the horizontal and vertical directions.
FIG. 5 shows a conceptual flow diagram of the smoothing and downsampling filtering process. The first step 43 is a vertical smoothing step carried out using a lowpass filter h identical to filter h_{1} in the first embodiment. The second step 44 is a vertical downsampling step that discards every other pixel in the vertical direction. The third step 45 is a horizontal smoothing step in which the remaining image data are filtered by the same filter h in the horizontal direction. The fourth step 46 is a horizontal downsampling step that discards every other pixel in the horizontal direction.
Needless to say, the values of pixels that will be discarded immediately following a filtering operation do not have to be computed in that filtering operation. The entire operation performed by the twodimensional smoothing and downsampling filter 42m can therefore be described by the following equation: ##EQU17##
This operation reduces the size of the image by a factor of two in both the horizontal and vertical directions. The same lowpass filter h is used in all the filters 42m (m=1 to n). It is not necessary to double the tap spacing by inserting zero coefficients at successive wavelet scales as in the first embodiment.
In the drawings, the symbol h_{d} denotes filtering by h followed by downsampling as described above.
FIG. 6 shows the digital image decoder of the second embodiment. The decoding section 29 is the same as in the first embodiment. The output of the reduced image decoder 32 is now denoted X'_{n}, but is similar to the output R in the first embodiment. The decoder employs filters e, f, g, and h that are identical to e_{1}, f_{1}, g_{1}, and h_{1} in the first embodiment. The subscripts d and u attached to these filters denote associated downsampling and upsampling operations, respectively.
The edge images Sv_{1} and Sh_{1} output by the edge image decoder 31 are filtered by a cascade of smoothing and downsampling filters 48m (m=1 to n1) identical to the smoothing and downsampling filters 42m in the encoder. Each filter 48m filters and smooths each input edge image Sv_{m} and Sh_{m} in both the vertical and horizontal directions, and downsamples the result by a factor of two in both directions to generate edge images Sv_{m+1} and Sh_{m+1}. Due to the downsampling, Sv_{m+1} is only half as wide and half as high as Sv_{m}, and Sh_{m+1} is only half as wide and half as high as Sh_{m}.
The decoded edge images Sv_{m} and Sh_{m} are input to an edge synthesizer 49m (m=1 to n). There, the vertical edge image Sv_{m} is downsampled by a factor of two in the horizontal direction, filtered by the abovementioned filter e in the vertical direction, then downsampled by a factor of two in the vertical direction, while the horizontal edge image Sh_{m} is downsampled by a factor of two in the vertical direction, filtered by filter e in the horizontal direction, then downsampled by a factor of two in the horizontal direction. These operations are depicted conceptually in FIG. 7. The resulting highfrequency images will be denoted Yh_{m} and Yv_{m} as in the first embodiment.
The highfrequency images Yh_{m} and Yv_{m} (m=1 to n) and the lowfrequency reduced image X'_{n} are input to an inverse wavelet transform processor 50 comprising cascaded inverse wavelet processors 50m (m=1 to n). Processors 50m use the conjugate filters f*, g*, and h* to perform the same computation as in the first embodiment, but each filtering step in the computation is preceded by an upsampling step. These upsampling steps double the image size by inserting zerovalued pixels between every pair of adjacent pixels in the appropriate direction. To prevent the zerovalued pixels from reducing the mean value of the output image, after each filtering step, the resulting pixel values are doubled.
Due to upsampling, the image X'_{m1} output from inverse wavelet processor 50m is twice as wide and twice as high as each of the three images X'_{m}, Yv_{m}, and Yh_{m} input to processor 50m. The three input images X'_{m}, Yv_{m}, and Yh_{m} all have the same size.
FIG. 8 depicts the operation performed by the inverse wavelet processor 50m schematically. As indicated at the top of this drawing, the partially reconstructed image X'_{m} received from the preceding processor 50(m+1) in the cascade is upsampled horizontally by interpolating zeros (reducing the mean pixel value by a factor of two) and filtered horizontally by h*; then the resulting pixel values are doubled (restoring the correct mean value). The image is now upsampled vertically, again by interpolating zeros, and filtered by h* vertically. Similarly, highfrequency image Yh_{m} is upsampled vertically, filtered by f* vertically, doubled in pixel value, upsampled horizontally, and filtered by g* horizontally. Highfrequency image Yv_{m} is upsampled horizontally, filtered by f* horizontally, doubled in pixel value, upsampled vertically, and filtered by g* vertically.
The three resulting images are then added together to produce a sum image, all pixel values in which are again doubled to counteract the effect of the second upsampling steps. The final result is the partially reconstructed image X'_{m1} which is passed to the next wavelet processor 50(m1), or the reconstructed image X'_{0} output to the input/output device 37.
The operation performed by inverse wavelet processor 50m can be described by the following equation. Upsampling is accomplished by dividing the values of i, j, k, and l by two. If either i/2+k/2 or j/2+1/2 is not an integer, the corresponding values of X'_{m}, Yh.sub._{m}, and Yv_{m} are taken to be zero. ##EQU18##
The second embodiment provides the same advantages as the first embodiment but requires less computation, because the amount of image data to be processed is reduced by a factor of four (by a factor of two horizontally and another factor of two vertically) at each wavelet scale. The computational load of the additional upsampling and downsampling steps, and of multiplying pixel values by two, is slight.
Third Embodiment
The third embodiment detects edges diagonally as well as horizontally and vertically. To prevent an increase in the amount of encoded data, some of the downsampling steps that were performed in the decoder in the second embodiment are moved into the encoder.
FIG. 9 is a block diagram of the digital image encoder of the third embodiment. The twodimensional smoothing and downsampling section 41 is the same as in the second embodiment. The original image X_{0} is also input to a onedimensional smoothing and downsampling section 51, which outputs the original image X_{0} together with a vertically smoothed and downsampled image Xh and a horizontally smoothed and downsampled image Xv to an edge detector 52. The edge detector 52 outputs three edge images Sh, Sv, and Sc to the encoding section 27, which is the same as in the first two embodiments.
FIG. 10 illustrates the processes carried out in the onedimensional smoothing and downsampling section 51. Image Xv is produced by filtering tile original image X_{0} with the lowpass filter h in the horizontal direction, then downsampling the resulting image by removing every other pixel in the horizontal direction. Image Xv accordingly contains only half as much data as image X_{0}.
Similarly, image Xh is produced by filtering X_{0} vertically with the lowpass filter h, then downsampling the result by removing every other pixel in the vertical direction. Image Xh also contains only half as much data as image X_{0}.
Images Xv and Xh can be described by the following equations: ##EQU19##
The edge detector 52 detects edges by the method described earlier: by taking differences between adjacent pixels, then taking differences between these differences. This method is applied in the directions that have not been downsampled. Accordingly, differences are taken horizontally in the vertically downsampled image Xh, producing a horizontal edge image Sh. Similarly, differences are taken vertically in the horizontally downsampled image Xv, producing a vertical edge image Sv.
Edges are detected diagonally by processing the original image X_{0} in both the horizontal and vertical directions. The differenceofdifferences operation is first performed on X_{0} in, for example, the horizontal direction, producing an intermediate image. The differenceofdifferences operation is next performed on this intermediate image in the vertical direction. Resulting pixels with absolute values equal to or less than a threshold value T are then set to zero, and other values are suitably quantized, producing a diagonal edge image Sc. Prior to quantization, Sc has the following values: ##EQU20##
The edge image encoder 24 can encode the diagonal edge image Sc by methods similar to those used for the vertical and horizontal edge images Sv and Sh. Sc has the same size as the original image X_{0} , while Sv and Sh are only half this size, so the total amount of data to be encoded is the same as in the first two embodiments.
FIG. 11 is a block diagram of the digital image decoder in the third embodiment. The decoding section 29 is the same as in the preceding embodiments, except that the edge image decoder 31 outputs a set of three edge images, Sv_{1}, Sh_{1}, and Sc_{1}, corresponding to the encoded edge images Sv, Sh, and Sc.
These images are input to a cascade of twodimensional smoothing and downsampling filters 53m (m=1 to n1). These are identical to the twodimensional smoothing and downsampling filters 48m in the second embodiment, except that each filter smooths and downsamples three separate edge images instead of two, producing vertical, horizontal, and diagonal edge images Sv_{m+1}, Sh_{m+1}, and Sc_{m+1}. Each of these edge images Sv_{m+1}, Sh_{m+1}, and Sc_{m+1} is only half as high and half as wide as the corresponding image Sv_{m}, Sh_{m} , or Sc_{m} in the preceding set.
These images are processed by corresponding edge synthesizers 54m as explained below to create sets of vertical, horizontal, and diagonal highfrequency images Yv_{m}, Yh_{m}, and Yc_{m}. These are supplied to an inverse wavelet transform processor 55, comprising cascaded inverse wavelet processors 55m (m=1 to n), which output reconstructed or partially reconstructed images X'_{m1}.
Edge synthesizer 54m operates as illustrated in FIG. 12, using the same edge synthesis filter e as in the second embodiment. The vertical edge image Sv_{m} is filtered vertically by e, then downsampled by removing every other pixel in the vertical direction to create al vertical highfrequency image Yv_{m}. Similarly, the horizontal edge image Sh_{m} is filtered horizontally by e, then downsampled horizontally to create a horizontal highfrequency image Yh_{m}. The diagonal edge image Sc_{m} is filtered vertically by e, downsampled in the vertical direction, filtered horizontally by e, then downsampled in the horizontal direction, creating a diagonal highfrequency image Yc_{m}.
Due to the combination of downsampling in the onedimensional smoothing and downsampling section 51 in the encoder and in the edge synthesizer 54m in the decoder, the three highfrequency images Yv_{m}, Yh_{m}, and Yc_{m} are all the same size. Each has been downsampled m times in both the horizontal and vertical directions, reducing the number of pixels by a total factor of 4^{m}.
Inverse wavelet processor 55m operates on images X'_{m}, Yv_{m}, Yh_{m}, and Yc_{m} as illustrated in FIG. 13, using the same complementary pair of highpass and lowpass filters g* and h* as in the second embodiment, but not using the completion filter f*. As in the second embodiment, each filtering step is preceded by an upsampling step and followed by a compensatory doubling of the pixel values. Upsampling is carried out by inserting zerovalued pixels.
Inverse wavelet processor 55m filters each of the four input images horizontally and vertically, and adds the four resulting images together to obtain the output image X'_{m1}. Image X'_{m} is filtered horizontally and vertically by h*; image Yh_{m} is filtered horizontally by g* and vertically by h*; image Yv_{m} is filtered horizontally by h* and vertically by g*; and image Yc_{m} is filtered horizontally and vertically by g*. For computational efficiency, the results of the four horizontal filtering operations are added pairwise, so that only two vertical filtering operations have to be performed.
The operation performed by inverse wavelet processor 55m can also be described by the following equation. As before, if either i/2+k/2 or j/2+1/2 is not an integer, the corresponding values of X'_{m}, Yh_{m}, Yv_{m} and Yc_{m} are zero. ##EQU21##
The third embodiment provides the same advantages as the second and offers improved rendition of highfrequency detail in the reconstructed image, due to diagonal edge detection.
Fourth Embodiment
The fourth embodiment is similar to the third embodiment, differing only in the operation of the edge detector 52. Block diagrams will be omitted.
The edge detector 52 of the fourth embodiment begins by detecting horizontal, vertical, and diagonal edge sharpness by the methods described in the third embodiment, using a threshold value T. It next constructs thickened edges by including contiguous points with edge sharpness values exceeding a lower threshold U. Finally, it trims the edges so that each edge has a uniform thickness (although some edges may be thicker than others). This process will be explained below for an edge in the horizontal edge image Sh.
FIG. 14 shows part of a typical horizontal edge image Sh, using stippling to denote pixels with sharpness values exceeding T, and hatching to denote pixels exceeding U but not exceeding T. Here, "exceeding" means having a positive edge sharpness value exceeding T (or U). Pixels with negative sharpness values are processed separately.
Edge points in the horizontal edge image Sh tend to be organized into chains with a generally vertical orientation. FIG. 14 shows one such chain, running generally from the top left toward the bottom right. This edge is irregular in shape, however, and if detected by threshold T, the edge is fragmented into two parts. Such irregularities and fragmentation greatly impair the efficiency which which the edge image can be encoded.
FIG. 15 shows the thickened version of this edge. The edge detector 52 constructs FIG. 15 from FIG. 14 by the following procedure.
Starting from the top left corner, the edge detector searches for a pixel in the horizontal edge image Sh in FIG. 14 exceeding threshold T. When the first such pixel is found, it is placed in the thickened edge image in FIG. 15 as the first pixel of the first layer of an edge. This first layer is then extended in the left and right directions by including any contiguous pixels having the same sign and exceeding threshold U. The extension continues as far in each direction as such contiguous pixels continue. In the drawings, pixel 56 is found first and pixels 57 and 58 are added to complete the first layer. If there were further pixels exceeding threshold U immediately to the left of pixel 57 or to the right of pixel 58, they would also be added in.
Next, if there any pixels exceeding threshold T immediately below the bottom layer of the edge so far, they are added to begin a new layer of the edge. This new layer is then extended to the left and right by adding contiguous pixels exceeding threshold U. In this extension process, all contiguous pixels exceeding threshold T are added in. Pixels exceeding threshold U but not exceeding threshold T are added in only if they are not disposed more than a certain limit extension distance beyond the ends of the preceding layer. In this example, the limit extension distance is one pixel. Thus pixel 59 is added, because it extends only one pixel past pixel 57, but the pixels to the left of pixel 59 would not be added to this layer unless they exceeded threshold T.
The edge detector continues in this manner, extending the thickened edge downward by adding new layers as long as it can find a pixel exceeding threshold T immediately below the bottom layer so far, and enlarging each layer horizontally to include contiguous pixels exceeding threshold T, as well as pixels exceeding threshold U out to the limit extension distance. In this example, pixel 60 is not added to the thickened edge because it would extend two pixels beyond the preceding layer, and the extension limit is one pixel.
When no pixels exceeding threshold T can be found below the bottom layer, the thickened edge is considered complete. The edge detector then searches for another pixel exceeding threshold T in absolute value to start a new thickened edge, and thickens this edge in the same way. This entire procedure is repeated until no more pixels exceeding threshold T can be found. A similar procedure is then followed to process pixels having negative sharpness values exceeding T and U.
Although the limit extension distance in the example above was one pixel, other limit extension distances such as zero, two, or three pixels can be used. The preferable limit extension distance depends in part on the threshold values T and U.
FIG. 16 shows the trimmed version of the thickened edge in FIG. 15. The trimmed edge is constructed by comparing adjacent layers in the thickened edge, and trimming the longer layer to match the length of the shorter layer. Thus pixel 59 is trimmed to reduce the length of its layer to three pixels, matching the length of the layer above. This trimming procedure is carried out from the top of the edge to the bottom, then repeated until all layers in the edge have been trimmed to the same length.
If the longer layer extends beyond the shorter layer at only one end, the extending end is trimmed. If the longer layer extends beyond the shorter layer at both ends, the edge detector 52 examines the preceding layer, and trims in such a way as to maintain a constant offset from one layer to the next. In FIG. 16, for example, after trimming layers 61 and 62, the edge detector 52 trims layer 63 so that it is offset one pixel to the right of layer 62, because layer 62 is similarly offset one pixel to the right of layer 61.
The trimmed edge in FIG. 16 is continuous, and has a more regular shape than the original edge in FIG. 14. These properties enable the edge in FIG. 16 to be encoded more efficiently than the edge in FIG. 14, even though it contains more pixels. In chain encoding, for example, after the position and length of the first layer have been specified, the position of the rest of the edge can be specified just by giving the offset of each layer from the preceding layer. In quantization of the edge sharpness values, pixels exceeding threshold U but not threshold T can be quantized to zero but then dequantized to a nonzero value in the digital image decoder.
In the vertical edge image Sv, edges are thickened and trimmed by a similar procedure, except that the thickening and trimming are done vertically, and the edge detector 52 works from left to right instead of from top to bottom.
Edges in the diagonal edge image Sc can be processed either horizontally or vertically. The choice depends on the way in which the diagonal edge image will be encoded. If vertical chain encoding is used, for example, the thickening and trimming should be done horizontally.
Edge thickening and trimming can also be added to the edge detection process in the first and second embodiments, and in any of the embodiments that follow. The invented edge thickening and trimming technique is moreover useful for regularizing edges in image encoding and image processing in general, and is not limited to edges detected by the differenceofdifferences method.
Fifth Embodiment
The fifth embodiment is similar to the second embodiment, but is adapted to process moving images comprising successive frames.
Referring to FIG. 17, the digital image encoder of the fifth embodiment has the same twodimensional smoothing and downsampling section 41 and edge detector 21 as in the second embodiment, but a different encoding section 65. The encoding section 65 comprises a pair of motion estimators 66 and 67, a motion vector encoder 68, a pair of motion compensators 69 and 70, and a pair of memories 71 and 72. It also has the same edge image encoder 24 and reduced image encoder 25 as in the digital image encoders of the preceding embodiments, and the same edge image decoder 31 and reduced image decoder 32 as in the digital image decoders of the preceding embodiments.
Motion estimator 66 receives the pair of edge images Sh and Sv from the edge detector 21. To simplify the drawing, this pair of edge images is denoted by the single letter S. Motion estimator 66 also receives a pair of previous edge images S" (a previous horizontal edge image and a previous vertical edge image) from memory 71. Motion estimator 66 outputs edge motion vectors Ms and a pair of horizontal and vertical edge difference images, denoted for simplicity by the single symbol Ds.
Motion estimator 67 receives the reduced image X_{n} from the last smoothing filter 42n and a previous reduced image X"_{n} from memory 72, and outputs reduced motion vectors Mr and a reduced difference image Dr. The motion vector encoder 68 encodes motion vectors Ms and Mr and outputs motion information Cm.
The edge image encoder 24 encodes the edge difference images Ds and outputs edge image information Cs. This information is decoded by the edge image decoder 31 to produce a pair of decoded edge difference images, which are denoted in the drawing by the single symbol D's. Motion compensator 69 receives this pair of images D's, the previous pair of images S", and the motion vectors Ms, and produces a pair of decoded edge images S_{1}, which it stores in memory 71.
Similarly, the reduced image encoder 25 encodes the reduced difference image Dr and outputs reduced image information Cr. The reduced image decoder 32 decodes Cr and outputs a decoded reduced difference image D'r. Motion compensator 70 receives this image D'r, the previous reduced image X"n, and the motion vectors Mr, and produces a decoded reduced image X'n which it stores in memory 72.
The multiplexer 73 multiplexes the edge image information Cs, motion information Cm, and reduced image information Cr for output as an encoded image C.
Referring to FIG. 18, the digital image decoder of the fifth embodiment has a decoding section 75 comprising a demultiplexer 76, a motion vector decoder 77, and decoders 31 and 32, motion compensators 69 and 70, and memories 71 and 72 identical to those in the digital image encoder in FIG. 17. The rest of the digital image decoder in FIG. 18 is the same as in the second embodiment, comprising smoothing and downsampling filters 48m (m=1 to n1), edge synthesizers 49m (m=1 to n), and an inverse wavelet transform processor 50.
The demultiplexer 76 receives the encoded image C from the input/output device 28 and demultiplexes it into edge image information Cs, motion information Cm, and reduced image information Cr. The motion vector decoder 77 decodes Cm to obtain decoded edge motion vectors M's and reduced motion vectors M'r. Other symbols in FIG. 18 are as already explained in FIGS. 6 and 17, with S_{m} representing the pair of edge images Sh_{m} and Sv_{m}, and Y_{m} the pair of highfrequency images Yh_{m} and Yv_{m}.
Next the operation of the fifth embodiment will be described. The description will be confined mainly to the encoding section 65 of the encoder.
Referring again to FIG. 17, motion estimator 67 estimates motion in the reduced image X_{n} by, for example, the wellknown blockmatching method. This method involves comparing a block of pixels in the image X_{n} with blocks displaced by k pixels horizontally and l pixels vertically from this block in the previous image X"_{n} in memory 72, and selecting the vector (k, l) that gives the closest match. These vectors (k, l) are output as motion vectors Mr, while the differences between the pixel values in X_{n} and the pixels values in X"_{n} at positions offset by these motion vectors (k, l) are output as the reduced difference image Dr. Thus the reduced difference image Dr is the difference between X_{n} and X"_{n} over and above the motion represented by the motion vectors Mr.
Motion estimator 66 carries out a similar process separately on the horizontal and vertical edge images to obtain the motion vectors Ms and the pair of edge difference images denoted Ds. The motion vector encoder 68 encodes the motion vectors Ms and Mr by a lossless encoding method such as a variablelength encoding method.
The encoders 24 and 25 encode the difference images Ds and Dr by, for example, the same methods as used in the preceding embodiments. The decoders 31 and 32 employ methods reverse to those methods to obtain the decoded difference images D'r and D's.
Motion compensator 70 adds the decoded reduced difference image D'r to the previous reduced image X"_{n} according to the motion vectors Mr, and stores the resulting image X'_{n} in memory 72. "According to the motion vectors Mr" means that each pixel value in D'r is added to a corresponding pixel value in the previous reduced image X"_{n} at a location offset by a motion vector (k, l) in Mr. The method by which X'_{n} is obtained from D'r is inverse to the method by which Dr is obtained from X_{n}, and the operations of encoder 25 and decoder 32 are mutually inverse, so image X'_{n} is substantially identical to image X_{n}.
Similarly, motion compensator 69 adds the two decoded edge difference images to the corresponding previous edge images according to the motion vectors Ms, and stores the resulting pair of decoded edge images S_{1} in memory 71. The pair of edge images S_{1} are substantially identical to the pair of edge images S output by the edge detector 21.
Memory 72 has space for storing at least two complete reduced images: the previous image X"_{n} supplied to motion estimator 67 and motion compensator 70, and the new image X'_{n} stored by motion compensator 70. At the end of each frame, the previous image X"_{n} is discarded, leaving the new image X'_{n} to be used as the previous image in the next frame. Memory 71 stores edge images in a similar fashion.
From time to time the motion estimators 66 and 67 may ignore the previous images S" and X"_{n} in the memories 71 and 72 and supply the edge images S and reduced image X_{n} directly to the encoders 24 and 25, instead of supplying difference images. The motion information Cm in this case consists of a message indicating that the current frame does not require motion compensation. The motion compensators 69 and 70 then simply output the images received from the decoders 31 and 32.
The motion compensators 69 and 70 and memories 71 and 72 in the digital image decoder operate in the same way as the motion compensators 69 and 70 and memories 71 and 72 in the digital image encoder, using the decoded motion vectors M's and M'r supplied by the motion vector decoder 77. Since a lossless encoding scheme is employed for the motion vectors, M's and M'r are exactly identical to Ms and Mr, and the images denoted S_{1} and X'_{n} in FIG. 18 are exactly identical to the images denoted S_{1} and X'_{n} in FIG. 17. Further operation of the digital image decoder is as already described in the second embodiment.
By removing temporal redundancy, motion compensation can dramatically increase the coding efficiency of moving images, but block matching conventionally requires considerable computation. One advantage of the fifth embodiment is that most of the block matching is performed in the edge images, which have been thresholded and tend to consist mostly of zerovalued pixels. The large number of zerovalued pixels greatly speeds tip the blockmatching process. Further gains in speed are possible by using motion vectors calculated for one of the two edge images to help find motion vectors in the other edge image.
Another advantage is that the motion estimators and compensators in the digital image encoder and digital image decoder use identical previous images S" and X"_{n}. This prevents the accumulation of quantization error and other image errors resulting from compressive coding in the encoders 24 and 25, and ensures consistency between motion estimation in the digital image encoder and motion compensation in the digital image decoder.
Other advantages of the fifth embodiment are as described in the second embodiment. The fifth embodiment can be employed in video teleconferencing and a wide variety of other multimedia applications.
Sixth Embodiment
The sixth embodiment is similar to the fifth embodiment, with further provisions for speeding up the computation of motion vectors.
Referring to FIG. 19, the encoding section 78 of the digital image encoder in the sixth embodiment is similar to the encoding section 65 in the fifth embodiment, but has a bidirectional interconnection 79 between motion estimators 66 and 67. This enables motion estimator 67 to receive the edge motion vectors Ms found by motion estimator 66, scale them down to the size of the lowfrequency reduced image X_{n} and restrict the search for motion vectors in image X_{n} to vectors close to the scaleddown edge motion vectors. In the extreme case, the search can be eliminated entirely: the reduced motion vectors Mr can be computed directly from the edge motion vectors Ms.
Alternatively, motion estimator 66 can receive the motion vectors Mr found by block matching in the lowfrequency reduced image X_{n}, scale them up to the size of the edge images Sh and Sv, and restrict the search for motion vectors in Sh and Sv to vectors close to the scaledup vectors. This scheme is particularly effective in detecting large, rapid motion.
Aside from this additional interconnection 79, the sixth embodiment has the same structure as the fifth embodiment and operates in the same way, so further description will be omitted. By narrowing down the search for motion vectors, the sixth embodiment enables frames to be encoded at a faster rate than in the fifth embodiment.
Seventh Embodiment
The seventh embodiment combines features of the third and fifth embodiments by detecting edges horizontally, vertically, and diagonally, and employing motion compensation.
Referring to FIG. 20, the digital image encoder of the seventh embodiment has the same twodimensional smoothing and downsampling section 41, onedimensional smoothing and downsampling section 51, and edge detector 52 as in the third embodiment (FIG. 8), and the same encoding section 65 as in the fifth embodiment (FIG. 17). The symbol Sc in FIG. 20 denotes the set of three edge images Sv, Sh, and So in FIG. 8. Each of the symbols S_{1}, S", Ds, and D's likewise denotes a set of three edge images.
Referring to FIG. 21, the digital image decoder of the seventh embodiment has the same decoding section 75 as the fifth embodiment FIG. 17), and the same twodimensional smoothing and downsampling filters 531 to 53(n1), edge synthesizers 541 to 54n, and inverse wavelet transform processor 55 as in the third embodiment (FIG. 11).
The seventh embodiment operates as explained in the third and fifth embodiments, and provides similar effects, so further description will be omitted.
Eighth Embodiment
The eighth embodiment combines features of the sixth and seventh embodiments.
Referring to FIG. 22, the digital image encoder of the eighth embodiment has the same configuration as in the seventh embodiment (FIG. 20), except for the addition of a bidirectional interconnection 79 between motion estimators 66 and 67 as in the sixth embodiment (FIG. 19). The eighth embodiment operates as explained in the sixth and seventh embodiments, so further description will be omitted.
Ninth Embodiment
The ninth embodiment performs motion estimation before edge detection, smoothing, and down sampling. To maintain consistency between the encoder and decoder, the encoder performs substantially all the processes carried out in the decoder, including edge synthesis and the inverse wavelet transform.
Referring to FIG. 23, the original image frames input at the input terminal 10 are supplied to a motion estimator 80, which performs motion estimation as described above, outputting a difference image D_{0} and motion vectors M. The motion vectors M are encoded by a motion vector encoder 68 as in the fifth to eighth embodiments. The difference image D_{0} is supplied to an edge detector 21 and twodimensional smoothing and downsampling section 41 in place of the original image X_{0}, and is processed in the same way that the original image X_{0} was processed in the second embodiment. The resulting pair of edge images will again be denoted by the symbol S, while the reduced image resulting from smoothing and downsampling of D_{0} will be denoted D_{n}. These images S and D_{n} are encoded by respective encoders 24 and 25 to produce encoded image information Cs and Cr, which is combined with the motion information Cm by a multiplexer 73 as in the preceding embodiments.
The encoded information Cs and Cr is also decoded by decoders 31 and 32 to produce decoded images S_{1} and D'_{n}. These are supplied to twodimensional smoothing and downsampling filters 48m (m=1 to n1), edge synthesizers 49m (m=1 to n), and an inverse wavelet transform processor 50 which are identical to the corresponding elements in the digital image decoder of the second embodiment, shown in FIG. 6. The inverse wavelet transform processor 50 outputs a reconstructed difference image D'_{0} to a motion compensator 81.
Using this image D'_{0}, the motion vectors M output by the motion estimator 80, and a previous image X"_{0} stored in a memory 82, the motion compensator 81 produces a new reconstructed image X'_{0}, which is stored in the memory 82. The previous image X"_{0} was also supplied to the motion estimator 80 for use in motion estimation. The reconstructed image X'_{0} will become the previous image in the next frame.
The motion estimator 80, motion compensator 81 and memory 82 in FIG. 23 operate like the motion estimator 67, motion compensator 70, and memory 72 in the fifth or seventh embodiment (FIG. 17 or 20), although they operate on the original image X_{0} instead of the reduced image X_{n}. The inverse waveform processors 50m in FIG. 23 operate like the inverse waveform processors 50m in the second embodiment (FIG. 6), although they process a series of partially reconstructed difference images D'_{m} instead of a series of partially reconstructed images X'_{m} (m=1 to n), and output a fully reconstructed difference image D'_{0} instead of a fully reconstructed image X'_{0}.
The encoding section 83 in the ninth embodiment comprises the motion vector encoder 68, edge image encoder 24, reduced image encoder 25, and multiplexer 73.
Referring to FIG. 24, the digital image decoder of the ninth embodiment has a decoding section 84 with a demultiplexer 76 and motion vector decoder 77 similar to those in the fifth to eighth embodiments. The other elements in FIG. 24 are similar to the elements with the same reference numerals in FIG. 23. The motion compensator 81 receives a fully reconstructed difference image D'_{0} from the inverse wavelet transform processor 50, and outputs a fully reconstructed image X'_{0} to the input/output device 37.
As in the fifth to eighth embodiments, the digital image encoder in FIG. 23 and digital image decoder in FIG. 24 operate with identical previous images X"_{0}, thereby maintaining consistency between motion estimation in the encoder and decoder. Performing motion estimation before edge detection can lead to higher compression ratios than in the fifth to eighth embodiments, because only one set of motion vectors has to be encoded.
Tenth Embodiment
The tenth embodiment combines features of the third and ninth embodiments. Three edge images are detected, as in the third embodiment. Edge detection, smoothing, and downsampling are preceded by motion estimation, and followed by all the processes performed in the digital image decoder, as in the ninth embodiment.
Referring to FIG. 25, the digital image encoder of the tenth embodiment has the motion estimator 80, motion compensator 81, memory 82, and encoding section 83 of the ninth embodiment (FIG. 23), and other elements as in the third embodiment (FIGS. 9 and 11). Referring to FIG. 26, the digital image decoder has the decoding section 84, motion compensator 81, and memory 82 of the ninth embodiment (FIG. 24), and other elements as in the third embodiment (FIG. 11).
The tenth embodiment operates as described in the third and ninth embodiments, and provides similar advantages. A detailed description will be omitted.
Eleventh Embodiment
The eleventh embodiment is a digital image encoder/decoder (codec) that combines the functions of the digital image encoder and decoder of the ninth embodiment. The eleventh embodiment thus provides one device that can operate in either an encoding mode or a decoding mode.
Referring to FIG. 27, to the digital image encoder of the ninth embodiment (FIG. 23), the eleventh embodiment adds the demultiplexer 76 and motion vector decoder 77 of the digital image decoder of the ninth embodiment (FIG. 24), a second input terminal 90, and three switches 91, 92, and 93.
To encode a moving image, switches 91, 92, and 93 are set to the position marked "a" in the drawing. Successive frames X_{0} are input at input terminal 10 and processed just as in the digital image encoder of the ninth embodiment, the encoded results being output from the demultiplexer 73 to input/output devise 28.
To decode an encoded moving image, switches 91, 92, and 93 are set to the position marked "b." The encoded moving image C is input from the second input terminal 90 to the demultiplexer 76 and processed just as in the digital image decoder of the ninth embodiment, the decoded results being output from the motion compensator 81 to input/output device 37.
The eleventh embodiment has the same advantages as the ninth embodiment, with the further advantage of allowing the encoding and decoding functions to share the same motion compensator 81, decoders 31 and 32, filters 48m, edge synthesizers 49m, and inverse wavelet transform processor 50, thereby avoiding needless duplication. This embodiment is particularly suited for videophones and other bidirectional video systems.
Twelfth Embodiment
The twelfth embodiment is a digital image encoder/decoder that combines the features of the tenth and eleventh embodiments.
Referring to FIG. 28, the twelfth embodiment adds to the digital image encoder of the tenth embodiment (FIG. 25) the same demultiplexer 76, motion vector decoder 77, second input terminal 90, and switches 91, 92, and 93 as in the eleventh embodiment. The twelfth embodiment operates like the eleventh, but has the further advantage of improved rendition of highfrequency image detail due to diagonal edge detection.
As described above, the invented encoding and decoding method and devices provide high compression rates with modest amounts of computation. Although there is some loss of detail, reconstructed images are not distorted, and because highfrequency information is synthesized in a single step instead of by iteration, the reconstruction process is reliably completed within a short and definite time. The invention can be usefully applied to both still and moving images for a wide variety of purposes, in a wide variety of systems.
The scope of the invention is not restricted to the embodiments described above, but includes such further variations as the following.
In the first, second, fifth, sixth, ninth, and eleventh embodiments, instead of producing separate horizontal and vertical edge images, the edge detector 21 can output a single edge image in which each edge point has both a sharpness value S(i, j) and direction angle θ (i, j), computed from Sh(i, j) and Sv(i, j) by, for example, the following formulas:
S(i, j)=sgn Sh(i, j)!· Sh(i, j).sup.2 +Sv(i, j).sup.2 !.sup.1/2
θ (i , j)=arctan Sv(i, j)/Sh(i, j)!
In these formulas, sgn denotes the algebraic sign (plus or minus one), and arctan the inverse tangent function. The encoding section then has only one set of edge positions to encode. The decoding section in the digital image decoder can recover Sh(i, j) and Sv(i, j) from S(i, j) and θ (i, j).
This method of encoding only a single edge image is particularly effective if chain encoding is employed, because the direction angle tends to change only slowly along a chain of edge points. Moreover, the angle tends to be perpendicular to the direction of the chain, so it is possible to have the edge detector calculate only the above edge sharpness S(i, j), and let the decoder infer the direction angle θ (i, j) of an edge point (i, j) from the direction of the chain in the vicinity of that point. The compression ratio can be significantly improved in this way, although with some loss of reconstruction accuracy.
This encoding method can also be applied to the horizontal and vertical edge images in the other embodiments, if the original image is not downsampled before horizontal and vertical edge detection. The necessary downsampling is then performed instead in the decoder, as in the second embodiment. In the fifth, sixth, seventh, and eighth embodiments, this encoding method has the further advantage of reducing the number of edge images for which motion vectors have to be computed.
In the digital image encoder in any of the embodiments, it is possible to have the twodimensional smoothing performed in a single step, instead of by a cascaded series of filters. The filters h_{1}, h_{2}, . . . , h_{n} in the first embodiment, for example, can be mutually convolved to yield a single filter that gives the same result as applying the cascaded series of filters. For use in the other embodiments, downsampling can be incorporated into this single filter simply by calculating the values of only 1/4^{n} of the output pixels.
The smoothing filters applied in the horizontal and vertical directions need not be identical. A smoothing filter with an even number of taps can be applied in one of these directions, and a smoothing filter with all odd number of taps in the other direction. The edge synthesis filters (e), completion filters (f), and highpass filters (g) employed for decoding then also require separate versions for the horizontal and vertical directions.
Edge detection is not limited to the differenceofdifferences method. This method is preferred, but other methods can be employed.
The filters employed were described as being symmetric around zero or 1/2, but this is only a matter of notation. The center of symmetry can be disposed at any convenient location.
The standard edge from which the edge synthesis filters are generated need not be an edge of the type shown in FIG. 3, in which pixel values change at first one constant rate, then another constant rate. Other types can be used, provided the standard edge has only one point which would be detected as an edge point by the edge detector.
Those skilled in the art will recognize that still further variations are possible without departing from the scope claimed below.
Claims (30)
Priority Applications (6)
Application Number  Priority Date  Filing Date  Title 

JP6264836  19941028  
JP26483694A JPH08123967A (en)  19941028  19941028  Edge detecting device for image 
JP6264835  19941028  
JP26483594  19941028  
JP15014295A JP3308767B2 (en)  19941028  19950616  Picture coding and decoding apparatus, image decoding apparatus 
JP7150142  19950616 
Applications Claiming Priority (4)
Application Number  Priority Date  Filing Date  Title 

US09069870 US5878172A (en)  19941028  19980430  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081104 US5991448A (en)  19941028  19980519  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081780 US5949910A (en)  19941028  19980520  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09082729 US6101277A (en)  19941028  19980521  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
Related Child Applications (4)
Application Number  Title  Priority Date  Filing Date 

US09069870 Division US5878172A (en)  19941028  19980430  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081104 Division US5991448A (en)  19941028  19980519  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081780 Division US5949910A (en)  19941028  19980520  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09082729 Division US6101277A (en)  19941028  19980521  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
Publications (1)
Publication Number  Publication Date 

US5761341A true US5761341A (en)  19980602 
Family
ID=27319874
Family Applications (5)
Application Number  Title  Priority Date  Filing Date 

US08548943 Expired  Lifetime US5761341A (en)  19941028  19951026  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09069870 Expired  Lifetime US5878172A (en)  19941028  19980430  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081104 Expired  Lifetime US5991448A (en)  19941028  19980519  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081780 Expired  Lifetime US5949910A (en)  19941028  19980520  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09082729 Expired  Lifetime US6101277A (en)  19941028  19980521  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
Family Applications After (4)
Application Number  Title  Priority Date  Filing Date 

US09069870 Expired  Lifetime US5878172A (en)  19941028  19980430  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081104 Expired  Lifetime US5991448A (en)  19941028  19980519  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09081780 Expired  Lifetime US5949910A (en)  19941028  19980520  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US09082729 Expired  Lifetime US6101277A (en)  19941028  19980521  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
Country Status (4)
Country  Link 

US (5)  US5761341A (en) 
EP (1)  EP0709809B1 (en) 
KR (1)  KR100388377B1 (en) 
DE (2)  DE69525127T2 (en) 
Cited By (34)
Publication number  Priority date  Publication date  Assignee  Title 

US5878172A (en) *  19941028  19990302  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US5933535A (en) *  19951005  19990803  Microsoft Corporation  Objectbased video compression process employing arbitrarilyshaped features 
US5949912A (en) *  19960628  19990907  Oki Electric Industry Co., Ltd.  Image coding method and apparatus 
US6037988A (en) *  19960322  20000314  Microsoft Corp  Method for generating sprites for objectbased coding sytems using masks and rounding average 
WO2000027642A1 (en) *  19981110  20000518  Lexmark International, Inc.  Low electromagnetic emissions and improved signal quality video drive architecture for laser printers 
WO2000033243A1 (en) *  19981130  20000608  Equator Technologies, Inc.  Image processing circuit and method for reducing a difference between pixel values across an image boundary 
US6073849A (en) *  19961101  20000613  Psc Scanning, Inc.  Electronic edge detection system using a second derivative signal processor 
US6122318A (en) *  19961031  20000919  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US6125211A (en) *  19970916  20000926  Lu; ChungYa  Progressive image transmission 
US6259738B1 (en) *  19961031  20010710  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US6348926B1 (en) *  19980122  20020219  Matsushita Electric Industrial Co., Ltd.  Image conversion method and conversion device 
US6400831B2 (en)  19980402  20020604  Microsoft Corporation  Semantic video object segmentation and tracking 
US20030043285A1 (en) *  20010905  20030306  Fuji Photo Film Co., Ltd.  Image data forming method and apparatus 
US6611627B1 (en) *  20000424  20030826  Eastman Kodak Company  Digital image processing method for edge shaping 
US20030161531A1 (en) *  20000321  20030828  Gianfranco De Grandi  Method of multitime filtering coherentsensor detected images 
US6671410B2 (en) *  19970324  20031230  Minolta Co., Ltd.  Image processing apparatus that can have picture quality improved in reproduced original image data 
US20040037472A1 (en) *  19981223  20040226  Xerox Corporation  System and method for directed acuity segmentation resolution compression and decompression 
US6711278B1 (en)  19980910  20040323  Microsoft Corporation  Tracking semantic objects in vector image sequences 
US20040141557A1 (en) *  20030116  20040722  Samsung Electronics Co. Ltd.  Methods and apparatus for removing blocking artifacts of MPEG signals in realtime video reception 
WO2004072802A2 (en) *  20030211  20040826  New Jersey Institute Of Technology  Face detection method and apparatus 
WO2005004040A1 (en) *  20030702  20050113  Celartem Technology Inc.  Image sharpening with region edge sharpness correction 
US20050265577A1 (en) *  20020226  20051201  Truelight Technologies, Llc  Realtime software video/audio transmission and display with content protection against camcorder piracy 
US20060093225A1 (en) *  20041029  20060504  Sandeep Bhatia  System, method, and apparatus for providing massively scaled down video using iconification 
US20060133510A1 (en) *  20041216  20060622  Rahul Saxena  Local macroblock information buffer 
US20070269120A1 (en) *  20060517  20071122  Motorola, Inc.  Video image compression using model plus difference image 
US20090003720A1 (en) *  20070628  20090101  Microsoft Corporation  Efficient image representation by edges and lowresolution signal 
US7558310B1 (en) *  20010109  20090707  Urbain Alfred von der Embse  Multiscale code division frequency/wavelet multiple access 
US20090284651A1 (en) *  20080519  20091119  Microsoft Corporation  Quantization and differential coding of alpha image data 
US7653255B2 (en)  20040602  20100126  Adobe Systems Incorporated  Image region of interest encoding 
US20100098338A1 (en) *  20081016  20100422  Keyence Corporation  Method for Deciding Image Data Reduction Ratio in Image Processing, Pattern Model Positioning Method in Image Processing, Pattern Model Creating Method in Image Processing, Image Processing Apparatus, Image Processing Program, and Computer Readable Recording Medium 
US20110103702A1 (en) *  20091104  20110505  Samsung Electronics Co., Ltd.  Apparatus and method of compressing and restoring image using filter information 
US20120250995A1 (en) *  20110401  20121004  Sony Corporation  Image processing device and image processing method 
US20160253779A1 (en) *  20150227  20160901  Samsung Electronics Co., Ltd.  Image processing apparatus and method 
US20170105007A1 (en) *  20020930  20170413  Samsung Electronics Co., Ltd.  Image coding method and apparatus using spatial predictive coding of chrominance and image decoding method apparatus 
Families Citing this family (31)
Publication number  Priority date  Publication date  Assignee  Title 

US6226414B1 (en) *  19940420  20010501  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US6148115A (en) *  19961108  20001114  Sony Corporation  Image processing apparatus and image processing method 
US6018368A (en) *  19970711  20000125  Samsung ElectroMechanics Co., Ltd.  Scalable encoding apparatus and method with improved function of scaling motion vector 
US6272256B1 (en) *  19971126  20010807  International Business Machines Corporation  Fast compression of periodic halftoned bitonal images 
US6898319B1 (en) *  19980911  20050524  Intel Corporation  Method and system for video frame enhancement using edge detection 
US6820202B1 (en) *  19981109  20041116  First Data Corporation  Account authority digital signature (AADS) system 
US7047416B2 (en) *  19981109  20060516  First Data Corporation  Accountbased digital signature (ABDS) system 
JP3206584B2 (en) *  19990205  20010910  日本電気株式会社  Image data processing method and apparatus 
WO2000060532A9 (en) *  19990401  20020620  Univ Maryland  Method and apparatus for restoration of low resolution images 
US6636633B2 (en)  19990503  20031021  Intel Corporation  Rendering of photorealistic computer graphics images 
US7170941B2 (en) *  19990813  20070130  Patapsco Designs Inc.  Temporal compression 
US6456663B1 (en) *  20000329  20020924  Matsushita Electric Industrial Co., Ltd.  DCT domain down conversion system that compensates for IDCT mismatch 
WO2001098747A3 (en) *  20000622  20020411  Univ Arizona  Agonist versus antagonist binding to g proteincoupled receptors 
US7552333B2 (en) *  20000804  20090623  First Data Corporation  Trusted authentication digital signature (tads) system 
EP1320953A4 (en) *  20000804  20061025  First Data Corp  Method and apparatus for access authentication entity 
US7248257B2 (en) *  20010214  20070724  Technion Research & Development Foundation Ltd.  Low bandwidth transmission of 3D graphical data 
GB0108080D0 (en) *  20010330  20010523  Univ Bath  Audio compression 
US7151844B2 (en) *  20011206  20061219  General Motors Corporation  Image sensor method and apparatus having hardware implemented edge detection processing 
JP4169522B2 (en) *  20020322  20081022  株式会社リコー  Image processing apparatus, image processing program, and storage medium for storing the program 
US6757434B2 (en) *  20021112  20040629  Nokia Corporation  Regionofinterest tracking method and device for waveletbased video coding 
GB0227743D0 (en) *  20021128  20030108  Rolls Royce Plc  Wavelet compression 
US20050105611A1 (en) *  20031117  20050519  Gisle Bjontegaard  Video compression method 
JP2005275900A (en) *  20040325  20051006  Noritsu Koki Co Ltd  Image processing method for particulate noise suppression, program and image processor executing this method 
JP4626473B2 (en) *  20051012  20110209  セイコーエプソン株式会社  Image processing apparatus, image output apparatus and method 
US7518618B2 (en) *  20051223  20090414  Xerox Corporation  Antialiased tagging using lookup table edge pixel identification 
US20100014777A1 (en) *  20080719  20100121  Headplay (Barbados) Inc.  System and method for improving the quality of compressed video signals by smoothing the entire frame and overlaying preserved detail 
US8428364B2 (en)  20100115  20130423  Dolby Laboratories Licensing Corporation  Edge enhancement for temporal scaling with metadata 
US9661301B2 (en) *  20110218  20170523  Sony Corporation  Image processing device and image processing method 
US20140286397A1 (en) *  20110929  20140925  Dolby Laboratories Licensing Corporation  Framecompatible fullresolution stereoscopic 3D video delivery with symmetric picture resolution and quality 
KR101538010B1 (en) *  20120831  20150722  주식회사 에이투텍  Apparatus and Method for Encoding Image, Apparatus and Method for Decoding Image 
US20180007362A1 (en) *  20160630  20180104  Sony Interactive Entertainment Inc.  Encoding/decoding digital frames by downsampling/upsampling with enhancement information 
Citations (7)
Publication number  Priority date  Publication date  Assignee  Title 

US5420636A (en) *  19920519  19950530  Sony Corporation  Apparatus for and method of transmitting video signal 
EP0679032A2 (en) *  19940420  19951025  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge systhesis and inverse wavelet transform 
US5481269A (en) *  19940527  19960102  Westinghouse Electric Corp.  General frame wavelet classifier 
US5546477A (en) *  19930330  19960813  Klics, Inc.  Data compression and decompression 
US5561724A (en) *  19930611  19961001  Hitachi Medical Corporation  Method conducting smoothing processing in a linear direction having a minimized magnitude of density change 
US5598481A (en) *  19940429  19970128  Arch Development Corporation  Computeraided method for image feature analysis and diagnosis in mammography 
US5661822A (en) *  19930330  19970826  Klics, Ltd.  Data compression and decompression 
Family Cites Families (5)
Publication number  Priority date  Publication date  Assignee  Title 

US4685143A (en) *  19850321  19870804  Texas Instruments Incorporated  Method and apparatus for detecting edge spectral features 
JP3182015B2 (en) *  19930127  20010703  テキサス インスツルメンツ インコーポレイテツド  The method of synthesizing optical images 
US5495292A (en) *  19930903  19960227  Gte Laboratories Incorporated  Interframe wavelet transform coder for color video compression 
KR0150150B1 (en) *  19940714  19981015  김광호  System of error diffusion algorithm with edge enhancement 
DE69525127T2 (en) *  19941028  20021002  Oki Electric Ind Co Ltd  Apparatus and method for encoding and decoding images using an edge synthesis and an inverse wavelet transform 
Patent Citations (7)
Publication number  Priority date  Publication date  Assignee  Title 

US5420636A (en) *  19920519  19950530  Sony Corporation  Apparatus for and method of transmitting video signal 
US5546477A (en) *  19930330  19960813  Klics, Inc.  Data compression and decompression 
US5661822A (en) *  19930330  19970826  Klics, Ltd.  Data compression and decompression 
US5561724A (en) *  19930611  19961001  Hitachi Medical Corporation  Method conducting smoothing processing in a linear direction having a minimized magnitude of density change 
EP0679032A2 (en) *  19940420  19951025  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge systhesis and inverse wavelet transform 
US5598481A (en) *  19940429  19970128  Arch Development Corporation  Computeraided method for image feature analysis and diagnosis in mammography 
US5481269A (en) *  19940527  19960102  Westinghouse Electric Corp.  General frame wavelet classifier 
NonPatent Citations (8)
Title 

ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing, Toronto, Ontario, Canada, 14 17 May 1991 pp. 2745 2748. Mallat et al. Compact Image Coding from Edges with Wavelets . * 
ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing, Toronto, Ontario, Canada, 1417 May 1991 pp. 27452748. Mallat et al. "Compact Image Coding from Edges with Wavelets". 
IEEE Transaction on Communications, vol. No. 4, Apr. 1983, pp. 532 540. P.J. Burt et al. The Laplacian Pyramid as a Compact Image Code . * 
IEEE Transaction on Communications, vol. No. 4, Apr. 1983, pp. 532540. P.J. Burt et al. "The Laplacian Pyramid as a Compact Image Code". 
R. Mont u far Chaveznava et al. Astronomical Image Coding using an Edge and Texture Model , Proceedings of the IEEE SP International Symposium on Time Frequency and Time Scale Analysis, 25 28 Oct. 1994, pp. 148 151. * 
R. MontufarChaveznava et al. "Astronomical Image Coding using an Edge and Texture Model", Proceedings of the IEEESP International Symposium on TimeFrequency and TimeScale Analysis, 2528 Oct. 1994, pp. 148151. 
Staphane Mallat et al, "Compact Image Coding from Edges with Wavelets", ICASSP Proceedings, IEEE, 1991, pp. 27452748. 
Staphane Mallat et al, Compact Image Coding from Edges with Wavelets , ICASSP Proceedings, IEEE, 1991, pp. 2745 2748. * 
Cited By (79)
Publication number  Priority date  Publication date  Assignee  Title 

US5878172A (en) *  19941028  19990302  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US6101277A (en) *  19941028  20000808  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US5991448A (en) *  19941028  19991123  Oki Electric Industry Co., Ltd.  Image encoding and decoding method and apparatus using edge synthesis and inverse wavelet transform 
US5959673A (en) *  19951005  19990928  Microsoft Corporation  Transform coding of dense motion vector fields for frame and object based video coding applications 
US6026182A (en) *  19951005  20000215  Microsoft Corporation  Feature segmentation 
US5933535A (en) *  19951005  19990803  Microsoft Corporation  Objectbased video compression process employing arbitrarilyshaped features 
US6037988A (en) *  19960322  20000314  Microsoft Corp  Method for generating sprites for objectbased coding sytems using masks and rounding average 
US5949912A (en) *  19960628  19990907  Oki Electric Industry Co., Ltd.  Image coding method and apparatus 
US6754269B1 (en) *  19961031  20040622  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US7167521B2 (en)  19961031  20070123  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US7215709B2 (en)  19961031  20070508  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US6122318A (en) *  19961031  20000919  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US7308031B2 (en)  19961031  20071211  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US20050084016A1 (en) *  19961031  20050421  Noboru Yamaguchi  Video encoding apparatus and video decoding apparatus 
US20050053154A1 (en) *  19961031  20050310  Noboru Yamaguchi  Video encoding apparatus and video decoding apparatus 
US6259738B1 (en) *  19961031  20010710  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US6292514B1 (en) *  19961031  20010918  Kabushiki Kaisha Toshiba  Video encoding apparatus and video decoding apparatus 
US20070147514A1 (en) *  19961031  20070628  Noboru Yamaguchi  Video encoding apparatus and video decoding apparatus 
US20040091049A1 (en) *  19961031  20040513  Noboru Yamaguchi  Video encoding apparatus and video decoding apparatus 
US6073849A (en) *  19961101  20000613  Psc Scanning, Inc.  Electronic edge detection system using a second derivative signal processor 
US6671410B2 (en) *  19970324  20031230  Minolta Co., Ltd.  Image processing apparatus that can have picture quality improved in reproduced original image data 
US6125211A (en) *  19970916  20000926  Lu; ChungYa  Progressive image transmission 
US6348926B1 (en) *  19980122  20020219  Matsushita Electric Industrial Co., Ltd.  Image conversion method and conversion device 
US6400831B2 (en)  19980402  20020604  Microsoft Corporation  Semantic video object segmentation and tracking 
US7088845B2 (en)  19980910  20060808  Microsoft Corporation  Region extraction in vector images 
US20040189863A1 (en) *  19980910  20040930  Microsoft Corporation  Tracking semantic objects in vector image sequences 
US7162055B2 (en)  19980910  20070109  Microsoft Corporation  Tracking semantic objects in vector image sequences 
US6711278B1 (en)  19980910  20040323  Microsoft Corporation  Tracking semantic objects in vector image sequences 
US6147699A (en) *  19981110  20001114  Lexmark International, Inc.  Low electromagnetic emissions and improved signal quality video drive architecture for laser printers 
WO2000027642A1 (en) *  19981110  20000518  Lexmark International, Inc.  Low electromagnetic emissions and improved signal quality video drive architecture for laser printers 
WO2000033243A1 (en) *  19981130  20000608  Equator Technologies, Inc.  Image processing circuit and method for reducing a difference between pixel values across an image boundary 
US6236764B1 (en) *  19981130  20010522  Equator Technologies, Inc.  Image processing circuit and method for reducing a difference between pixel values across an image boundary 
US7123771B2 (en) *  19981223  20061017  Xerox Corporation  System and method for directed acuity segmentation resolution compression and decompression 
US6771827B1 (en)  19981223  20040803  Xerox Corporation  System and method for directed acuity segmentation resolution compression and decompression 
US6853754B2 (en)  19981223  20050208  Xerox Corporation  System and method for directed acuity segmentation resolution compression and decompression 
US20040037472A1 (en) *  19981223  20040226  Xerox Corporation  System and method for directed acuity segmentation resolution compression and decompression 
US20030161531A1 (en) *  20000321  20030828  Gianfranco De Grandi  Method of multitime filtering coherentsensor detected images 
US6611627B1 (en) *  20000424  20030826  Eastman Kodak Company  Digital image processing method for edge shaping 
US7558310B1 (en) *  20010109  20090707  Urbain Alfred von der Embse  Multiscale code division frequency/wavelet multiple access 
US20030043285A1 (en) *  20010905  20030306  Fuji Photo Film Co., Ltd.  Image data forming method and apparatus 
US7085428B2 (en) *  20010905  20060801  Fuji Photo Film Co., Ltd.  Image data forming method and apparatus 
US20050265577A1 (en) *  20020226  20051201  Truelight Technologies, Llc  Realtime software video/audio transmission and display with content protection against camcorder piracy 
US8068683B2 (en)  20020226  20111129  Amof Advance Limited Liability Company  Video/audio transmission and display 
US7630569B2 (en)  20020226  20091208  Decegama Angel  Realtime software video/audio transmission and display with content protection against camcorder piracy 
US20100054701A1 (en) *  20020226  20100304  Decegama Angel  Realtime software video/audio transmission and display with content protection against camcorder piracy 
US20170105007A1 (en) *  20020930  20170413  Samsung Electronics Co., Ltd.  Image coding method and apparatus using spatial predictive coding of chrominance and image decoding method apparatus 
US9866842B2 (en) *  20020930  20180109  Samsung Electronics Co., Ltd.  Image coding method and apparatus using spatial predictive coding of chrominance and image decoding method apparatus 
US7463688B2 (en) *  20030116  20081209  Samsung Electronics Co., Ltd.  Methods and apparatus for removing blocking artifacts of MPEG signals in realtime video reception 
US20040141557A1 (en) *  20030116  20040722  Samsung Electronics Co. Ltd.  Methods and apparatus for removing blocking artifacts of MPEG signals in realtime video reception 
US7162076B2 (en) *  20030211  20070109  New Jersey Institute Of Technology  Face detection method and apparatus 
WO2004072802A2 (en) *  20030211  20040826  New Jersey Institute Of Technology  Face detection method and apparatus 
WO2004072802A3 (en) *  20030211  20050609  New Jersey Tech Inst  Face detection method and apparatus 
US20040190760A1 (en) *  20030211  20040930  Chengjun Liu  Face detection method and apparatus 
WO2005004040A1 (en) *  20030702  20050113  Celartem Technology Inc.  Image sharpening with region edge sharpness correction 
US20050025383A1 (en) *  20030702  20050203  Celartem Technology, Inc.  Image sharpening with region edge sharpness correction 
US7653255B2 (en)  20040602  20100126  Adobe Systems Incorporated  Image region of interest encoding 
US20060093225A1 (en) *  20041029  20060504  Sandeep Bhatia  System, method, and apparatus for providing massively scaled down video using iconification 
US7574059B2 (en) *  20041029  20090811  Broadcom Corporation  System, method, and apparatus for providing massively scaled down video using iconification 
US8363730B2 (en) *  20041216  20130129  Intel Corporation  Local macroblock information buffer 
US20150103913A1 (en) *  20041216  20150416  Rahul Saxena  Local macroblock information buffer 
US8811494B2 (en)  20041216  20140819  Intel Corporation  Local macroblock information buffer 
US9172969B2 (en) *  20041216  20151027  Intel Corporation  Local macroblock information buffer 
US20060133510A1 (en) *  20041216  20060622  Rahul Saxena  Local macroblock information buffer 
US20070269120A1 (en) *  20060517  20071122  Motorola, Inc.  Video image compression using model plus difference image 
WO2007136930A2 (en) *  20060517  20071129  Motorola, Inc.  Video/image compression using model plus difference image 
WO2007136930A3 (en) *  20060517  20080508  Motorola Inc  Video/image compression using model plus difference image 
US8116581B2 (en) *  20070628  20120214  Microsoft Corporation  Efficient image representation by edges and lowresolution signal 
CN101689297B (en)  20070628  20120905  微软公司  Efficient image representation by edges and lowresolution signal 
US20090003720A1 (en) *  20070628  20090101  Microsoft Corporation  Efficient image representation by edges and lowresolution signal 
WO2009006056A3 (en) *  20070628  20090305  Microsoft Corp  Efficient image representation by edges and lowresolution signal 
US8208543B2 (en)  20080519  20120626  Microsoft Corporation  Quantization and differential coding of alpha image data 
US20090284651A1 (en) *  20080519  20091119  Microsoft Corporation  Quantization and differential coding of alpha image data 
US20100098338A1 (en) *  20081016  20100422  Keyence Corporation  Method for Deciding Image Data Reduction Ratio in Image Processing, Pattern Model Positioning Method in Image Processing, Pattern Model Creating Method in Image Processing, Image Processing Apparatus, Image Processing Program, and Computer Readable Recording Medium 
US8155473B2 (en) *  20081016  20120410  Keyence Corporation  Method for deciding image data reduction ratio in image processing, pattern model positioning method in image processing, pattern model creating method in image processing, image processing apparatus, image processing program, and computer readable recording medium 
US9172974B2 (en) *  20091104  20151027  Samsung Electronics Co., Ltd.  Apparatus and method of compressing and restoring image using filter information 
US20110103702A1 (en) *  20091104  20110505  Samsung Electronics Co., Ltd.  Apparatus and method of compressing and restoring image using filter information 
US8625893B2 (en) *  20110401  20140107  Sony Corporation  Image processing device and image processing method 
US20120250995A1 (en) *  20110401  20121004  Sony Corporation  Image processing device and image processing method 
US20160253779A1 (en) *  20150227  20160901  Samsung Electronics Co., Ltd.  Image processing apparatus and method 
Also Published As
Publication number  Publication date  Type 

DE69525127T2 (en)  20021002  grant 
US6101277A (en)  20000808  grant 
EP0709809A2 (en)  19960501  application 
DE69525127D1 (en)  20020314  grant 
KR100388377B1 (en)  20031128  grant 
US5878172A (en)  19990302  grant 
US5949910A (en)  19990907  grant 
US5991448A (en)  19991123  grant 
EP0709809B1 (en)  20020123  grant 
EP0709809A3 (en)  19970205  application 
Similar Documents
Publication  Publication Date  Title 

Schuster et al.  RateDistortion based video compression: optimal video frame compression and object boundary encoding  
Chang  Compresseddomain techniques for image/video indexing and manipulation  
US5610657A (en)  Video compression using an iterative error data coding method  
US6466698B1 (en)  Efficient embedded image and video compression system using lifted wavelets  
US5592228A (en)  Video encoder using global motion estimation and polygonal patch motion estimation  
US5805739A (en)  Lapped orthogonal vector quantization  
US5917609A (en)  Hybrid waveform and modelbased encoding and decoding of image signals  
Zandi et al.  CREW: Compression with reversible embedded wavelets  
US5841473A (en)  Image sequence compression and decompression  
US20030206582A1 (en)  2D transforms for image and video coding  
Uz et al.  Interpolative multiresolution coding of advance television with compatible subchannels  
US5546477A (en)  Data compression and decompression  
US20060239345A1 (en)  Method of signalling motion information for efficient scalable video compression  
US5659363A (en)  Coding and decoding of video signals  
US5661822A (en)  Data compression and decompression  
US20070091997A1 (en)  Method And Apparatus For Scalable Video Decoder Using An Enhancement Stream  
US6219458B1 (en)  Overlapped reversible transforms for unified lossless/lossy compression  
US5903679A (en)  Method for decoding signal vectorquantized by blocks  
US20020090028A1 (en)  Codec system and method for spatially scalable video data  
US5512956A (en)  Adaptive spatialtemporal postprocessing for low bitrate coded image sequences  
US6873734B1 (en)  Method and apparatus for compression using reversible wavelet transforms and an embedded codestream  
US20050084166A1 (en)  Image processing using probabilistic local behavior assumptions  
US20030133500A1 (en)  Method and apparatus for subband encoding and decoding  
US6996186B2 (en)  Programmable horizontal filter with noise reduction and image scaling for video encoding system  
US20060088098A1 (en)  Method and arrangement for reducing the volume or rate of an encoded digital video bitstream 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: OKI ELECTRIC CO., LTD., JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GO, SHIYU;REEL/FRAME:007765/0190 Effective date: 19951002 

FPAY  Fee payment 
Year of fee payment: 4 

FPAY  Fee payment 
Year of fee payment: 8 

FPAY  Fee payment 
Year of fee payment: 12 

AS  Assignment 
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OKI ELECTRIC INDUSTRY CO., LTD.;REEL/FRAME:033546/0400 Owner name: GLOBAL D, LLC., JAPAN Effective date: 20140724 

AS  Assignment 
Owner name: INPHI CORPORATION, CALIFORNIA Effective date: 20140729 Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GLOBAL D, LLC.;REEL/FRAME:034193/0116 