IL100589A - Apparatus and method for smoothing images - Google Patents

Description

l60l4skl 13.12. 2 FIELD OF THE INVENTION The present invention relates to apparatus and methods for smoothing images and reducing noise. 3ACKGR0UND OF THE INVENTION A known problem is digital image noise reduction in the face of (a) randomly distributed noise, which is often additive, (b) fixed pattern noise due to imaging detector response non-uniformities, (c) analog recording noise of video signals due to video standard bandwidth limitations and luminance/chrominance signal formats, and (d) compression noise such as block noise and edge noise or mosquito noise created by block transform coding.

The need for image restoration in the face of noise exists in a wide range of applications such as electronic imaging and scanning, video recording equipment and analog and digital TV displays. Imaging sensors such as CCD-TV continuous and still cameras and medical imaging systems often face low light level situations, in which the image quality deteriorates due to reduced signal to noise ratios. Significant amplification of such video signals amplifies the various noise effects to the point where they are visible and disturbing to the observer. Electronic noise in still-video images is usually perceived as high frequency noise. In image sequences, electronic noise fluctuates randomly due to its random statistical nature, and can therefore be reduced by temporal integration.

Photo response non-uniformi ies of imaging detectors, such as CCD imagers, CCD image scanners and image facsimile machines, result in fixed-pattern noise. Its spatial structure depends on the internal design characteristics of the detector. CCD scanner detectors, for example, suffer from fixed-pattern noise caused by nonuni form ties in the detector element responsivi ties.. These are only partially correctable using digital calibrated processing schemes, the residual fixed-pattern noise remaining visible.

Fixed-pattern noise is particularly disturbing in still imagery. These effects are usually masked and not visually perceived in high contrast textured images. However, in low light level imaging situations where extensive signal amplification is required in order to perceive low contrasts, the fixed pattern noise effects are clearly visible and disturbing to the observer.

Image noise also appears in medical imaging applications, for example in ultrasound, and in photon-counting imaging systems. Image scanning applications also often require noise reduction, depending on the lighting conditions, and on the type of scanned data (imagery and text on paper or film).

Existing digital image noise reduction techniques can generally be categorized into three classes: (a) Spatial smoothing operators which utilize only spatial image information for reducing image noise, (b) temporal image integration operators which prolong the effective exposure time of an image changing over time hence reducing temporal random fluctuations of image noise, and (c) combinations of the techniques (a) and (b) .

Linear spatial smoothing operators, such as low pass filters, usually result in subjectively unacceptable blurring of essential high frequency image detail such as edges, lines and contours. More advanced filtering techniques such as Wiener filters adapt to local estimates of signal and noise according to statistical models of the signal noise processes, which are often difficulat to define a-priori. This type of technique is discussed in Mahesh, B. et al , "Adaptive estimators for filtering noisy images" .Optical engineering, 29(5). PP · 488 - 9k; 1990.

A Wiener filter is an example of a more general class of filters known as Kalman filters, described in Gelb, A. (ed. ) Applied optimal estimation , Technical staff, Analytic sciences corporation, MIT Press, Cambridge, MA, USA, 197^· Kalman filters require more intensive computation for local estimation of second order statistical parameters in the image. Kalman filtering techniques also rely on signal and noise models which are generally not appropriate for all images.

Other operators, such as median filters, do not require any a-priori knowledge of signal and noise models, and are designed to preserve high frequency edge signals while at the same time reducing the noise in smooth image regions. However, such operators introduce unwanted image noise effects due to the statistical nature of their pixel replication. This type of operator is discussed in Chin, R. T.; and Yeh, C. L., "Quantitative evaluation of some edge preseving noise-smoothing techniques", Computer vision , graphics and image processing , 23, ρ^^ 67 - 9 1 . 19δ 3 ■ Chin and Yeh also compare the operator to ther edge preserving operators.

Temporal image noise is often reduced by image ntegration techniques, for example by use of recursive running-verage filtering techniques, which are discussed in the above-eferenced publication by Gelb and in Rabiner, L. R. & Gold, B. heory and application of digital signal processing, Prentice-all, Englewood Cliffs, NJ, USA, particularly pp. 295 - 209 , 975 · However, in situations where motion occurs in the image, ue to camera motion and/or motion of an object in the scene, igh frequency image detail is usually compensated and blurred ue to the prolonged effective exposure time. Therefore, such ethods are unsuitable for many applications.

Two-directional low pass filtering techniques are iscussed in the context of dynamic range compression of images, η Guissin, R. "Adaptive dynamic range compression for FLIR magery", SPIE -- 6 th meeting in Israel on optical engineerin , ol. 1038 , pp. 299 - 306 , 1988 .

A theoretical and more' general treatment of two-irectional filtering of images is provided in the above-eferenced Rabiner and Gold publication. However, the described echniques do not provide visually pleasing results.

Heuristic techniques using fuzzy logic formulations ave been applied to noise reduction problems with limited uccess, as explained in Pal, S. K. and Majumder, D. K. D. Fuzzy athmatical approach to pattern recognitio , Halsted Press, John iley & Sons, NY, USA, 1986 .

A general .text providing background to the technology 100589/3 s™wn and described herein is Papoulis, A. Probabili ty , random variables and stochas tic processes , McGraw-Hill, Kogakusha Ltd. , 1 5.

The disclosures of all the above references is incorporated herein by reference.

I 100589/3 7 SUMMARY OF THE INVENTION The present invention seeks to provide improved noise eduction apparatus and also apparatus for noise reduction in njunction with one or several of the following functions: edge nhancement, temporal interpolation, magnification adjustment by patial interpolation, and dynamic range compression (DRC) .

The present invention also seeks to provide apparatus or image noise reduction which employs an adaptive, acuity-reserving, multi-directional and multi dimensional smoothing ethod. The method and apparatus of the present invention are pplicable, inter alia, for (a) adaptive spatial noise reduction n still images, (b) adaptive temporal noise reduction in time hanging image sequences, and (c) adaptive spatio-temporal noise eduction by combining the first two approaches (a) and (b) .

The image noise effects which may be reduced using the pparatus of the present invention include random photon and lectronic noise, fixed pattern noise, and analog recording noise rom a source such as video equipment. In spatial smoothing pplications , the present invention utilizes one-directional and wo-directional filtering schemes, which employ adaptive eighting of noisy measurements determined by easily computed ixel-based signal to noise measures, and preferably also tilizes precomputed steady state Kalman filter estimation gain arameters .

The signal to noise measures employed by the present nvention are designed to discriminate, in the presence of noise-nduced uncertainties, between occurrences of (a) edge signals, 100589/3 (b) line and contour signals, and (c) smooth brightness signals. In smooth image regions, also termed small uncertainties, extensive smoothing results in a dramatic reduction in image noise. In other locations where edge and line occurrences are hypothesized (high uncertainties), smoothing is miniaized so as to avoid blurring of sharp image features.

The spatial smoothing scheme of the present invention combines one-directional and two-direc ional adaptive filtering methods in a variety of one- and two-dimensional processing configurations. The configurations shown and described herein allow iterated computations in the presence of excessive noise, and may be implemented in various real-time imaging and scanning applications using an efficient pipeline arch tecture. When temporally smoothing image sequences, the same adaptive weighting schemes may be applied in the time domain, resulting in adaptive, running-average image integration configurations. The spatial and temporal noise reduction schemes may also be combined in spatio-temporal smoothing conf gurations by combining, for example, two-directional current image estimates and accumulated estimates : of previous images.

The image noise reduction method provided by the current invention provides effective noise reduction solutions both spatially and temporally. The present invention seeks to provide a general method of adaptive image smoothing, which can be adapted with a high degree of flexibility and on a pixel-by-pixel basis, according to a simple local signal and noise measure.

In accordance with the present invention, extensive 100589/3 smoothing is applied to certain image regions without degrading image quality as perceived by the human visual system. Appropriate adaptation of the smoothing mechanism is provided in transition areas between differently characterized image regions, so that abrupt brightness changes or edges are preserved without introducing unwanted visible noisy edge effects.

The proposed method utilizes adaptive one-directional and two-directional processing to extract, on a pixel -by-pixel basis, a criterion which determines a smoothing procedure suitable for the pixel signal and noise behavior. The intermediate results of the one-directional and two-directional processing may then be combined in any of various one- and two-dimensional spatial processing configurations, and three-dimensional spatio-temporal processing configurations disclosed herein .

A preferred embodiment of the present invention employs Kalman filter theory to provide an estimation gain parameter, as explained in detail below. Kalman filter theory is discussed in the above-referenced publication by Gelb. Alternatively, heuristic 'approaches may be employed to provide the estimation gain parameter, such as fuzzy logic theory, discussed in the above-referenced publication by Pal and Majumder.

The present invention seeks to provide an effective method for adaptive noise reduction in electronic images. The method incorporates an adaptive smoothing technique which determines, at each pixel in the image, the most suitable weighting of the current pixel measurement and its recursively computed estimates of neighboring pixels. The recursively 100589/3 computed estimates of neighboring pixels are determined by one- irectional and two-directional estimation filtering processes. eighboring pixels are each estimated on the basis of a different et of pixels. The sets are respectively arranged along ifferent directions relative to the current pixel.

Recursive estimates of adjoining pixels in the one- and wo-directional methods may be computed adaptively by means of a imply computed inage intensity signal to noise measure. A ocally computed edge signal measure normalized by an estimated mage noise measure such as a standard deviation estimate rovides an indication of pixel signal to noise ratio. The per- ixel computed signal to noise ratio is preferably employed to elect a smooth weighting function which is suitable for each of he following: (a) edge signals, (b) lines and contours, and (c) mooth surfaces.

An adaptive weighting function is computed recursively -priori for a range of signal-to-noise ratio values, preferably mploying a simplified, steady state, Kalman filter estimation ain parameter foraulation. The result of the operation of the alman filter -may be stored in lookup tables for rapid, easy ccess. If fuzzy logic methods are used instead of Kalman ilters, the adaptive weights stored in the lookup tables may be ermed "membership functions", as discussed in the above- eferenced publication by Pal and Majumder.

The one-directional and two-directional estimation echniques shown and described herein may be extended to multi- irectional processing. Also, the embodiments shown and described 100589/3 ^^rein may be extended to operate in two and three spatial dimensions, where previously smoothed pixels in adjoining image pixels and lines are incorporated in the smoothing process. Two- and three-dimensional, spatio-temporal noise reduction processing methods are also disclosed which combine previously smoothed images, such as video sequences with a spatially smoothed current image in order to provide a good quality estimate of the current image in the presence of uncertainties due to noise and motion.

The method of the present invention is applicable to a variety of image processing applications, including image enhancement, dynamic range compression, coding and compression, interpolation and electronic zoom applications.

A particular feature of the image smoothing devices shown and described herein is that the output therefrom is generally nonlinear relative to the input thereto.

One embodiment of the present invention is described in Appendix A, appended hereto. Other preferred embodiments of the present invention are described herein with reference to the figures .

In accordance with a preferred embodiment of the present invention, there is provided a method for acuity-preserving image smoothing including the steps of proceeding along a first dimension of received image pixels in a first direction and computing a first sequence of estimated pixel values from the received image pixels defined along the first direction, proceeding along the first dimension of received image pixels in a second opposite direction and 'computing a second sequence of estimated pixel values defined along the second 100589 3 and coaparing, for each individual pixel along the first dimension, an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the step of proceeding and computing includes, for an individual pixel in the first sequence, the steps of computing a difference value between the received image pixel and an adjacently preceding estimated pixel value in the first sequence, employing the received image pixel and at least one preceding estimated pixel values in the first sequence to estimate a first directional signal to noise ratio, generating an adjusted difference value to reflect the first directional signal to noise ratio, and employing the adjusted difference value to update an adjacently preceding estimated pixel value, thereby to compute the estimated pixel value of the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the step of comparing includes the steps of computing a difference value between the received image pixel and a function of adjacently preceding estimated pixel values in the first and second sequences, employing preceding estimated pixel values in the first and second sequences to estimate a two-directional signal to noise ratio, adjusting the difference value to reflect the signal to noise ratio, and employing the adjusted difference value to update a function of adjacently preceding e ^.mated pixel values in the first and second sequences, thereby to compute the improved estimated pixel value of the individual pixel.

Still further in accordance with a preferred embodiment of the present invention, the step of comparing includes the step of comparing the adjacently preceding estimated pixel values in the first and second sequences to at least one of the following parameters of the individual location: the estimated pixel value in the first sequence corresponding to the individual pixel, the estimated pixel value in the second sequence corresponding to the individual pixel, and the received image pixel.

Additionally ' n accordance with a preferred embodiment of the present invention, the method includes the steps of repeating, for at least a second dimension of received image pixels, the steps of proceeding in first and second directions and the step of comparing, thereby to compute at least one additional improved estimated pixel value for each individual pixel, and combining the at least two improved estimated pixel values, thereby to obtain a further improved two-dimensional estimated pixel value.

Further in accordance with a preferred embodiment of the present invention, the second dimension includes a time dimension .

Still further in accordance with a preferred embodiment of the present invention, the method includes the steps of proceeding along a second dimension of received image pixels in a first direction and computing a first sequence of second dimension estimated pixel values from the improved estimated i el values of the first dimension, proceeding along the second dimension of received image pixels in a second opposite direction and computing a second sequence of second dimension estimated pixel values from the improved estimated pixel values of the first dimension, and comparing, for each individual pixel along the second dimension, an adjacently preceding second dimension estimated pixel value in the first sequence of second dimension estimated pixel values and an adjacently preceding second dimension estimated pixel value in the second sequence of second dimension estimated pixel values, thereby to compute a further improved estimated pixel value for the individual pixel.

Additionally in accordance with a preferred embodiment of the present invention, the received image defines a second dimension thereof and a scanning direction in which the image is received along the second dimension, and the method also includes the steps of proceeding along the second dimension of received image pixels in the scanned direction and computing a sequence of second dimension estimated pixel values from the improved' estimated pixel values of the first dimension, and comparing, for each individual pixel along the second dimension, an adjacently preceding second dimension estimated pixel value in the sequence of second dimension estimated pixel values and an improved estimated pixel value of the first dimension which adjacently proceeds the individual pixel along the second dimension, thereby to compute a further improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of invention, the method includes the step of adjusting the results of the second dimension steps in order to reflect the difference between the received image and results of the first dimension steps.

Still further in accordance with a preferred embodiment of the present invention, the method also includes the step of scanning an image using an electronic scanner, thereby to define the received image pixels.

Additionally in accordance with a preferred embodiment f the present invention, the method also includes the step of eceiving the received image pixels from a video system.

There is also provided in accordance with a preferred mbodiment of the present invention apparatus for acuity- reserving image smoothing including apparatus for proceeding long a first dimension of received image pixels in a first irection and for computing a first sequence of estimated pixel alues from the received image pixels defined along the first irection, apparatus for proceeding along the first dimension of eceived image pixels in a second opposite direction and for omputing a second sequence of estimated pixel values defined long the second direction, and apparatus for comparing, for each ndividual pixel along the first dimension, an adjacently receding estimated pixel value in the first sequence of stimated pixel values to an adjacently preceding estimated pixel alue in the second sequence of estimated pixel values, thereby o compute an improved estimated pixel value for the individual ixel. ' Further in accordance with a preferred embodiment of 100589/3 ,he present invention, the apparatus for proceeding ar.d computing includes, for an individual pixel in the first sequence, apparatus for computing a difference value between the received image pixel and an adjacently preceding estimated pixel value in the first sequence, apparatus for employing the received image pixel and at least one preceding estimated pixel values in the first sequence to estimate a first directional signal to noise ratio, apparatus for generating an adjusted difference value to reflect the first directional signal to noise ratio, and apparatus for employing the adjusted difference value to update an adjacently preceding estimated pixel value, thereby to compute the estimated pixel value of the individual pixel.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for comparing includes apparatus for computing a difference value between the received image pixel and a function of adjacently preceding estimated pixel values in the first and second sequences, apparatus for employing preceding estimated pixel values in the first and second sequences to estimate a two-directional signal to noise ratio, apparatus for adjusting the difference value to reflect the signal to noise ratio, and apparatus for employing the adjusted difference value to update a function of adjacently preceding estimated pixel values in the first and second sequences, thereby to compute the improved estimated pixel value of the individual pixel.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for comparing includes 100589/3 apparatus for comparing the adjacently preceding estiaated pixel values in the first: and second sequences to at least one of the following parameters of the individual location: the estimated pixel value in the first sequence corresponding to the individual pixel, the estimated pixel value in the second sequence corresponding to the individual pixel, and the received image pixel .

Further in accordance with a preferred embodiment of the present invention, the image smoothing apparatus includes apparatus for proceeding, for at least a second dimension of received image pixels, in first and second directions in order to compute first and second sequences, respectively, of estimated pixel values and for comparing adjacently preceding estimated pixel values in the first and second sequences, thereby to compute at least one additional improved estimated pixel value for each individual pixel, and apparatus for combining the at least two. improved estimated pixel values, thereby to obtain a further improved two-dimensional estimated pixel value.

Still further in accordance with a preferred embodiment of the present invention, the second dimension is a time dimension.

Additionally in accordance with a preferred embodiment of the present invention, the image smoothing apparatus includes apparatus for proceeding along a second dimension of received image pixels in a first direction and for computing a first sequence of second dimension estimated pixel values from the improved estimated pixel values of the first 'dimension, apparatus for proceeding along the second dimension of received image 100589/3 pixels in a second opposite direction and for computing a second sequence of second dimension estimated pixel values from the improved estimated pixel values of the first dimension, and apparatus for comparing, for each individual pixel along the second dimension, an adjacently preceding second dimension estimated pixel value in the first sequence of second dimension estimated pixel values and an adjacently preceding second dimension estimated pixel value in the second sequence of second dimension estimated pixel values, thereby to compute a further improved estimated pixel value for the individual pixel.

Additionally in accordance with a preferred embodiment of the present invention, the received image defines a second dimension thereof and a scanning direction in which the image is received along the second dimension and the image smoothing apparatus also includes apparatus for proceeding along the second dimension of received image pixels in the scanned direction and for computing a sequence of second dimension estimated pixel values from the improved estimated pixel values of the first dimension, and apparatus for ^c^mparing , for each individual pixel along the second dimension, an adjacently preceding second dimension estimated pixel value in the sequence of second dimension estimated pixel values and an improved estimated pixel value of the first dimension which adjacently proceeds the individual pixel along the second dimension, thereby to compute a further improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the image smoothing apparatus includes 100589/3 apparatus for adjusting the results of the second dimension steps in order to reflect the difference between the received image and results of the first dimension steps.

Still further in accordance with a preferred embodiment of the present invention, the image smoothing apparatus also includes an electronic scanner for scanning an image, thereby to provide the received image pixels.

Additionally in accordance with a preferred embodiment of the present invention, the image smoothing apparatus also includes a video system for providing the received image pixels.

There is also provided in accordance with another preferred embodiment of the present invention apparatus for one-directional time domain smoothing of a current image which was preceded by a sequence of images, the apparatus including apparatus for computing a difference function between the raw value of an individual pixel of the current image and a smoothed value corresponding to at least one pixels of at least one preceding image, and apparatus for generating a smoothed pixel value for the individual pixel of the current image by computing a weighted sum of the smoothed value and of the raw value of the individual pixel of the current image wherein the weights are a function of the difference function.

There is also provided in accordance with another preferred embodiment of the present invention a method for one-directional time domain smoothing of a current image which was preceded by a sequence of images, the method including the steps of computing a difference function between the raw value of an individual pixel of the current image and a smoothed value 100589/3 corresponding to a: least one pixels of at least one preceding image, and generating a smoothed pixel value for the individual pixel of the current image by computing a weighted sum of the smoothed value and of the raw value of the individual pixel of the current image wherein the weights are a function of the difference function.

There is also provided, in accordance with another preferred embodiment of the present invention, a method for acuity preserving iaage smoothing including the steps of proceeding along a first dimension of received image pixels in a first direction and computing a first sequence of estimated pixel values from the received image pixels defined along the first direction, proceeding along the first dimension of received image pixels in a second opposite direction and computing a second sequence of estimated pixel values defined along the second direction, and for each individual pixel along the first dimension, comparing an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding .es timated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated-pixel value for the individual pixel, wherein, in at least one of the above steps of proceeding, proceeding and comparing, each pixel value estimation relies on the received value for that pixel and on estimated values for at least one preceding pixel along at least one direction, and determining, prior to at least one of the above steps of proceeding, proceeding and comparing, the extent to which the current received image pixel is taken 100589/3 into account relative to estimated preceding pixel values.

Further in accordance with a preferred embodiment of the present invention, the step of determining takes into account image derived inforaation other than the pixel value of the pixel to be estimated.

'Still further in accordance with a preferred embodiment of the present invention, the step of determining takes into account image derived information other than the signal to noise ratio .

Additionally in accordance with a preferred embodiment of the present invention, the step of determining takes into account image derived information other than the autocorrelation of the image.

Further in accordance with a preferred embodiment of the present invention, the image derived information pertains to the image as a whole.

Still further in accordance with a preferred embodiment of the present invention, the image derived information includes the location within the image of the pixel whose value is to be estimated.

Additionally in accordance with a preferred embodiment of the present invention, the image derived information includes the location of the pixel whose value is to be estimated relative to at least one user-designated location within the image.

Further in accordance with a preferred embodiment of the present invention, the step of determining takes into account at least one characteristic of process by which the image is acquired. 2G Still further in accordance with a preferred embodiment of the present invention, the at least one process characteristic includes at least one of the following group: lighting under which the image was acquired, characteristics of the device employed to acquire the image, characteristics of the device employed to record the image, and characteristics of the method employed to compress and decompress the image.

In accordance with another preferred embodiment of the present invention, there is provided a method for enhancement of contrast in an image including the steps of extracting the high frequency detail from the image signal, amplifying the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of at least one of the following image features: the amplitude of the high frequency detail signal and local image brightness, and combining the amplified high frequency detail with at least a portion of the image signal.

Further in accordance with a preferred embodiment of the present invention, the step of amplifying includes the step of amplifying the extracted high frequency detail signal to an extent which is partly determined by the user.

Still further in accordance with a preferred embodiment of the present invention, the method also includes the step of reducing noise in the image so as to prevent enhancement of noise artifacts.

Further in accordance with a preferred embodiment of the present invention, the noise reduction step includes the following steps: proceeding along a first dimension cf received image pixels in a first direction and computing a first sequence of estimated pixel values from the received image prxels defined along the first direction, proceeding along the first dimension of received image pixels in a second opposite direction and computing a second sequence of estimated pixel values defined along the second direction, and for each individual pixel along the first dimension, comparing an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Still further in accordance with a preferred embodiment of the present invention, in at least one of the above steps of proceeding, proceeding and comparing, estimation of each pixel value relies on the received value for that pixel and on estimated values for at least one preceding pixel alor.g at least one direction, and the noise reduction step also includes- the step of determining, prior to at least one of the above steps of proceeding, proceeding and comparing, the extent to which the current received image pixel is taken into account relative to estimated preceding pixel values.

There is also provided, in accordance with another preferred embodiment of the present invention, a method for dynamic range compression of an image, the method including the steps of reducing noise in the image to different extents in different portions of the image, such that noise is substantially reduced in -a first portion of the image which is expected to 100589/3 become noisy as a result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and compressing the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the noise reduction step includes the following steps: proceeding along a first dimension of received image pixels in a first direction and computing a first sequence of estimated pixel values from the received image pixels defined along the first direction, proceeding along the first dimension of received image pixels in a second opposite direction and computing a second sequence of estimated pixel values defined along the second direction, and, for each individual pixel along the first dimension, comparing an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

There is also provided, in accordance with another preferred embodiment of the present invention, a method for dynamic range compression of an image, the method including the steps of enhancing contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second 100589/3 portion of the image which is expected to lose less contrast than the first portion of the image, and compressing the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the step of compressing includes the step of linearly compressing the dynamic range of the image.

Still further in accordance with a preferred embodiment of the present invention, the step of compressing includes the step of nonlinearly compressing the dynamic range of the image.

There is also provided, in accordance with another preferred embodiment of the present invention, a method for enlarging an image including the steps of substantially replicating pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and reducing noise in the image such that artifactual differences between adjacent blocks tends to be smoothed.

Further in accordance with a preferred embodiment of the present invention, the steps of replicating and reducing are repeated iteratively such that, in each step, blocks are replaced with bigger blocks.

Still further in accordance with a preferred embodiment of the present invention, the first step of replicating is preceded by a step of reducing noise in the image.

Further in accordance with a preferred embodiment of the present invention, the method includes, following the last of the replicating and noise reducing steps, the step of restoring to the enlarged image, high frequency detail which was removed from the image in the noise reducing step, at locations in the 2 H 100589/3 enlarged image which correspond to the locations of the high frequency detail in the original image.

Still further in accordance with a preferred embodiment of the present invention, the step of reducing noise includes the steps of proceeding along a first dimension of received image pixels in a first direction and computing a first sequence of estimated pixel values from the received image pixels defined along the first direction, proceeding along the first dimension of received image pixels in a second opposite direction and computing a second sequence of estimated pixel values defined along the second direction, and, for each individual pixel along the first dimension, comparing an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

There is also provided, in accordance with another preferred embodiment of the present invention, apparatus for acuity-preserving image smoothing including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel, wherein estimation of at least one pixel value relies on the received value for that pixel and on estimated values for at least one preceding pixel along at least one direction, and- -a weight determining unit which is operative to determine the extent to which the current received image pixel is taken into account relative to estimated preceding pixel values.

There is also provided, in accordance with another preferred embodiment of the present invention, apparatus for enhancement of contrast in an image including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of at least one of the following image features: the amplitude of the high frequency detail signal and local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

There is further provided, in accordance with another preferred embodiment of the present invention, apparatus for dynamic ' range compression of an image, including a differential noise reducer operative to reduce noise in the image to different extents in different portions of the image, such that noise is, substantially reduced in a first portion of the image which is expected to become noisy as a result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynaaic range of the image.

There is also provided, in accordance with: another preferred embodiment of the present invention, apparatus for dynamic range compression of an image, the apparatus including a differential contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

There is also provided, in accordance with another preferred embodiment of the present invention, apparatus for enlarging an image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and a noise reduction unit operative to reduce noise in the image such that artifactual differences between adjacent blocks tends to be smoothed.

Further in accordance with a preferred embodiment of the ' present invention, the apparatus is at least partly implemented in soft·.% are.

Still furcher in accordance with a preferred embodiment of the present invention, the apparatus is at least partly implemented in hardware.

Additionally in accordance with a preferred embodiment of the present invention, the apparatus is at least partly implemented in VLSI hardware.

Further in accordance with a preferred embodiment of the present invention, the apparatus is at least partly implemented in discrete electronic component hardware.

Still further in accordance with a preferred embodiment of the present invention, the apparatus is at least partly implemented using DSP technology.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video camera, for acquiring a video image, and noise reduction apparatus for reducing noise in the video image including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel lvalues defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise 'includes spatial noise reducing apparatus for reducing noise within a single video image.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes temporal noise reducing apparatus for reducing noise over a sequence of video images.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatio-temporal noise reduction apparatus for performing, in combination, spatial noise reduction and temporal noise reduction functions.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video camera for acquiring a video image, and contrast enhancement apparatus for enhancing contrast in the video image including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of the amplitude of the high frequency detail signal, and/or local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a porcion of the image signal.

There is also provided, in accordance with a preferred embodiment of the present invention, a video system including a video camera for acquiring a video image, and image enlargement apparatus for enlarging the video image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and a noise reduction unit operative to reduce noise in the image such that artifactual differences between adjacent blocks tends to be smoothed.

There is also provided, in accordance with a preferred embodiment of the present invention, a video system including a video camera for acquiring a video image, and dynamic range compression apparatus for compressing the dynamic range of the video image including a differential noise reducer operative to reduce noise in the image to different extents in different portions of the image, such that noise is subs antially reduced in a first portion of the image which is expected to become noisy as a. result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynamic range of the image.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video camera for acquiring a video image, and dynamic range compression apparatus for compressing the dynamic range of the video image including a differential contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the video camera includes one of the following types of video camera: an analog still video camera, a digital still video camera, an analog moving video camera, and a digital moving video camera.

Further in accordance with a preferred embodiment of the present invention, the video camera includes one of the following types of video camera: an analog camcorder, and a digital camcorder.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video player for playing a video image, and noise reduction apparatus for reducing noise in the video image including a first directional pixel value estimator proceeding along a first diaension of received image pixels in a first direction which is operative to compute a first- sequence of estimated pixel values from the received image pixels defined along the fir.st direction, a second directional pixel . value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel..

Further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatial noise reducing apparatus for reducing noise within a single video image.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes temporal noise reducing apparatus for reducing . noise over a sequence of video images.

Further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatio-temporal noise reduction apparatus for performing, in combination, spatial noise reduction and temporal noise reduction functions .

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video player for playing a video image, and contrast enhancement apparatus for enhancing contrast in the video image including a detail extractor operative to extract high frequency detail from the image signal, a. nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of at least one of the following image features: the amplitude of the high frequency detail signal, and local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video player for acquiring a video image, and image enlargement apparatus for enlarging the video image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and a noise reduction unit operative to reduce noise in the image such that artifactual differences between adjacent blocks tends to be smoothed.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video player for acquiring a video image, and dynamic range compression apparatus for compressing the dynamic range of the video image including a differential noise reducer operative to reduce noise in the image to different extents in different portions of the image, such that noise is substantially reduced in a first portion of the image which is expected to become noisy as a result of dynanic range compression and noise 'is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynamic range of the image.

There is also provided, in accordance with another preferred embodiment of the present invention, a video system including a video player for acquiring a video image, and dynamic range compression apparatus for- compressing the dynamic range of the video image including a differential contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the video player includes one of the following: an analog still video player, a digital still video player, a VCR, a digital video recorder/player, a video playing unit integrally formed with a video camera, a compact disk video recorder/player, and a video disk recorder/player.

There is also provided, in accordance with another preferred embodiment of the present invention, an image display system including apparatus for providing a pictorial display of an image, and noise reduction apparatus for reducing noise in the displayed image including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatial noise reducing apparatus for reducing noise within a single image.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes temporal noise reducing apparatus for reducing noise over a sequence of images .

Further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatio-temporal noise reduction apparatus for performing, in combination, spatial noise reduction and temporal no ise reduction func tions .

There is also provided, in accordance with another preferred embodiment of the present invention, an image display system including apparatus for providing a pictorial display of an image, and contrast enhancement apparatus for enhancing contrast in the displayed image including a detail extractor operative to extract high frequency detail from the iaage signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of at least one of the following image features: the amplitude of the high frequency detail signal, and local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

There is also provided, in accordance with another preferred embodiment of the present invention, an image display system including apparatus for providing a pictorial display of an image, and image enlargement apparatus for enlarging the image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and a noise reduction unit operative to reduce noise in the image such that artifactual differences between adjacent blocks tends to be smoo thed .

There is also provided, in accordance with another preferred embodiment of the present invention, an image display system including apparatus for providing a pictorial display of an image, and dynamic range compression apparatus for compressing the dynamic range of the displayed image including a differential noise reducer operative to reduce noise in the image to different extents in different portions of the image, such that noise is substantially reduced in a first portion of the image which is expected to become noisy as a result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynamic range of the image.

There is also provided, in accordance with another preferred embodiment of the present invention, an image display system including apparatus for providing a pictorial display of an image, and dynamic range compression apparatus for compressing the dynamic range of the displayed image including a differential contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the display providing apparatus includes one of the following: a TV display monitor, a TV receiver, a still video printer, an image printer, an image proofer, a fax printer, a computer including a video display board, a computer including a ax image display board.

There is also provided, in accordance with another preferred embodiment of the present invention, an image transmitting system including apparatus for transmitting an image to a remote location, and image improvement apparatus for reducing noise in the transmission process including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatial noise reducing apparatus for reducing noise within a single image.

Still further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes temporal noise reducing apparatus for reducing noise over a sequence of images.

Additionally in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatio- temporal noise reduction apparatus for performing, in combination, spatial noise reduction and temporal noise reduction functions.

There is also provided, in accordance with another preferred embodiment of the present invention, an image transmitting system including apparatus for transmitting an image to a remote location, and image improvement apparatus for enhancing contrast in the transmitted image including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of the amplitude of the high frequency detail signal, and/or local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

There is also provided, in accordance with another preferred embodiment of the present invention, an image transmitting system including apparatus for transmitting an image to a remote location, and image improvement apparatus for enlarging the image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and a noise reduction unit operative to reduce noise in the image such that artifact al differences between adjacent blocks tends to be smoothed.

There is also provided, in accordance with another preferred embodiment of the present invention, an image transmitting system including apparatus for transmitting an image to a remote location, and image improvement apparatus for compressing the dynamic range of the transmitted image including a differential noise reducer operative to reduce noise in -the image to different extents in different portions of the iaage , such that noise is substantially reduced in a first portion of the image which is expected to become noisy as a result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynamic range of the image.

There is also provided, in accordance with a preferred embodiment of the present invention, an image transmitting system including apparatus for transmitting an image to a remote location, and image improvement apparatus for compressing the dynamic range of the transmitted image including a differential contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the apparatus for transmitting includes a TV transmitter.

Still further in accordance with a preferred embodiment of the present invention, the image improvement apparatus is operative in the course of encoding an image prior to transmission thereof .

Additionally in accordance with a preferred embodiment of the present invention, the image improvement apparatus is operative in the course of transmission of the image.

Further in accordance with a preferred embodiment of the present invention, the noise reduction apparatus is incorporated in a relay station which is operatively intermediate the transmitting location and the remote receiving location.

Still further in accordance with a preferred embodiment of the present invention, the image improvement apparatus is operative in the course of decoding a transmitted image.

There is also provided, in accordance with another preferred embodiment of the present invention, an image scanning system including an electro-optic scanner for scanning an image, noise reduction apparatus for reducing noise in the scanned image including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a seccr.d opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in , accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatial noise reducing apparatus for reducing noise within a single image.

There is also provided, in accordance with another preferred embodiment of the present invention, an image scanning system including an electro-optic scanner for scanning an image, contrast enhancement apparatus for enhancing contrast in the scanned image including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of the amplitude of the high frequency detail signal, and/or local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

There is also provided, in accordance with another preferred embodiment of the present invention, an image scanning system including an electro-optic scanner for scanning an image, image enlargement apparatus for enlarging the image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of subs antially identical pixels, and a noise reduction unit operative to reduce noise in the image such that ar'tifactual differences between adjacent blocks tends to be smoothed.

There is also provided, in accordance with another preferred embodiment of the present invention, an image scanning system including an electro-optic scanner for scanning an image, dynamic range compression apparatus for compressing the dynamic range of the scanned image including a differential noise reducer operative to reduce noise in the image to different extents in different portions of the image, such that noise is substantially reduced in a first portion of the image which is expected to become noisy as a result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynamic range. of the image.

There is also provided, in accordance with another preferred embodiment of the present invention., an image scanning system including an electro-optic scanner for scanning an image, dynamic range compression apparatus for compressing the dynamic range of the scanned image including a di ferencial contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the electro-optic scanner includes one o the following: a flat-bed scanner, a drum scanner, a fax scanner, a manually fed document scanner, and an imaging scanner which may, for example, employ reflective optical elements.

There is also provided, in accordance with another preferred embodiment of the present invention, an image processing system including an image modifying computer for modifying an image, and noise reduction apparatus for reducing noise in the modified image including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel val es to an adjacently preceding estimated pixel' value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present inventicn, the apparatus for reducing noise -includes spatial noise reducing apparatus for reducing noise- within a s ingle image .

Still further in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise include temporal noise reducing apparatus for reducing noise over a sequence of images.

Additionally in accordance with a preferred embodiment of the present invention, the apparatus for reducing noise includes spatio-temporal noise reduction apparatus for performing, in combination, spatial noise reduction and temporal noise reduction functions.

There is also provided, in accordance with another preferred embodiment of the present invention, an image processing system including an image modifying coaputer for modifying an image, and contrast enhancement apparatus for enhancing contrast in the modified image including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined a≤ a nonlinear function of the amplitude of the high frequency detail signal, and/or local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

There is also provided, in accordance with another preferred embodiment of the present invention, an image processing system including an image modifying computer for modifying an image, and image enlargement apparatus for enlarging the image including a pixel replicator operative to substantially replicate pixels of the image, thereby to replace each individual pixel with a block of substantially identical pixels, and a noise reduction unit operative to reduce noise in the image such that artifactual differences between adjacent blocks tends to be smoothed .

There is also provided, in accordance with another preferred embodiment of the present invention, an image processing system including an image modifying computer for modifying an image, and dynamic range compression apparatus for compressing the dynamic range of the scanned image including a differential noise reducer operative to reduce noise in the image to different extents in different portions of the image, such that noise is substantially reduced in a first portion of the image which is expected to become noisy as a result of dynamic range compression and noise is only mildly reduced in a second portion of the image which is expected to be less noisy than the first portion of the image as a result of dynamic range compression, and a dynamic range compression unit operative to compress the dynamic range of the image. i+6 There is also provided, in accordance with another preferred embodiment of the present invention, an image processing system including an image modifying computer for modifying an image, and dynamic range compression apparatus for compressing the dynamic range of the scanned image including a differential contrast enhancer operative to enhance contrast in the image to different extents in different portions of the image, such that contrast is substantially enhanced in .a first portion of the image which is expected to lose considerable contrast as a result of dynamic range compression and contrast is only mildly enhanced in a second portion of the image which is expected to lose less contrast than the first portion of the image, and a dynamic range compression unit operative to compress the dynamic range of the image.

Further in accordance with a preferred embodiment of the present invention, the image modifying computer includes one of the following: a workstation, a personal computer, a mainframe computer, a desktop publishing system.

There is also provided, in accordance with another preferred embodiment of the present invention, image compression apparatus for compressing an image including an, image compressing unit, and a noise reducing unit for reducing noise in the image prior to compression thereof, the noise reducing unit including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received iaage pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel, and an image compressing unit for compressing the noise-reduced image.

There is also provided, in accordance with another preferred embodiment of the present invention, image decompression apparatus for decompressing an image including an image decompressing unit, and a noise reducing unit for reducing noise in the decompressed image, the noise reducing unit including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently i*8 preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

Further in accordance with a preferred embodiment of the present invention, the noise reducing unit is operative to reduce block transform compression noise.

Still further in accordance with a preferred embodiment of the present invention, the noise reducing unit is operative to reduce discrete cosine transform compression noise.

Additionally in accordance with a preferred embodiment of the present invention, the apparatus is operative in accordance with the JPEG standard or the MPEG standard or the CCITT H.26I standard.

In accordance with a preferred embodiment of the present invention, there is also provided, for use with a Karaoke recording system including a video camera and a video editing system for editing a video image sequence generated by the video camera, an MPEG image encoder for compressing the edited video image sequence onto a compact disk, the encoder including a noise reduction unit including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pixel values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along " the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacently preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby to compute an improved estimated pixel value for the individual pixel.

In accordance with a preferred embodiment of the present invention, there is also provided, for use with a Karaoke recording system including a video camera and a video editing system for editing a video image sequence generated by the video camera, an MPEG image encoder for compressing the edited video image sequence onto a compact disk, the encoder including a contrast enhancement unit including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of the amplitude of the high frequency detail signal, and/or local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

In accordance with a preferred embodiment of the present invention, there is also provided a Karaoke playback system including a Karaoke compact disk player for receiving and playing back a Karaoke compact disk storing a compressed video sequence, and an MPEG image decoder for decoding the compressed video sequence, including a noise reduction unit including a first directional pixel value estimator proceeding along a first dimension of received image pixels in a first direction which is operative to compute a first sequence of estimated pi el values from the received image pixels defined along the first direction, a second directional pixel value estimator proceeding along the first dimension of received image pixels in a second opposite direction which is operative to compute a second sequence of estimated pixel values defined along the second direction, and an improved estimate generator operative, for each individual pixel along the first dimension, to compare an adjacentl preceding estimated pixel value in the first sequence of estimated pixel values to an adjacently preceding estimated pixel value in the second sequence of estimated pixel values, thereby. to compute an improved estimated pixel value for the individual pixel.

In accordance with a preferred embodiment of the present invention, there is also provided a Karaoke playback system including a Karaoke compact disk player for receiving and playing back a Karaoke compact disk storing a compressed video sequence, and an MPEG image decoder for decoding the compressed video sequence, including a contrast enhancement unit including a detail extractor operative to extract high frequency detail from the image signal, a nonlinear detail amplifier operative to amplify the extracted high frequency detail signal to an extent at least partly determined as a nonlinear function of the amplitude of the high frequency detail signal and/or local image brightness, and a detail-image combiner operative to combine the amplified high frequency detail with at least a portion of the image signal.

In the present specification and claims, the terms "image smoothing" and "noise reduction" are for the most part used interchangeably.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be understood and appreciated from the following detailed description, taken in conjunction with the drawings in which: Fig. 1 is a simplified block diagram of one-dimensional two-directional iaage smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention; Fig. 2 is a simplified block diagram of, one-direc tional smoothing unit 24 of Fig. 1; Fig. 3 is a simplified block diagram of two-directional smoothing unit 28 of Fig. 1, constructed ' and operative in accordance with a first embodiment of the present invention; Fig. 4 is a simplified block diagram of smoothing unit 28 of Fig. 1, constructed and operative in accordance with a second embodiment of the present invention; Fig. 5 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a first modification of the apparatus of Fig. 1; Fig. 6 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a second modification of the apparatus of Fig. 1; Fig. 7 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a third modification of the apparatus of Fig. 1; Fig. 8 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a fourth modification of the apparatus of Fig. 1; Fig. 9 s a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a fifth modification of the apparatus of Fig. 1; Fig. 10 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a sixth modification of the apparatus of Fig. 1 ; Fig. 11 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a seventh modification of the apparatus of Fig. 1; Fig. 12 is a simplified block diagram of smoothing apparatus for smoothing a sequence of images; Fig. 13 is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention which is a modification of the apparatus of Fig. 12; Fig. 14 is a simplified block diagram of one-dimensional two-directional image smoothing apparatus which is a modification of the apparatus of Fig. 1 in that computation of the estimation gain parameter (EGP) is carried out externally of the two-directional processor 16; Fig. 15 a simplified block diagram of spatial noise reducing apparatus which combines the features of Figs. 8 and 10; Fig. 16 is a simplified block diagram of estimation gain parameter determining apparatus which may replace units 310 and 318 of Fig. 14; Fig. 17 is a simplified block diagram of estimation gain parameter determining apparatus which is a modification to the apparatus of Fis. 16; Fig. l8 is a simplified block diagram of an estimation gain parameter adjustment unit 500 constructed and operative in accordance with a first alternative embodiment of the present invention which may replace estimation gain parameter adjustment unit 320 of Fig. 14; Fig. 19 s a simplified block diagram of an estimation gain parameter adjustment unit 550 constructed and operative in accordance with a second alternative embodiment of the present invention which may replace estimation gain parameter adjustment unit 320 of Fig. Ik; Fig. 20 is a simplified block diagram of an estimation gain parameter adjustment unit 600 constructed and operative in accordance with a third alternative embodiment of the present invention which may replace estimation gain parameter adjustment unit 320 of Fig. 14; Fig. 21 is a simplified block diagram of apparatus for combined spatial noise reduction and enhancement of an image; Fig. 22 is a simplified block diagram of an enhancement unit in Fig. 21; Fig. 23 i≤ a simplified block diagram of dynamic range compression apparacus operative in the spatial domain; Fig. 2 is a simplified block diagram of combined spatial noise reduction and spatial interpolation apparatus constructed and operative in accordance with a preferred embodiment of the present invention; Fig. 25 is a simplified block diagram of spatio-temporal noise reduction apparatus which is operative to. provide spatial noise reduction and one-directional temporal noise reduction ; Fig. 26 is a simplified block diagram of a modification of the apparatus of Fig. 25 in which the temporal noise reduction is "pseudo 2-directional " instead of one-directional; Fig. 27 is a simplified block diagram of apparatus for combined spatial noise reduction, temporal noise reduction, enhancement and dynamic range compression; Fig. 28 is a simplified block diagram of improved analog still video equipment incorporating the apparatus for image smoothing, enhancing and interpolating shown and described hereinabove with reference to Figs. 1 - 27 ; Fig. 29 is a simplified block diagram of improved digital still video equipment incorporating the apparatus for image smoothing, enhancing and interpolating shown and described hereinabove with reference to Figs. 1 - 27 ; Fig. 30 is a simplified block diagram of improved analog and digital moving video equipment incorporating the apparatus for image smoothing, enhancing and (interpolating shown and described hereinabove with reference to Figs. 1 - 27 ; Fig. 31 is a simplified block diagram of improved image scanning equipment incorporating the apparatus for image smoothing, enhancing and interpolating shown and described hereinabove with reference to Figs. 1 - 27; Fig. 32 is a simplified block diagram of improved facsimile equipment incorporating the apparatus for image smoothing, enhancing and interpolating' shown and described hereinabove with reference to Figs. 1 - 27; · Fig. 33 is a simplified block diagram of improved teleconferencing and videophone equipment incorporating the apparatus for image smoothing, enhancing and interpola ing shown and described hereinabove with reference to Figs. 1 - 27; and Fig- 3^ s a simplified block diagram of improved equipment for providing Karaoke entertainment, incorporating the apparatus for image smoothing, enhancing and interpolating shown and described hereinabove with reference to Figs. 1 - 27· DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Reference is now made to Fig. 1 which is a simplified block diagram of or.e-dimens ional two-directional image smoothing apparatus, referenced generally 10, which is constructed and operative in accordance with a first preferred embodiaent of the present invention.

The image smoothing apparatus 10 includes an image preprocessing unit 12, an estimation gain parameter computation unit l4 and a two-directional processor 16 which receives image input from the image preprocessing unit 12 and which includes an estimation gain parameter LUT 18 which is loaded by estimation gain parameter computation unit 14.

Image preprocessing unit 12 is operative to receive analog image data from a suitable device such as a video camera or video recorder and perform an analog to digital conversion of the analog image data. The resulting digital image data may be stored in a frame buffer if suitable, for example if it is necessary to accomodate a data input rate which differs from the processing rate of the apparatus of Fig. 1. The output of the image preprocessing unit 12, also termed herein "raw image data", is provided, line by line, to a raw data line buffer 22 in two-directional processor 16.

The tera "line", as employed in the present specification, refers to a one-dimensional unit of an image, such as an image row, image column, or diagonal one-dimensional array of pixels within the image. Selection of a dimension of the image along which to process preferably takes into account characteris ics of ; e image such as image edges arranged along a particular dimension and charac eristics of the noise such as a high probability that noise of a particular statistical character will occur along a particular dimension.

The index "i" is used herein as an index of the pixels within a' line (for each line, i = 1 I) . "X(i)" as used herein denotes the raw data image value for pixel i ar.d includes a signal portion and a noise portion as defined in Equations 1 A and IB of Appendix B. The minimum mean square estimate, S(i) , of the signal portion S (i) of the raw data image is defined by Equation 2 of Appendix B, as discussed in the above-referenced publication by Papoulis .

It is appreciated that a one-dimensional line may be processed in either of two opposite directions. For example, an image row may be processed from right to left or from left to right. An image column may be processed from top to bottom or from bottom to top.

The two-directional processor 16 includes, apart from LUT 18 and raw data line buffer 22 , a pair of one-directional smoothing units 24 and 26 and a two-directional smoothing unit 28 . Units 24 and 26 smooth the raw image data in raw image line buffer 22 , proceeding in first and second opposite directions respectively. Two-directional smoothing unit 28 receives delayed output data from one-directional smoothing unit 24 and 26 . Two-directional smoother 28 combines image data which has undergone one-directional smoothing in both the first and second one-directional smoothers by performing a two-directional smoothing process thereupon.

One-directional smoothers 24 and 26 receive raw image data X(i) from image preprocessing unit 12 and from raw data line buffer 22 respectively, and further receive an estimation gain parameter K(i) from a suitable source such as LUT 18.. One-directional smoothers 24 and 26 each compute a respective approximation, termed herein S+(i) and S_(i) respectively and defined in Equations 3A and 3B respectively of Appendix B, to the minimum mean square error estimate S(i), defined in Equation 2 of Appendix B.

It is a particular feature of the present invention that each one-directional smoother, when computing an estimate S+(i) or S_(i) , respectively, of the signal portion of the raw image value X(i) of pixel i, employs only information regarding pixel i and pixels precedin i in the direction of smoothing.

Another particular feature of the embodiment of Fig. 1 as well as of the other embodiments shown and described herein with reference to foregoing figures is that the outputs of the image smoothing devices shown and described herein are normally nonlinear relative to the inputs thereto.

For example, for the purposes of simplification, line buffer 22 will be assumed to store image rows, one-directional smoother 24 will be assumed to smooth from left to right and one-directional smoother 26 will be assumed to smooth from right to left. It is appreciated that the above example is not intended to be limiting. In this example, left-to-right smoother 24, when ♦ computing an estimate of the signal portion S (i) of pixel i, employs only the raw image values of pixel i and pixels to the left of pixel i. Le f t- to - righ t smoother 24 does not employ raw image values of pixels to the right of pixel i to estimate the signal portion of pixel i. In contrast, righ t - to - le smoother 26, when computing an estimate of the signal portion of pixel i, employs only the raw image values of pixel i and pixels to the right of" pixel i. Righ t- to-lef smoother 26 does not employ raw image values of pixels to the left of pixel i to estimate the signal portion of pixel i.

A particular advantage of the above characteristics of one-directional smoothers 24 and 26 is that for each pixel i, the signal estimates S+(i) and S_(i) generated by smoothers 24 and 26 respectively are substantially independent of one another. Also, in the present example, S+(i), the signal estimate of . one-directional smoother 24 for pixel i, may be assumed to be "uncontaminated" by image effects occurring to the right of pixel i. Similarly, the signal estimate S_(i) of one-directional smoother 26 for pixel i may be assumed to be "uncontaninated" by image effects occurring to the left of pixel i.

Equation 3A of Appendix B is a preferred recursive equation which may be employed by one-directional smoother 24 for computing a signal estimate in a first direction for pixel i, S+(i), using the signal estimate S+(i-l) of the (i-l)th pixel. The (i-l)th pixel is the pixel which precedes the current pixel i in the (+) direction of smoothing. Equation 3B is a preferred recursive equation which may be employed by one-directional smoother 26 for computing the signal estimate for pixel i, S_(i), using the signal estimate S_(i+1) of the .(i+l)th pixel. The (i+l)th pixel precedes the current pixel i in the (-) direction of smoothing.

In Equations 3A and 3B, K+(i) and K_(i) refer respectively to estimation gain parameters provided to one-directional smoothers 24 and 26 respectively by LUT 18. As shown in Fig. 1, smoothers 2k and 26 address LUT 18 by means of the magnitudes of parameters d+(i) and d_(i), respectively. These parameters are both generated from the raw image data and each comprise a respective directional estimate of signal strength of an edge at pixel i. d+(i) and d_(i) are defined with reference to Equations 3A and 3B of Appendix B.

Preferably, as explained above, estimation gain parameter K(i) is stored in LUT 18 which is constructed by estimation gain parameter computation unit lk . Unit lk preferably receives two external values, sigman and rs . rg is the correlation coefficient of the signal and is theoretically defined by Equation 2 of Appendix B. Sigman, the standard deviation, is defined by Equation 1A in the present example, however, it is appreciated that the applicability of the present method extends to a wide variety of noise distributions and is not limited to Gaussian noise distributions. Unit lk may include means for accepting manual input from a user, in which case sigman and rg may be input by hand.

Any suitable initial value for rg may be selected by the user, such as a value within the range 0.6 - 0.8. Any suitable initial value for sigman may' be selected by the user, such as a value between 0 and 10 gray levels. Once initial values for rs and sigman have been determined, the apparatus of Fig. 1 may be employed as explained herein in order to obtain an output image. Upon viewing the output image, if the user finds the output image to be too smooth or blurry, he may decrease the value of rs and/cr decrease the value of sigman. If the user views the output i::age and finds it to be too noisy or choppy, he may increase the value of rg and/or increase the value of sigman.

Estimaticr. gain parameter computation unit 14 computes K parameters as a function of d parameters and stores pairs of K and d parameters ir. estimation gain parameter LUT 18 . Estimation gain parameter LUT 18 is addressed by the magnitudes of d+ and d_ values arriving from one-directional smoothers 24' and 26 respectively and ccuputes K+ and K_ parameters which are supplied back to one-directional smoothers 24 and 26 respectively. Estimation gain parameter LUT 18 also provides K+_ values to two-directional smoother 28 , which are addressed in accordance with the magnitudes of dT_ values provided by unit 28 , as described in detail below.

Gain estimation parameter computation unit 14 may be implemented in accordance with Equations 4 and 5 of Appendix B, of which equation is a recursive formula and equation 5 is an initial formula with which the recursive process may be initiated.

Preferably, K is computed for each of a plurality of d values', corresponding to a plurality of snr values, such as all snr values in a range of 0 - 100 , at' a resolution of 0 . 02 - 0 . 1 . LUT 18 comprises, therefore, a table of 1000 - 5000 pairs of d and K values. Since, for each value of d, reaches a steady state after a relatively small number of recursions, only a single K value need be stored for each d value. A suitable number of iterations of equation 4 may be performed for each d value, such as 25 - 50 iterations, and the single steady state K value which results may be stored in association with the corresponding d value.

It is appreciated that LUT l8 need not be constructed in accordance with equations 4 and 5· Alternatively, for example, good approximations to the values obtained by using equations 4 and 5 may be generated by linearization and Taylor series expansion. Also, the values obtained by employing equations 4 and 5 or by any other method may be thresholded or otherwise modified in order to avoid computational error due to limited accuracy.

It is believed that the computation of Equations 4 and 5 may be replaced by heuristic methods of generating K such as fuzzy logic methods, in which case the functions stored in LUT l8 would be more appropriately termed "fuzzy membership functions". Fuzzy logic methods are described in the above-referenced publication by Pal and Majumder.

In accordance with a preferred embodiment of the present invention, a second LUT may be provided which, instead of storing pairs of K and d values in LUT l8, stores pairs of Kxd and d values, thereby eliminating the need to multiply the K outpu of the LUT by d when employing Equations 3A and 3B pertaining to smoothers 24 and 26 respectively. The original LUT l8 is preferably retained to subserve smoother 28 which does not employ the product K x d, as shown by Equation 7· One-directional smoother 24 stores the signal estimate S+(i) for all pixels i in a signal estimate line buffer 32 which interfaces with two-directional smoother 28. One-directional smoother 24 also stores the d+ (i) values computed for each pixel i in a d+(i) line buffer 34 which also interfaces with two-directional smoother 28.

Two-directional smoother 28 is operative to receive one-directional signal estimate values S+(i-l) and one-directional d+(i)values from one-directional smoother 24, via line buffers 3 and 34 respectively, and also to receive the corresponding one-directional values S_(i+1) and d_(i) directly from one-directional smoother 26, which proceeds in the opposite direction relative to one-directional smoother 24. Two-directional smoother 28 computes a two-directional d value, d+_ (i), using Equation 6 of Appendix B, which value is used to address LUT 18. The resulting K+_(i) value is employed by two-directional smoother 28 to compute a two-directional signal estimate value, S+_(i), for each pixel i, which is the output of the two-directional processor i6. Equation 7 of Appendix B is a preferred formula for the computation of S+_.

Preferably, the output of two-directional saoother 28 also includes the two-directional difference value, d+_(i), as well as a value Suml(i) , defined by Equation 8 of Appendix B, which are useful in certain applications, as described in detail below with reference to Figs. 10 and 11.

Reference is made briefly to Fig. 2 which is a simplified block diagram of a one-directional smoother, such as one-directional smoother 24 of Fig. 1, constructed and operative in accordance with a preferred embodiment of the present invention. It is appreciated that the apparatus of Fig. 2 is suitable for implementing recursive Equation 3A of Appendix B. One-directional smoother 26 of Fig. 1 may be identical to one-directional smoother 2 Ά of Fig. 2 except that one-directional smoother 26 proceeds in the - direction rather than the + direction such that the pixel preceding pixel .i is pixel (i + 1) rather than pixel (i-1) .

A particular advantage of the apparatus of Fig. 2 is that large signal discontinuities occurring along the dimension of processing are preserved. Disadvantages of the apparatus of Fig. 2 are that high amplitude noise fluctuation and spikes may be preserved and that phase delays may be introduced due to the directional and recursive nature of the apparatus of Fig. 2.

Reference is made' to Fig. 3 which is a simplified block diagram of two-directional smoother 28 of Fig. 1, constructed and operative in accordance with one embodiment of the present invention. It is appreciated that the apparatus of Fig. 3 is suitable for implementing Equations 6 and 7 of Appendix B. A particular advantage of the apparatus of Fig. 3 is that one-directional smoothed results from neighbors symmetrically disposed on both sides of the current pixel are employed to estimate the strength of the edge signal at the current pixel, and also to effectively smooth noise spikes.

Reference is made to Fig. k which is a simplified block diagram of two-directional smoother 28 of Fig. 1, constructed and operative in accordance with another embodiment of the present invention. The apparatus of Fig. is similar to the apparatus of Fig. 3 except that a different value addresses LUT 18. In Fig. 3, d+_(i) addresses LUT 18 and this address is generated in accordance with Equation 6 of Appendix B. In Fig. 4, dm+ (i) addresses LUT l8 and this address is generated in accordance with Equation of Appendix B.

A particular advantage of the apparatus of Fig. 4, relative to the apparatus of Fig. 3, is that two separate instances are identified and differently handled. In- the first instance, the current input image value, X(i) , falls outside of the intensity range delimited by S+(i-l) and S_(i+1). In the second instance, the current input image value, X(i) , falls between S+(i-l) and S_(i+1) . In the first instance, the outputs generated by the apparatus of Figs. 3 and 4 are the same, because both apparatus "hypothesize" the occurrence of a spike coinciding with an image edge. In the second instance, however, the outputs generated by the apparatus of Figs. 3 and 4 are not the same, because the apparatus of Fig. hypothesizes a surface and consequently, increases the degree of smoothing.

In accordance with a preferred embodiment of the present invention, two-directional processor 16 of Fig. 1 may be augmented with one or more additional two-directional processors, each being substantially identical to two-directional processor l6. Figs. 5, 6 and 7 are simplified block diagrams of smoothing apparatus constructed and operative in accordance with three alternative embodiments of the present invention, respectively, each of which comprise 2 two-directional processors.

Reference is now made specifically to Fig. 5. which is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention. The apparatus of Fig. 5 is similar to the apparatus of Fig. 1 except that it includes 2 two-directional processors 40 and 42, each of which may be substantially identical to the single two-directional processor 1 6 of Fig. 1 . Two-directional processor 40 receives raw data X(l,i) line by line and generates a two-directional signal estimate, Sh+_(l-l,i), with a one line delay.

The signal estimates generated by two-directional processor 0 is stored in a line buffer 44 of two-directional processor 2, which may be substantially identical to raw data line buffer 22 of Fig. 1 . The data in line buffer 44 is received by a smoothing unit 46 in two-directional processor 42, which comprises units which may be substantially identical to units 18 , 24, 26 , 28 , 32 and 34 . A particular advantage of the apparatus of Fig. 5 is that no intermediate memory buffer need be provided between two-directional processors 40 and 42 .

Processors 40 and 42 operate along the same dimension, which may be any dimension such as the horizontal dimension. In Fig. 5 , the output of two-directional processor 40 is termed Sh+_ to indicate that, in the present example, processor 40 proceeds along the horizontal dimension, and the output of two-directional processor 42 is termed S +_, to indicate that processor 42 provides output which has twice been processed along the same dimension as employed by processor 40 .

In Fig. 5 and in subsequent figures, 1 is an index for image lines (rows, columns, diagonal one-dimensional units, or other types of one-dimensional arrays) .

The raw data input to the apparatus of Fig. 5 is designated X(l,i) whereas the output is designated S**1 ( 1 - 2 , i ) to indicate that the apparatus of Fig. 5 operates substantially in real-time, with a delay of only two lines.

Reference is now made to Fig. 6 which illustrates image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention. The apparatus of Fig. 6 is similar to the apparatus of Fig. 5 except that an intermediate image memory buffer 48 is provided between two-directional processors 40 and 42 which stores Sh+_ values for all image pixels. A particular advantage of the apparatus of Fig. 6 is that, due to the provision of image buffer 48 , two-directional processors 40 and 42 need not process along the same dimension of image data.

For example, as shown in Fig. 6 , two-directional processor 40 may process the image horizontally, row by row, as indicated by the superscript "h" of the output of processor 40 . Two-directional processor 42 may process the image vertically column by column, as indicated by the superscript "v" of the output of processor 42 . The indices of the output of processor 40 are indicated as 1 and i in Fig. 6 , whereas the indices of the input of processor 42 are indicated as m and j, because the two inputs may be read in along different dimensions and therefore are assigned different indices.

Reference is now made to Fig. 7 t which is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a preferred embodiment of the present invention. The apparatus of Fig. 7 may be similar to the apparatus of Fig. 1 except that it includes two two-directional processors 0 and 52 , each of which may be substant ally identical to two-directional processor 16 of Fig. 1 . Unlike in Figs. 5 and 6 , both two -di rec ional processors 50 and 52 in Fig. 7 are arranged in parallel and therefore both operate on raw data X ( 1 , i ) .

It is appreciated that two -direc tional processors 50 and 52 of Fig. 7 may process the image along the same dimension but using different input parameters sigman and rg . For example, two-directional processor 0 may process the image using K values suitable for excessive Smoothing whereas two-directional processor 2 may process the image using K values suitable for providing a choppy image.

The apparatus of Fig. 7 also includes an arithmetic unit 54 which is operative to combine the estimated signals S^+_ ( 1 - 1 , i) and S +_(l-l.i) , generated by two-directional processors 50 and 52 respectively, into an enhanced estimated signal S^ + _ ( 1 - 1 , ) . For example, the outputs of units 50 and may be suitably weighted and then added by unit 5^ *n order to obtain an indication of a high frequency enhancement. Alternatively, the combination operation of unit ^ m y comprise a weighted subtraction resulting in a bandpass frequency filter.

In Fig. 7 . the raw data input to the apparatus of Fig.

A R 7 is designated X(l,i) whereas the output is designated S + _ ( 1 -l,i) to indicate that the apparatus of Fig. 7 operates substantially in real-time, with a delay of only one line.

Reference is now made to Fig. 8 which is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with another embodiment of the present invention. The apparatus of Fig. 8 is similar to the apparatus of Fig. 6 except that the apparatus of Fig. 8 may operate in real time and in order to allow real time operation, the processing of the second dimension is not two -di re c tional in the sa^e sense as in Fig . 6. · ■' As shown, two-directional processor 42 of Fig. 6 is replaced by a "pseudo two-directional smoother" 80. Pseudo two-directional smoother 80 receives S^+_(l+l,i) output values from two-directional processor 40. These values are two-directional, as indicated by the subscript "+-", and were processed along a first dimension such as the horizontal dimension, as indicated by the superscript h. It is appreciated that the first dimension need not be the horizontal dimension and in fact may be the vertical dimension or a diagonal dimension oriented at some degree to the horizontal such as but not limited to degrees, or a time dimension. More generally, in all the embodiments illustrated herein, identification of a particular dimension with a particular orientation is not intended to be limiting.

Pseudo two-directional smoother 80 smoothes the output values of two-directional smoother 40 along a second dimension in accordance with Equations 10 - 15 of Appendix B. Two-dimensional smoother 80 is termed herein "pseudo two-directional" because of the difference between the first directiqn or top- to-bottom recursive estimation employed by smoother 80, defined by equation 10 , and the second direction or bo C orn- to- top recursive estimation employed by smoother 80 , as defined by equation 12 . The top- to-bo t torn estimation of equation 10 employs the second dimensional one-directional estimation of the previous row (one above the current row) .

In Equation 10 , Kv+(l-l,i) is the steady state estimation gain parameter as defined in Equation 4 of Appendix B for given re and snr for: snr = ( dv+ ( 1 - 1 , i ) /sigman) and dv+(l-l,i) = as defined in Equation 11 of Appendix B.

In contrast, the bottom- to- top estimation of equation 12 does not employ a second dimensional one-directional estimation of the previous row (one below the current row) since this procedure would necessitate storing of substantially the entire image and would introduce considerable delay. Instead, the bottom- to- top second dimensional estimation of equation 12 is based upon the first dimensional two-directional or horizontally smoothed estimation of the row below the current row. In other words, the second directional estimate for a current row is based only upon a single row preceding the current row rather than being based upon all rows preceding the current row. The advantage of using Equation 12 is that the delay introduced is only a one line delay.

Reference is now made to Fig. which is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with another embodiment of the present invention which is particularly useful in applications in which it is desired to preserve high frequency detail along a first dimension such as a horizontal dimension and to prevent the high frequency detail from being smoothed in the course of a second dimensional smoothing process such as a smoothing process along a vertical dimension.

The apparatus of Fig. 9 includes units 12 , 14 , 40 and 80 of Fig. 8. In addition, the apparatus of Fig. 9 includes a high frequency detail signal preserving unit 100. High frequency detail preserving unit 100 receives the signal outputs S*1 + _ ( 1 + 1 , i) of two-directional processor 40 and subtracts it from the corresponding original input image values X(l+l,i) in order to obtain values a +_(l+l,i) for the horizontal high frequency fluctuations. These fluctuations, in certain applications, are not considered undesired noise but rather indicate high frequency detail along the horizontal or first dimension which should be preserved. A mathematical definition of the ah+_(l,i) values is provided in Equation 16 of Appendix B.

As shown, the high frequency detail values, a^+_(l,i) , are preserved by storing in a line buffer 102 and do not enter the second dimensional smoothing process carried out by pseudo two-directional smoother 80. An arithmetic unit 106 is provided which combines the high frequency detail values of the first dimension with the two-dimensionally smoothed values of smoother 80 . A preferred equation according to which arithmetic unit 106 may be implemented is Equation 17 of Appendix B.

In equation 17 , g(l,i) is a high frequency gain factor which may be a constant or, alternatively, may vary over individual pixels, g determines the weight assigned to the ah+_ (l,i) values, relative to the two-dimensionally smoothed output values of unit 80. If g is too large, the high frequency detail will appear over-ecohasized in the output image, relative to the vertical smoothed information. If g is too small, the high frequency detail will appear to be insufficiently emphasized. Therefore, g may be initially set to a predetermined value such as 1 and may subsequently be changed to a different constant value which may be selected by visual inspection of the output image .

Alternatively, g may be computed as a function of individual pixels using a suitable method such as Wiener filters. Wiener filters are described in the abovereferenced blication by Mahesh et al.

The apparatus of Figs. 8 and 9 are useful in a wide variety of . applications . Two sample applications are described herein which are exemplary of possible applications.

Example 1 : Linear scanning detectors, such as CCD image scanners have response non-uniformi ies. Often, the detectors are calibrated and a large portion of the non-uniforniities are corrected by appropriate circuitry. However, such corrective measures are limited in accuracy, and residual non-uniformities on the order of 1% to % usually remain in the image. Such non-uniformities are perceived' as disturbing intensity differences or stripes between adjoining lines along the image scanning dimens ion .

In accordance with the present invention, such an image may be scanned in a dimension such as the horizontal dimension and intermediately stored in preprocessing unit 12. In the embodiment of Fig. 9, the stored image is first smoothed two-directionally along the horizontal dimension. The high frequency detail signal ah+_(l,i) is computed by differencing the incoming signal from the smoothed result and is stored in line buffer unit 102. The high frequency signal ah+_(l,i) is uncorrupted by overshoots and' ripples which usually occur in linear filtering since the low pass filter used is an edge preserving two-directional smoother.

The two-directional horizontally smoothed signal is then vertically smoothed by unit 80, and the result S +_(l,i) is added to the high frequency preserved signal a^"+ _(l,i) by arithmetic unit 106. In this example, the noise to be effectively reduced is mainly in the vertical direction due to line to line non-uniformities of scanning detector elements. The vertical non-uniformities appear as spikes as the apparatus of Fig. 9 proceeds along the vertical dimension of processing and consequently are significantly reduced.

Example 2 : A known problem is analog recording noise which appears in pre-recorded video images as horizontal stripes and streaks which are normally perceived as being colored. The streaks appear due to the PAL and NTSC video standards and prerecorded playback limitations. The streaking effects are often perceived as stripes due to brightness and color differences between adjacent video lines in various locations along the video lines, and detract from the quality of video and still video imagery. < The edge preserving two-directional smoother unit 40 of Fig. 9 nay operate as a high frequency extractor to line buffer 102. Unit 40 also provides an edge-preserved low frequency horizontal signal comprising all vertical non-uni orm ies which are to be reduced by pseudo two-directional smoother unit 80. If the image is a color image, such as an RGB image, the above described process may be applied to each of the three color images separately to achieve the final RGB image result.- Reference is now made to Fig. 10 which is a , simplified block diagram of image smoothing apparatus constructed and operative in accordance with a further preferred embodiment of the present invention. The apparatus of Fig. 10 is siailar to the apparatus of Fig. 5 in which one two-directional processor 42 processes the output of another two-directional processor 40 except that two-directional processor 40 is replaced by a three-dimensional processor 110.

Three-dimensional processor 110 provides S^(l,i) output for a current line to two-directional processor 42, which . may operate in a suitable dimension such as the horizontal. The S^(l,i) output for a current line 1 which is generated by three-dimensional processor 110 is a function of Sh · ( 1-1 , i ) output, for at least one pixel of a previous line, which is provided by two-directional processor 42. Preferably, the S output provided to processor 110 by processor 42 pertains to the vertical neighbor and the two diagonal neighbors of the current pixel, all three of which are located in the previous row.

In the above example, processor 110 is three-dimensional, the three dimensions being the vertical and both 45- degree diagonals. Al ernatively, processor 1 10 may be one- or two-dimensional. The processing carried out by processor 110 along each of the dimensions is one-directional, such as top-to-bottom in the present example.

A particular advantage of the above-described embodiment is that the reliability of the pre-es timated value of the estimated signal is enhanced by using previous estimates of neighbors of a current pixel as well as input regarding the current pixel .

Suitable equations for implementing units 110 and 42 of Fig. 10 are Equations 18 - 22 of Appendix B.

Preferably, the output of unit 42 in Fig. 10 includes two values, Sumlh · 3 ( l-l , ± ) and dh · ^ + _ ( 1 - 1 , i ) , as defined above with reference to Fig. 1 . These values are useful in certain applications, as explained below with reference to Fig. 1 1 .

Reference is now made to Fig. 11 which is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with another embodiment of the present invention which is particularly suitable for applications in which it is desired to preserve thin lines, such as lines whose width is only one pixel, rather than treating the thin lines as noise and smoothing them out.

The apparatus of Fig. 11 is similar to the apparatus of Fig. 10 except that a thin line preserving unit 120 is provided which interfaces with two-directional unit 42. Thin line preserving unit 120 includes an arithmetic unit 122 which receives at least one Sumlh,^+. value of the current line, rom unit 42 . Preferably arithmetic unit 122 receives three Suml '·> + _ values from the previous line for each current pixel, corresponding to the vertical neighbor and two diagonal neighbors of the current pixel in the previous line. Arithmetic unit 122 provides an address for LUT 18 in unit 42.

The Sumlh * + _ input of arithmetic unit 122, as defined with reference to Equation 23 of Appendix B, is the sum of the two one-directional differences for a particular pixel and therefore is an indication of the presence of a high ^frequency detail signal in that pixel. A logical equation suitable for implementing arithmetic unit 122 of Fig. 1 1 is Equation 23 of Appendix B. Equation 23 is operative to generate an output suitable for addressing LUT 18 by incrementing the LUT address d+_ if a thin line is found to extend from a current pixel i in row 1-1 to a^t least one of pixels i- 1 , i and i + 1 in row 1-2. The LUT address is incremented when a thin line is encountered because increasing the value of a LUT address has the effect of decreasing the amount of smoothing and a low level of smoothing is desirable when a thin line is encountered.

Reference is now made to Fig. 12 which is a simplified block diagram of image smoothing apparatus constructed and operative in accordance with a further embodiment of the present invention. The apparatus of Fig. 12 includes an · image preprocessing unit 140 which may be identical to image preprocessing unit 12 of Fig. 1. The preprocessed output o,f preprocessor l40 is provided to a one-directional time domain smoothing unit 142. Unit 142 computes a one-directional time domain estimate dt+(l,i,m) of the difference between the raw value of a pixel (l,i) in image m and between a temporally smoothed value of the corresponding pixel in image, m- 1 . The pixels of image m-1 are stored in a suitable memory unit such as image buffer 144.

The difference estimate dt+(l,i,m) is used to address LUT 18 which provides a value Kt+(l,i,m) which is employed as a weight as defined in Equation 24 of Appendix B.

Reference is now made to Fig. 13 which is a simplified block diagram of two-directional time domain image smoothing apparatus for smoothing a sequence of images which is constructed and operative in accordance with a further embodiment of the present invention. The apparatus of Fig. 13 includes a preprocessing unit 1 0 which may be identical to preprocessing unit 12 of Fig. 1 .

Preferably, a one-dimensional two-directional processor 152 receives the sequence of preprocessed images from preprocessing unit 150 and performs a spatial smoothing operation along lines of each image, which lines may comprise rows of each image .

The preprocessed spatially smoothed output of 2 -directional processor 15 or alternatively the preprocessed output of preprocessor 150 is received by a "pseudo-two directional" time domain smoothing unit 1 4 .

Preferred equations for implementing time domain smoothing unit 15 ^ are Equations 25 - 30 of Appendix B. Equations 2 - 30 assume that the input of time domain smoothing unit 154 arrives directly from preprocessor 150 . If a ,unit 1 2 is provided and the input of unit 1 4 arrives from unit 152 then the value x(l,i,m+l) is replaced by Sh+ _ ( 1 , i , m + 1 ) .

Time donain smoothing unit 154 computes an estimated signal value S +_(l.i,m) for pixel (l,i) of current image m using the estimated signal value 3 + _ ( 1. i , m-1 ) for pixel (l.i) of preceding, image m-1 and a raw value or two-direc tionally spatially smoothed value for pixel (l,i) of the current image m and for the same pixel of proceeding image m+1. Unit 15k is termed herein "pseudo 2- di rec tional " because the estimated signal values for image m are recursive functions of: (a) the estimated signal values for preceding images, given by S + ( 1 , i , m-1 ) in accordance with Equation 25 of Appendix B, and corresponding to a first direction of processing; and of (b) a less high-quality indication of the signal values for the proceeding image, given by St_(l,i,m+1) in accordance with Equation 27 of Appendix B, and corresponding to a second "pseudo-direction" of processing.

Conventional approaches to temporal noise reduction such as running average algorithms, discussed in the above-referenced publications by Gelb and by Rabiner and Gold have the disadvantage of blurring images of moving objects and of non-stationary images. The embodiments of Figs. 12 and 13 are operative to adapt the integration parameter on a pixel-by-pixel basis. Adaptation is in accordance with the measured difference between the current pixel and the smoothed result of the corresponding pixel in the previous image, as may be appreciated with reference to Equations 2 k - 30 of Appendix B.

It is believed that the applicability of the apparatus and methods show- and described herein are not restricted to smoothing of visual images and may also be useful ir. smoothing other types of data such as but not limited to audio signals and ultrasound signals. Also, the particular delay structure shown and described herein are merely exemplary of possible delay structure's. Any suitable conventional delay structure may be employed to implement each of the embodiments shown ar.d described herein.

Fig. l4 illustrates one-dimensional two-directional image smoothing apparatus which is a modification of the apparatus of Fig. 1 in that computation of the estimation gain parameter (EGP) is carried out externally of the two-directional processor 16 and in that a more sophisticated unit is employed to compute the EGP. It will be appreciated by persons skilled in the art that these features may be provided separately or, as illustrated, in combination and that one or both of these features may be provided as modif cations of the previously described embodiments, wherever suitable.

Units 212, 222, 224, 226, 228, 232 and 23^ of Fig. 14 are respectively similar to units 12, 22, 24, 26, 28, 3 and 3^ of Fig. 1. However, units 14 and 18 of Fig. 1, which are operative to provide an estimation gain parameter (EGP), referenced herein K, are replaced by an EGP computation unit 300 which is external to two-directional processor 216.

EGP computation unit 300 comprises an EGP address computation unit 310 which is similar to unit 14 of Fig. 1 and an EGP LUT 318 which is similar to unit l8 of F,ig. 1 and which is operative to provide an EGP value as a function of local signal characteristics, ich, in the illustrated embodiments, are operationalized as values sigman and rg .

It is appreciated that the LUTs shown and described in the present application, such as LUTs 18 and 318, may be replaced by any other suitable computational units.

Preferabl , the EGP unit 300 also provides per-pixel adjustment of the EG? by means of a cascaded per-pixel adjustment unit 320 which receives EGP output from the EGP LUT 3l8 and adjusts the EGP output in accordance with the geometric location of the current pixel within the image. This is particularly useful in applications where different iaage regions are known to require different degrees of smoothing. In these applications, the locations of these regions may be defined by a user. For example, in prepress and publishing applications, during retouching procedures such as cut-and-paste and blending, a user may identify regions which require strong smoothing and regions which require weak smoothing..

Typically, adjacent differently smoothed image regions need to blend smoothly into one another so as to avoid visually disturbing image discontinuity artifacts. Therefore, the geometric function which alters K as a function of image location must be smooth. For example, an adjusted K can be computed from an initial K value. K , received from a K LUT such as LUT 318, * using Equation 31 of Appendix B. K is adjusted to take into account the distance between the processed pixel coordinate i, and the designated point at which the desired noise reduction process is to operate. As the processed image point departs from the designated poi t, g(a,d,i) of equation 31 approaches unity, and the EGP K(i) approaches unity. From equations 3A, 3B and 7 of Appendix B it is apparent that as K(i) approaches unity, the smoothing operation becomes transparent. In other words, it does not affect the processed image, hence achieving a smooth transition between adjoining image regions.

More generally, EGP adjustment unit 320 may be utilized to adapt the EGP in accordance with any type of feature- of the image and/or in accordance with any other empirical or heuristic, application-specific or other information which may be available. Suitable image-derived control information which may be received and utilized by EGP adjustment unit 320 may be obtained by performing additional local and global computations on the incoming image signal X. Control information may also be user-determined.

In the illustrated embodiment, EGP unit 300 receives three difference inputs (d inputs) from smoothers 22k, 226 and 228 , respectively, and provides three EGP ' s to the same three smoothers respectively. Alternatively, however, unit 300 may be operative to process a single incoming d input by sequentially multiplexing the three d signals into unit 300 . The respective results may then be output in the same order.

Fig. 15 illustrates spatial noise reducing apparatus, referenced generally 332 , which combines the features of Figs. 8 and 10 . The spatial noise reduction apparatus 332 of Fig. 15 includes an image preprocessing unit 3 ^0 which may be similar to image preprocessing unit 12 of Fig. 1 and a spatial noise reduction unit 330 which combines the features of the spatial- noise reduction systems of Figs. 8 and 10.

Specifically, spatial noise reduction unit 330 typically includes the following units: a. a 3-dimensional processor 50 which is substantially similar to 3 -dimens ional processor 110 of Fig. 10 except that EGP computation is preferably external so that processor 350 provides a multiplexed difference value dm to, and receives a multiplexed EGP value K__ from, an EGP computation unit 352. EGP computation unit 352 may be similar to EGP computation unit 300 of Fig. 14. b. a 2-directional processor 36Ο which is substantially similar to 2-directional processor 40 of Fig. 8 except that EGP computation is preferably external so that 2-directional processor 36Ο provides a multiplexed difference value dm to, and receives a multiplexed EGP value Km from, an EGP computation unit 362. EGP computation unit 362 may be similar to EGP computation unit 300 of Fig. 14. c. a "pseudo 2-directional" processor 370 which is substantially similar to pseudo 2-directional processor 80 of Fig. 8 except that EGP computation is preferably external so that processor 370 provides a multiplexed difference value dm to, and receives a multiplexed EGP value Km from, an EGP computation unit 372. EGP computation unit 372 may be similar to EGP computation unit 3OO of Fig. 14.

It is appreciated that the three EGP computation units 352, 362 and 372 may be replaced by a single EGP computation unit serving all three processors 350. 36Ο and 370, in parallel or multiplexed mode. 100589/3 The combined operation of the various conponents of noise reducer 330 in Fig. 15 is now described. 3-dimensional processor 350 uses previously smoothed estimates of adjacent image lines to compute an improved estimate of pixels in a current line. Subsequently, 2-directional processing is applied by 2-directional processor 360 which applies strong smoothing in a first dimension to narrow noise spikes in a first dimension whose width is a single pixel.

If unit 360 operates independently, independently computed two-directional computations of adjacent image lines (rows or columns) may develop phase offsets which are seen by human viewers as noise stripe effects. However, these effects are reduced by the operation of 3~dimensional processor 3 0 which correlates the estimates of adjacent lines.

Finally, pseudo 2-directional unit 370 applies strong smoothing to narrow noise spikes of single pixel width, along a second dimension.

Fig. l6 illustrates estimation gain parameter computation apparatus, referenced generally 400, which is a variation on unit 300 in Fig. 14. Although the apparatus of Fig. 16 is useful in Fig. I , more generally, the apparatus of Fig. 16 may be used to provide EGP ' s to any of . the image processors in any of the figures shown and described herein. In unit 400, rather- than directly modifying K values as in unit 300, sigman values are modified as a function of pixel location coordinates.

The EGP computation apparatus 400 receives image point coordinates i and difference data d(i), usin notation developed above with reference to previous figures. A modified value for 100589/3 standard deviation of noise, sigman , is computed by sigman computation unit 10 in accordance with Equation 32 of Appendix ♦ The sigman output of computation unit 4 10 is received by a sigman LUT 420 which provides per-pixel adjustment of sigman in accordance with the coordinate i of the current image pixel X(i). For example, the LUT 420 may be arranged so as. to # decrease sigman for pixels which lie far from a user-designated point of interest and to increase - sigman for pixels which lie close to the user-designated point of interest, as in Equation 32 of Appendix B.

A signal-to-noise (snr) computation unit 30 is operative to compute a pixel-specific snr value, by using received pixel-specific sigman values to normalize corresponding pixel-specific d values . in accordance with Equation 3 of Appendix B.

EGP values K, for a variety of snr values and correlation values rg , are computed by a K computation unit 440 , using Equation -of Appendix B with steady state values for K and for snr, from Equation 3 of Appendix B. The EGP values K are stored in a LUT 450. Retrieval from LUT 450 is controlled by snr values arriving from snr computation unit 430. The output of EGP computation apparatus 400 is a pixel-specific EGP value K(i).

Fig. 17 illustrates estimation gain parameter computation apparatus, referenced generally 460, which is a variation on unit 400 in Fig. l6.

The EGP computation apparatus 460 receives raw image 100589/3 * * standard deviation of noise, sigman , is computed by sigman computation unit lO in accordance with Equation 32 of Appendix B .

♦ The sigman output of computation unit 4 10 is received by a sigman LUT 420 which provides per-pixel adjustment of sigman in accordance with the coordinate i of the current image pixel X(i) . For example, the LUT 420 may be arranged so as to decrease sigman for pixels which lie far from a user-designated point of interest and to increase sigman for pixels which lie close to the user-designated point of interest, as in Equation 32 of Appendix B.

A ' signal- to-noise (snr) computation unit 430 is operative to compute a pixel-specific snr value, by using received pixel-specific sigman values to normalize corresponding pixel-speci ic d values . in accordance with Equation 34 of Appendix B.

EGP values K, for a variety of snr values and correlation values rg , are computed by a K computation unit 440 , using Equation ' 4 of Appendix B with steady state values for K and for snr, from Equation 3 ^ of Appendix B. The EGP values are stored in a LUT 450 . Retrieval from LUT 50 is controlled by snr values arriving from snr computation unit 430 . The output of EGP computation apparatus 40Q is a pixel-specific EGP value K(i).

Fig. 17 illustrates estimation gain parameter computation apparatus, referenced generally 460 , which is a variation on unit 400 in Fig. 16 .

The EGP computation apparatus 4 60 receives raw image 100589/3 data coordinates ( i } and difference data d(i), using notation developed above with reference to previous figures. A modified value for standard deviation of noise, sigman, is computed by sigman computation unit jQ in accordance with Equation 33 of Appendix B. Alternatively, a user-determined sigman value may be employed.' The sigman output of computation unit 470 is received by a sigman LUT 480 which provides per-pixel adjustment of sigman in accordance with the current pixel X(i) of the raw image. For example, the LUT 480 may be arranged', so as to increase sigman for bright pixels and to decrease sigman for dark pixels, as in Equation 33 o Appendix B.

A signal-to-noise (snr) computation unit 490 is operative to compute a pixel-specific snr value, by using received pixel-specific sigman values to normalize corresponding pixel-specific d values in accordance with Equation 35 of Appendix B.

EGP values K, for a variety of snr values and correlation values rg , are computed by a K computation unit 440, using Equation 4 of Appendix B with steady state values for K and for snr, from Equation 34 of Appendix B. The EGP values K are stored in a LUT 450 . Retrieval from LUT 4 0 is controlled by snr values arriving from snr computation unit 490 . The output of EGP computation apparatus 460 is a pixel-specific EGP value K(i).

The apparatus of Fig. 17 is particularly suited to applications where an image has various regions of different brightnesses - and different noise levels. For example, in CCD imaging, which may be carried out by a one-dimensional scanner or 100589/3 data coordinates X ( i ) and difference data d ( i ) ·, using notation developed above with reference to previous figures. A modified value for standard deviation of noise, sigman, is computed by sigman computation unit 470 in accordance with Equation 33 of Appendix B. Alternatively, a user-determined sigman value may be employed." The sigaan output of computation unit 470 is received by a sigman LUT 480 which provides per-pixel adjustment of sigman in accordance with the current pixel X(i) of the raw image. For example, the LUT 480 may be arranged so as to increase sigman for bright pixels and to decrease sigman for dark pixels, as in Equation 33 of Appendix B.

A signal-to-noise (snr) computation unit 90 is operative to compute a pixel-specific snr value, by using received pixel -specific s.igman values to normalize corresponding pixel-specific d values in accordance with Equation 35 of Appendix B.

EGP values K, for a variety of snr values and correlation values rg , are computed by a K computation unit 440 , using Equation 4 of Appendix B with steady state values for K and for snr, from Equation 34 of Appendix B. The EGP values K are stored in a LUT 450. Retrieval from LUT 4 0 is controlled by snr values arriving from snr computation unit 490 . The output of EGP computation apparatus 460 is a pixel-specific EGP value K(i).

The apparatus of Fig. 17 is particularly suited to applications where an image has various regions of different brightnesses and different noise levels. For example, in CCD imaging, which may be carried out by a one-dimensional scanner or 87 100589/3 a two-dimensional camera, the dynamic range of the sensing process is typically divided into three signal ranges, each of which has a different dominant noise. At very low levels of signal, dark noise dominates. At midrange intensities, shot or photon noise dominates. At high intensity levels, photoresponse nonuniformities or fixed pattern noise dominates. The noise measured in this type of application is roughly a root sum of squares of the three categories of noise. Therefore, each CCD may be calibrated so as to take into account a different noise level for each level of illumination.

Alternatively, sigman LUT 480 may be preset by a user, as in applications in which it is desired to control the degree of smoothing as a function of levels of luminance or of color.

Fig. 18 illustrates an estimation gain parameter adjustment unit 500 constructed and operative in accordance with a first alternative embodiment of the present invention which is a variation of estimation gain parameter adjustment unit 320 of Fig. Ik. The EGP adjustment unit 500 receives raw data, X(i), and an initial EGP, K(i)*, from LUT 18 of Fig. 1 or from LUT 318 of Fig. 14. - Replacement of estimation gain parameter adjustment unit 320 of Fig. 14 with corresponding unit 500 renders estimation gain parameter computation unit 300 particularly useful in reducing certain types of image compression noise. Specifically, unit 300 in which unit 500 replaces unit 320 is particularly useful in reducing image compression noise which is often created by widely used block transform coding schemes such as Discrete 88 100589/3 Cosine Transform (DCT) based JPEG, MPEG and CCITT H.261. In the encoding process of such schemes, the input image pixel samples are grouped typically into. 8x8 blocks, each block transformed by the DCT into a set of 64 values referred to as DCT coefficients. One of these values (the block average brightness) is referred to as the DC coefficient, and the remaining 63 as the AC coefficients. The compression of the image is achieved by quantizing each of these coefficients using one of 64 corresponding values from a quantization table. The best performing quantization tables, in terms of visual fidelity of the decompressed images, are generally characterized by a higher degree of quantization at the higher frequency DCT coefficients as compared to. the lower frequencies in each block. The lost information due to the quantization operation is often perceived by human observers in the form of blocky artifacts in image regions having smooth brightness surfaces, and in the form of edge noise ("mosquito noise") at pixels which are located in or adjacent to blocks having high contrast quantized edges.

The general problem in DCT compressed images is to remove or reduce the Block noise and edge noise without impairing the quality of the image, thereby retaining both continuity of smooth surfaces and contrast of edgy and textured regions . This is done by taking advantage of the fact that block noise is known to have a well defined geometric structure, namely, the shape of the block. Therefore, given the size of the DCT block, for each image point, a geometric relationship with respect to the grid can be determined. This information can be used to .control the estimation gain parameter K, using unit 400. For example, varying 89 100589/3 degrees of noise reduction smoothing may be desirable along boundaries of adjacent blocks as compared to the smoothing within each individual block.

In addition to using knowledge of the block geometry for improved control of K values in unit 400, additional image derived information on the decompressed image signal contents within each block can be utilized to deal with compression noise such as block noise and edge noise. The amplitudes of the noise artifacts are a function of the degree to which the DCT coefficients are quantized. Given the compression ratio for a given image, the expected noise quantization noise level can be roughly predicted and used in units 310 and 318 of the estimation gain parameter computation unit 300. Morever, image regions in which block noise prevails may be identified by comparing the high frequency signal content to the low frequency signal content on a pixel by pixel basis (absolute amplitude ratios) or on a block by block basis (rms ratios computed separately for each block or for groups of adjoining blocks) .

Referring again to Fig. 18, computation unit 510 is operative to compute the average DC value of X within each block, XN£, using Equation 36 of Appendix B. The per-block average DC values are then employed by computation unit 5 0 to compute the AC component of the raw data in the block, XAC(i) , by computing the difference between the input and the per-block DC value, as set forth in Equation 37 of Appendix B. The output of computation unit 5 0 is the standard deviation of The standard deviation of the AC component, si ma^Q, of 90 100589/3 block is utilized by units 510 and 520 to compute and create a coefficient fl using equation 38 of Appendix B or alternatively f2 using equation 39 of Appendix B. Either of coefficients fl and f2 may be used to adjust the estimation gain parameter K*(i) previously computed by unit 18 or 318. The adjustment in this instance is such that blocks having relatively low sigma_ac, should be smoothed to a higher degree since block noise is expected in those regions. On the other hand, for blocks having a high sigma_ac (textured regions for example), the noise reduction operation is preferably tuned to the expected noise level for which the K*(i) was previously computed. In this case, both factors fl and f2 approach unity.

Computation unit 5^0 receives values for CI and C2 by default or by the user. CI can be selected on the order of sig-man, the expected noise in the image, such that if sigma^^ is much smaller than the expected noise, then high degree of smoothing is desired even in cases where local d signals are high because it is expected that these are due mainly to block noise which should be smoothed out. If' however sigma^Q is much greater than sigman, ; then the noise reduction should use the computed K*(i) factor computed based on the expected noise level. In the f2 factor alternative, C2 is selected so as to provide a maximum f2 value in cases where sigma_ac is very small. In those cases, f2 = C2/C1. A typical value for C2 is between Cl/3 and Cl/5.

Fig. 1 illustrates an estimation gain parameter adjustment unit 550 which is a variation of the apparatus of Fig. 18. EGP adjustment unit 0 includes units 560, 570 and 58Ο which operationalize equations 36, 37. ^0 an l. More generally, the 91 100589/3 •apparatus of Fig. 19 employs locally derived image signal high and low frequency measures sigmaAC and sigmanQ to indicate the degree to which block noise is expected to be visible to the observer. Average DC values for each block of raw values are computed,, and sigma^, the standard deviation of the average DC values, over blocks, is also computed. Sigma^Q is the standard deviation of the average AC values within each block.

A particular feature of the apparatus of Fig. 1 is that, regardless of the AC standard deviation in current or adjacent blocks, the EPG remains essentially unchanged if the DC standard deviation is small compared to the the AC standard deviation and a fine tuning constant C3. If, however, sigman(^ >> sigma^Q, then 3 decreases and this decreases the K smoothing parameter which increases the degree of smoothing. C3 is a fine tuning constant which varies the degree at which the s gma^Q affects the f 3 factor in the presence of given levels of sigmaAC< C3 is to be set by a user, with a default value of zero or unity. The fine tuning is achieved by observing the displayed image and modifying C'3 until the best result is achieved.

Fig. 20 illustrates another variation of an estimation gain parameter adjustment unit, referenced generally 600 . Equations for units 610 and 620 are numbered as Equations 37. 40 and 42, respectively, of Appendix B. Like the apparatus of Fig. 19 » the apparatus of Fig. 20 also employs locally derived image signal high and low frequency measures to indicate the degree to which block noise is expected to be visible to the observer. However, in Fig. 20 , instead of applying an f factor which is 92 100589/3 within each block, the f factor varies over pixels within a single block. However, if the DC standard deviation is small compared to the sum of | XAC | and a constant C4. then the EPG remains essentially equal to K . The computation of the constant C4 is similar to the computation of constant C3 in unit 570.

It is appreciated that the alternative methods for fine-tuning of the EPG described above with reference to Figs. 18 - 20 merely exemplify the various possibilities for implementing fine-tuning unit 320 of. Fig. 14 and are not . intended to be limiting. For example, more smoothing may be applied to image regions having weak AC signal components, compared to local DC variations, and less smoothing may be applied to image regions having strong AC components, compared to local DC variations. The amount of DC variations depends, inter alia, on the size of the blocks which typically is 8x8 or 16 x 16.

Fig. 21 illustrates apparatus, referenced generally 65Ο, for combined spatial noise reduction and enhancement of an image. The apparatus 65Ο of Fig. 21 is operative to perform one and/or two-dimensional contrast enhancement on an image which has been presmoothed using any of the image smoothing techniques provided in accordance with the present invention. Synergism is provided between the smoothing and contrast enhancement functions such that: (a) Some of the contrast lost due to the smoothing process is recovered by the enhancement process, whereas introduction of artifacts; and 1 (b) False signal overshoots are avoided by avoiding 93 100589/3 enhancement in the vicinity of those edges whose contrast is such that enhancement is unnecessary.

The apparatus includes a preprocessing unit 660 which may be similar to preprocessing unit 12 of Fig. 1, a spatial noise reduction unit 670, and an image enhancement unit 680. Spatial noise reduction unit 70 may be similar to unit 330 of Fig. 15· Alternatively, unit 670 may comprise a spatial noise reduction unit in which individual features shown and described above in the context of any of Figs. 1 - 11 and 14 - 20 are suitably combined. Image enhancement unit 68Ο is described in detail below with reference to Fig. 22.

If spatial noise reduction unit 670 performs only one-dimensional smoothing, one-dimensional enhancement may be employed .

An alternative to the illustrated embodiment is that spatial noise reduction unit 67Ο may be eliminated and image enhancement may be applied directly to the raw preprocessed image. For example, when the image is a high snr image which is blurred as < a- result of imaging lens defocus, it is sometimes unneccessary- to presmooth the image prior to enhancement.

Fig. 22 is a simplified block diagram of the enhancement unit 680 of Fig. 21. A high pass filter 700 extracts high frequency detail signals from the incoming image. Units 720 and 75 provide gain parameters which are employed by multiplier units 740 and 770 to amplify the high frequency detail. The amplified high frequency result is added to the original incoming image signal by adding unit 780. 4 100589/3 If enhancement is one-dimensional, high pass filter 700 may be operationaiized as the difference between an incoming current image pixel X(i) and a 1, 2, 1 weighted average of pixel i and its two neighbors along the single dimension. If enhancement is two-dimensional, the high pass filter may be operationaiized as the difference between an incoming image pixel and a weighted average of the pixel's vicinity, using suitable weights such as the following matrix of weights: 1 2 1 2 2 1 2 1, where rows of weights correspond to respective lines in the image, and where the current pixel is located in the center.

The high pass signal may then be given by the weighted operator -1 -2 -1 -2 12 -2 -1 -2 -1, normalized by 16, which is a simple bit shift. The above numerical values have been found to give good visual results, to be easily implementable , and to be easily embeddable in 3" dimensional smoothing unit 110 of Fig. 10. However, the specific numerical values are not intended to be limiting.

Still with .reference to Fig 22, the amplification gain parameter g is intended to provide visually pleasing amplification of the high frequency detail signal H, in that the gain g is limited to some preset maximum value in cases of very small amplitudes of H, thereby avoiding enhancement of low snr edges. Also, the gain g is nearly inversely proportional to the amplitude of H(i) for increasing values of |H(i)| , thereby avoiding overenhancing artifacts of already contrasty 'edges .

A gain function g which is often used, in the literature 95 100589/3 is given in equation 43 of Appendix B, as a function of locally computed standard deviation of the detail signal H, sigma^. Large computation windows for s gmajj often introduce artifacts into the enhanced image. Alternatively, as the window size diminishes, the accuracy of the sigma^ estimation is reduced, but the parameter relates more directly to the pixel in question, that is pixel in the center of the window. At the limit, with window size being that of a single pixel, sigmaH ( i j = | H ( i ) | , and g is given in equation 44 of Appendix B. The reduced accuracy in using small windows for computing sigmac is improved by pre-smcothing the incoming image as illustrated in Fig 21. Spatial noise reduction unit 67Ο is applied prior to enhancement unit 680 , such that presmoothed values |H(i)J comprise better estimates which can then be used directly in computing g. In this case unit 710 simply computes the absolute value of the detail signal H, a considerable simplification over sigma^ computations.

From equation 44 of Appendix B, for small values of |H(i) | , the value of g approaches MAX, a parameter defined by default or ·· user, and typically ranges between 1 to 2 . For IH ( i ) I>>(C5 MAX) , . the amplitude of the amplified signal |H(i) |*g(i) is maximum and approaches C5- Typical settings for the parameter C5 range between 8 to 32 gray levels (in 256 gray level display systems).

An improved gain function g which further trims the amplified response at large contrast edges is computed using equation 45 of Appendix B. In this case, as the edge contrast increases, the amplified response diminishes and approaches zero. 96 flmax| is the maximum expected edge contrast in a given image, and can in certain cases be assumed to be the entire image signal range. Further tuning of the gain g function can be achieved using equation 46 of Appendix B, where the parameter p has the effect of altering the |H| amplitude at which the maximum amplification amplitude of |H(i)|*g(i) occurs.

Referring again to enhancement unit 680 in Fig 22, the amplified signal H(i)*g(i) is further amplified by a second gain factor w(i), which is determined by w LUT unit 750 . The w gain values are determined by w computation unit 760 using equation 47 of Appendix B, with W a tuning scaling constant which can be user defined according to the particular image, and the wp parameter determining the degree of nonlinearity in the w(i) function as a function of brightness X(i). The w gain parameter is intended to compensate for the assumed logarithmic contrast response of the human visual system, whereby contrast detection threshold is known to vary linearly with image brightness levels. As the brightness in the image increases, the amplification w must also increase such that the observer can perceive similar edge gray level differences at all brightness levels of image regions. The enhanced image signal output from unit 680 is given in equation 48 of Appendix B .

Fig 23 illustrates apparatus, referenced generally 1000 , for dynamic range compression useful in instances where the image display unit may have a narrower display signal dynamic range as compared to that of the incoming image. The dynamic range of imaging sensors such as CCD devices is generally expressed in terms of the ratio of maximum edge contrast which can be imaged 97 by the camera without it being saturated, to the minimal level of noise in the signal. This ratio indicates the extreme situation in which both sides of a maximum contrast edge (black and white) must be imaged, while retaining sufficient sensitivity to sense the minimal signal noise level which generally occurs along the low brightness part of the edge.

Display devices such as TV monitors and hard copy prints typically have display ranges of 64:1 to 256:1, while CCD imaging cameras typically have dynamic ranges of 1000:1 to 2000:1. One obvious and trivial method of reducing the acquired image dynamic range to the display range is to simply apply a linear or nonlinear transformation to the incoming image signal. Such methods generally reduce the sensitivity of the imaging system, in that low contrast edges may disappear in the compressed display image. Linear compression allocates display ranges equally throughout the image dynamic range. Nonlinear compression often allocate larger display ranges to selected image signal ranges such as low brightness regions such that low contrast detail inforaation may be visible. in the displayed image. This may have the effect of amplifying this low brightness image region and its corresponding noise. Allocating larger display ranges to selected image signal ranges comes at the cost of reducing the display ranges of the remaining image signal ranges as is often the case in high brightness regions. In such regions it is expected that the detectivity of low contrast detail signals will be reduced.

In remedy of the noise amplification in low brightness regions and loss of sensitivity in high brightness regions, the 98 dynamic range compression unit 1500 which preferably consists of a LUT which values are computed and loaded by computation unit I55O, is preceded by noise reduction unit 670 which can be tuned for example by unit 470 of Fig. 17 to reduce noise in selected image regions as a function of image brightness level. Morever unit 1000' is preceded by enhancement unit 67Ο which can be tuned for example by unit 750 in Fig 22 to enhance image regions in compensation of expected reduction in sensitivity due to a given dynamic range compression transformation.

Fig 24 illustrates apparatus, referenced generally 2200, for electronic image magnification. Image magnification involves the interpolation of pixel values using adjoining pixel values. Image magnif cation often tends to increase the visiblity of image artifacts such as aliasing effects in the original magnification image as well as noise. The method illustrated in Fig -24 incorporates repeated smoothing and pixel replication to achieve image magnification while avoiding interpolating (smoothing) high contrast edges. The pre-smoothing operation is utilized both as an anti-aliasing filter which preserves edges of width larger than one pixel along the dimension of processing, and for the reduction of noise which would otherwise be perceived, upon magnification, as blurred spots in the image.

A single magni ication step involves unit 2000 which is composed of a spatial noise reduction step computed by unit 670, followed by pixel replication unit 2100 and thereof followed again by a noise smoothing unit 67Ο with modified noise setting parameters. Multiple magnification steps involve repeated applications of unit 25OO. Optionally, the signal removed by the first 99 100589/3 application of unit 670 may, rather than being discarded, be added back to the magnified image in order to restore detail information .

Fig 25 illustrates method and apparatus, referenced generally 2600, for spatio-temporal image noise reduction. Apparatus 26ΟΟ comprises spatial noise reduction unit 670 followed by temporal noise reduction unit 142 which interacts with estimation gain parameter computation unit 300· It is intended here that improved spatio-temporal noise reduction may be achieved by incorporating spatial signal information such as sigma^,-. in unit 55Ο or |Xac(i) | in unit 600 in the control of K+t(l,i,a) of unit 142. For example in decompressed JPEG or MPEG image sequences, in-block AC signal measures can be compared to temporal difference signals d+t(l,i,m). In instances where d+ t ( 1 , i , m ) >> IXac ( 1 , i , m ) | , there is indication that the d+fc signal is caused by block quantization noise rather than by image motion and this may decrease the setting of K+t(l,i,m) such that increased temporal smoothing occurs at that pixel.

Fig 2,6 illustrates method and apparatus, referenced generally 2700, for spatio-temporal image noise reduction. Apparatus 27ΟΟ differs from apparatus 2600 in that it uses temporal smoothing unit 15^· Otherwise all considerations are similar.

Fig 27 illustrates general block diagram for spatio-temporal dynamic range compression whereby spatial pre-smoothing of the image precedes temporal noise reduction units 142 or 15^· The output of the temporal noise reduction unit is then enhanced by enhancement unit 680. By combining spatio-temporal noise 100 100589/3 uction with enhancement and dynamic range compression operations, spatial information may be employed to control temporal filtering and vice versa.

It is appreciated that the apparatus and methods for image smoothing, enhancing and interpolating, shown and described herein, are useful in a very wide variety of applications. Sample applications are now described.

Reference is made to Fig. 28 which is a simplified block diagram of improved analog still video equipment incorporating the apparatus for image smoothing, enhancing and interpolating and dynamic range compression shown and described herein.

The apparatus of Fig. 28 includes an analog still video camera 3010 which photographs a scene 3000. The analog video signal generated by camera 3010 may be stored on a suitable medium, such as a video floppy disk 3020, for subsequent replay on a suitable device such as a still video player 3030.

The analog video signal generated by analog still video camera 3010 or the signal stored on video floppy disk 3020 may be provided to a computer 3040 equipped with a digitizing video board which is operative to digitize the signal and then to further modify the digitized signal as per user instructions. For example, the computer 30^0 may perform color modifications on the digitized signal.

The original signal as stored on video floppy disk 3020 or as provided on-the-fly, or the modified digital signal provided by computer 3040, is provided to an output device such as a TV display monitor 3050, a still video printer 3Ο6Ο or a VCR 101 100589/3 30J0. Al ernatively, the signal may be remotely transmitted, via a TV transmitter 3080, to a remote output device for immediate display or for storage on a suitable medium such as a video floppy disk.

Any of the image smoothing, enhancing and interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 may be incorporated into the apparatus of Fig. 28 at any one of the following points: a. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27, including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be incorporated as IC's and/or VLSI's within any of the following components of Fig. 28: camera 3010, video player 3030, digitizing computer 3040 , TV display monitor 3050 , still video printer 3060 , VCR 307O and TV transmitter 3Ο8Ο. b. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27, apart from the analog to digital conversion in preprocessing unit 12, but including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be implemented in software and incorporated into the software of digitizing computer 3040 .

Reference is made to Fig. 2 which is a simplified block diagram of improved digital still video equipment 102 100589/3 incorporating the apparatus for image smoothing, enhancing and interpolating shown and described herein.

The apparatus of Fig. 29 includes a digital still video camera 3 0 which photographs a scene 3100 and digitizes and stores the acquired digital image onto digital floppy disk 3120 in uncompressed or compressed mode such as JPEG image compression standard, for subsequent decompression and replay on a suitable device such as a digital still video player 3130.

The digital compressed or uncompressed video image signal generated by digital still video camera 3110 or the signal stored on digital floppy disk 3120 may be provided to a computer 3l40 equipped with an appropriate digital interface, which is operative to further decompress and modify the digital signal as per user instructions. For example, the computer 31^0 aay perform color modifications on the digital decompressed signal. The original signal as stored on digital floppy disk 3120 or as provided directly from digital still video camera 3H0, or the modified digital signal provided by computer 31^0, is provided to an output device such as a TV display monitor 3150. a still video printer 3l60 or a VCR 3170. Alternatively, the signal may be remotely transmitted, via a TV- transmitter in the form of standard modem equipment, to a remote output device for immediate display or for storage on a suitable medium. The image smoothing, enhancing, interpolating and dynamic range compression shown and described herein with reference to Figs. 1-27 may be incorporated into the apparatus of Fig. 2 at any one of the following junctures: . a. Any. of the image smoothing, enhancing, interpolating 103 100589/3 and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27, including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be incorporated as integrated circuits and VLSI's within any of the following components of Fig. 29: camera 3H0. video player 3130. computer 31^0. TV display monitor 3150, still video printer 316O, VCR 317O and TV transmitter 3180. b. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27, apart from the analog to digital conversion in preprocessing unit 12, but including variations on the systems specifically shown and described which ccabine features of the various systems specifically shown and described, can be implemented in software and incorporated into software of computer 31^0.

Reference is made to Fig. 30 which is a simplified block diagram of improved analog and digital moving video equipment incorporating the apparatus for image smoothing, enhancing, interpolating and dynamic range compression shown and described herein. The apparatus of Fig. 30 includes an analog or digital video camera 3210 (and camcorder 3220) which photograph a scene 32ΟΟ and record the acquired analog video onto VCR 3250 (onto built in recorder of camcorder) or digital video onto digital VCR 325Ο in uncompressed or compressed mode such as JPEG or MPEG image compression standard. The analog video recording is useful for subsequent display on TV display 3270 or digitization onto 104 100589/3 computer 3230 or CD recorder 3240 and later display onto TV display 3270 . The digital video recording is useful for display onto digital TV display 3 70 and storage onto CD recorder 3240 and computer 3230 . Computer 3230 and CD player 3240 can decompress the stored aoving video and display the video onto TV display 3270 . Recorded video in all formats can then be replayed and edited and improved for example by computer 3230 on MPEG compressed video sequences for authoring applications. The image smoothing, enhancing, interpolating and dynamic range compression shown and described herein with reference to Figs.l-27 may be incorporated into the apparatus of Fig. 30 at any one of the following junctures: a. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 , including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be incorporated as integrated circuits and VLSI's within any of the following components of Fig. 30 : cameras 3210 and 3220 , VCR 3250 , computer 3230 , CD recorder/player 3 40 , TV display monitor 3 70 . b. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 , apart from the analog to digital conversion in preprocessing unit 12 , but including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be implemented in software and incorporated into software of 105 100589/3 v digitizing computer 3230.

Reference is made to Fig. 31 which is a simplified block diagram of improved image scanning, display and reproducing incorporating the apparatus for image smoothing, enhancing, interpolating and dynamic range compression shown and described herein. The apparatus of Fig. 31 includes an electro-optic scanner 3310 which scans a transparency image 3300 or hard copy image 3305 · and stores the scanned image in a digital file 3320 . The digital image file 3320 is useful for subsequent printout on printer 3330 of improved harcopy 3350 , and for further image processing on workstation 33^0 . Improved image can then- be displayed on TV display 3360 , printout on printer 3330 and image storage arcive 3370 . The image smoothing,, enhancing, interpolating and dynamic range compression shown and described herein with reference to Figs. 1 - 27 may be incorporated into the apparatus of Fig. 31 at any one of the following junctures: a. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 , including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be incorporated as integrated circuits and VLSI ' s within any of the following components of Fig. 31 : scanner 3310 , image printer 3330 , workstation 33^0 , TV display monitor 3270 . b. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 , apart from the analog to digital 106 100589/3 conversion in preprocessing unit 12, but including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be implemented in software and incorporated into software of workstation 33^0.

Reference is made to Fig. 32 which is a simplified block diagram of improved fax image scanning, coding, transmission, decoding, processing, display and storage incorporating the apparatus for image smoothing, enhancing, interpolating and dynamic range compression shown and described herein. The apparatus of Fig. 32 includes an electro-optic fax scanner 34lO which scans a hard copy image 3^00, and stores the scanned image in a digital file 3^20. The digital image file 3^20 is useful for subsequent coding 3*30 and transmission. Upon reception, the the encoded image file is decoded by 3^0 or 3^70, for further improvement, processing and analysis in computer 3^3. and fax printout 3^60 and fax image archiving 3^90. The image smoothing, enhancing, interpolating and dynamic range compression shown and described herein with refernce to Figs. 1-27 may be incorporated into the apparatus of Fig. 32 at any one of the following junctures: a. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27, including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be incorporated as integrated circuits and VLSI "s within any of the following components of Fig. 32: fax scanner 3^10, f coder 107 100589/3 3 ^ 30 . b. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 , apart from the analog to digital conversion in preprocessing unit 12 , but including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be implemented in software and incorporated into software and DSP software of fax scanner 3410 and coder 3^ 30 .

Reference is made to Fig. 33 which is a simplified block diagram of improved teleconferencing system of televising, coding, transmtting, receiving, decoding, analysi, display and recording incorporating the apparatus for image smoothing, enhancing, interpolating and dynamic range compression shown and described herein. The apparatus of Fig. 33 includes an imaging system 3510 . which televises a typical conference scene 3500 , and compresses the video information via CCITT H.261 or MPEG or JPEG coding schemes 3530 . The coded data is transmitted via ISDN or satelite or telephone lines to a remote location, where it is decoded : 35^ 0 , and displayed onto TV display 3560 , recorded on VCR 3570 or further analyzed on computer 3580 for subsequent display and recording. The. image smoothing, enhancing, interpolating and dynamic range compression shown and described herein with refernce to Figs.1 - 27 may be incorporated into the apparatus of Fig. 33 at any one of the following junctures: a. Any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein 108 100589/3 with reference tc Figs. 1 - 27 , including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be incorporated as integrated circuits and VLSI's within any of the following components of Fig. 33 : imaging system 3510 , coder 3530 , decoder 3 ^ 0 . computer 358Ο . b. Any of t e image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27 , apart from the analog to digital conversion in preprocessing unit 12 , but including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be implemented in software and incorporated into software and DSP software of computer 3580 .

Reference is made to Fig. 34 which is a simplified block diagram of improved Karaoke entertainment system of televising, video recording, video editing, MPEG encoding, compact disk storage, compact disk playback on jukebox, decoding, MPEG decoding and TV display incorporating the apparatus for image smoothing, enhancing, interpolating and dynamic range compression shown and described herein. The apparatus of Fig. 3 ^ includes an imaging system 4010 , which televises a typical romantic and musical scene 4000 , records it on video cassette 4020 , and video edited on 4030 resulting in edited video clips 4 θ4θ . Selected clips are then MPEG encoded in 4070, and stored on CDs 4 θ8θ . The coded data is selectively read and MPEG decoded in 4 100 , for display 413O. , The image smoothing, enhancing, interpolating and 109 100589/3 dynamic range compression shown and described herein with reference to Figs. 1-27 may be incorporated into the apparatus of Fig. 34. Specifically, any of the image smoothing, enhancing, interpolating and dynamic range compression systems shown and described herein with reference to Figs. 1 - 27, including variations on the systems specifically shown and described which combine features of the various systems specifically shown and described, can be implemented in DSPs and incorporated as integrated circuits and VLSI ' s within any of the following components of Fig. 34: image noise reduction 4050, image contrast enhancement 4060, MPEG encoder 4070, MPEG decoder 4100, image noise reduction 4110, image contrast enhancement 4120, and TV display 4l3'0.

Appendices C-l to C-6, appended hereto, are software listings of six software implementations of six respective embodiments of the present invention. Appendices C-7 and C-8 are software listings of two alternative procedures for creating LUTs in accordance with a preferred embodiment of the present invention, which LUTs are accessed by the procedures of Appendices C-l .to C-6. The listings are appended hereto merely to provide an extremely detailed disclosure of the present invention. However, it is appreciated that, as described herein, the present invention need not be implemented in software.

The procedure of listing C-l is operative to perform two-directional noise reduction similar to that performed by unit 16 of Fig. 1 or by unit 21.6 of Fig. 14.

The procedure of listing C-2 is operative to perform 110 100589/3 three-dimensional noise reduction similar to that performed by unit 1 10 of Fig. 10 , wherein various multiplication operations are eliminated and are replaced by reference to suitable LUTs .

The procedure of listing C-3 is operative to perform pseudo two-dimensional noise reduction similar to that performed by the apparatus of Fig. .

The procedure of listing C-4 is operative to perform spatial and temporal noise reduction similar to that performed by the apparatus of Fig. 25 , in which the spatial noise reduction is at least three dimensional and the temporal noise reduction is one-dimensional .

The procedure of listing C- 5 is operative to perform three-dimensional noise reduction similar to that performed by unit 110 of Fig. 10 , wherein the multiplication operations eliminated in C- 2 are actually performed.

The procedure of listing C-6 is operative to perform 3-dimensional spatial noise reduction similar to that performed by unit 110 of Fig. 10 and to perform enhancement similar to that performed by unit 680 of Fig. 22 , wherein the noise reduction and enhancement mechanisms are embedded so as to achieve savings in computations. The embedding is implemented by adjusting the relative weights of previously estimated pixels in previously processed lines of the three-dimensional spatial noise reduction mechanism.

The software listings are written in the "C" programming language and operate under MS DOS Version 5 . 0 . The software listed herein may be run on any suitable conventional computerized image processing system such as an IBM AT 486 1 1 1 100589/3 equipped with: (a) a FG 100 iaage processing AT 1024 board, commercially available from Imaging Technology, 600 W. Cummings Park, Woburn, MA, 01801, USA; (b) a software package, incorporating utilities and a callable library known as IMTOOL.LIB, which is marketed as Imagetool Model 1000, No. 13050 71, commercially available from Werner Frei Associates, Santa Monica, CA, USA; and (c) a conventional TV display.

Prior to running any of software routines C-l to C-6, a W-. DAT file is created specifying the dimensions of the image. Also, a LUT or LUTs are created which are accessed by the routine.

To create the two LUTs required for each of procedures C-l and C-2, which LUTs are termed herein CREATLUT1.DAT and CREATLUT2.DAT, run the C-7 routine, using any suitable parameters such as sigman=4, rg = 0.8.

To run the C-7 routine, load the routine onto the equipment described above and enter the following instruction: CREATLUT CREATLUT1.DAT CREATLUT2.DAT To create the four LUTs (a- CREATLUT1.DAT and a CREAT-LUT2.DAT for each of the two dimensions in which the image is processed) required for each of procedures C-3 and C-4, run the C-7 routine twice. When first running the C-7 routine, use any suitable parameters such as sigman=4, rg = 0.8 to estimate the standard deviation of the noise and the autocorrelation, respectively, along the first dimension of the image. When second 112 100589/3 running the C-7 routine, use any suitable parameters such as sigman=4, rs = 0.8 to estimate the standard deviation of the noise and the autocorrelation, respectively, along the second dimension of the image. The parameters employed for the first and second dimensions of the image are preferably not equal if the noise and autocorrelation behave differently along the two dimensions .

To run the C~7 routine the first time, load the routine onto the equipment described above and enter the following instruction: CREATLUT CREATLUT1.DAT CREATLUT2.DAT To run the C-7 routine the second time, load the routine onto the equipment described above and enter the following instruction: CREATLUT CREATLUT3.DAT CREATLUT4.DAT To create the two LUTs required for C~5 or C-6, termed herein CRLUTM1.DAT and CRLUT 2.DAT, run the C-8 routine, using any suitable parameters such as sigman=4, rg = 0.8. To run the C-8 routine, load the routine onto the equipment described' above and enter the following instruction: CRLUTM CRLUTM1.DAT CRLUTM2.DAT To run any of software routines C-l to C-6, once the necessary LUTs and W.DAT file have been constructed, the routine may be loaded onto the equipment described above, linked to the IMT00L.LIB library function, and run, using the following instructions : For C-l: ADSM1D1 CRLUT1.DAT CRLUT2.DAT W.DAT ' For C-2: CREATLU-Tl CRLUT1.DAT CRLUT2.DAT W.DAT- 113 100589/3 For C- 3 : ADF RTEV CRLUT1.DAT CRLUT2.DAT CRLUT3.DAT CRLUT4.DAT W.DAT For C-4: ADSMT3 CRLUT1.DAT CRLUT2.DAT CRLUT3.DAT CRLUT^ . DAT W.DAT For C-5: ADS 1_M CRLUTM1.DAT W.DAT For C-6: ADSMEN_M CRLUTM1.DAT W.DAT Appendix D, appended hereto, describes applications in which the apparatus and methods described herein are useful. It is appreciated, however, that the apparatus and methods shown and described herein are useful in any situation hich involves image processing, so that the applications set forth in Appendix D are not intended to be limiting.

Appendix E provides details of a preferred VLSI implementation of reduction unit 670 of Fig, 21 and of a portion of enhancement unit 680 of Fig. 21.

The apparatus and methods shown and described herein are suitable for processing color images- represented in any suitable format, such as RGB, CMYK, YUV or other conventional color coordinate spaces .

Instead of processing each component separately, such as each of the R, G, and B components of an RGB-represented color image, it is sometimes preferable to employ the Y component of a corresponding YUV image to generate EGP's and other parameters and to employ these parameters, pixel by pixel, to process each of the R, G and B separations. 114 100589/3 It is appreciated that the various features of the various embodiments shown and described herein may be combined in any suitable manner so as to provide variations on the embodiments particularly shown and described herein.

It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention is defined only by the claims that ollow : 115

Claims (32)

100589/4 C L A I M S

1. Apparatus for one-directional time domain smoothing of a current image which was preceded by a sequence of images, the apparatus comprising: apparatus for computing a difference function between the raw value of an individual pixel of the current image and a smoothed preceding image value corresponding to at least one pixels of at least one preceding image; and apparatus for generating a smoothed pixel value for the individual pixel of the current image by computing a weighted sum of the smoothed preceding image value and of the raw value of the individual pixel of the current image wherein the weights are a function of said difference function and of at least one characteristic of at least a portion of the current image other than the difference between the individual pixel value and at least one pixels of at least one preceding image.

2. A method for one-directional time domain smoothing of a current image which was preceded by a sequence of images, the method comprising: computing a difference function between the raw value of an individual pixel of the current image and a smoothed preceding image value corresponding to at least one pixels of at least one preceding image; and generating a smoothed preceding image pixel value for the individual pixel of the current image by computing a weighted sum of the smoothed value and of the raw value of the individual pixel of the current image wherein the weights are a function of said difference function and of at least one characteristic of at least a portion of the current image other than the difference between the individual pixel value and at least one pixels of at least one preceding image.

3. A method according to claim 2 wherein said at least one characteristic comprises the location of boundaries of blocks employed for block transform image coding. 116 100589/4

4. A method according to claim 2 wherein said at least one characteristic comprises average of image brightness.

5. A method according to claim 2 and also comprising: storing the current image; and. generating a smoothed pixel value for at least one individual pixel in the current image based on: at least one smoothed pixel value of said current image; and at least a portion of an image following said current image in a temporal sequence.

6. A method according to claim 5 and. also comprising spatially smoothing the following image and wherein the smoothed current image pixel value is based on at least one spatially smoothed portion of said following image.

7. A method for acuity-preserving image smoothing comprising: proceeding along at least a portion of a first dimension of received image pixels in a first direction and recursively computing a first sequence of estimated pixel values from the received image pixels defined along the first direction; proceeding along at least a portion of the first dimension of received image pixels in a second direction and recursively computing a second sequence of estimated pixel values from the received image pixels defined along the second direction; and for at least one individual pixel along the first dimension, computing an improved estimated pixel value for the individual pixel based on a data dependent combination of at least estimated pixel values in the first and second sequences. 117 ψ 100589/4

8. A method according to claim 7 wherein, for an individual pixel in the first sequence, said proceeding and computing comprises: computing a difference value between the received image pixel and an adjacently preceding estimated pixel value in the first sequence; employing at least the recei ved image pixel and at least one preceding estimated pixel values in the first sequence to estimate a first directional signal to noise ratio; generating an adjusted difference value to reflect the first directional signal to noise ratio; and employing the adjusted difference value to update an adjacently preceding estimated pixel value, thereby to compute the estimated pixel value of the individual pixel.

9. A method according to claim 8 wherein the signal to noise ratio depends on locally weighted signal to noise ratio estimates of at least one pixel in the vicinity of the individual pixel.

10. A method according to claim 9 wherein the vicinity is included within a coding block boundary.

11. 1 1. A method according to claim 7 wherein said computing an estimated pixel value comprises: computing a difference value between the received image pixel and a function of at least adjacently preceding estimated pixel values in the first and second sequences; employing preceding estimated pixel values in the first and second sequences to estimate a two-directional signal to noise ratio; adjusting the difference value to reflect the signal to noise ratio; and 118 100589/4 employing the adjusted difference value to update a function of at least adjacently preceding estimated pixel values in the first and second, sequences, thereby to compute the improved estimated pixel value of the individual pixel.

12. A method according to claim 11 wherein the signal to noise ratio depends on locally weighted signal to noise ratio estimates of at least one pixel in the vicinity of the individual pixel.

13. A method according to claim 7 wherein the received image pixels comprise estimated pixel values and each individual estimated pixel value is computed by combining at least three previous estimated pixel values arranged along three respective dimensions relative to the individual estimated pixel value.

14. A method according to claim 7 and also comprising: repeating, for at least a second dimension of received image pixels, said proceeding in first and second directions and said computing an improved estimated pixel value, thereby to compute at least one additional improved estimated pixel value for each individual pixel; and combining the at least two improved estimated pixel values, thereby to obtain a further improved at least two-dimensional estimated pixel value.

15. A method according to claim 7 and also comprising: performing the proceeding and the improved estimated pixel value computation at least once more, using a second dimension as said first dimension.

16. A method according to claim 15 and also comprising: 119 100589/4 adjusting outputs of the proceeding using the second dimension and of the improved estimated pixel value computation using the second dimension in order to reflect the difference between the received image and outputs of the proceeding using the first dimension and of the improved estimated pixel value computation using the first dimension.

17. A method according to claim 7 wherein the received image defines a second dimension thereof and a scanning direction in which the image is received along the second dimension, the method also comprising: proceeding along at least a portion of the second dimension of received image pixels in the scanned direction and computing a sequence of second dimension estimated pixel values from the improved estimated pixel values of the first dimension; and for each individual pixel along the second dimension, comparing an adjacently preceding second dimension estimated pixel value in the sequence of second dimension estimated pixel values and an improved estimated pixel value of the first dimension which adjacently proceeds the individual pixel along the second dimension, thereby to compute a further improved estimated pixel value for the individual pixel.

18. A method according to claim 17 and also comprising: adjusting outputs of the proceeding using the second dimension and of the improved estimated pixel value computation using the second dimension in order to reflect the difference between the received image and outputs of the proceeding using the first dimension and of the improved estimated pixel value computation using the first dimension.

19. 1 . A method according to claim 7 and also comprising: 120 100589/4 adjusting outputs of each estimated pixel value computation to preserve narrow elongated contours.

20. A method according to claim 7 wherein said first dimension comprises a temporal dimension.

21. A method according to claim 7 wherein said first dimension comprises a spatial dimension.

22. A method according to claim 7 and also comprising imaging an image using an electronic imager, thereby to define the received image pixels.

23. A method according to claim 7 and also comprising scanning an image using an electronic scanner, thereby to define the received image pixels.

24. A method according to claim 7 and also comprising receiving said received image pixels from a video system.

25. A method according to claim 7 wherein recursively computing the first sequence comprises computing at least one data dependent combination of received image pixels.

26. A method according to claim 25 wherein recursively computing the second sequence comprises computing at least one data dependent combination of received image pixels.

27. A method according to claim 7 wherein said data dependent combination comprises an image-dependent weighted sum in which the weights depend on at least a portion of the image. 121 100589/4

28. A method according to claim 7 wherein said data on which said combination depends comprises the location of boundaries of blocks employed for block transform image coding.

29. A method according to claim 7 wherein said data on which said combination depends comprises a local average of image brightness.

30. Apparatus for acuity-preserving image smoothing comprising: a first-direction recursive pixel value estimator operative to proceed along at least a portion of a first dimension of received image pixels in a first direction and to recursively compute a first sequence of estimated pixel values from the received image pixels defined along the first direction; a second-direction recursive pixel value estimator operative to proceed along at least a portion of the first dimension of received image pixels in a second direction and to recursively compute a second sequence of estimated pixel values from the received image pixels defined along the second direction; and a data dependent pixel value estimator operative, for at least one individual pixel along the first dimension, to compute an improved estimated pixel value for the individual pixel based on a data dependent combination of at least estimated pixel values in the first and second sequences.

31. Apparatus according to claim 30 wherein, for an individual pixel in the first sequence, said first-direction pixel value estimator comprises: a estimate-received value difference computer operative to compute a difference value between the received image pixel and an adjacently preceding estimated pixel value in the first sequence; 122 100589/4 a SN ratio estimator operative to employ at least the received image pixel and at least one preceding estimated pixel values in the first sequence to estimate a first directional signal to noise ratio; a difference adjuster operative to generate an adjusted difference value to reflect the first directional signal to noise ratio; and a pixel value estimator operative to employ the adjusted difference value to update an adjacently preceding estimated pixel value, thereby to compute the estimated pixel value of the individual pixel.

32. Apparatus according to claim 30 wherein said data dependent pixel value estimator comprises: an estimate-received value difference computer operative to compute a difference value between the received image pixel and a function of at least adjacently preceding estimated pixel values in the first and second sequences; an SNR ratio estimator operative to employ preceding estimated pixel values in the first and second sequences to estimate a two-directional signal to noise ratio; a difference adjuster operative to adjust the difference value to reflect the signal to noise ratio; and a pixel value estimator operative to employ the adjusted difference value to update a function of at least adjacently preceding estimated pixel values in the first and second sequences, thereby to compute the improved estimated pixel value of the individual pixel. C: 13870 123