EP1535239A2 - Traitement d'images utilisant des hypotheses probabilistes sur un comportement localise - Google Patents

Traitement d'images utilisant des hypotheses probabilistes sur un comportement localise

Info

Publication number
EP1535239A2
EP1535239A2 EP03730463A EP03730463A EP1535239A2 EP 1535239 A2 EP1535239 A2 EP 1535239A2 EP 03730463 A EP03730463 A EP 03730463A EP 03730463 A EP03730463 A EP 03730463A EP 1535239 A2 EP1535239 A2 EP 1535239A2
Authority
EP
European Patent Office
Prior art keywords
image
low pass
component
components
level
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP03730463A
Other languages
German (de)
English (en)
Inventor
Ran Boneh
Yotam Toib
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Quix Technologies Ltd
Original Assignee
Quix Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Quix Technologies Ltd filed Critical Quix Technologies Ltd
Publication of EP1535239A2 publication Critical patent/EP1535239A2/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/20—Contour coding, e.g. using detection of edges
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/1883—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit relating to sub-band structure, e.g. hierarchical level, directional tree, e.g. low-high [LH], high-low [HL], high-high [HH]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/34—Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
    • H04N19/619—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding the transform being operated outside the prediction loop
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/63—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using sub-band based transform, e.g. wavelets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/63—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using sub-band based transform, e.g. wavelets
    • H04N19/635—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using sub-band based transform, e.g. wavelets characterised by filter definition or implementation details
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation

Definitions

  • the present invention relates to image and video processing and, more particularly, but not exclusively to image and video processing for upsampling, compression, decompression, motion detection and like applications, the processing being based on assumptions about localized behavior of the image data in the probabilistic sense.
  • Video and image compression reduce the amount of data that needs to be stored or transferred and therefore are instrumental in making feasible numerous digital media applications such as digital TV, streaming over the Internet, digital cameras for personal use and for surveillance, and video for cellular phones.
  • the theoretical topics involved in video and image compression include information theory, signal processing and psycho-visual aspects of human vision.
  • Codec scalability refers to the ability to adapt to variable bit-rates, say for varying levels of available bandwidth.
  • a connection say over the Internet may vary over time in the amount of available bandwidth, and a single image or video source file encoded using a scalable codec can be streamed and played-back at different bit- rates for different end devices or over different connections.
  • the ability to provide codec scalability today is limited.
  • Image compression generally works by exploiting redundancy within the image and video compression generally exploits both inter and intra-frame redundancy.
  • DCT based applications to which most current commercial compression products belong, for example MPEG 1, 2, & 4; H263, L, 4; Microsoft, and RealNetworks, etc
  • Wavelets based applications for example JPEG2000, and
  • DCT based algorithms are most common today and include the industry leaders in the sense of quality per bit rate (QBR). Nevertheless they are far from being a sufficient solution for the crucial bandwidth and storage issues with which the industry is faced.
  • DCT based algorithms lack a natural solution for other important properties such as spatial and temporal scalability and editing capabilities. It is also believed that DCT is at the end of its improvement curve, so that dramatic improvement may only be expected from non-DCT solutions.
  • Wavelet based applications claim the potential to be comparable to DCT solutions in terms of QBR and also to be scalable. But the potential is not yet implemented in video, mainly because of the shift invariance problem, which is discussed in greater detail below.
  • a device and technique optimized for all of the above requirements may be expected to reduce the infrastructure requirement for storage and transport of multiple bit-rate files or expensive trans-rating per channel, and provide best streaming quality per available bandwidth and end device limitations.
  • image transform apparatus for processing pixel image data,' comprising: at least one low pass extractor for extracting a low pass component from a series of image pixels, at least one low pass downsampler associated with the at least one low pass extractor for producing downsampled versions of respective extracted low pass components, at least one high pass extractor for extracting a high pass component from a series of image pixels, at least one high pass downsampler associated with the at least one high pass extractor for producing downsampled versions of respective extracted low pass components, and control and iteration circuitry associated with the low pass extractor and the high pass extractor for: feeding complementary sets of pixels from an image to the respective extractors, receiving respective low and high pass components from respective high and low pass downsan plers, and iteratively feeding complementary sets of pixels from a downsampled version of a most recently formed low pass component to respective high and low pass extractors and receiving further respective low and high pass components until a stop condition is reached.
  • the apparatus preferably comprises data storage circuitry for storing each high pass component and a final low pass component.
  • the apparatus preferably comprises a stop condition input for setting the stop condition.
  • the stop condition input is associated with a user interface.
  • the stop condition input is operable to receive data regarding at least one of image importance, image part importance, and available bandwidth, and to use the data to set the stop condition dynamically.
  • the apparatus preferably comprises video frame processing circuitry for providing the pixel image data from a video frame.
  • the frame is one of a series of frames, the apparatus being operable to obtain the high and low pass components for each frame in the series and further comprising frame selection circuitry to select video frames for feeding to a compressor to compress the frames as a video stream , thereby to produce a video stream of a required frame rate.
  • the pixel image data and the stop condition are set such that the processing comprises lossless image compression.
  • the stop condition is variable such that the processing provides different levels of lossy compression to the pixel image data.
  • image transform apparatus for recovering an original image from a series of successively obtained high pass components and a final low pass component, the apparatus comprising: an estimator comprising: a component combiner for combining a current level high pass component and a current low pass component to form a combination, an upsample guess unit for estimating an upsampled version of the combination and iteratively correcting the version to converge on a next level image, the upsampler comprising an edge enhancement unit configured with a step function to scan the upsampled image, the step function being able to identify and enhance step like local behavior over the image, thereby to enhance resolution of the upsampled image, the image providing a next level low pass component, and a control and iteration unit for iteratively feeding to the estimator a next one of the successively obtained high pass components and a respective next level low pass component until a
  • the upsample guess unit is configured to use the high pass components to identify the step-like local behavior, thereby to carry out the enhancing.
  • the step function is a non-linear step function.
  • the step function comprises an edge enhancement function.
  • control and iteration unit further comprises upsampling functionality for controlling the estimator to operate the edge unit on the zeroth level low pass component to provide an enhancement thereof.
  • the upsampling functionality is usable on an original image to provide an enhanced resolution version thereof.
  • a method of transforming a pixilated image comprising: levelwise iterative stages of: extracting, downsampling and retaining high frequency information components, and generating low frequency components for extraction therefrom of further high frequency components, and downsampling the low frequency components, for a next level of the iteration, until a stop condition is reached, and retaining a last one of the low frequency information components.
  • a method of reconstructing an image from retained high frequency components of successive downsamplings and a low frequency component corresponding to a last one of the successively retained high frequency components comprising: combining the low frequency component with its corresponding high frequency component to form an initial combination, iteratively enhancing the combination by enhancing step-like local behavior over the combination to form an enhanced combination, using the enhanced combination together with a next one of the high frequency components to form a new initial combination, repeating the steps of combining, and iteratively enhancing to form a new enhanced combination, and further forming new enhanced combination until all of the high frequency components are used, and outputting a final enhanced combination as a reconstructed image.
  • an image enhancer for improving resolution of a pixilated image comprising: an upsampler for upsampling the image, and an edge enhancer configured with a step function to scan the upsampled image, the step function being able to identify and enhance step like local behavior over the image, thereby to enhance resolution of the upsampled image.
  • an image storage apparatus for reducing the amount of storage capacity needed for storage of a given pixilated image, the apparatus comprising: a downsampler for downsampling an image, a low pass extractor associated with the downsampler for extracting a low pass component of the downsampled image, the low pass component being for storage, an upsampler for upsampling the low pass component and an enhancer configured with a step function to scan the upsampled image, the step function being able to identify and enhance step like local behavior over the image, thereby to restore the pixilated image.
  • FIG. 1 is a simplified block diagram illustrating apparatus for performing a transform on a pixel-based image according to a first preferred embodiment of the present invention
  • FIG. 2 is a simplified block diagram illustrating apparatus for performing the inverse of the transform of Fig. 1 in order to retrieve the original image
  • FIG. 3 is a flow chart illustrating a simplified procedure for carrying out the transform of Fig. 1;
  • FIG. 4 is a schematic diagram illustrating a block of four pixels and indicating how the pixels may be encoded as symbols in between the pixels, each symbol containing data for each of its surrounding pixels;
  • FIG. 5 is a simplified diagram illustrating a possible set of four filters forming together a quadrat of the transform of the present invention.
  • the quadrat support may be larger then the 2x2 support described in the figure;
  • FIG. 6 is a simplified flow chart illustrating the inverse transform for use with the apparatus of Fig. 2;
  • FIG. 7 is a simplified flow chart illustrating the upsampling guess stage of Fig. 6 for a single level
  • FIG. 8 is a simplified diagram illustrating the reverse transform in the case where range quantization has been used, and showing how dequantization is inco ⁇ orated into the body of the reverse transform;
  • FIG. 9 is a simplified flow chart illustrating edge estimation for the reverse transform procedure of Fig. 6;
  • FIG. 10 is a simplified diagram illustrating the pull and push notions and a series of pixel arrangements for push that are examples of isotropic- variance (IV) and non-isotropic variance operators;
  • FIG. 11 shows screen pixel and symbol arrangements during the transform according to preferred embodiments of the present invention.
  • FIG. 12 is a simplified diagram illustrating a compression-decompression scheme based on the above transform and inverse transform, and showing how the transform leaves freedom to set the scalability;
  • FIG. 13 is a simplified flow chart illustrating a procedure for upsampling, for example to improve the resolution, of an existing image according to a preferred embodiment of the present invention
  • FIG. 14A is a simplified flow chart illustrating a procedure according to one preferred embodiment of the present invention for encoding and subsequently decoding a video stream
  • FIG. 14B is a simplified block diagram illustrating apparatus for carrying out encoding of successive frames of a video stream according to a preferred embodiment of the present invention
  • FIG. 14C is a simplified block diagram of apparatus for carrying out decoding of a video frame encoded by the apparatus of FIG. 14A;
  • FIG. 15 is a simplified flow diagram illustrating a generalized method of carrying out scalability, that is the transform of the above embodiments with variable levels of precision;
  • FIG. 16A is simplified flow diagram illustrating a first scalability method comprising using the transform of the above embodiments with variable levels of quantization
  • FIG. 16B is a simplified flow diagram illustrating another scalability method involving resolution levels
  • FIG. 17 is a simplified flow diagram illustrating a method of using the transform of the present embodiments with adjustable frame rates
  • FIG. 18 is a simplified diagram illustrating apparatus for carrying out the transform of the present embodiments as an enhancement layer to existing image or video standards.
  • FIG.19 is a simplified flow diagram illustrating a procedure for storing data according to a preferred embodiment of the present invention.
  • FIG.20 is a simplified flow diagram illustrating a storage procedure according to a further preferred embodiment of the present invention.
  • FIG. 21 is a simplified flow diagram illustrating a procedure according to a preferred embodiment of the present invention for enhancing image precision to more closely match screen precision.
  • FIG. 22 is a simplified diagram illustrating the true-construct relation. DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • the present embodiments show a method of image processing which is based on an understanding of pixel behavior within an image known herein as the image assumption.
  • the image assumption states that an image is distinguished from any arbitrary assignment of values to pixels by demonstrating step-like behavior in the probabilistic sense. Such step-like behavior could in principle be used to confirm that data being investigated is indeed an image, or as in the present embodiments, can be used as an underlying assumption about the data which can be used within a data transform procedure that supports compression.
  • the present invention describes a transform that processes the entropy of a given image and thus provides an especially suitable input for any one of many known compression processes.
  • the transform provides a set of ready processed input data that can be selected from to provide compression at different data rates.
  • the transform is not in itself a compression process.
  • a transform is carried out in which an image GO is decomposed into a low-pass component LI, and a high pass component To, which may be referred to as the texture.
  • the two components are downsampled and the high pass component is saved.
  • the process may be repeated any desired number of times, each time saving the texture component and using the low pass component for the next level.
  • the low pass component is saved.
  • the original image is thus transformed into a multi-level pyramid of a texture component for each level and a low pass component for the final level.
  • the reverse transform is likewise an iterative process and is discussed hereinbelow.
  • the transformed data is generally obtained up to a certain predetermined precision s, which is selected in accordance with the need to quantize, encode and decode. Given the single-level-map
  • the present embodiments provide a method for computing a pseudo-inverse operator for the map, which depends continuously on the precision s, and is optimal in the sense of the image assumption described above.
  • the preferred embodiments provide an exact inverse operator.
  • the embodiments provide an optimal upsampling operator, as will be discussed in detail hereinbelow.
  • T transforms G into n texture images with sizes S/2, S/4, S/8, .. , S/2 n and one Lowpass image of size S/2 n .
  • TO, Tl, T2,.., Tn-l,Ln may be constrained under a set of conditions, examples of which are listed hereinbelow.
  • the reverse transform T "1 works iteratively by reconstruction of the n-level image from the n+1 level lowpass component and n level texture component.
  • the n- level image is then used as the low pass component for the next level together with the saved texture component corresponding to that level.
  • each iteration uses a prediction step that is based on the above-described image assumption.
  • the prediction step may use linear or non-linear guesses or a combination of both. It may be seen as upsampling the lowpass level and improve the upsampling guess using hints taken from the texture component.
  • the prediction step is thus is referred to hereinbelow as the improved upsampling guess or Ups++.
  • the prediction step is itself part of an iterative sub-process in which the predicted image is decomposed into lowpass and texture sub-images that are compared to the original lowpass and texture images.
  • the sub-process continues additively with the difference images until a stopping condition is reached, as will be explained below.
  • the data presented to the reverse transform is quantized.
  • the reverse transform or data reconstruction process may incorporate a range dequantization technique into the procedure, in which the same image assumption is used to recover detail lost in the quantization process.
  • the process is continued using the image assumption to make guesses about the original image.
  • the process is continued using the image assumption to make guesses about the original image.
  • the transform T and its inverse provide a scalable image compression-decompression technique which can be applied to still image data or to video image data.
  • the variable parameters used in the technique and the nature of the image combine to provide lossless compression in certain cases and lossy compression with a wide range of possible compression levels in image data in general.
  • Figure 1 is a simplified diagram which illustrates image transform apparatus for processing pixel image data according to a first preferred embodiment of the present invention.
  • the apparatus of Fig. 1 illustrates an iterative device using a single high pass extractor and a single low pass extractor.
  • Image transform apparatus 10 comprises an input 11 for receiving an input image, , a low pass extractor 12 for extracting a low pass component from a series of image pixels, downsampler 13 for downsampling the low pass component received from the low pass extractor 12 by a factor of two a high pass extractor 14 for extracting a high pass component from a series of image pixels, a downsampler 16 connected at the output of the high pass extractor 14 for downsampling the high pass component, and control and iteration circuitry 16.
  • control and iteration circuitry 16 is first of all to feed complementary sets of pixels from the original image or the downsampler to the low pass extractor 12 and high pass extractor 14, and to feed the downsampled low pass image at each stage to the extractors for the next stage.
  • the respective extractors then extract the low and high pass components from the respective sets.
  • the downsampled high pass components are sent directly to output 18, whereas the low frequency components are passed back as described above, for use in place of the original image in a further iteration of the extraction process.
  • Complementary sets of pixels from a downsampling of the low pass components of the previous stage are then sent to the extractors and the process is repeated to generate new high and low pass components.
  • the number of iterations can be set by the user or by the software as desired.
  • the original image is reduced to output content 20 which comprises a series of high pass components, one for each stage.
  • the final low pass component is also retained.
  • the stop condition can be set in accordance with such factors as importance of the image itself or perhaps in accordance with detail within the image that it is desired to preserve.
  • the stop condition sets the number of layers in the transform, which is not the same as the extent of compression, the number of layers in the transform is significant.
  • Such is particularly useful in applications such as image compression.
  • software may identify a detailed part of the image.
  • software may be preset to identify a central part of the image as the most important part.
  • the apparatus then retains greater precision in these more important regions and carries out a higher level of compression in other regions.
  • Differential compression levels of parts of images are particularly useful in applications involving compression of moving image data, and, as will be explained below, the transform of the present embodiments retains the freedom to apply scalability to the compression.
  • Available bandwidth for image transfer may also be measured and used to set the stop condition.
  • the apparatus is useful for real time image compression prior to transfer of the image over a communication infrastructure such as the Internet.
  • video streams at a plurality of frame rates may be available.
  • a subsequent compression stage of the apparatus may comprise frame selection circuitry associated with available bandwidth detection circuitry 24, and the selection circuitry chooses between streams to produce a variable rate data stream which may thus comprise a low frame rate at low available bandwidth and a high frame rate at high available bandwidth, thus in each case making most effective use of the connection.
  • certain pixel image data can be classified as well-behaved under the image assumption.
  • the entire step behavior of the image can be reconstructed from the low and high pass components by application of the inverse of the above-described transform.
  • the apparatus together with an appropriate stop condition provides lossless image compression. In most cases however the stop condition is varied as desired.
  • Fig. 2 is a simplified diagram illustrating image transform apparatus for recovering an original image from a series of successively obtained high pass components and a final low pass component, such as may be provided by the output of the apparatus of Fig. 1.
  • the principle behind the apparatus of Fig. 2 is that the tendency of an image towards localized step or edge behavior allows us to recover missing information from the image components that are available simply by reinforcing the edge behavior that can be detected.
  • the apparatus 30 comprises an input 32 for receiving the high pass and low pass component data 20, and an estimator 34 for carrying out estimation based on the image assumption to recover the original image.
  • the estimator 34 comprises a component combiner 36 for combining a current level high pass component and a current low pass component into an image, and which also maps the pixels onto a next level pixel and an edge unit 38 which operates on the image from the combiner to fill in the spaces in the pixel space.
  • the edge unit uses a step function to determine and enhance steps or edges indicated in the image.
  • Any suitable step function may be used, but in a preferred embodiment a non-linear step function is used which operates on a given pixel by measuring differences between the surrounding pixel intensity values and using the changes in intensity vertically and horizontally around the given pixel to apply a new value to the given pixel.
  • the non-linear step function is described in greater detail hereinbelow.
  • the edge unit operates iteratively, as will be described in greater detail hereinbelow.
  • the estimator as a whole also operates iteratively and the edge enhanced output of each iteration is used together with the next available high frequency component to carry out a further combining and enhancing stage until all of the available high frequency components have been used up.
  • the final image should be a reasonable reconstruction of the original.
  • a further stage of image enhancement can be carried out on the final image by upsampling and then placing the upsampled image in the edge unit, this time without a high frequency component as there is none available.
  • the step function is then used to enhance the final image in the same way as before to provide a final result which, provided there is edge behavior to enhance, is of higher resolution than the original.
  • Upsampling may also be carried out directly on original images on which no transform has been carried out, in order to increase image resolution. The latter may desirable to provide a zoom feature or to take full advantage of a high resolution display with a lower resolution image.
  • Apparatus 30 preferably comprises a control and iteration unit 40 for iteratively feeding the correct high and low pass components to the estimator at each stage until a zeroth level low pass component is reached, or in the case of upsampling until the desired improved resolution image is produced.
  • Fig. 3 is a simplified diagram illustrating the operation of the apparatus of Fig. 1.
  • an initial image Go is obtained in the form of an array of pixels.
  • One set of pixels from the image is directed to a low frequency branch and another, complementary, set of pixels is directed to a high frequency branch.
  • the sets of pixels are each filtered in their respective branches and then downsampled.
  • the downsampled high frequency component is saved.
  • the low frequency component is used as the image in a new iteration. When the desired number of iterations is reached then the final low frequency component is also saved.
  • the low frequency component is then further encoded followed by the preserved high frequency components in the order of high to low level.
  • Such an order of encoding is particularly useful in entropy encoding and like schemes.
  • Fig. 4 shows a group of four neighboring pixels XI ...X4 arranged in a block of 2 x 2.
  • a symbol S At the center of the block is a symbol S.
  • the symbol S can be derived from values of the surrounding pixels in a number of ways, and the formation of the value S is a way of filtering the pixels.
  • V (xl - x2 + x3 - x4) / 4
  • L, H, V, and T are mutually orthogonal.
  • L, H, V, and T are known as quadrat decomposition filters. It will be appreciated that the filters L, H, V, and T are mere examples, and numerous other sets of orthogonal filters could be used. In particular there is no reason to be limited to operations on 2x2 blocks.
  • each pixel can be recovered from the four surrounding quadrats according to the following formula:
  • G2Q the operator that decomposes a given image G into quadrat Q
  • Q2G may be defined as the operator that reconstructs G from Q.
  • a given image G can be seen as a chessboard of pixels.
  • Gd half of the pixels of G
  • Gs complementary half
  • step 2 we take the result over the whole image of the low pass filter L and downsample it, by half
  • G is transformed into four texture (high pass) images with sizes S/2, S/4, S/8, S/16 and one Lowpass image of size S/16 where S is the size of original image G.
  • the zebra space is defined by span (Zh, Zv), that is, vectors of the form a ' Zh + b ' Zv. The vectors are as shown in the following figure.
  • the reconstruction quality varies from upsampling quality (quantization step equals infinite) to full reconstruction (no quantization - quantization step equals zero). That is, the degree of improvement of the upsampling quality depends on the texture granularity, in a continuous and monotonically decreasing fashion. This property translates to fine granularity scalability in video and image encoder/decoder, as will be discussed hereinbelow.
  • Fig. 6 is a simplified flow chart illustrating a preferred embodiment of the reverse transform.
  • the reverse transform begins with the material saved in the transform process of Fig. 3, namely the low pass image at level n and high pass images for each of the levels from 0 to n.
  • the process begins by setting the low pass image at level n as the given Gn+1 at level n+1, and proceeds as follows: - Guess a quadrat Qn using Lsn and Tdn
  • the guessing stage is discussed in greater detail below, but in general terms it works by trying to locate edges of Gn from the information given by Gn+1.
  • Such a process is based on the image assumption, which may be stated as the assumption that an image posses step-like local behavior in a probabilistic sense. While such an image assumption may not be true for every possible image, it enables good and reliable guesses for typical images.
  • Tdn represents the type of data that is not represented by the lowpass filter.
  • a convenient way to describe or carry out upsampling, that is improving the resolution of an image, not necessarily as part of the reverse transform, is through the use of quadrats, although such a method is not optimized in terms of performance.
  • the benefits of describing upsampling using quadrats include a unified basis for linear and non-linear guesses, which will be described below, and providing a unified basis for the two features of upsampling and decompression.
  • the non-linear guess technique of the preferred embodiments is based on the use of quadrats
  • linear guesses do not necessarily demand such filters and in practice may not use any of the four kinds of filter or any other kind of quadrat.
  • an effective embodiment can be made in which linear guesses do involve guessing quadrats, resulting in a more unified system.
  • Fig. 7 is a simplified flow chart illustrating upsampling using guessing of quadrats.
  • upsampling replaces expression (1) from the procedure described in respect of Fig. 6, in which the loop test comparison was made between Gn+1* and Gn+1. In practice the loop runs for a pre-determined number of iterations as was discussed above. Upsampling is different from image reconstruction in that the guess is made on the basis of the lower level without any independent high frequency component. Thus upsampling can be used on an original image to improve resolution based on the image assumption or may be used as an extra stage after image decompression to compensate for detail lost in the compression-decompression process by providing a further increase in resolution based on the infonnation that is present. Upsampling may also be used independently to improve the resolution of an image, as will be described hereinbelow.
  • Image decompression using quadrats is carried out in a manner similar to the above upsampling routine as will be shown in the next section, with the difference being that the available high frequency component is used to improve the upsampling guess.
  • Compression and Decompression Compression is achieved following the transform of Fig. 3, with the steps of quantizing and entropy encoding the texture or high frequency image components and the n th level lowpass image component.
  • Entropy coding and decoding are especially applicable to video encoding where a stream of related frames are involved.
  • Entropy coding the encoding of a fragment is conditioned by the inter-frame, inter-level and intra frame neighborhood of the fragment.
  • Fig. 8 is a simplified flow chart illustrating the combined process of range dequantization according to a preferred embodiment of the present invention.
  • the concept of range dequantization is based on the non-orthogonality of data elements, and take advantage of the image assumption.
  • Range dequantization suggests a way of interpretation of quantized information.
  • a bit-stream is decoded into quantization symbols that represent ranges of values and are dequantized into representative values within the range.
  • quantized information is interpreted as a range rather than a value within this range.
  • the reconstruction process which predicts G iteratively from Ls and Td, decomposes the predicted result into Td* and Ls*, and continues with the difference images, comparing Td and Td* up to a range. This results in the ability of the improved upsampling guess to set any Td* value within the dequantization range.
  • a quantizer As the skilled person will appreciate, the role of a quantizer is to represent a broad range of values by a small group of symbols, and detail is lost in the quantization process.
  • the dequantizer then operates to map symbols to certain values on the original range, which may or may not coincide with the original image before quantization.
  • the image assumption based predictor advises the dequantization process as follows.
  • Range dequantization may be carried out in the following way:
  • Td values in the Td components are set to dequantization values D (q)
  • G* is predicted from Ls and Td and decomposed into Ls* and Td*
  • a series of modification iterations are then carried out in which Td is compared to Td* modulo range, that is, if Tdn* and Tdn are in the same range, the process continues as usual, otherwise the process continues with what is known as the
  • PIR function a function which is the coincides with the identity function across the range, but for an input outside the range provides an output on the closest boundary.
  • Td values are set to zero.
  • G* is predicted by Ls and Td and decomposed into Ls* and Td*
  • the dequantization phase and the reverse transform phase are not separated.
  • the dequantization itself is done to a range rather than to a representative value. At the end of the process, the value within the range may be determined by the predictor.
  • Tdn D (q) (say middle of quantization range) Guess a quadrat Qn out of Lsn and Tdn
  • a consistent element in the inverse transform process is the formation of the guess.
  • An example of a simple linear guess of a quadrat for upsampling involves a central pixel X4 surrounded by pixels X0 - X3 and X5 - X8:
  • Tdn can be guessed as zero or any other educated guess may be made. If Tdn is guessed as zero it may be abandoned in the calculation for performance reasons. If the range dequantization approach is taken, Tdn is NOT compared and not rectified per iteration, as any value obtained during the process complies with the infinite range, and hence any value is legitimate and there is no need or justification for correction. in the following, the object is to make a non-linear guess of the quadrat of S symbol X4. A preferred embodiment of a non-linear guess procedure is now described in greater detail with reference to the flow chart of Fig. 9.
  • NLG NLG which operates on an image to emphasize edge behavior.
  • V NLG (L3-M, L5-M) + NLG ((T0-T6)/2, (T2-T8)/2)
  • V NLG ((L0+L6V2-M, (L2+L8)/2 -M) + NLG (T7, Tl)
  • the transform preferably has the following characteristics both from the push and the pull aspects, the concepts of pull and push being discussed below:
  • N - Normalization — filter weights are summed to 1, for both pull and push.
  • C - Centralization - center of mass of the operator is in the pushed symbol/pulled pixel, for both pull and push.
  • the downsampling map D G0->G1 which is a part of the proposed transform, should fulfill a set of constraints.
  • the map can be represented as a combination of low-pass filtering and grid dilution by a factor of 2, known as the "PULL" representation.
  • the pull representation requires:
  • Isotropic Variance The variance of each push filter should be equal for any possible 2d direction.
  • the quantized material defined by the transform is of homogenous type (T or
  • HH in Wavelets terminology as opposed to the 3 types of quantized material used in separable Wavelets (HL, LH, HH).
  • non-compensation property and the narrowness of the downsample filter enable accurate pixel compression, that is, compressing arbitrary shaped regions at different compression qualities.
  • Quadrat manipulation and special features can be added by performing manipulations (multiply, addition, Gamma correction, etc) on the quadrat members during the guessing process. For example, sharpening can be implemented by factoring the high-pass members of the quadrat (H, V, T) by a factor > 1, while contrasting can be implemented by factoring L. Such manipulations can be achieved with negligible toll on CPU performance.
  • N (x, y, z, w) N (x,0,0,0).
  • Nr andNa A preferred mode for 4-level upsampling uses a combination of non-linear guesses.
  • S is a user set sharpening parameter having a default value of 1.
  • S is a user set sharpening parameter whose default value is 1.
  • Scalability Scalability is the result of the multi-resolution nature of the transform.
  • Each level is divided into four sectors: DO, Dl, SO, SI as indicated in the tables referred to.
  • bit band It is possible to compress/decompress each sector independently, which enables refined sector granularity scalability. In each sector it is possible to operate separately on each bit band. It is appreciated that the role of the bit band depends on the actual quantization method used if any, and a common method is to let bit 0 determine whether the quantized pixel value is 0 or not, bit 1 determines its sign and the following bits locate its value.
  • the decompression process may be stopped after each bit band and the resultant quality will then be uniform over the entire image.
  • the decompressed bits may be further operated on using the upsample improvement referred to above, and when the decompressor has used up all available bits the upsampler then takes over alone.
  • Quantization may be organized, during the initial compression, according to bit importance as referred to above. In such case the scalability is pre-tuned. Another way of providing scalability is to traverse the image sector by sector, and such a method is particularly beneficial when using entropy coding, for example for video streams.
  • scalability is only one way to deal with differential importance levels within the image data.
  • the encoding process may dynamically alter quantization levels, allocating more bits to more important sets, just as higher priority levels may be set in the scalability order. Dynamic allocation in this way is well known for image encoding systems which rely on object detection technology. In other applications, it is known to concentrate more on the center of the image, on the assumption that the more important aspects of the image are located there. Indeed in video it is quite common that the center of the image is the only part that is completely in focus. The importance of certain parts of the data may also be derived from considerations of the algorithmic past, motion information, user preference or any other source.
  • Fig. 11 is a schematic diagram illustrating the relationship between the pixels and symbols at the different processing levels.
  • the grid in the series shows the following:
  • L0 indicates pixels at level 0
  • SL1 indicates symbols at level 1
  • L2 indicates pixels at level 2.
  • the image boundary problem is avoided by treating an image as having the topology of a toroid, i.e. being cyclically continuous in the x-axis and in the y-axis.
  • mapping M G * ⁇ Ls, Td ⁇ , where
  • D,
  • D/2.
  • the kernel dimension represents the number of independent vectors that are zeroed under M.
  • the reverse transform reconstructs quadrats that are true-construct and agree on the input Lsn and Tdn.
  • this was done by iteratively operating the functions G2Q and Q2G.
  • the quadrat space is 4 times larger then the true-construct quadrat space suggest that true-construct quadrats are constrained under certain true-construction equations.
  • the 2-level operator is the result of multiplying a level-0 operator by a level- 1 operator as shown in the following table:
  • the resulting 1 -dimension 2-level operator obtained by summing along the multiplied operator columns, is the following vector:
  • the transform of the present embodiments is reversible up to zebra space.
  • the analysis set out hereinabove is based on the toroidal solution for the image margin problem. That is, as mentioned elsewhere herein the edge pixels are considered as continuations of the opposite margin.
  • the 3 pixels line (1,5,8) is extended into the 7 pixels line (5,8,1,5,8,1,5).
  • the compression works by partitioning a given image into Ls-swiss and Td- direct, saving the Td"s and continue with the Ls-swiss.
  • the reconstruction makes use of the fact that the input image is zero-margined.
  • the right-down corner symbol is supported by four pixels of which three are zero.
  • the right-down corner pixel extracted from the symbol is simply the original right-down pixel.
  • the next symbol to the left represents a difference between two pixels of which one was already extracted and so on.
  • a simple solution to the problem is to quantize the input image (possibly with error diffusion) so that there is no need for quantization, and hence no quantization errors, of the pyramid components.
  • Such a solution demands that filters are calculated without dividing the symbol values.
  • the above solution also demands lossless compression of the final low pass component, hereinafter "gbase”.
  • the above-described LT-diffusion can work for a single level but demands lossless compression of higher levels and the base level.
  • Fig. 12 is a simplified diagram illustrating a scalable compression-decompression scheme based on the above transform and inverse transform.
  • the concept of scalability is that it should be possible to carry out a single compression process of the data, which can then be decompressed either completely or partially up to the user band width requirements.
  • the pyramid components are then encoded for maximal rate or precision, but in a layered fashion, so that individual users need only take data they need up to the rate or precision that they are able to accept.
  • the data is typically made available on the server in reverse order so that everyone gets the base precision or data rate, and then the higher layers are taken only as needed.
  • Fig. 13 is a simplified flow chart illustrating a generalized scheme for upsampling and which uses the procedure outlined above in Fig. 7 as its guess stage.
  • upsampling is a special case of decompression in which there is no information on Tdn.
  • Tdn can be thought of as quantized with an infinite quantization step. Either way, the next level Qn is constructed based on Lsn alone. In order to produce such a quadrat it is additionally possible to take advantage of the range dequantization approach described with reference to Fig. 8 above.
  • Upsampling is particularly useful for providing a zoom feature, for increasing image resolution for display on a high resolution screen and is applicable to video streams.
  • upsampling is useful to upgrade broadcast quality television pictures for the new high resolution screens that are becoming available.
  • Fig. 14 A is a simplified diagram illustrating the application of the above-described transform to the field of video encoding.
  • the move from an image codec to a video codec requires the ability to deal with motion redundancy.
  • Motion redundancy can be dealt with in the same way it is dealt with using commonly known DCT applications, that is, calculating and coding motion vectors and apply the transform to the motion compensated image, or it can be handled without using motion compensation and with reduced ,or even without, motion vectors coding as will be described.
  • Conventional video encoding operates by compressing the motion compensated images based on the difference between the original image and the motion-estimated images, and the transforms of the present invention can be used in the same manner.
  • Fig. 14B is a simplified block diagram showing a preferred embodiment of a video frame encoder that uses the method shown in Fig. 14A.
  • the video frame encoder 100 determines motion between two successive frames 102 and 104. Each of the frames is separately applied to the transform of the present embodiments.
  • a series of operators 106 carry out low pass filtering and downsampling to produce the successive lowpass components at each level of the transform and at each level, high pass operators 112 extract the high pass components and carry out downsampling thereof.
  • the high pass components and the final low pass component are preserved and the intermediate low pass components are fed to the following lowpass and highpass operators for use in the next level.
  • Circles on the right are the output of the transform modeling (and may be quantized).
  • Motion estimation is carried out on the successive untransformed versions of the frames at motion estimator 114, and the motion estimation data is used at encoder 116 to encode the components produced by the compression transform.
  • encoding of the components produced by the transform uses motion estimation of previous frames which are available from the video stream.
  • motion estimation a preferred way of dealing with motion redundancy in the sense of scalability and editing properties, as indicated in the figure, is to use the available motion information for conditional entropy coding of images.
  • each image fragment is coded independently, by contrast with difference or error image coding, but the coding is provided given its estimation based on its algorithmic past, a term that includes all the information that is already known in the decompression stage, such as former (and possibly later) frames in the stream, former decoded levels and former decoded neighborhood within the current frame.
  • a problem known as the shift-invariance problem causes wavelet-based video applications to be incapable of effectively exploiting single pixel object shift due to motion.
  • the shift invariance problem applies to shifts of any odd number of pixels.
  • the problem is to a great extent solved, that is rendered practically non-existent, when using the transform of the present invention, as the quadrat reconstruction at each level is invariant under single pixel shifts.
  • the quality of the upsampling used in the present embodiments enables efficient motion estimation. For example, motion estimation at level 0 can be carried out by coding motion vectors at a higher level (say level 2) and upsampling the motion-estimated level.
  • the coded motion information is reduced as fewer motion vectors are needed to describe the motion of the higher level.
  • motion information of level 2 is about four times cheaper then motion information of level 0.
  • motion information for each level n can extracted from the lower level n+1 by upsampling level n+1 of the current frame and using level n of the previous frames to create motion vectors for level n. Thus no motion information is encoded.
  • Fig. 14C is a simplified block diagram illustrating a video frame decoder in accordance with a preferred embodiment of the present invention.
  • the Video frame decoder 120 uses previous frame 122 as a basis for interpreting the motion data.
  • the previous frame 122 and the incoming data both serve as inputs to decoder unit 124 which obtains the Ln and Tn-1...To components.
  • the Ls2 and Td2 components are fed to upsampling guess component 126 which produces the lowpass component Lsl for the first level, in accordance with the procedure outlined hereinabove.
  • Lsl and the next highpass component Tdl are passed to the next upsampling guess component 128 which produces the next lowpass component Lso.
  • Lso is then passed to the final upsampling guess component 130 together with the first highpass component Tdo and the frame 132 is recovered.
  • upsampling may be carried out on the final frame.
  • Fig. 14C the circles on the left represent the decoded data and the input to the reverse transform (pyramid reconstruction). They include base lowpass information and texture hints - that is the highpass components for each level.
  • Rectangles on the right represent the reconstructed pyramid at each level, created by upsampling the lowpass from the preceding level and possibly using the texture information to assist the step function operator to create the next level.
  • Decoding at the decoder 124 is carried out using motion estimation of previous frames (in the case of a video stream)
  • the 'Upsampling with hints' nature of the compression and the non- compensation property enables three types of scalability, all three of which use the available bit budget or bandwidth in different ways to decide on the level of decompression. Then where necessary, upsampling is used to compensate for loss of information.
  • FIG. 15 A generalized scaling method is illustrated in Fig. 15. The number of levels of the transform is selected to be maximal. Then compression is ordered by visual importance order and the user simply takes components up to his bandwidth capacity.
  • Quantization scalability One method of providing scalability is known as Quantization scalability and is illustrated in appended Fig. 16 A.
  • the transform is carried out as per the previous embodiments and then the data for steadily decreasing granularities is encoded. The user simply takes the data for successive granularity until the bit budget expires.
  • Fig. 16 A One method of providing scalability is known as Quantization scalability and is illustrated in appended Fig. 16 A. The transform is carried out as per the previous embodiments and then the data for steadily decreasing granularities is encoded. The user simply takes the data for successive granularity until the bit budget expires.
  • 16B shows a similar process in which, in place of quantization granularities the same procedure is applied to provided resolution levels. All users download the data for the coarsest resolution, but only those having sufficient bit budget obtain the data that provides successively higher resolutions.
  • a third method is frame rate scalability and is illustrated in Fig. 17.
  • the method is particularly intended for video streams and requires the simultaneous availability of multiple frame rate streams.
  • the method involves switching between different frame rates, that is between different streams, during compression as dictated by the available bandwidth.
  • the transform encoding of different frames is essentially independent.
  • the initial transformed data is itself formed into several compressed streams, each compressed with a different frame rate.
  • the quantized images are compressed into 30 frames per second stream (fps) and into another stream of 10 fps.
  • fps frames per second stream
  • the only difference is that the second stream encodes each third image. It is also possible to construct a variable rate stream.
  • An advantage of the present embodiments is that they make it easier to edit material after compression has been carried out. Assuming a single frame is to be modified or removed from a compressed video stream, current motion compensated solutions implement such a task by recompression — decompression, frame removal and compressing again. Putting aside the complexity of the operation itself, the procedure results in a loss of quality and error propagation since quantized material is re-quantized. Should editing need to be repeated the effect becomes steadily worse.
  • decoding the stream from the nearest key frame results in a quantized representation of each frame, so that a certain frame can be modified or removed and the resulting quantized stream coded again without affecting the quality of other frames. Otherwise stated, the frames remain mathematically unchanged in their rendered form, only their bit-encoded form may be different.
  • Fig. 18 is a simplified block diagram illustrating a codec in which a transform according to the present embodiments is applied as an enhancement layer over a conventional image or video codec.
  • Conventional codec 180 may be an H.26L for video or JPEG-2000 for images.
  • the conventional codec 180 is connected to an enhancement layer 182 which carries out the transform of the present embodiments.
  • an enhancement layer 182 which carries out the transform of the present embodiments.
  • a complementary arrangement is provided in which a conventional or base layer decoder 186 of the present embodiments is connected to decompressor 188.
  • the base or conventional layer 180 is used to encode the lowpass image and possibly other features such as the motion vectors and rate-distortion mechanism and the enhancement layer performs the remainder of the transform as described hereinabove.
  • a solution based on enhancement layers such as that shown in Fig. 18 provides compatibility with existing standards, and indeed accords with a compatibility approach suggested by the MPEG committee in that it is additive only to the MPEG standard.
  • Fig. 19 is a simplified flow chart that illustrates a process flow for operating a storage mechanism in accordance with a preferred embodiment of the present invention.
  • the storage mechanism is based on the presumption that decomposing a given image into a quadrat and then upsampling the Ls part (the s-pixels of L) gives results which are visually lossless.
  • the storage mechanism forms and then stores the Ls part of the input images rather than the original image. As a result, the amount of storage required is reduced by a factor of 2.
  • the storage mechanism may be extended to any kind of image and to more then one-level of upsampling. However, in such a case the storage is lossy and depends on the amount of visual distortion that the user is willing to accept.
  • Fig. 20 is a simplified flow chart that illustrates a further preferred embodiment of the storage mechanism.
  • the storage mechanism is combined with another compression application as a pre-process, thus effectively reducing the source by a factor of 2 before the compression is carried out.
  • the storage mechanism is appended as a post-process to provide one step upsampling the conventional decompression has been completed.
  • Fig. 21 is a simplified flowchart illustrating a process for using the transforms of the present invention for screen resolution enhancement.
  • Screen resolution is often higher, and sometimes much higher, than image resolution, and in such a case the upsampler can be used to fill available but unused screen spots to enhance screen resolution by a factor of 2. In practice the upsampler can be used beyond the factor of two to any desired resolution although with steadily diminishing effectiveness.
  • the diagram shows a procedure in which the available screen resolution and the image resolution are compared. If there is additional available resolution on the screen then upsampling is carried out as described above with respect to Figs. 7 and
  • Fig. 22 illustrates the concept of "true- construct" equation that states the sum of directional derivatives along close curve equal zero. This is further described at the true-construction equations section above.
  • Applications of the technique and codec of the present invention are applicable inter alia to the following applications: a. A digital scalable video encoder/decoder b. A digital image upsampler for zoom or image size enlargement. c. A digital video upsampler for zoom or video frame size enlargement. d. A digital image resolution enhancement mechanism that increases the resolution of an image to be presented on a higher resolution display. e. A digital video resolution enhancement mechanism that increases the resolution of digital video to be presented on a higher resolution display. f.
  • Enhancement layers for a digital video encoder/decoder g. Enhancement layers for a digital image encoder/decoder. h. A digital scalable image encoder/decoder. i. A digital scalable live video encoder/decoder. j. A digital edit-able video encoder/decoder enabling loss-less editing
  • An efficient storage mechanism which may store an image of a certain type in a lossless fashion while reducing the number of pixels by a factor of two (2).
  • m Storage format enabling storage of high-resolution and high frame rate video while enabling reduction of any or both factors (resolution and frame rate) with time, thus utilizing storage more effectively. There could be automatic mechanisms that degrade the stored content quality (resolution and or frame rate) per a pre-defined specification.
  • n Efficient motion estimation mechanism.
  • An application that enables viewing a streaming video in various combinations of resolution and frame rate and thus enables a pre-view for a stream using low bit rate and then per user demand, or automatically based on certain characteristics of the stream such as motion, increase resolution and or frame rate.
  • VCR capabilities such as multiple speed fast forward or fast backward, for a video stream in streaming time.
  • q Application enabling decompression of a compressed video or image file with real time settings of special features such as sharpening, contrasting and gamma correction.
  • r Application enabling upsampling in post process of a video or image file with real time settings of special features such as sharpening, contrasting and gamma correction.
  • s An enhancement to the preceding two applications, in which there are provided pre-configuration or template-based attribute settings, for example any or all of: sharpening, contrasting and gamma correction.
  • the configuration templates could be adjusted to specific content types (faces, scenario etc.).
  • the user or, optionally a real time mechanism could choose the best template according to the content.
  • 2x2 upsampling as described above, successively, it is possible to interpolate to any coordinate Ls of the form k/2 m , and hence to any desirable precision.
  • the grid size is 1, and m and k are integers. Examples for such an application include: non-integral upsampling, format conversions, graphic applications, image rectification and others.
  • An alternative compression mechanism that uses two frame sizes, a target frame size and a smaller frame size.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un appareil de transformation d'image qui permet de traiter des données d'image de pixels. Cet appareil comprend : au moins un extracteur passe-bas qui permet d'extraire un composant passe-bas d'une série de pixels d'image, au moins un sous-échantillonneur passe-bas associé à l'extracteur passe-bas pour produire des versions sous-échantillonnées de composants passe-bas extraits respectifs, au moins un extracteur passe-haut qui permet d'extraire un composant passe-haut d'une série de pixels image, au moins un sous-échantillonneur passe-haut associé à l'extracteur passe-haut pour produire des versions sous-échantillonnées de composants passe-bas extraits respectifs, enfin des circuits de commande et itératifs associés à l'extracteur passe-bas et l'extracteur passe-haut pour transmettre des ensembles complémentaires de pixels d'une image aux extracteurs respectifs, pour recevoir des composants respectifs passe-bas et passe-haut provenant de sous-échantillonneurs respectifs passe-bas et passe-haut et pour transmettre de manière itérative des ensembles complémentaires de pixels d'une version sous-échantillonnée du composant passe-bas le plus récent à des extracteurs respectifs passe-bas et passe-haut et pour recevoir d'autres composants passe-bas et passe-haut respectifs jusqu'à ce qu'une condition d'arrêt soit atteinte. La transformée inverse correspondante se fonde sur des hypothèses concernant une étape localisée ou un comportement des contours dans l'image.
EP03730463A 2002-06-25 2003-06-25 Traitement d'images utilisant des hypotheses probabilistes sur un comportement localise Withdrawn EP1535239A2 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US39104702P 2002-06-25 2002-06-25
US391047P 2002-06-25
PCT/IL2003/000535 WO2004001666A2 (fr) 2002-06-25 2003-06-25 Traitement d'images utilisant des hypotheses probabilistes sur un comportement localise

Publications (1)

Publication Number Publication Date
EP1535239A2 true EP1535239A2 (fr) 2005-06-01

Family

ID=30000660

Family Applications (1)

Application Number Title Priority Date Filing Date
EP03730463A Withdrawn EP1535239A2 (fr) 2002-06-25 2003-06-25 Traitement d'images utilisant des hypotheses probabilistes sur un comportement localise

Country Status (4)

Country Link
US (1) US7474794B2 (fr)
EP (1) EP1535239A2 (fr)
AU (1) AU2003241143A1 (fr)
WO (1) WO2004001666A2 (fr)

Families Citing this family (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7062445B2 (en) * 2001-01-26 2006-06-13 Microsoft Corporation Quantization loop with heuristic approach
US7027982B2 (en) * 2001-12-14 2006-04-11 Microsoft Corporation Quality and rate control strategy for digital audio
US6980695B2 (en) * 2002-06-28 2005-12-27 Microsoft Corporation Rate allocation for mixed content video
US7383180B2 (en) * 2003-07-18 2008-06-03 Microsoft Corporation Constant bitrate media encoding techniques
US7609763B2 (en) * 2003-07-18 2009-10-27 Microsoft Corporation Advanced bi-directional predictive coding of video frames
US7343291B2 (en) 2003-07-18 2008-03-11 Microsoft Corporation Multi-pass variable bitrate media encoding
US8064520B2 (en) * 2003-09-07 2011-11-22 Microsoft Corporation Advanced bi-directional predictive coding of interlaced video
US20050162268A1 (en) * 2003-11-18 2005-07-28 Integraph Software Technologies Company Digital video surveillance
US8868772B2 (en) 2004-04-30 2014-10-21 Echostar Technologies L.L.C. Apparatus, system, and method for adaptive-rate shifting of streaming content
US7818444B2 (en) 2004-04-30 2010-10-19 Move Networks, Inc. Apparatus, system, and method for multi-bitrate content streaming
US20060233262A1 (en) * 2005-04-13 2006-10-19 Nokia Corporation Signaling of bit stream ordering in scalable video coding
US8370514B2 (en) 2005-04-28 2013-02-05 DISH Digital L.L.C. System and method of minimizing network bandwidth retrieved from an external network
US8683066B2 (en) 2007-08-06 2014-03-25 DISH Digital L.L.C. Apparatus, system, and method for multi-bitrate content streaming
US7725593B2 (en) * 2005-07-15 2010-05-25 Sony Corporation Scalable video coding (SVC) file format
US20080049841A1 (en) * 2006-08-24 2008-02-28 Eric Jeffrey Hardware Method for Performing Real Time Multi-Level Wavelet Decomposition
FR2905539B1 (fr) * 2006-09-01 2008-11-28 Canon Res Ct France Soc Par Ac Procede et dispositif de codage de signaux numeriques multidimensionnels
US20080115175A1 (en) * 2006-11-13 2008-05-15 Rodriguez Arturo A System and method for signaling characteristics of pictures' interdependencies
US8875199B2 (en) * 2006-11-13 2014-10-28 Cisco Technology, Inc. Indicating picture usefulness for playback optimization
US8155207B2 (en) 2008-01-09 2012-04-10 Cisco Technology, Inc. Processing and managing pictures at the concatenation of two video streams
US7705889B2 (en) * 2007-06-15 2010-04-27 Sony Corporation Shutter time compensation
US8254455B2 (en) * 2007-06-30 2012-08-28 Microsoft Corporation Computing collocated macroblock information for direct mode macroblocks
US8346002B2 (en) * 2007-07-20 2013-01-01 Microsoft Corporation High dynamic range image hallucination
US8804845B2 (en) * 2007-07-31 2014-08-12 Cisco Technology, Inc. Non-enhancing media redundancy coding for mitigating transmission impairments
US8958486B2 (en) * 2007-07-31 2015-02-17 Cisco Technology, Inc. Simultaneous processing of media and redundancy streams for mitigating impairments
EP2213097A2 (fr) * 2007-10-16 2010-08-04 Cisco Technology, Inc. Transfert de propriétés de concaténation et d'ordre d'images dans un flux vidéo
US20090141809A1 (en) * 2007-12-04 2009-06-04 Sony Corporation And Sony Electronics Inc. Extension to the AVC standard to support the encoding and storage of high resolution digital still pictures in parallel with video
US8170097B2 (en) * 2007-12-04 2012-05-01 Sony Corporation Extension to the AVC standard to support the encoding and storage of high resolution digital still pictures in series with video
US8718388B2 (en) * 2007-12-11 2014-05-06 Cisco Technology, Inc. Video processing with tiered interdependencies of pictures
JP5230483B2 (ja) * 2008-03-18 2013-07-10 キヤノン株式会社 撮像装置及びその制御方法
US8325800B2 (en) 2008-05-07 2012-12-04 Microsoft Corporation Encoding streaming media as a high bit rate layer, a low bit rate layer, and one or more intermediate bit rate layers
US8379851B2 (en) 2008-05-12 2013-02-19 Microsoft Corporation Optimized client side rate control and indexed file layout for streaming media
US7925774B2 (en) 2008-05-30 2011-04-12 Microsoft Corporation Media streaming using an index file
WO2009152450A1 (fr) 2008-06-12 2009-12-17 Cisco Technology, Inc. Signaux d’interdépendances d’images dans le contexte du mmco pour aider à manipuler un flux
US8705631B2 (en) * 2008-06-17 2014-04-22 Cisco Technology, Inc. Time-shifted transport of multi-latticed video for resiliency from burst-error effects
US8971402B2 (en) 2008-06-17 2015-03-03 Cisco Technology, Inc. Processing of impaired and incomplete multi-latticed video streams
US8699578B2 (en) * 2008-06-17 2014-04-15 Cisco Technology, Inc. Methods and systems for processing multi-latticed video streams
EP2297964A4 (fr) * 2008-06-25 2017-01-18 Cisco Technology, Inc. Support pour bloquer des opérations de mode truqué
US8265140B2 (en) * 2008-09-30 2012-09-11 Microsoft Corporation Fine-grained client-side control of scalable media delivery
US8320465B2 (en) * 2008-11-12 2012-11-27 Cisco Technology, Inc. Error concealment of plural processed representations of a single video signal received in a video program
WO2010073582A1 (fr) * 2008-12-22 2010-07-01 パナソニック株式会社 Dispositif, procédé, circuit intégré et programme d'agrandissement d'image
US9036693B2 (en) * 2009-01-08 2015-05-19 Sri International Method and system for providing region-of-interest video compression
US8189666B2 (en) 2009-02-02 2012-05-29 Microsoft Corporation Local picture identifier and computation of co-located information
US20100218232A1 (en) * 2009-02-25 2010-08-26 Cisco Technology, Inc. Signalling of auxiliary information that assists processing of video according to various formats
US8949883B2 (en) 2009-05-12 2015-02-03 Cisco Technology, Inc. Signalling buffer characteristics for splicing operations of video streams
US8279926B2 (en) 2009-06-18 2012-10-02 Cisco Technology, Inc. Dynamic streaming with latticed representations of video
US9510029B2 (en) 2010-02-11 2016-11-29 Echostar Advanced Technologies L.L.C. Systems and methods to provide trick play during streaming playback
US20170171271A1 (en) * 2015-12-09 2017-06-15 International Business Machines Corporation Video streaming

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5374995A (en) * 1993-03-22 1994-12-20 Eastman Kodak Company Method and apparatus for enhancing sharpness of a sequence of images subject to continuous zoom
US5483153A (en) * 1994-03-24 1996-01-09 Massachusetts Institute Of Technology Transient event detector for use in nonintrusive load monitoring systems
US5587711A (en) * 1994-09-30 1996-12-24 Apple Computer, Inc. Method and system for reconstructing quantized signals
JP3466032B2 (ja) * 1996-10-24 2003-11-10 富士通株式会社 動画像符号化装置および復号化装置
US6442301B1 (en) * 1997-01-06 2002-08-27 Applied Science Fiction, Inc. Apparatus and method for defect channel nulling
US6091777A (en) * 1997-09-18 2000-07-18 Cubic Video Technologies, Inc. Continuously adaptive digital video compression system and method for a web streamer
US6563964B1 (en) * 1999-02-08 2003-05-13 Sharp Laboratories Of America, Inc. Image downsampling using redundant pixel removal
US7082394B2 (en) * 2002-06-25 2006-07-25 Microsoft Corporation Noise-robust feature extraction using multi-layer principal component analysis

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2004001666A2 *

Also Published As

Publication number Publication date
AU2003241143A1 (en) 2004-01-06
AU2003241143A8 (en) 2004-01-06
US7474794B2 (en) 2009-01-06
WO2004001666A3 (fr) 2004-03-18
US20050084166A1 (en) 2005-04-21
WO2004001666A2 (fr) 2003-12-31

Similar Documents

Publication Publication Date Title
US7474794B2 (en) Image processing using probabilistic local behavior assumptions
US12166987B2 (en) Decomposition of residual data during signal encoding, decoding and reconstruction in a tiered hierarchy
US6904176B1 (en) System and method for tiled multiresolution encoding/decoding and communication with lossless selective regions of interest via data reuse
CN108028941B (zh) 用于通过超像素编码和解码数字图像的方法和装置
Jasmi et al. Comparison of image compression techniques using huffman coding, DWT and fractal algorithm
US6597739B1 (en) Three-dimensional shape-adaptive wavelet transform for efficient object-based video coding
US20100316119A1 (en) Preserving text quality in video encoding
US20240040160A1 (en) Video encoding using pre-processing
JP2012095310A (ja) ビデオ・コード変換のための変換ドメイン・サブ−サンプリング
WO2022125180A1 (fr) Prédiction de chrominance à partir de luminance, pour un codage vidéo
RU2518932C2 (ru) Уменьшенное рассогласование коэффициентов усиления постоянной состовляющей (dc) и dc-утечки при обработке преобразования с перекрытием
US12230001B2 (en) Encoder, decoder and related methods
Baeza et al. ROI-based procedures for progressive transmission of digital images: A comparison
Minewaki et al. Near lossless coding of sparse histogram images based on zero-skip quantization
EP2370934A1 (fr) Systèmes et procédés pour la compression, la transmission et la décompression de codecs vidéo
US20240314329A1 (en) Intra Prediction Using Downscaling
US7630568B2 (en) System and method for low-resolution signal rendering from a hierarchical transform representation
US20030165197A1 (en) Programmable vertical filter for video encoding
US20030198395A1 (en) Wavelet transform system, method and computer program product
Singh et al. A brief introduction on image compression techniques and standards
Singh et al. Comparative studies of various techniques for image compression algorithm
AU740066B2 (en) Method and apparatus for hierarchical encoding or decoding video images
Devaraj Inter-Intra Frame Coding in Motion Picture Compensation Using New Wavelet Bi-Orthogonal Coefficients
Ng Sub-band coding for visual communications
Li A line-based lossless backward coding of wavelet trees (BCWT) and BCWT improvements for application

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20050125

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK

DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20100104